Repository: explosion/spacy-llm (language: python)

Issue:
Sync dev with main
<!--- Provide a general summary of your changes in the title. -->

## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
Sync dev with main.

### Corresponding documentation PR
<!--- Add the link to the corresponding documentation PR here, if applicable. -->
\-

### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
Chore.

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I confirm that I have the right to submit this contribution under the project's MIT license.
- [ ] I ran all tests in `tests` and `usage_examples/tests`, and all new and existing tests passed. This includes
  - all external tests (i. e. `pytest` ran with `--external`)
  - all tests requiring a GPU (i. e. `pytest` ran with `--gpu`)
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.



Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 57b80a8b..d87e1978 100644
--- a/README.md
+++ b/README.md
@@ -12,17 +12,17 @@ This package integrates Large Language Models (LLMs) into [spaCy](https://spacy.
 
 - Serializable `llm` **component** to integrate prompts into your spaCy pipeline
 - **Modular functions** to define the [**task**](https://spacy.io/api/large-language-models#tasks) (prompting and parsing) and [**model**](https://spacy.io/api/large-language-models#models)
-- Interfaces with the APIs of 
+- Interfaces with the APIs of
   - **[OpenAI](https://platform.openai.com/docs/api-reference/)**
   - **[Cohere](https://docs.cohere.com/reference/generate)**
   - **[Anthropic](https://docs.anthropic.com/claude/reference/)**
 - Supports open-source LLMs hosted on Hugging Face 🤗:
   - **[Falcon](https://huggingface.co/tiiuae)**
   - **[Dolly](https://huggingface.co/databricks)**
-  - **[Llama 2](https://huggingface.co/meta-llama)**  
+  - **[Llama 2](https://huggingface.co/meta-llama)**
   - **[OpenLLaMA](https://huggingface.co/openlm-research)**
   - **[StableLM](https://huggingface.co/stabilityai)**
-- Integration with [LangChain](https://github.com/hwchase17/langchain) 🦜️🔗 - all `langchain` models and features can be used in `spacy-llm`  
+- Integration with [LangChain](https://github.com/hwchase17/langchain) 🦜️🔗 - all `langchain` models and features can be used in `spacy-llm`
 - Tasks available out of the box:
   - Named Entity Recognition
   - Text classification
@@ -58,13 +58,40 @@ python -m pip install spacy-llm
 ## 🐍 Quickstart
 
 The task and the model have to be supplied to the `llm` pipeline component using [spaCy's config
-system](https://spacy.io/api/data-formats#config). 
+system](https://spacy.io/api/data-formats#config).
 
-Let's run some text classification using a GPT-4 model from OpenAI. If you're using hosted APIs (as opposed to local 
+Let's run some text classification using a GPT-4 model from OpenAI. If you're using hosted APIs (as opposed to local
 models like Falcon, Dolly or LLaMA), ensure to that your API keys are set as environmental variables.
 
-Create a config file `config.cfg` containing at least the following
-(or see the full example [here](usage_examples/textcat_openai)):
+Create a new API key from openai.com or fetch an existing one, and ensure the
+keys are set as environmental variables. For more background information, see
+the documentation around setting [API keys](https://spacy.io/api/large-language-models#api-keys).
+
+### In Python code
+
+To do some quick experiments, from 0.5.0 onwards you can run:
+
+```python
+import spacy
+
+nlp = spacy.blank("en")
+llm = nlp.add_pipe("llm_textcat")
+llm.add_label("INSULT")
+llm.add_label("COMPLIMENT")
+doc = nlp("You look gorgeous!")
+print(doc.cats)
+# {"COMPLIMENT": 1.0, "INSULT": 0.0}
+```
+
+By using the `llm_textcat` factory, the latest version of the built-in textcat task is used, 
+as well as the default GPT-3-5 model from OpenAI.
+
+### Using a config file
+
+To get more control over the various parameters of the `llm` pipeline, 
+create a config file `config.cfg` containing at least the following (or see the
+full example
+[here](https://github.com/explosion/spacy-llm/tree/main/usage_examples/textcat_openai)):
 
 ```ini
 [nlp]
@@ -77,12 +104,11 @@ pipeline = ["llm"]
 factory = "llm"
 
 [components.llm.task]
-@llm_tasks = "spacy.TextCat.v2"
+@llm_tasks = "spacy.TextCat.v3"
 labels = ["COMPLIMENT", "INSULT"]
 
 [components.llm.model]
-@llm_models = "spacy.GPT-4.v1"
-config = {"temperature": 0.3}
+@llm_models = "spacy.GPT-4.v2"
 ```
 
 Now run:
@@ -93,11 +119,13 @@ from spacy_llm.util import assemble
 nlp = assemble("config.cfg")
 doc = nlp("You look gorgeous!")
 print(doc.cats)
+# {"COMPLIMENT": 1.0, "INSULT": 0.0}
 ```
 
-That's it! There's a lot of other features - prompt templating, more tasks, logging etc. For more information on how to 
+That's it! There's a lot of other features - prompt templating, more tasks, logging etc. For more information on how to
 use those, check out https://spacy.io/api/large-language-models.
 
+
 ## 🚀 Ongoing work
 
 In the near future, we will
@@ -110,7 +138,7 @@ PRs are always welcome!
 
 ## 📝️ Reporting issues
 
-If you have questions regarding the usage of `spacy-llm`, or want to give us feedback after giving it a spin, please use 
+If you have questions regarding the usage of `spacy-llm`, or want to give us feedback after giving it a spin, please use
 the [discussion board](https://github.com/explosion/spacy-llm/discussions).
 Bug reports can be filed on the [spaCy issue tracker](https://github.com/explosion/spacy-llm/issues). Thank you!
 
diff --git a/pyproject.toml b/pyproject.toml
index fbc12c14..54def2f1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,7 +20,11 @@ filterwarnings = [
     "ignore:^.*You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use.*:UserWarning",
     "ignore:^.*Xformers is not installed correctly.*:",
     "ignore:^.*The 'warn' method is deprecated, use 'warning' instead.*:DeprecationWarning",
-    "ignore:^.*Support for class-based `config` is deprecated.*:"
+    "ignore:^.*Support for class-based `config` is deprecated.*:",
+    "ignore:^.*The `dict` method is deprecated; use `model_dump` instead.*",
+    "ignore:^.*The `parse_obj` method is deprecated; use `model_validate` instead.*",
+    "ignore:^.*`__get_validators__` is deprecated.*",
+    "ignore:^.*The `construct` method is deprecated.*"
 ]
 markers = [
     "external: interacts with a (potentially cost-incurring) third-party API",
diff --git a/spacy_llm/compat.py b/spacy_llm/compat.py
index e63e33b5..76148b77 100644
--- a/spacy_llm/compat.py
+++ b/spacy_llm/compat.py
@@ -4,13 +4,19 @@
 if sys.version_info[:2] >= (3, 8):  # Python 3.8+
     from typing import Literal, Protocol, runtime_checkable
 else:
-    from typing_extensions import Literal, Protocol, runtime_checkable  # noqa: F401
+    from typing_extensions import runtime_checkable  # noqa: F401
+    from typing_extensions import Literal, Protocol  # noqa: F401
 
-if sys.version_info[:2] >= (3, 9):  # Python 3.8+
+if sys.version_info[:2] >= (3, 9):  # Python 3.9+
     from typing import TypedDict  # noqa: F401
 else:
     from typing_extensions import TypedDict  # noqa: F401
 
+if sys.version_info[:2] >= (3, 11):  # Python 3.11+
+    from typing import Self  # noqa: F401
+else:
+    from typing_extensions import Self  # noqa: F401
+
 try:
     import langchain
 
@@ -42,3 +48,13 @@
 except ImportError:
     accelerate = None
     has_accelerate = False
+
+
+from pydantic import VERSION
+
+PYDANTIC_V2 = VERSION.startswith("2.")
+
+if PYDANTIC_V2:
+    from pydantic.v1 import BaseModel, ValidationError, validator  # noqa: F401
+else:
+    from pydantic import BaseModel, ValidationError, validator  # noqa: F401
diff --git a/spacy_llm/models/hf/dolly.py b/spacy_llm/models/hf/dolly.py
index f622e3c7..db6599b4 100644
--- a/spacy_llm/models/hf/dolly.py
+++ b/spacy_llm/models/hf/dolly.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Tuple
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import Literal, transformers
 from ...registry.util import registry
diff --git a/spacy_llm/models/hf/falcon.py b/spacy_llm/models/hf/falcon.py
index b551b401..00fcff9f 100644
--- a/spacy_llm/models/hf/falcon.py
+++ b/spacy_llm/models/hf/falcon.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Tuple
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import Literal, transformers
 from ...registry.util import registry
diff --git a/spacy_llm/models/hf/llama2.py b/spacy_llm/models/hf/llama2.py
index f6f4bc5c..6adadc9b 100644
--- a/spacy_llm/models/hf/llama2.py
+++ b/spacy_llm/models/hf/llama2.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Tuple
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import Literal, transformers
 from ...registry.util import registry
diff --git a/spacy_llm/models/hf/openllama.py b/spacy_llm/models/hf/openllama.py
index c926fd94..6f10204c 100644
--- a/spacy_llm/models/hf/openllama.py
+++ b/spacy_llm/models/hf/openllama.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Tuple
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import Literal, torch, transformers
 from ...registry.util import registry
diff --git a/spacy_llm/models/hf/stablelm.py b/spacy_llm/models/hf/stablelm.py
index 3eb7cf7f..5aca0f52 100644
--- a/spacy_llm/models/hf/stablelm.py
+++ b/spacy_llm/models/hf/stablelm.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Tuple
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import Literal, has_transformers, torch, transformers
 from ...registry.util import registry
diff --git a/spacy_llm/models/langchain/model.py b/spacy_llm/models/langchain/model.py
index 57e80a44..b09d1788 100644
--- a/spacy_llm/models/langchain/model.py
+++ b/spacy_llm/models/langchain/model.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable, Optional, Type
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ...compat import has_langchain, langchain
 from ...registry import registry
diff --git a/spacy_llm/models/rest/anthropic/model.py b/spacy_llm/models/rest/anthropic/model.py
index 4210ee1e..943233de 100644
--- a/spacy_llm/models/rest/anthropic/model.py
+++ b/spacy_llm/models/rest/anthropic/model.py
@@ -32,7 +32,7 @@ def credentials(self) -> Dict[str, str]:
             warnings.warn(
                 "Could not find the API key to access the Anthropic Claude API. Ensure you have an API key "
                 "set up via the Anthropic console (https://console.anthropic.com/), then make it available as "
-                "an environment variable 'ANTHROPIC_API_KEY."
+                "an environment variable 'ANTHROPIC_API_KEY'."
             )
 
         return {"X-API-Key": api_key if api_key else ""}
@@ -110,7 +110,7 @@ def _request(json_data: Dict[str, Any]) -> Dict[str, Any]:
     @classmethod
     def get_model_names(cls) -> Tuple[str, ...]:
         return (
-            # claude-1
+            # claude-2
             "claude-2",
             "claude-2-100k",
             # claude-1
diff --git a/spacy_llm/models/rest/anthropic/registry.py b/spacy_llm/models/rest/anthropic/registry.py
index dfe922b3..504da15a 100644
--- a/spacy_llm/models/rest/anthropic/registry.py
+++ b/spacy_llm/models/rest/anthropic/registry.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ....compat import Literal
 from ....registry import registry
@@ -32,7 +32,7 @@ def anthropic_claude_2(
     """
     return Anthropic(
         name=name,
-        endpoint=Endpoints.COMPLETIONS,
+        endpoint=Endpoints.COMPLETIONS.value,
         config=config,
         strict=strict,
         max_tries=max_tries,
diff --git a/spacy_llm/models/rest/cohere/model.py b/spacy_llm/models/rest/cohere/model.py
index 5a2ea636..293ed92b 100644
--- a/spacy_llm/models/rest/cohere/model.py
+++ b/spacy_llm/models/rest/cohere/model.py
@@ -62,9 +62,24 @@ def _request(json_data: Dict[str, Any]) -> Dict[str, Any]:
             except HTTPError as ex:
                 res_content = srsly.json_loads(r.content.decode("utf-8"))
                 # Include specific error message in exception.
-                raise ValueError(
-                    f"Request to Cohere API failed: {res_content.get('message', {})}"
-                ) from ex
+                error_message = res_content.get("message", {})
+                # Catch 'blocked output' and 'blocked input' errors from Cohere
+                # This usually happens when it detects violations in their Usage guidelines.
+                # Unfortunately Cohere returns this as an HTTPError, so it cannot be caught in the response.
+                if "blocked" in error_message:
+                    # Only raise an error when strict. If strict is False, do
+                    # nothing and parse the response as usual.
+                    if self._strict:
+                        raise ValueError(
+                            f"Cohere API returned a blocking error. {error_message}. "
+                            "If you wish to ignore and continue, you can pass 'False' to the 'strict' argument of this model. "
+                            "However, note that this will affect how spacy-llm parses the response."
+                        ) from ex
+                else:
+                    # Catching other types of HTTPErrors (e.g., "429: too many requests")
+                    raise ValueError(
+                        f"Request to Cohere API failed: {error_message}"
+                    ) from ex
             response = r.json()
 
             # Cohere returns a 'message' key when there is an error
@@ -83,16 +98,18 @@ def _request(json_data: Dict[str, Any]) -> Dict[str, Any]:
         # timeout is larger.
         responses = [_request({"prompt": prompt}) for prompt in prompts]
         for response in responses:
-            for result in response["generations"]:
-                if "text" in result:
-                    # Although you can set the number of completions in Cohere
-                    # to be greater than 1, we only need to return a single value.
-                    # In this case, we will just return the very first output.
-                    api_responses.append(result["text"])
-                    break
-                else:
-                    api_responses.append(srsly.json_dumps(response))
-
+            if "generations" in response:
+                for result in response["generations"]:
+                    if "text" in result:
+                        # Although you can set the number of completions in Cohere
+                        # to be greater than 1, we only need to return a single value.
+                        # In this case, we will just return the very first output.
+                        api_responses.append(result["text"])
+                        break
+                    else:
+                        api_responses.append(srsly.json_dumps(response))
+            else:
+                api_responses.append(srsly.json_dumps(response))
         return api_responses
 
     @classmethod
diff --git a/spacy_llm/models/rest/cohere/registry.py b/spacy_llm/models/rest/cohere/registry.py
index d657f40d..3279bf4f 100644
--- a/spacy_llm/models/rest/cohere/registry.py
+++ b/spacy_llm/models/rest/cohere/registry.py
@@ -1,6 +1,6 @@
 from typing import Any, Callable, Dict, Iterable
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ....compat import Literal
 from ....registry import registry
diff --git a/spacy_llm/models/rest/openai/__init__.py b/spacy_llm/models/rest/openai/__init__.py
index 5671c910..e1782596 100644
--- a/spacy_llm/models/rest/openai/__init__.py
+++ b/spacy_llm/models/rest/openai/__init__.py
@@ -1,19 +1,35 @@
 from .model import Endpoints, OpenAI
-from .registry import openai_ada, openai_babbage, openai_curie, openai_davinci
-from .registry import openai_gpt_3_5, openai_gpt_4, openai_text_ada
-from .registry import openai_text_babbage, openai_text_curie, openai_text_davinci
+from .registry import openai_ada, openai_ada_v2, openai_babbage, openai_babbage_v2
+from .registry import openai_code_davinci, openai_code_davinci_v2, openai_curie
+from .registry import openai_curie_v2, openai_davinci, openai_davinci_v2
+from .registry import openai_gpt_3_5, openai_gpt_3_5_v2, openai_gpt_4, openai_gpt_4_v2
+from .registry import openai_text_ada, openai_text_ada_v2, openai_text_babbage
+from .registry import openai_text_babbage_v2, openai_text_curie, openai_text_curie_v2
+from .registry import openai_text_davinci, openai_text_davinci_v2
 
 __all__ = [
     "OpenAI",
     "Endpoints",
     "openai_ada",
+    "openai_ada_v2",
     "openai_babbage",
+    "openai_babbage_v2",
+    "openai_code_davinci",
+    "openai_code_davinci_v2",
     "openai_curie",
+    "openai_curie_v2",
     "openai_davinci",
+    "openai_davinci_v2",
     "openai_gpt_3_5",
+    "openai_gpt_3_5_v2",
     "openai_gpt_4",
+    "openai_gpt_4_v2",
     "openai_text_ada",
+    "openai_text_ada_v2",
     "openai_text_babbage",
+    "openai_text_babbage_v2",
     "openai_text_curie",
+    "openai_text_curie_v2",
     "openai_text_davinci",
+    "openai_text_davinci_v2",
 ]
diff --git a/spacy_llm/models/rest/openai/model.py b/spacy_llm/models/rest/openai/model.py
index 28e7afd5..e653beac 100644
--- a/spacy_llm/models/rest/openai/model.py
+++ b/spacy_llm/models/rest/openai/model.py
@@ -25,7 +25,7 @@ def credentials(self) -> Dict[str, str]:
             warnings.warn(
                 "Could not find the API key to access the OpenAI API. Ensure you have an API key "
                 "set up via https://platform.openai.com/account/api-keys, then make it available as "
-                "an environment variable 'OPENAI_API_KEY."
+                "an environment variable 'OPENAI_API_KEY'."
             )
 
         # Check the access and get a list of available models to verify the model argument (if not None)
diff --git a/spacy_llm/models/rest/openai/registry.py b/spacy_llm/models/rest/openai/registry.py
index d8b500b0..3777ebee 100644
--- a/spacy_llm/models/rest/openai/registry.py
+++ b/spacy_llm/models/rest/openai/registry.py
@@ -1,11 +1,55 @@
 from typing import Any, Callable, Dict, Iterable
 
-from spacy.util import SimpleFrozenDict
+from confection import SimpleFrozenDict
 
 from ....compat import Literal
 from ....registry import registry
 from .model import Endpoints, OpenAI
 
+_DEFAULT_TEMPERATURE = 0.0
+
+"""
+Parameter explanations:
+    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
+        or other response object that does not conform to the expectation of how a well-formed response object from
+        this API should look like). If False, the API error responses are returned by __call__(), but no error will
+        be raised.
+    max_tries (int): Max. number of tries for API request.
+    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
+        at each retry.
+    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
+"""
+
+
+@registry.llm_models("spacy.GPT-4.v2")
+def openai_gpt_4_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(temperature=_DEFAULT_TEMPERATURE),
+    name: Literal[
+        "gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314"
+    ] = "gpt-4",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'gpt-4' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["0314", "32k", "32k-0314"]]): Model to use. Base 'gpt-4' model by default.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-4' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
 
 @registry.llm_models("spacy.GPT-4.v1")
 def openai_gpt_4(
@@ -19,18 +63,45 @@ def openai_gpt_4(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'gpt-4' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["0314", "32k", "32k-0314"]]): Model to use. Base 'gpt-4' model by default.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-4' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-4' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.GPT-3-5.v2")
+def openai_gpt_3_5_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(temperature=_DEFAULT_TEMPERATURE),
+    name: Literal[
+        "gpt-3.5-turbo",
+        "gpt-3.5-turbo-16k",
+        "gpt-3.5-turbo-0613",
+        "gpt-3.5-turbo-0613-16k",
+    ] = "gpt-3.5-turbo",  # noqa: F722,F821
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'gpt-3.5' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Literal["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k"]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-3.5' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -58,18 +129,12 @@ def openai_gpt_3_5(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'gpt-3.5' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Literal["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k"]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-3.5' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'gpt-3.5' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -82,6 +147,38 @@ def openai_gpt_3_5(
     )
 
 
+@registry.llm_models("spacy.Text-Davinci.v2")
+def openai_text_davinci_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=1000, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal[
+        "text-davinci-002", "text-davinci-003"
+    ] = "text-davinci-003",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'text-davinci' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["text-davinci-002", "text-davinci-003"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
 @registry.llm_models("spacy.Text-Davinci.v1")
 def openai_text_davinci(
     config: Dict[Any, Any] = SimpleFrozenDict(),
@@ -94,18 +191,42 @@ def openai_text_davinci(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'text-davinci' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["text-davinci-002", "text-davinci-003"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-davinci' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Code-Davinci.v2")
+def openai_code_davinci_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["code-davinci-002"] = "code-davinci-002",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'code-davinci' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["code-davinci-002"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'code-davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -128,18 +249,42 @@ def openai_code_davinci(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'code-davinci' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["code-davinci-002"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'code-davinci' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'code-davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Text-Curie.v2")
+def openai_text_curie_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["text-curie-001"] = "text-curie-001",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'text-curie' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["text-curie-001"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-curie' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -162,18 +307,42 @@ def openai_text_curie(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'text-curie' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["text-curie-001"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-curie' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-curie' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Text-Babbage.v2")
+def openai_text_babbage_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["text-babbage-001"] = "text-babbage-001",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'text-babbage' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["text-babbage-001"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-babbage' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -196,18 +365,42 @@ def openai_text_babbage(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'text-babbage' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["text-babbage-001"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-babbage' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-babbage' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Text-Ada.v2")
+def openai_text_ada_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["text-ada-001"] = "text-ada-001",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'text-ada' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["text-ada-001"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'text-ada' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -230,18 +423,42 @@ def openai_text_ada(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'text-ada' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["text-ada-001"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'claude-1' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'text-ada' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Davinci.v2")
+def openai_davinci_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["davinci"] = "davinci",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'davinci' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["davinci"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -264,18 +481,42 @@ def openai_davinci(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'davinci' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["davinci"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'claude-1' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'davinci' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Curie.v2")
+def openai_curie_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["curie"] = "curie",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'curie' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["curie"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'curie' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -298,18 +539,42 @@ def openai_curie(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'curie' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["curie"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'claude-1' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'curie' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Babbage.v2")
+def openai_babbage_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["babbage"] = "babbage",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'babbage' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["babbage"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'babbage' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -332,18 +597,42 @@ def openai_babbage(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'babbage' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["babbage"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'claude-1' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'babbage' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
+    """
+    return OpenAI(
+        name=name,
+        endpoint=Endpoints.NON_CHAT.value,
+        config=config,
+        strict=strict,
+        max_tries=max_tries,
+        interval=interval,
+        max_request_time=max_request_time,
+    )
+
+
+@registry.llm_models("spacy.Ada.v2")
+def openai_ada_v2(
+    config: Dict[Any, Any] = SimpleFrozenDict(
+        max_tokens=500, temperature=_DEFAULT_TEMPERATURE
+    ),
+    name: Literal["ada"] = "ada",  # noqa: F722
+    strict: bool = OpenAI.DEFAULT_STRICT,
+    max_tries: int = OpenAI.DEFAULT_MAX_TRIES,
+    interval: float = OpenAI.DEFAULT_INTERVAL,
+    max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
+) -> Callable[[Iterable[str]], Iterable[str]]:
+    """Returns OpenAI instance for 'ada' model using REST to prompt API.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
+    name (Optional[Literal["ada"]]): Model to use.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'ada' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
@@ -366,18 +655,12 @@ def openai_ada(
     max_request_time: float = OpenAI.DEFAULT_MAX_REQUEST_TIME,
 ) -> Callable[[Iterable[str]], Iterable[str]]:
     """Returns OpenAI instance for 'ada' model using REST to prompt API.
-    config (Dict[Any, Any]): LLM config arguments passed on to the initialization of the model instance.
+
+    config (Dict[Any, Any]): LLM config passed on to the model's initialization.
     name (Optional[Literal["ada"]]): Model to use.
-    strict (bool): If True, ValueError is raised if the LLM API returns a malformed response (i. e. any kind of JSON
-        or other response object that does not conform to the expectation of how a well-formed response object from
-        this API should look like). If False, the API error responses are returned by __call__(), but no error will
-        be raised.
-    max_tries (int): Max. number of tries for API request.
-    interval (float): Time interval (in seconds) for API retries in seconds. We implement a base 2 exponential backoff
-        at each retry.
-    max_request_time (float): Max. time (in seconds) to wait for request to terminate before raising an exception.
-    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): Anthropic instance for 'claude-1' model using REST to
-        prompt API.
+    RETURNS (Callable[[Iterable[str]], Iterable[str]]]): OpenAI instance for 'ada' model
+
+    DOCS: https://spacy.io/api/large-language-models#models
     """
     return OpenAI(
         name=name,
diff --git a/spacy_llm/pipeline/llm.py b/spacy_llm/pipeline/llm.py
index cdc448cc..b48db6ae 100644
--- a/spacy_llm/pipeline/llm.py
+++ b/spacy_llm/pipeline/llm.py
@@ -15,12 +15,26 @@
 
 from .. import registry  # noqa: F401
 from ..compat import TypedDict
-from ..ty import Cache, Labeled, LLMTask, PromptExecutor, Scorable, Serializable
-from ..ty import validate_type_consistency
+from ..ty import Cache, LabeledTask, LLMTask, PromptExecutorType, ScorableTask
+from ..ty import Serializable, validate_type_consistency
 
 logger = logging.getLogger("spacy_llm")
 logger.addHandler(logging.NullHandler())
 
+DEFAULT_MODEL_CONFIG = {
+    "@llm_models": "spacy.GPT-3-5.v2",
+    "strict": True,
+}
+DEFAULT_CACHE_CONFIG = {
+    "@llm_misc": "spacy.BatchCache.v1",
+    "path": None,
+    "batch_size": 64,
+    "max_batches_in_mem": 4,
+}
+
+DEFAULT_SAVE_IO = False
+DEFAULT_VALIDATE_TYPES = True
+
 
 class CacheConfigType(TypedDict):
     path: Optional[Path]
@@ -34,25 +48,17 @@ class CacheConfigType(TypedDict):
     assigns=[],
     default_config={
         "task": None,
-        "model": {
-            "@llm_models": "spacy.GPT-3-5.v1",
-            "strict": True,
-        },
-        "cache": {
-            "@llm_misc": "spacy.BatchCache.v1",
-            "path": None,
-            "batch_size": 64,
-            "max_batches_in_mem": 4,
-        },
-        "save_io": False,
-        "validate_types": True,
+        "model": DEFAULT_MODEL_CONFIG,
+        "cache": DEFAULT_CACHE_CONFIG,
+        "save_io": DEFAULT_SAVE_IO,
+        "validate_types": DEFAULT_VALIDATE_TYPES,
     },
 )
 def make_llm(
     nlp: Language,
     name: str,
     task: Optional[LLMTask],
-    model: PromptExecutor,
+    model: PromptExecutorType,
     cache: Cache,
     save_io: bool,
     validate_types: bool,
@@ -72,7 +78,7 @@ def make_llm(
     if task is None:
         raise ValueError(
             "Argument `task` has not been specified, but is required (e. g. {'@llm_tasks': "
-            "'spacy.NER.v2'})."
+            "'spacy.NER.v3'})."
         )
     if validate_types:
         validate_type_consistency(task, model)
@@ -96,7 +102,7 @@ def __init__(
         *,
         vocab: Vocab,
         task: LLMTask,
-        model: PromptExecutor,
+        model: PromptExecutorType,
         cache: Cache,
         save_io: bool,
     ) -> None:
@@ -128,10 +134,15 @@ def __init__(
     @property
     def labels(self) -> Tuple[str, ...]:
         labels: Tuple[str, ...] = tuple()
-        if isinstance(self._task, Labeled):
+        if isinstance(self._task, LabeledTask):
             labels = self._task.labels
         return labels
 
+    def add_label(self, label: str) -> int:
+        if not isinstance(self._task, LabeledTask):
+            raise ValueError("The task of this LLM component does not have labels.")
+        return self._task.add_label(label)
+
     @property
     def task(self) -> LLMTask:
         return self._task
@@ -156,7 +167,7 @@ def score(
 
         DOCS: https://spacy.io/api/pipe#score
         """
-        if isinstance(self._task, Scorable):
+        if isinstance(self._task, ScorableTask):
             return self._task.scorer(examples)
         return {}
 
diff --git a/spacy_llm/registry/reader.py b/spacy_llm/registry/reader.py
index 45b3d13a..8809fb94 100644
--- a/spacy_llm/registry/reader.py
+++ b/spacy_llm/registry/reader.py
@@ -39,7 +39,8 @@ def _fewshot_reader(eg_path: Path) -> Iterable[Dict[str, Any]]:
     else:
         if not eg_path.exists():
             raise ValueError(
-                "Specified file path doesn't exist. Please ensure to provide a valid file path."
+                f"Specified file path: {str(eg_path)} doesn't exist. "
+                "Please ensure to provide a valid file path."
             )
 
         suffix = eg_path.suffix.replace("yaml", "yml")
diff --git a/spacy_llm/tasks/__init__.py b/spacy_llm/tasks/__init__.py
index 09e6beae..6551c536 100644
--- a/spacy_llm/tasks/__init__.py
+++ b/spacy_llm/tasks/__init__.py
@@ -1,54 +1,50 @@
 from spacy import Language
 
-from ..pipeline.llm import make_llm
+from ..pipeline.llm import DEFAULT_CACHE_CONFIG, DEFAULT_MODEL_CONFIG, DEFAULT_SAVE_IO
+from ..pipeline.llm import DEFAULT_VALIDATE_TYPES, make_llm
+from .builtin_task import BuiltinTask
 from .lemma import LemmaTask, make_lemma_task
-from .ner import NERTask, make_ner_task, make_ner_task_v2
+from .ner import NERTask, make_ner_task_v3
 from .noop import NoopTask, make_noop_task
 from .rel import RELTask, make_rel_task
 from .sentiment import SentimentTask, make_sentiment_task
-from .spancat import SpanCatTask, make_spancat_task, make_spancat_task_v2
+from .spancat import SpanCatTask, make_spancat_task_v3
 from .summarization import SummarizationTask, make_summarization_task
 from .textcat import TextCatTask, make_textcat_task
 
 _LATEST_TASKS = (
-    "spacy.NER.v2",
+    "spacy.NER.v3",
     "spacy.REL.v1",
     "spacy.Sentiment.v1",
-    "spacy.SpanCat.v2",
+    "spacy.SpanCat.v3",
     "spacy.Summarization.v1",
     "spacy.TextCat.v3",
 )
 
-# Register llm_TASK factories with NoOp models.
+# Register llm_TASK factories with default models.
 for task_handle in _LATEST_TASKS:
     Language.factory(
         name=f"llm_{task_handle.split('.')[1].lower()}",
         default_config={
             "task": {"@llm_tasks": task_handle},
-            "model": {"@llm_models": "spacy.NoOp.v1"},
-            "cache": {
-                "@llm_misc": "spacy.BatchCache.v1",
-                "path": None,
-                "batch_size": 64,
-                "max_batches_in_mem": 4,
-            },
-            "save_io": False,
-            "validate_types": True,
+            "model": DEFAULT_MODEL_CONFIG,
+            "cache": DEFAULT_CACHE_CONFIG,
+            "save_io": DEFAULT_SAVE_IO,
+            "validate_types": DEFAULT_VALIDATE_TYPES,
         },
         func=make_llm,
     )
 
 __all__ = [
     "make_lemma_task",
-    "make_ner_task",
-    "make_ner_task_v2",
+    "make_ner_task_v3",
     "make_noop_task",
     "make_rel_task",
     "make_sentiment_task",
-    "make_spancat_task",
-    "make_spancat_task_v2",
+    "make_spancat_task_v3",
     "make_summarization_task",
     "make_textcat_task",
+    "BuiltinTask",
     "LemmaTask",
     "NERTask",
     "NoopTask",
diff --git a/spacy_llm/tasks/builtin_task.py b/spacy_llm/tasks/builtin_task.py
new file mode 100644
index 00000000..fa565a97
--- /dev/null
+++ b/spacy_llm/tasks/builtin_task.py
@@ -0,0 +1,333 @@
+import abc
+from pathlib import Path
+from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Type, cast
+
+import jinja2
+import srsly
+from spacy import Language, util, Errors
+from spacy.tokens import Doc
+from spacy.training import Example
+
+from ..registry import lowercase_normalizer
+from ..ty import FewshotExample, TaskResponseParser
+
+
+class BuiltinTask(abc.ABC):
+    """Abstract base task implementing interfaces and/or functionality expected from all built-in tasks:
+        - working prompt template strings
+        - swappable response parsers
+        - swappable prompt example type
+        - integration of fewshot example into the fully rendered prompt
+        - initializable (in line with other spaCy components)
+        - (de-)serialization
+
+    On the relation of BuiltinTask to LLMTask: the latter specifies the minimal contract a task implementation
+    has to fulfill, whereas a BuiltinTask requires (and offers) functionality beyond that. The rationale behind that is
+    that built-in tasks should provide as smooth a usage experience as possible while still making it as easy as possible
+    for users to write their own, custom tasks.
+    """
+
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser,
+        prompt_example_type: Type[FewshotExample],
+        template: str,
+        prompt_examples: Optional[List[FewshotExample]],
+    ):
+        """Initializes task.
+        parse_responses (TaskResponseParser): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        template (str): Prompt template passed to the model.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        """
+        self._parse_responses = parse_responses
+        self._prompt_examples = prompt_examples or []
+        self._template = template
+        self._prompt_example_type = prompt_example_type
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[Any]:
+        """Generate prompts from docs.
+        docs (Iterable[Doc]): Docs to generate prompts from.
+        RETURNS (Iterable[Any]): Iterable with one prompt per doc.
+        """
+        environment = jinja2.Environment()
+        _template = environment.from_string(self._template)
+        for doc in docs:
+            prompt = _template.render(
+                text=doc.text, prompt_examples=self._prompt_examples, **kwargs
+            )
+            yield prompt
+
+    @abc.abstractmethod
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[Any]
+    ) -> Iterable[Doc]:
+        """
+        Parses LLM responses.
+        docs (Iterable[Doc]): Docs to map responses into.
+        responses ([Iterable[Any]]): LLM responses.
+        RETURNS (Iterable[Doc]]): Updated docs.
+        """
+
+    def _initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        n_prompt_examples: int = 0,
+        **kwargs: Any,
+    ) -> None:
+        """Initializes prompt examples from Doc examples.
+        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
+            for initialization.
+        nlp (Language): Language instance.
+        n_prompt_examples (int): How many prompt examples to infer from the provided Example objects.
+            0 by default. Takes all examples if set to -1.
+        """
+        for eg in get_examples():
+            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
+                self._prompt_examples.append(
+                    self._prompt_example_type.generate(eg, **kwargs)
+                )
+
+    def get_cfg(self) -> Dict[str, Any]:
+        """Serialize the task's configuration attributes."""
+        cfg = {key: getattr(self, key) for key in self._cfg_keys}
+        return cfg
+
+    def set_cfg(self, cfg: Dict[str, Any]) -> None:
+        """Deserialize the task's configuration attributes.
+
+        cfg (Dict[str, Any]): dictionary containing configuration attributes.
+        """
+        for key, value in cfg.items():
+            setattr(self, key, value)
+
+    def _get_prompt_examples(self) -> List[Dict[str, Any]]:
+        """Serialize examples."""
+        examples = [eg.dict() for eg in self._prompt_examples]
+        return examples
+
+    def _set_prompt_examples(self, examples: List[Dict[str, Any]]) -> None:
+        """Set prompt examples.
+
+        examples (List[Dict[str, Any]]): prompt examples.
+        """
+        self._prompt_examples = [
+            self._prompt_example_type.parse_obj(eg) for eg in examples
+        ]
+
+    def to_bytes(
+        self,
+        *,
+        exclude: Tuple[str] = cast(Tuple[str], tuple()),
+    ) -> bytes:
+        """Serialize the BuiltinTask to a bytestring.
+
+        exclude (Tuple): Names of properties to exclude from serialization.
+        RETURNS (bytes): The serialized object.
+        """
+        serialize = {
+            "cfg": lambda: srsly.json_dumps(self.get_cfg()),
+            "prompt_examples": lambda: srsly.msgpack_dumps(self._get_prompt_examples()),
+        }
+
+        return util.to_bytes(serialize, exclude)
+
+    def from_bytes(
+        self,
+        bytes_data: bytes,
+        *,
+        exclude: Tuple[str] = cast(Tuple[str], tuple()),
+    ) -> "BuiltinTask":
+        """Load the BuiltinTask from a bytestring.
+
+        bytes_data (bytes): The data to load.
+        exclude (Tuple[str]): Names of properties to exclude from deserialization.
+        RETURNS (BuiltinTask): Modified BuiltinTask instance.
+        """
+
+        deserialize = {
+            "cfg": lambda b: self.set_cfg(srsly.json_loads(b)),
+            "prompt_examples": lambda b: self._set_prompt_examples(
+                srsly.msgpack_loads(b)
+            ),
+        }
+
+        util.from_bytes(bytes_data, deserialize, exclude)
+        return self
+
+    def to_disk(
+        self,
+        path: Path,
+        *,
+        exclude: Tuple[str] = cast(Tuple[str], tuple()),
+    ) -> None:
+        """Serialize the task to disk.
+
+        path (Path): A path (currently unused).
+        exclude (Tuple): Names of properties to exclude from serialization.
+        """
+
+        serialize = {
+            "cfg": lambda p: srsly.write_json(p, self.get_cfg()),
+            "prompt_examples": lambda p: srsly.write_msgpack(
+                p, self._get_prompt_examples()
+            ),
+        }
+
+        util.to_disk(path, serialize, exclude)
+
+    def from_disk(
+        self,
+        path: Path,
+        *,
+        exclude: Tuple[str] = cast(Tuple[str], tuple()),
+    ) -> "BuiltinTask":
+        """Deserialize the task from disk.
+
+        path (Path): A path (currently unused).
+        exclude (Tuple): Names of properties to exclude from serialization.
+        RETURNS (BuiltinTask): The BuiltinTask instance.
+        """
+
+        deserialize = {
+            "cfg": lambda p: self.set_cfg(srsly.read_json(p)),
+            "prompt_examples": lambda p: self._set_prompt_examples(
+                srsly.read_msgpack(p)
+            ),
+        }
+
+        util.from_disk(path, deserialize, exclude)
+        return self
+
+    @property
+    def prompt_template(self) -> str:
+        return self._template
+
+    @property
+    @abc.abstractmethod
+    def _cfg_keys(self) -> List[str]:
+        """A list of configuration attributes to serialize."""
+        pass
+
+    @classmethod
+    def _check_extension(cls, extension: str) -> None:
+        """Add extension if need be.
+        extension (str): Extension to check/add.
+        """
+        if not Doc.has_extension(extension):
+            Doc.set_extension(extension, default=[])
+
+
+class BuiltinTaskWithLabels(BuiltinTask, abc.ABC):
+    """Built-in tasks with labels."""
+
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser,
+        prompt_example_type: Type[FewshotExample],
+        template: str,
+        prompt_examples: Optional[List[FewshotExample]],
+        labels: List[str],
+        label_definitions: Optional[Dict[str, str]],
+        normalizer: Optional[Callable[[str], str]],
+    ):
+        """Built-in task with labels.
+
+        parse_responses (TaskResponseParser): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        template (str): Prompt template passed to the model.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        labels (List[str]): List of labels to pass to the template.
+            Leave empty to (optionally) populate it at initialization time.
+        label_definitions (Optional[Dict[str, str]]): Map of label -> description
+            of the label to help the language model output the entities wanted.
+            It is usually easier to provide these definitions rather than
+            full examples, although both can be provided.
+        normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+        )
+        self._normalizer = normalizer if normalizer else lowercase_normalizer()
+        self._label_dict = {
+            self._normalizer(label): label for label in sorted(set(labels))
+        }
+        self._label_definitions = label_definitions
+
+    def _initialize(  # type: ignore[override]
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        labels: List[str] = [],
+        n_prompt_examples: int = 0,
+        **kwargs,
+    ) -> None:
+        """Supports initialization of tasks with labels by auto-discovering labels and returning the derived few-shot
+        examples and label dict.
+
+        Labels can be set through, by order of precedence:
+
+        - the `[initialize]` section of the pipeline configuration
+        - the `labels` argument supplied to the task factory
+        - the labels found in the examples
+
+        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
+            for initialization.
+        nlp (Language): Language instance.
+        labels (List[str]): Optional list of labels.
+        n_prompt_examples (int): How many prompt examples to infer from the Example objects.
+            0 by default. Takes all examples if set to -1.
+        """
+        if not labels:
+            labels = list(self._label_dict.values())
+        infer_labels = not labels
+
+        if infer_labels:
+            labels = []
+
+        for eg in get_examples():
+            if infer_labels:
+                labels.extend(self._extract_labels_from_example(eg))
+            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
+                self._prompt_examples.append(
+                    self._prompt_example_type.generate(eg, **kwargs)
+                )
+
+        self._label_dict = {
+            self._normalizer(label): label for label in sorted(set(labels))
+        }
+
+    @abc.abstractmethod
+    def _extract_labels_from_example(self, example: Example) -> List[str]:
+        """Extracts labels from Example instance.
+        example (Example): Example to extract labels from.
+        RETURNS (List[str]): Labels extracted from Example instance.
+        """
+
+    @property
+    def labels(self) -> Tuple[str, ...]:
+        return tuple(self._label_dict.values())
+
+    def add_label(self, label: str) -> int:
+        if not isinstance(label, str):
+            raise ValueError(Errors.E187)
+        if label in self.labels:
+            return 0
+        self._label_dict[self._normalizer(label)] = label
+        return 1
+
+    @property
+    def normalizer(self) -> Callable[[str], str]:
+        return self._normalizer
+
+    @property
+    def label_dict(self) -> Dict[str, str]:
+        return self._label_dict
+
+    @property
+    def label_definitions(self) -> Optional[Dict[str, str]]:
+        return self._label_definitions
diff --git a/spacy_llm/tasks/lemma.py b/spacy_llm/tasks/lemma.py
deleted file mode 100644
index 19ee2cc1..00000000
--- a/spacy_llm/tasks/lemma.py
+++ /dev/null
@@ -1,136 +0,0 @@
-from typing import Any, Callable, Dict, Iterable, List, Optional, Type
-
-import jinja2
-from pydantic import BaseModel
-from spacy.language import Language
-from spacy.scorer import Scorer
-from spacy.tokens import Doc
-from spacy.training import Example
-
-from ..registry import registry
-from ..ty import ExamplesConfigType
-from .templates import read_template
-from .util import SerializableTask
-
-_DEFAULT_LEMMA_TEMPLATE_V1 = read_template("lemma")
-
-
-class LemmaExample(BaseModel):
-    text: str
-    lemmas: List[Dict[str, str]]
-
-
-@registry.llm_tasks("spacy.Lemma.v1")
-def make_lemma_task(
-    template: str = _DEFAULT_LEMMA_TEMPLATE_V1,
-    examples: ExamplesConfigType = None,
-):
-    """Lemma.v1 task factory.
-
-    template (str): Prompt template passed to the model.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    """
-    raw_examples = examples() if callable(examples) else examples
-    lemma_examples = (
-        [LemmaExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-
-    return LemmaTask(template=template, examples=lemma_examples)
-
-
-class LemmaTask(SerializableTask[LemmaExample]):
-    def __init__(
-        self,
-        template: str = _DEFAULT_LEMMA_TEMPLATE_V1,
-        examples: Optional[List[LemmaExample]] = None,
-    ):
-        """Default lemmatization task.
-
-        template (str): Prompt template passed to the model.
-        examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        """
-        self._template = template
-        self._prompt_examples = examples or []
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initializes prompt examples from Doc examples.
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        n_prompt_examples (int): How many prompt examples to infer from the provided Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        for eg in get_examples():
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(self._create_prompt_example(eg))
-
-    @property
-    def prompt_template(self) -> str:
-        return self._template
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=doc.text,
-                examples=self._prompt_examples,
-            )
-            yield prompt
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        for doc, prompt_response in zip(docs, responses):
-            parsed_response = [
-                [pr_part.strip() for pr_part in pr.split(":")]
-                for pr in prompt_response.replace("Lemmatized text:", "")
-                .replace("'''", "")
-                .strip()
-                .split("\n")
-            ]
-            tokens = [token for token in doc]
-
-            # If numbers of tokens recognized by spaCy and returned by LLM don't match, we don't attempt a partial
-            # match.
-            if len(tokens) != len(parsed_response):
-                yield doc
-
-            # Assign lemmas.
-            for token, lemma_info in zip(tokens, parsed_response):
-                if len(lemma_info) > 0:
-                    token.lemma_ = lemma_info[1]
-
-            yield doc
-
-    def scorer(
-        self,
-        examples: Iterable[Example],
-    ) -> Dict[str, Any]:
-        """Scores lemmatization accuracy on provided examples.
-        examples (Iterable[Example]): Examples to determine score against.
-        """
-        return Scorer.score_token_attr(examples, "pos")
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return ["_template"]
-
-    @property
-    def _Example(self) -> Type[LemmaExample]:
-        return LemmaExample
-
-    def _create_prompt_example(self, example: Example) -> LemmaExample:
-        """Create a lemma prompt example from a spaCy example."""
-        lemma_dict = [{t.text: t.lemma_} for t in example.reference]
-        return LemmaExample(text=example.reference.text, lemmas=lemma_dict)
diff --git a/spacy_llm/tasks/lemma/__init__.py b/spacy_llm/tasks/lemma/__init__.py
new file mode 100644
index 00000000..9e2617d0
--- /dev/null
+++ b/spacy_llm/tasks/lemma/__init__.py
@@ -0,0 +1,5 @@
+from .registry import make_lemma_task
+from .task import LemmaTask
+from .util import LemmaExample
+
+__all__ = ["make_lemma_task", "LemmaExample", "LemmaTask"]
diff --git a/spacy_llm/tasks/lemma/parser.py b/spacy_llm/tasks/lemma/parser.py
new file mode 100644
index 00000000..9505f9d1
--- /dev/null
+++ b/spacy_llm/tasks/lemma/parser.py
@@ -0,0 +1,24 @@
+from typing import Iterable, List
+
+from spacy.tokens import Doc
+
+from .task import LemmaTask
+
+
+def parse_responses_v1(
+    task: LemmaTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[List[List[str]]]:
+    """Parses LLM responses for spacy.Lemma.v1.
+    task (LemmaTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Iterable[List[str]]): Lists of 2-lists (token: lemmatized token) per doc/response.
+    """
+    for prompt_response in responses:
+        yield [
+            [pr_part.strip() for pr_part in pr.split(":")]
+            for pr in prompt_response.replace("Lemmatized text:", "")
+            .replace("'''", "")
+            .strip()
+            .split("\n")
+        ]
diff --git a/spacy_llm/tasks/lemma/registry.py b/spacy_llm/tasks/lemma/registry.py
new file mode 100644
index 00000000..e39a0f84
--- /dev/null
+++ b/spacy_llm/tasks/lemma/registry.py
@@ -0,0 +1,49 @@
+from typing import Optional, Type
+
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, Scorer, TaskResponseParser
+from .parser import parse_responses_v1
+from .task import DEFAULT_LEMMA_TEMPLATE_V1, LemmaTask
+from .util import LemmaExample, score
+
+
+@registry.llm_misc("spacy.LemmaParser.v1")
+def make_lemma_parser() -> TaskResponseParser[LemmaTask]:
+    return parse_responses_v1
+
+
+@registry.llm_misc("spacy.LemmaScorer.v1")
+def make_lemma_scorer() -> Scorer:
+    return score
+
+
+@registry.llm_tasks("spacy.Lemma.v1")
+def make_lemma_task(
+    template: str = DEFAULT_LEMMA_TEMPLATE_V1,
+    parse_responses: Optional[TaskResponseParser[LemmaTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    examples: ExamplesConfigType = None,
+    scorer: Optional[Scorer] = None,
+):
+    """Lemma.v1 task factory.
+
+    template (str): Prompt template passed to the model.
+    parse_responses (Optional[TaskResponseParser]): Callable for parsing LLM responses for this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or LemmaExample
+    lemma_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return LemmaTask(
+        template=template,
+        parse_responses=parse_responses or parse_responses_v1,
+        prompt_example_type=example_type,
+        prompt_examples=lemma_examples,
+        scorer=scorer or score,
+    )
diff --git a/spacy_llm/tasks/lemma/task.py b/spacy_llm/tasks/lemma/task.py
new file mode 100644
index 00000000..3223af56
--- /dev/null
+++ b/spacy_llm/tasks/lemma/task.py
@@ -0,0 +1,72 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type
+
+from spacy import Language
+from spacy.tokens import Doc
+from spacy.training import Example
+
+from ...compat import Self
+from ...ty import FewshotExample, Scorer, TaskResponseParser
+from ..builtin_task import BuiltinTask
+from ..templates import read_template
+
+DEFAULT_LEMMA_TEMPLATE_V1 = read_template("lemma.v1")
+
+
+class LemmaTask(BuiltinTask):
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        prompt_examples: Optional[List[FewshotExample]],
+        template: str,
+        scorer: Scorer,
+    ):
+        """Default lemmatization task.
+
+        parse_responses (TaskResponseParser[Self]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        template (str): Prompt template passed to the model.
+        scorer (Scorer): Scorer function.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+        )
+        self._scorer = scorer
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        for doc, lemmas in zip(docs, self._parse_responses(self, docs, responses)):
+            tokens = [token for token in doc]
+            # If numbers of tokens recognized by spaCy and returned by LLM don't match, we don't attempt a partial
+            # match.
+            if len(tokens) != len(lemmas):
+                yield doc
+
+            # Assign lemmas.
+            for token, lemma_info in zip(tokens, lemmas):
+                if len(lemma_info) > 0:
+                    token.lemma_ = lemma_info[1]
+
+            yield doc
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        n_prompt_examples: int = 0,
+    ) -> None:
+        super()._initialize(
+            get_examples=get_examples, nlp=nlp, n_prompt_examples=n_prompt_examples
+        )
+
+    def scorer(self, examples: Iterable[Example]) -> Dict[str, Any]:
+        return self._scorer(examples)
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return ["_template"]
diff --git a/spacy_llm/tasks/lemma/util.py b/spacy_llm/tasks/lemma/util.py
new file mode 100644
index 00000000..c95a2ece
--- /dev/null
+++ b/spacy_llm/tasks/lemma/util.py
@@ -0,0 +1,25 @@
+from typing import Any, Dict, Iterable, List
+
+from spacy.scorer import Scorer
+from spacy.training import Example
+
+from ...compat import Self
+from ...ty import FewshotExample
+
+
+class LemmaExample(FewshotExample):
+    text: str
+    lemmas: List[Dict[str, str]]
+
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        lemma_dict = [{t.text: t.lemma_} for t in example.reference]
+        return LemmaExample(text=example.reference.text, lemmas=lemma_dict)
+
+
+def score(examples: Iterable[Example], **kwargs) -> Dict[str, Any]:
+    """Score lemmatization accuracy in examples.
+    examples (Iterable[Example]): Examples to score.
+    RETURNS (Dict[str, Any]): Dict with metric name -> score.
+    """
+    return Scorer.score_token_attr(examples, "lemma")
diff --git a/spacy_llm/tasks/ner.py b/spacy_llm/tasks/ner.py
deleted file mode 100644
index 365aba98..00000000
--- a/spacy_llm/tasks/ner.py
+++ /dev/null
@@ -1,212 +0,0 @@
-from collections import defaultdict
-from typing import Any, Callable, Dict, Iterable, List, Optional, Union
-
-from spacy.language import Language
-from spacy.scorer import get_ner_prf
-from spacy.tokens import Doc, Span
-from spacy.training import Example
-from spacy.util import filter_spans
-
-from ..compat import Literal
-from ..registry import registry
-from ..ty import ExamplesConfigType
-from ..util import split_labels
-from .span import SpanExample, SpanTask
-from .templates import read_template
-
-_DEFAULT_NER_TEMPLATE_V1 = read_template("ner")
-_DEFAULT_NER_TEMPLATE_V2 = read_template("ner.v2")
-
-
-@registry.llm_tasks("spacy.NER.v1")
-def make_ner_task(
-    labels: str = "",
-    examples: Optional[Callable[[], Iterable[Any]]] = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-    case_sensitive_matching: bool = False,
-    single_match: bool = False,
-):
-    """NER.v1 task factory.
-
-    labels (str): Comma-separated list of labels to pass to the template.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Map of label -> description
-        of the label to help the language model output the entities wanted.
-        It is usually easier to provide these definitions rather than
-        full examples, although both can be provided.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-    alignment_mode (str): "strict", "contract" or "expand".
-    case_sensitive: Whether to search without case sensitivity.
-    single_match (bool): If False, allow one substring to match multiple times in
-        the text. If True, returns the first hit.
-    """
-    labels_list = split_labels(labels)
-
-    span_examples = (
-        [SpanExample(**eg) for eg in examples()] if callable(examples) else examples
-    )
-    return NERTask(
-        labels=labels_list,
-        template=_DEFAULT_NER_TEMPLATE_V1,
-        prompt_examples=span_examples,
-        normalizer=normalizer,
-        alignment_mode=alignment_mode,
-        case_sensitive_matching=case_sensitive_matching,
-        single_match=single_match,
-    )
-
-
-@registry.llm_tasks("spacy.NER.v2")
-def make_ner_task_v2(
-    labels: Union[List[str], str] = [],
-    template: str = _DEFAULT_NER_TEMPLATE_V2,
-    label_definitions: Optional[Dict[str, str]] = None,
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-    case_sensitive_matching: bool = False,
-    single_match: bool = False,
-):
-    """NER.v2 task factory.
-
-    labels (Union[str, List[str]]): List of labels to pass to the template,
-        either an actual list or a comma-separated string.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Map of label -> description
-        of the label to help the language model output the entities wanted.
-        It is usually easier to provide these definitions rather than
-        full examples, although both can be provided.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-    alignment_mode (str): "strict", "contract" or "expand".
-    case_sensitive: Whether to search without case sensitivity.
-    single_match (bool): If False, allow one substring to match multiple times in
-        the text. If True, returns the first hit.
-    """
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    span_examples = [SpanExample(**eg) for eg in raw_examples] if raw_examples else None
-
-    return NERTask(
-        labels=labels_list,
-        template=template,
-        label_definitions=label_definitions,
-        prompt_examples=span_examples,
-        normalizer=normalizer,
-        alignment_mode=alignment_mode,
-        case_sensitive_matching=case_sensitive_matching,
-        single_match=single_match,
-    )
-
-
-class NERTask(SpanTask):
-    def __init__(
-        self,
-        labels: List[str] = [],
-        template: str = _DEFAULT_NER_TEMPLATE_V2,
-        label_definitions: Optional[Dict[str, str]] = None,
-        prompt_examples: Optional[List[SpanExample]] = None,
-        normalizer: Optional[Callable[[str], str]] = None,
-        alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-        case_sensitive_matching: bool = False,
-        single_match: bool = False,
-    ):
-        """Default NER task.
-
-        labels (List[str]): List of labels to pass to the template.
-            Leave empty to populate it at initialization time (only if examples are provided).
-        template (str): Prompt template passed to the model.
-        label_definitions (Optional[Dict[str, str]]): Map of label -> description
-            of the label to help the language model output the entities wanted.
-            It is usually easier to provide these definitions rather than
-            full examples, although both can be provided.
-        examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-        alignment_mode (str): "strict", "contract" or "expand".
-        case_sensitive: Whether to search without case sensitivity.
-        single_match (bool): If False, allow one substring to match multiple times in
-            the text. If True, returns the first hit.
-        """
-        super().__init__(
-            labels=labels,
-            template=template,
-            label_definitions=label_definitions,
-            prompt_examples=prompt_examples,
-            normalizer=normalizer,
-            alignment_mode=alignment_mode,
-            case_sensitive_matching=case_sensitive_matching,
-            single_match=single_match,
-        )
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        labels: List[str] = [],
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initialize the NER task, by auto-discovering labels.
-
-        Labels can be set through, by order of precedence:
-
-        - the `[initialize]` section of the pipeline configuration
-        - the `labels` argument supplied to the task factory
-        - the labels found in the examples
-
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        labels (List[str]): Optional list of labels.
-        n_prompt_examples (int): How many prompt examples to infer from the Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        if not labels:
-            labels = list(self._label_dict.values())
-        infer_labels = not labels
-
-        if infer_labels:
-            labels = []
-
-        for eg in get_examples():
-            if infer_labels:
-                for ent in eg.reference.ents:
-                    labels.append(ent.label_)
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(self._create_prompt_example(eg))
-
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-
-    def assign_spans(
-        self,
-        doc: Doc,
-        spans: List[Span],
-    ) -> None:
-        """Assign spans to the document."""
-        doc.set_ents(filter_spans(spans))
-
-    def scorer(
-        self,
-        examples: Iterable[Example],
-    ) -> Dict[str, Any]:
-        return get_ner_prf(examples)
-
-    def _create_prompt_example(self, example: Example) -> SpanExample:
-        """Create an NER prompt example from a spaCy example."""
-        entities = defaultdict(list)
-        for ent in example.reference.ents:
-            entities[ent.label_].append(ent.text)
-
-        return SpanExample(text=example.reference.text, entities=entities)
diff --git a/spacy_llm/tasks/ner/__init__.py b/spacy_llm/tasks/ner/__init__.py
new file mode 100644
index 00000000..a29ebe91
--- /dev/null
+++ b/spacy_llm/tasks/ner/__init__.py
@@ -0,0 +1,11 @@
+from .registry import make_ner_task, make_ner_task_v2, make_ner_task_v3
+from .task import NERTask
+from .util import NERExample
+
+__all__ = [
+    "make_ner_task",
+    "make_ner_task_v2",
+    "make_ner_task_v3",
+    "NERExample",
+    "NERTask",
+]
diff --git a/spacy_llm/tasks/ner/registry.py b/spacy_llm/tasks/ner/registry.py
new file mode 100644
index 00000000..c7fbfae4
--- /dev/null
+++ b/spacy_llm/tasks/ner/registry.py
@@ -0,0 +1,179 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type, Union
+
+from ...compat import Literal
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, Scorer, TaskResponseParser
+from ...util import split_labels
+from ..span import parse_responses as parse_span_responses
+from ..span import parse_responses_cot as parse_span_responses_cot
+from ..span.util import check_label_consistency, check_label_consistency_cot
+from .task import DEFAULT_NER_TEMPLATE_V1, DEFAULT_NER_TEMPLATE_V2
+from .task import DEFAULT_NER_TEMPLATE_V3, NERTask, SpanTask
+from .util import NERCoTExample, NERExample, score
+
+
+@registry.llm_tasks("spacy.NER.v1")
+def make_ner_task(
+    parse_responses: Optional[TaskResponseParser[SpanTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: str = "",
+    examples: Optional[Callable[[], Iterable[Any]]] = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    single_match: bool = False,
+    scorer: Optional[Scorer] = None,
+):
+    """NER.v1 task factory.
+
+    parse_responses (Optional[TaskResponseParser[SpanTask]]): Callable for parsing LLM responses for this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (str): Comma-separated list of labels to pass to the template.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching: Whether to search without case sensitivity.
+    single_match (bool): If False, allow one substring to match multiple times in
+        the text. If True, returns the first hit.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    example_type = prompt_example_type or NERExample
+    span_examples = (
+        [example_type(**eg) for eg in examples()] if callable(examples) else examples
+    )
+
+    return NERTask(
+        parse_responses=parse_responses or parse_span_responses,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=DEFAULT_NER_TEMPLATE_V1,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=single_match,
+        label_definitions=None,
+        scorer=scorer or score,
+        description=None,
+        check_label_consistency=check_label_consistency,
+    )
+
+
+@registry.llm_tasks("spacy.NER.v2")
+def make_ner_task_v2(
+    parse_responses: Optional[TaskResponseParser[SpanTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_NER_TEMPLATE_V2,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    single_match: bool = False,
+    scorer: Optional[Scorer] = None,
+):
+    """NER.v2 task factory.
+
+    parse_responses (Optional[TaskResponseParser[SpanTask]]): Callable for parsing LLM responses for this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[str, List[str]]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    label_definitions (Optional[Dict[str, str]]): Map of label -> description
+        of the label to help the language model output the entities wanted.
+        It is usually easier to provide these definitions rather than
+        full examples, although both can be provided.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching (bool): Whether to search without case sensitivity.
+    single_match (bool): If False, allow one substring to match multiple times in
+        the text. If True, returns the first hit.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or NERExample
+    span_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return NERTask(
+        parse_responses=parse_responses or parse_span_responses,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=single_match,
+        scorer=scorer or score,
+        description=None,
+        check_label_consistency=check_label_consistency,
+    )
+
+
+@registry.llm_tasks("spacy.NER.v3")
+def make_ner_task_v3(
+    parse_responses: Optional[TaskResponseParser[SpanTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_NER_TEMPLATE_V3,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    scorer: Optional[Scorer] = None,
+    description: Optional[str] = None,
+):
+    """NER.v3 task factory, with chain-of-thought prompting.
+
+    parse_responses (Optional[TaskResponseParser[SpanTask]]): Callable for parsing LLM responses for this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[str, List[str]]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    description (str): A description of what to recognize or not recognize as entities.
+    label_definitions (Optional[Dict[str, str]]): Map of label -> description
+        of the label to help the language model output the entities wanted.
+        It is usually easier to provide these definitions rather than
+        full examples, although both can be provided.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching (bool): Whether to search without case sensitivity.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or NERCoTExample
+    span_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return NERTask(
+        parse_responses=parse_responses or parse_span_responses_cot,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=False,
+        scorer=scorer or score,
+        description=description,
+        check_label_consistency=check_label_consistency_cot,
+    )
diff --git a/spacy_llm/tasks/ner/task.py b/spacy_llm/tasks/ner/task.py
new file mode 100644
index 00000000..3b1b8774
--- /dev/null
+++ b/spacy_llm/tasks/ner/task.py
@@ -0,0 +1,100 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type
+
+from spacy.language import Language
+from spacy.tokens import Doc, Span
+from spacy.training import Example
+from spacy.util import filter_spans
+
+from ...compat import Literal, Self
+from ...ty import FewshotExample, Scorer, TaskResponseParser
+from ..span import SpanTask
+from ..span.task import SpanTaskLabelCheck
+from ..templates import read_template
+
+DEFAULT_NER_TEMPLATE_V1 = read_template("ner.v1")
+DEFAULT_NER_TEMPLATE_V2 = read_template("ner.v2")
+DEFAULT_NER_TEMPLATE_V3 = read_template("ner.v3")
+
+
+class NERTask(SpanTask):
+    def __init__(
+        self,
+        labels: List[str],
+        template: str,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        label_definitions: Optional[Dict[str, str]],
+        prompt_examples: Optional[List[FewshotExample]],
+        normalizer: Optional[Callable[[str], str]],
+        alignment_mode: Literal["strict", "contract", "expand"],
+        case_sensitive_matching: bool,
+        single_match: bool,
+        scorer: Scorer,
+        description: Optional[str],
+        check_label_consistency: SpanTaskLabelCheck[Self],
+    ):
+        """Default NER task.
+
+        labels (List[str]): List of labels to pass to the template.
+            Leave empty to populate it at initialization time (only if examples are provided).
+        template (str): Prompt template passed to the model.
+        parse_responses (TaskResponseParser[SpanTask]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        label_definitions (Optional[Dict[str, str]]): Map of label -> description
+            of the label to help the language model output the entities wanted.
+            It is usually easier to provide these definitions rather than
+            full examples, although both can be provided.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+        alignment_mode (str): "strict", "contract" or "expand".
+        case_sensitive_matching (bool): Whether to search without case sensitivity.
+        single_match (bool): If False, allow one substring to match multiple times in
+            the text. If True, returns the first hit.
+        scorer (Scorer): Scorer function.
+        description (str): A description of what to recognize or not recognize as entities.
+        check_label_consistency (SpanTaskLabelCheck): Callable to check label consistency.
+        """
+        super().__init__(
+            labels=labels,
+            template=template,
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            label_definitions=label_definitions,
+            prompt_examples=prompt_examples,
+            normalizer=normalizer,
+            alignment_mode=alignment_mode,
+            case_sensitive_matching=case_sensitive_matching,
+            single_match=single_match,
+            description=description,
+            allow_overlap=False,
+            check_label_consistency=check_label_consistency,
+        )
+        self._scorer = scorer
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        labels: List[str] = [],
+        n_prompt_examples: int = 0,
+    ) -> None:
+        super()._initialize(
+            get_examples=get_examples,
+            nlp=nlp,
+            labels=labels,
+            n_prompt_examples=n_prompt_examples,
+        )
+
+    def assign_spans(
+        self,
+        doc: Doc,
+        spans: List[Span],
+    ) -> None:
+        """Assign spans to the document."""
+        doc.set_ents(filter_spans(spans))
+
+    def scorer(self, examples: Iterable[Example]) -> Dict[str, Any]:
+        return self._scorer(examples)
+
+    def _extract_labels_from_example(self, example: Example) -> List[str]:
+        return [ent.label_ for ent in example.reference.ents]
diff --git a/spacy_llm/tasks/ner/util.py b/spacy_llm/tasks/ner/util.py
new file mode 100644
index 00000000..86072f90
--- /dev/null
+++ b/spacy_llm/tasks/ner/util.py
@@ -0,0 +1,36 @@
+from collections import defaultdict
+from typing import Any, Dict, Iterable
+
+from spacy.scorer import get_ner_prf
+from spacy.training import Example
+
+from ...compat import Self
+from ..span import SpanExample
+from ..span.examples import SpanCoTExample
+
+
+class NERExample(SpanExample):
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        entities = defaultdict(list)
+        for ent in example.reference.ents:
+            entities[ent.label_].append(ent.text)
+
+        return cls(text=example.reference.text, entities=entities)
+
+
+class NERCoTExample(SpanCoTExample):
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        return cls(
+            text=example.reference.text,
+            spans=SpanCoTExample._extract_span_reasons(example.reference.ents),
+        )
+
+
+def score(examples: Iterable[Example], **kwargs) -> Dict[str, Any]:
+    """Score NER accuracy in examples.
+    examples (Iterable[Example]): Examples to score.
+    RETURNS (Dict[str, Any]): Dict with metric name -> score.
+    """
+    return get_ner_prf(examples)
diff --git a/spacy_llm/tasks/rel.py b/spacy_llm/tasks/rel.py
deleted file mode 100644
index 85884da9..00000000
--- a/spacy_llm/tasks/rel.py
+++ /dev/null
@@ -1,268 +0,0 @@
-from typing import Callable, Dict, Iterable, List, Optional, Tuple, Type, Union
-
-import jinja2
-
-try:
-    from pydantic.v1 import BaseModel, Field, ValidationError, validator
-except ImportError:
-    from pydantic import BaseModel, Field, ValidationError, validator
-
-from spacy.language import Language
-from spacy.tokens import Doc
-from spacy.training import Example
-from wasabi import msg
-
-from ..registry import lowercase_normalizer, registry
-from ..ty import ExamplesConfigType
-from ..util import split_labels
-from .templates import read_template
-from .util import SerializableTask
-
-
-class RelationItem(BaseModel):
-    dep: int
-    dest: int
-    relation: str
-
-    @validator("dep", "dest", pre=True)
-    def clean_ent(cls, value):
-        if isinstance(value, str):
-            value = value.strip("ENT")
-        return value
-
-
-class EntityItem(BaseModel):
-    start_char: int
-    end_char: int
-    label_: str = Field(alias="label")
-
-
-class RELExample(BaseModel):
-    text: str
-    ents: List[EntityItem]
-    relations: List[RelationItem]
-
-
-def _preannotate(doc: Union[Doc, RELExample]) -> str:
-    """Creates a text version of the document with annotated entities."""
-    offset = 0
-
-    text = doc.text
-
-    for i, ent in enumerate(doc.ents):
-        end = ent.end_char
-        before, after = text[: end + offset], text[end + offset :]
-
-        annotation = f"[ENT{i}:{ent.label_}]"
-        offset += len(annotation)
-
-        text = f"{before}{annotation}{after}"
-
-    return text
-
-
-_DEFAULT_REL_TEMPLATE = read_template("rel")
-
-
-@registry.llm_tasks("spacy.REL.v1")
-def make_rel_task(
-    labels: Union[List[str], str] = [],
-    template: str = _DEFAULT_REL_TEMPLATE,
-    label_definitions: Optional[Dict[str, str]] = None,
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    verbose: bool = False,
-) -> "RELTask":
-    """REL.v1 task factory.
-
-    The REL task populates a `Doc._.rel` custom attribute.
-
-    labels (List[str]): List of labels to pass to the template,
-        either an actual list or a comma-separated string.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Map of label -> description
-        of the label to help the language model output the entities wanted.
-        It is usually easier to provide these definitions rather than
-        full examples, although both can be provided.
-    examples (Optional[Callable[[], List[RELExample]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-    verbose (bool): Controls the verbosity of the task.
-    """
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    rel_examples = [RELExample(**eg) for eg in raw_examples] if raw_examples else None
-    return RELTask(
-        labels=labels_list,
-        template=template,
-        label_definitions=label_definitions,
-        prompt_examples=rel_examples,
-        normalizer=normalizer,
-        verbose=verbose,
-    )
-
-
-class RELTask(SerializableTask[RELExample]):
-    def __init__(
-        self,
-        labels: List[str] = [],
-        template: str = _DEFAULT_REL_TEMPLATE,
-        label_definitions: Optional[Dict[str, str]] = None,
-        prompt_examples: Optional[List[RELExample]] = None,
-        normalizer: Optional[Callable[[str], str]] = None,
-        verbose: bool = False,
-    ):
-        """Default REL task. Populates a `Doc._.rel` custom attribute.
-
-        labels (List[str]): List of labels to pass to the template.
-            Leave empty to populate it at initialization time (only if examples are provided).
-        template (str): Prompt template passed to the model.
-        label_definitions (Optional[Dict[str, str]]): Map of label -> description
-            of the label to help the language model output the entities wanted.
-            It is usually easier to provide these definitions rather than
-            full examples, although both can be provided.
-        prompt_examples (Optional[Callable[[], List[RELExample]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-        verbose (bool): Controls the verbosity of the task.
-        """
-        self._normalizer = normalizer if normalizer else lowercase_normalizer()
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-        self._template = template
-        self._label_definitions = label_definitions
-        self._prompt_examples = prompt_examples or []
-        self._verbose = verbose
-
-    @classmethod
-    def _check_rel_extension(cls):
-        """Add `rel` extension if need be."""
-        if not Doc.has_extension("rel"):
-            Doc.set_extension("rel", default=[])
-
-    @property
-    def labels(self) -> Tuple[str, ...]:
-        return tuple(self._label_dict.values())
-
-    @property
-    def prompt_template(self) -> str:
-        return self._template
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=_preannotate(doc),
-                labels=list(self._label_dict.values()),
-                label_definitions=self._label_definitions,
-                examples=self._prompt_examples,
-                preannotate=_preannotate,
-            )
-            yield prompt
-
-    def _format_response(self, response: str, doc: Doc) -> List[RelationItem]:
-        """Parse raw string response into a structured format"""
-        relations = []
-        for line in response.strip().split("\n"):
-            try:
-                rel_item = RelationItem.parse_raw(line)
-                if 0 <= rel_item.dep < len(doc.ents) and 0 <= rel_item.dest < len(
-                    doc.ents
-                ):
-                    relations.append(rel_item)
-            except ValidationError:
-                msg.warn(
-                    "Validation issue",
-                    line,
-                    show=self._verbose,
-                )
-        return relations
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        self._check_rel_extension()
-
-        for doc, prompt_response in zip(docs, responses):
-            rels = self._format_response(prompt_response, doc)
-            doc._.rel = rels
-            yield doc
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        labels: List[str] = [],
-        n_prompt_examples: int = 0,
-    ) -> None:
-        """Initialize the SpanCat task, by auto-discovering labels.
-
-        Labels can be set through, by order of precedence:
-
-        - the `[initialize]` section of the pipeline configuration
-        - the `labels` argument supplied to the task factory
-        - the labels found in the examples
-
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        labels (List[str]): Optional list of labels.
-        n_prompt_examples (int): How many prompt examples to infer from the Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        self._check_rel_extension()
-
-        if not labels:
-            labels = list(self._label_dict.values())
-        infer_labels = not labels
-
-        if infer_labels:
-            labels = []
-
-        for eg in get_examples():
-            if infer_labels:
-                rels: List[RelationItem] = eg.reference._.rel
-                for rel in rels:
-                    labels.append(rel.relation)
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(self._create_prompt_example(eg))
-
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return [
-            "_label_dict",
-            "_template",
-            "_label_definitions",
-            "_verbose",
-        ]
-
-    @property
-    def _Example(self) -> Type[RELExample]:
-        return RELExample
-
-    def _create_prompt_example(self, example: Example) -> RELExample:
-        """Create a REL prompt example from a spaCy example."""
-        entities = [
-            EntityItem(
-                start_char=ent.start_char,
-                end_char=ent.end_char,
-                label=ent.label_,
-            )
-            for ent in example.reference.ents
-        ]
-
-        rel_example = RELExample(
-            text=example.reference.text,
-            ents=entities,
-            relations=example.reference._.rel,
-        )
-        return rel_example
diff --git a/spacy_llm/tasks/rel/__init__.py b/spacy_llm/tasks/rel/__init__.py
new file mode 100644
index 00000000..324126a8
--- /dev/null
+++ b/spacy_llm/tasks/rel/__init__.py
@@ -0,0 +1,11 @@
+from .registry import make_rel_task
+from .task import DEFAULT_REL_TEMPLATE, RELTask
+from .util import RelationItem, RELExample
+
+__all__ = [
+    "DEFAULT_REL_TEMPLATE",
+    "make_rel_task",
+    "RelationItem",
+    "RELExample",
+    "RELTask",
+]
diff --git a/spacy_llm/tasks/rel/parser.py b/spacy_llm/tasks/rel/parser.py
new file mode 100644
index 00000000..890a6aac
--- /dev/null
+++ b/spacy_llm/tasks/rel/parser.py
@@ -0,0 +1,36 @@
+from typing import Iterable, List
+
+from spacy.tokens import Doc
+from wasabi import msg
+
+from ...compat import ValidationError
+from .task import RELTask
+from .util import RelationItem
+
+
+def parse_responses_v1(
+    task: RELTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[List[RelationItem]]:
+    """Parses LLM responses for spacy.REL.v1.
+    task (RELTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Iterable[List[RelationItem]]): List of RelationItem instances per doc/response.
+    """
+    for response, doc in zip(responses, docs):
+        relations: List[RelationItem] = []
+        for line in response.strip().split("\n"):
+            try:
+                rel_item = RelationItem.parse_raw(line)
+                if 0 <= rel_item.dep < len(doc.ents) and 0 <= rel_item.dest < len(
+                    doc.ents
+                ):
+                    relations.append(rel_item)
+            except ValidationError:
+                msg.warn(
+                    "Validation issue",
+                    line,
+                    show=task.verbose,
+                )
+
+        yield relations
diff --git a/spacy_llm/tasks/rel/registry.py b/spacy_llm/tasks/rel/registry.py
new file mode 100644
index 00000000..17207629
--- /dev/null
+++ b/spacy_llm/tasks/rel/registry.py
@@ -0,0 +1,55 @@
+from typing import Callable, Dict, List, Optional, Type, Union
+
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, TaskResponseParser
+from ...util import split_labels
+from .parser import parse_responses_v1
+from .task import DEFAULT_REL_TEMPLATE, RELTask
+from .util import RELExample
+
+
+@registry.llm_tasks("spacy.REL.v1")
+def make_rel_task(
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_REL_TEMPLATE,
+    parse_responses: Optional[TaskResponseParser[RELTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    verbose: bool = False,
+) -> "RELTask":
+    """REL.v1 task factory.
+
+    The REL task populates a `Doc._.rel` custom attribute.
+
+    labels (List[str]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    parse_responses (Optional[TaskResponseParser[RELTask]]): Callable for parsing LLM responses for this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    label_definitions (Optional[Dict[str, str]]): Map of label -> description
+        of the label to help the language model output the entities wanted.
+        It is usually easier to provide these definitions rather than
+        full examples, although both can be provided.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+    verbose (bool): Controls the verbosity of the task.
+    """
+    labels_list = split_labels(labels)
+    example_type = prompt_example_type or RELExample
+    raw_examples = examples() if callable(examples) else examples
+    rel_examples = [example_type(**eg) for eg in raw_examples] if raw_examples else None
+
+    return RELTask(
+        parse_responses=parse_responses or parse_responses_v1,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=rel_examples,
+        normalizer=normalizer,
+        verbose=verbose,
+    )
diff --git a/spacy_llm/tasks/rel/task.py b/spacy_llm/tasks/rel/task.py
new file mode 100644
index 00000000..fe9f6e4e
--- /dev/null
+++ b/spacy_llm/tasks/rel/task.py
@@ -0,0 +1,123 @@
+from typing import Callable, Dict, Iterable, List, Optional, Type, Union
+
+from spacy.language import Language
+from spacy.tokens import Doc
+from spacy.training import Example
+
+from ...compat import Self
+from ...ty import FewshotExample, TaskResponseParser
+from ..builtin_task import BuiltinTaskWithLabels
+from ..templates import read_template
+from .util import RelationItem, RELExample
+
+DEFAULT_REL_TEMPLATE: str = read_template("rel.v1")
+
+
+class RELTask(BuiltinTaskWithLabels):
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        labels: List[str],
+        template: str,
+        label_definitions: Optional[Dict[str, str]],
+        prompt_examples: Optional[List[FewshotExample]],
+        normalizer: Optional[Callable[[str], str]],
+        verbose: bool,
+    ):
+        """Default REL task. Populates a `Doc._.rel` custom attribute.
+
+        parse_responses (TaskResponseParser[Self]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        labels (List[str]): List of labels to pass to the template.
+            Leave empty to populate it at initialization time (only if examples are provided).
+        template (str): Prompt template passed to the model.
+        label_definitions (Optional[Dict[str, str]]): Map of label -> description
+            of the label to help the language model output the entities wanted.
+            It is usually easier to provide these definitions rather than
+            full examples, although both can be provided.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+        verbose (bool): Controls the verbosity of the task.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+            labels=labels,
+            label_definitions=label_definitions,
+            normalizer=normalizer,
+        )
+        self._verbose = verbose
+        self._field = "rel"
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[str]:
+        return super().generate_prompts(
+            docs=[
+                Doc(doc.vocab, words=RELTask._preannotate(doc).split()) for doc in docs
+            ],
+            labels=list(self._label_dict.values()),
+            label_definitions=self._label_definitions,
+            preannotate=RELTask._preannotate,
+        )
+
+    @staticmethod
+    def _preannotate(doc: Union[Doc, RELExample]) -> str:
+        """Creates a text version of the document with annotated entities."""
+        offset = 0
+        text = doc.text
+
+        for i, ent in enumerate(doc.ents):
+            end = ent.end_char
+            before, after = text[: end + offset], text[end + offset :]
+            annotation = f"[ENT{i}:{ent.label}]"
+            offset += len(annotation)
+            text = f"{before}{annotation}{after}"
+
+        return text
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        self._check_extension(self._field)
+
+        for doc, rel_items in zip(docs, self._parse_responses(self, docs, responses)):
+            doc._.rel = rel_items
+            yield doc
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        labels: List[str] = [],
+        n_prompt_examples: int = 0,
+    ) -> None:
+        self._check_extension(self._field)
+        super()._initialize(
+            get_examples=get_examples,
+            nlp=nlp,
+            labels=labels,
+            n_prompt_examples=n_prompt_examples,
+        )
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return [
+            "_label_dict",
+            "_template",
+            "_label_definitions",
+            "_verbose",
+        ]
+
+    def _extract_labels_from_example(self, example: Example) -> List[str]:
+        rels: List[RelationItem] = example.reference._.rel
+        return [rel.relation for rel in rels]
+
+    @property
+    def verbose(self) -> bool:
+        return self._verbose
+
+    @property
+    def field(self) -> str:
+        return self._field
diff --git a/spacy_llm/tasks/rel/util.py b/spacy_llm/tasks/rel/util.py
new file mode 100644
index 00000000..72815490
--- /dev/null
+++ b/spacy_llm/tasks/rel/util.py
@@ -0,0 +1,49 @@
+from typing import List
+
+from spacy.training import Example
+
+from ...compat import BaseModel, Self, validator
+from ...ty import FewshotExample
+
+
+class RelationItem(BaseModel):
+    dep: int
+    dest: int
+    relation: str
+
+    @validator("dep", "dest", pre=True)
+    def clean_ent(cls, value):
+        if isinstance(value, str):
+            value = value.strip("ENT")
+        return value
+
+
+class EntityItem(BaseModel):
+    start_char: int
+    end_char: int
+    label: str
+
+
+class RELExample(FewshotExample):
+    text: str
+    ents: List[EntityItem]
+    relations: List[RelationItem]
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        entities = [
+            EntityItem(
+                start_char=ent.start_char,
+                end_char=ent.end_char,
+                label=ent.label_,
+            )
+            for ent in example.reference.ents
+        ]
+        return cls(
+            text=example.reference.text,
+            ents=entities,
+            relations=example.reference._.rel,
+        )
diff --git a/spacy_llm/tasks/sentiment.py b/spacy_llm/tasks/sentiment.py
deleted file mode 100644
index eeb75b68..00000000
--- a/spacy_llm/tasks/sentiment.py
+++ /dev/null
@@ -1,141 +0,0 @@
-from typing import Any, Callable, Dict, Iterable, List, Optional, Type
-
-import jinja2
-from pydantic import BaseModel
-from spacy.language import Language
-from spacy.tokens import Doc
-from spacy.training import Example
-
-from ..registry import registry
-from ..ty import ExamplesConfigType
-from .templates import read_template
-from .util import SerializableTask
-from .util.serialization import ExampleType
-
-_DEFAULT_SENTIMENT_TEMPLATE_V1 = read_template("sentiment")
-
-
-class SentimentExample(BaseModel):
-    text: str
-    score: float
-
-
-@registry.llm_tasks("spacy.Sentiment.v1")
-def make_sentiment_task(
-    template: str = _DEFAULT_SENTIMENT_TEMPLATE_V1,
-    examples: ExamplesConfigType = None,
-    field: str = "sentiment",
-):
-    """Sentiment.v1 task factory.
-
-    template (str): Prompt template passed to the model.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    field (str): The name of the doc extension in which to store the summary.
-    """
-    raw_examples = examples() if callable(examples) else examples
-    sentiment_examples = (
-        [SentimentExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-
-    return SentimentTask(template=template, examples=sentiment_examples, field=field)
-
-
-class SentimentTask(SerializableTask[SentimentExample]):
-    def __init__(
-        self,
-        template: str = _DEFAULT_SENTIMENT_TEMPLATE_V1,
-        examples: Optional[List[SentimentExample]] = None,
-        field: str = "sentiment",
-    ):
-        """Sentiment analysis task.
-
-        template (str): Prompt template passed to the model.
-        examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        field (str): The name of the doc extension in which to store the summary.
-        """
-        self._template = template
-        self._prompt_examples = examples or []
-        self._field = field
-        self._check_doc_extension()
-
-    def _check_doc_extension(self):
-        """Add extension if need be."""
-        if not Doc.has_extension(self._field):
-            Doc.set_extension(self._field, default=None)
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initialize sentiment task.
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        labels (List[str]): Optional list of labels.
-        n_prompt_examples (int): How many prompt examples to infer from the provided Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        self._check_doc_extension()
-
-        for eg in get_examples():
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(
-                    SentimentExample(
-                        text=eg.reference.text,
-                        score=getattr(eg.reference._, self._field),
-                    )
-                )
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=doc.text,
-                examples=self._prompt_examples,
-            )
-            yield prompt
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        self._check_doc_extension()
-
-        for doc, prompt_response in zip(docs, responses):
-            try:
-                setattr(
-                    doc._,
-                    self._field,
-                    float(
-                        "".join(prompt_response.replace("Answer:", "").strip().split())
-                    ),
-                )
-            except ValueError:
-                setattr(doc._, self._field, None)
-
-            yield doc
-
-    def scorer(
-        self,
-        examples: Iterable[Example],
-    ) -> Dict[str, Any]:
-        """Scores lemmatization accuracy on provided examples.
-        examples (Iterable[Example]): Examples to determine score against.
-        """
-        # todo
-        return {"sentiment_diff": 0}
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return ["_template"]
-
-    @property
-    def _Example(self) -> Type[ExampleType]:
-        return SentimentExample
diff --git a/spacy_llm/tasks/sentiment/__init__.py b/spacy_llm/tasks/sentiment/__init__.py
new file mode 100644
index 00000000..39876947
--- /dev/null
+++ b/spacy_llm/tasks/sentiment/__init__.py
@@ -0,0 +1,5 @@
+from .registry import make_sentiment_task
+from .task import SentimentTask
+from .util import SentimentExample
+
+__all__ = ["make_sentiment_task", "SentimentExample", "SentimentTask"]
diff --git a/spacy_llm/tasks/sentiment/parser.py b/spacy_llm/tasks/sentiment/parser.py
new file mode 100644
index 00000000..52082742
--- /dev/null
+++ b/spacy_llm/tasks/sentiment/parser.py
@@ -0,0 +1,22 @@
+from typing import Iterable, Optional
+
+from spacy.tokens import Doc
+
+from .task import SentimentTask
+
+
+def parse_responses_v1(
+    task: SentimentTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[Optional[float]]:
+    """Parses LLM responses for spacy.Sentiment.v1.
+    task (SentimentTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    field (str): Field to store responses in.
+    RETURNS (Iterable[Optional[float]]): Sentiment score per doc/response. None on parsing error.
+    """
+    for prompt_response in responses:
+        try:
+            yield float("".join(prompt_response.replace("Answer:", "").strip().split()))
+        except ValueError:
+            yield None
diff --git a/spacy_llm/tasks/sentiment/registry.py b/spacy_llm/tasks/sentiment/registry.py
new file mode 100644
index 00000000..13f248c0
--- /dev/null
+++ b/spacy_llm/tasks/sentiment/registry.py
@@ -0,0 +1,40 @@
+from typing import Optional, Type
+
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, TaskResponseParser
+from .parser import parse_responses_v1
+from .task import DEFAULT_SENTIMENT_TEMPLATE_V1, SentimentTask
+from .util import SentimentExample
+
+
+@registry.llm_tasks("spacy.Sentiment.v1")
+def make_sentiment_task(
+    template: str = DEFAULT_SENTIMENT_TEMPLATE_V1,
+    parse_responses: Optional[TaskResponseParser[SentimentTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    examples: ExamplesConfigType = None,
+    field: str = "sentiment",
+):
+    """Sentiment.v1 task factory.
+
+    template (str): Prompt template passed to the model.
+    parse_responses (Optional[TaskResponseParser[SentimentTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    field (str): The name of the doc extension in which to store the summary.
+    """
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or SentimentExample
+    sentiment_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return SentimentTask(
+        template=template,
+        parse_responses=parse_responses or parse_responses_v1,
+        prompt_example_type=example_type,
+        prompt_examples=sentiment_examples,
+        field=field,
+    )
diff --git a/spacy_llm/tasks/sentiment/task.py b/spacy_llm/tasks/sentiment/task.py
new file mode 100644
index 00000000..54c82572
--- /dev/null
+++ b/spacy_llm/tasks/sentiment/task.py
@@ -0,0 +1,88 @@
+from typing import Callable, Iterable, List, Optional, Type
+
+from spacy.language import Language
+from spacy.tokens import Doc
+from spacy.training import Example
+
+from ...ty import FewshotExample, Self, TaskResponseParser
+from ..builtin_task import BuiltinTask
+from ..templates import read_template
+from .util import SentimentExample
+
+DEFAULT_SENTIMENT_TEMPLATE_V1 = read_template("sentiment.v1")
+
+
+class SentimentTask(BuiltinTask):
+    def __init__(
+        self,
+        template: str,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        field: str,
+        prompt_examples: Optional[List[SentimentExample]],
+    ):
+        """Sentiment analysis task.
+
+        template (str): Prompt template passed to the model.
+        parse_responses (TaskResponseParser[Self]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        field (str): The name of the doc extension in which to store the sentiment score.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+        )
+        self._field = field
+        self._check_doc_extension()
+
+    def _check_doc_extension(self):
+        """Add extension if need be."""
+        if not Doc.has_extension(self._field):
+            Doc.set_extension(self._field, default=None)
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        n_prompt_examples: int = 0,
+    ) -> None:
+        """Initialize sentiment task.
+        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
+            for initialization.
+        nlp (Language): Language instance.
+        n_prompt_examples (int): How many prompt examples to infer from the provided Example objects.
+            0 by default. Takes all examples if set to -1.
+        """
+        self._check_doc_extension()
+        super()._initialize(
+            get_examples=get_examples, nlp=nlp, n_prompt_examples=n_prompt_examples
+        )
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[str]:
+        return super().generate_prompts(docs=docs)
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        self._check_doc_extension()
+
+        for doc, sentiment_score in zip(
+            docs, self._parse_responses(self, docs, responses)
+        ):
+            try:
+                setattr(doc._, self._field, sentiment_score)
+            except ValueError:
+                setattr(doc._, self._field, None)
+
+            yield doc
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return ["_template"]
+
+    @property
+    def field(self) -> str:
+        return self._field
diff --git a/spacy_llm/tasks/sentiment/util.py b/spacy_llm/tasks/sentiment/util.py
new file mode 100644
index 00000000..f718ff57
--- /dev/null
+++ b/spacy_llm/tasks/sentiment/util.py
@@ -0,0 +1,16 @@
+from spacy.training import Example
+
+from ...compat import Self
+from ...ty import FewshotExample
+
+
+class SentimentExample(FewshotExample):
+    text: str
+    score: float
+
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        return cls(
+            text=example.reference.text,
+            score=getattr(example.reference._, kwargs["field"]),
+        )
diff --git a/spacy_llm/tasks/span.py b/spacy_llm/tasks/span.py
deleted file mode 100644
index 708e1565..00000000
--- a/spacy_llm/tasks/span.py
+++ /dev/null
@@ -1,183 +0,0 @@
-import warnings
-from typing import Callable, Dict, Iterable, List, Optional, Tuple, Type
-
-import jinja2
-from pydantic import BaseModel
-from spacy.tokens import Doc, Span
-
-from ..compat import Literal
-from ..registry import lowercase_normalizer
-from .util.parsing import find_substrings
-from .util.serialization import SerializableTask
-
-
-class SpanExample(BaseModel):
-    text: str
-    entities: Dict[str, List[str]]
-
-
-class SpanTask(SerializableTask[SpanExample]):
-    """Base class for Span-related tasks, eg NER and SpanCat."""
-
-    def __init__(
-        self,
-        labels: List[str],
-        template: str,
-        label_definitions: Optional[Dict[str, str]] = {},
-        prompt_examples: Optional[List[SpanExample]] = None,
-        normalizer: Optional[Callable[[str], str]] = None,
-        alignment_mode: Literal[
-            "strict", "contract", "expand"  # noqa: F821
-        ] = "contract",
-        case_sensitive_matching: bool = False,
-        single_match: bool = False,
-    ):
-        self._normalizer = normalizer if normalizer else lowercase_normalizer()
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-        self._template = template
-        self._label_definitions = label_definitions
-        self._prompt_examples = prompt_examples or []
-        self._validate_alignment(alignment_mode)
-        self._alignment_mode = alignment_mode
-        self._case_sensitive_matching = case_sensitive_matching
-        self._single_match = single_match
-
-        if self._prompt_examples:
-            self._prompt_examples = self._check_label_consistency()
-
-    def _check_label_consistency(self) -> List[SpanExample]:
-        """Checks consistency of labels between examples and defined labels. Emits warning on inconsistency.
-        RETURNS (List[SpanExample]): List of SpanExamples with valid labels.
-        """
-        assert self._prompt_examples
-        example_labels = {
-            self._normalizer(key): key
-            for example in self._prompt_examples
-            for key in example.entities
-        }
-        unspecified_labels = {
-            example_labels[key]
-            for key in (set(example_labels.keys()) - set(self._label_dict.keys()))
-        }
-        if not set(example_labels.keys()) <= set(self._label_dict.keys()):
-            warnings.warn(
-                f"Examples contain labels that are not specified in the task configuration. The latter contains the "
-                f"following labels: {sorted(list(set(self._label_dict.values())))}. Labels in examples missing from "
-                f"the task configuration: {sorted(list(unspecified_labels))}. Please ensure your label specification "
-                f"and example labels are consistent."
-            )
-
-        # Return examples without non-declared labels. If an example only has undeclared labels, it is discarded.
-        return [
-            example
-            for example in [
-                SpanExample(
-                    text=example.text,
-                    entities={
-                        label: entities
-                        for label, entities in example.entities.items()
-                        if self._normalizer(label) in self._label_dict
-                    },
-                )
-                for example in self._prompt_examples
-            ]
-            if len(example.entities)
-        ]
-
-    @property
-    def labels(self) -> Tuple[str, ...]:
-        return tuple(self._label_dict.values())
-
-    @property
-    def prompt_template(self) -> str:
-        return self._template
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=doc.text,
-                labels=list(self._label_dict.values()),
-                label_definitions=self._label_definitions,
-                examples=self._prompt_examples,
-            )
-            yield prompt
-
-    def _format_response(self, response: str) -> Iterable[Tuple[str, Iterable[str]]]:
-        """Parse raw string response into a structured format"""
-        output = []
-        assert self._normalizer is not None
-        for line in response.strip().split("\n"):
-            # Check if the formatting we want exists
-            # <entity label>: ent1, ent2
-            if line and ":" in line:
-                label, phrases = line.split(":", 1)
-                norm_label = self._normalizer(label)
-                if norm_label in self._label_dict:
-                    # Get the phrases / spans for each entity
-                    if phrases.strip():
-                        _phrases = [p.strip() for p in phrases.strip().split(",")]
-                        output.append((self._label_dict[norm_label], _phrases))
-        return output
-
-    @staticmethod
-    def _validate_alignment(alignment_mode: str):
-        """Raises error if specified alignment_mode is not supported.
-        alignment_mode (str): Alignment mode to check.
-        """
-        # ideally, this list should be taken from spaCy, but it's not currently exposed from doc.pyx.
-        alignment_modes = ("strict", "contract", "expand")
-        if alignment_mode not in alignment_modes:
-            raise ValueError(
-                f"Unsupported alignment mode '{alignment_mode}'. Supported modes: {', '.join(alignment_modes)}"
-            )
-
-    def assign_spans(
-        self,
-        doc: Doc,
-        spans: List[Span],
-    ) -> None:
-        """Assign spans to the document."""
-        raise NotImplementedError()
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        for doc, prompt_response in zip(docs, responses):
-            spans = []
-            for label, phrases in self._format_response(prompt_response):
-                # For each phrase, find the substrings in the text
-                # and create a Span
-                offsets = find_substrings(
-                    doc.text,
-                    phrases,
-                    case_sensitive=self._case_sensitive_matching,
-                    single_match=self._single_match,
-                )
-                for start, end in offsets:
-                    span = doc.char_span(
-                        start, end, alignment_mode=self._alignment_mode, label=label
-                    )
-                    if span is not None:
-                        spans.append(span)
-
-            self.assign_spans(doc, spans)
-            yield doc
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return [
-            "_label_dict",
-            "_template",
-            "_label_definitions",
-            "_alignment_mode",
-            "_case_sensitive_matching",
-            "_single_match",
-        ]
-
-    @property
-    def _Example(self) -> Type[SpanExample]:
-        return SpanExample
diff --git a/spacy_llm/tasks/span/__init__.py b/spacy_llm/tasks/span/__init__.py
new file mode 100644
index 00000000..39852311
--- /dev/null
+++ b/spacy_llm/tasks/span/__init__.py
@@ -0,0 +1,14 @@
+from .examples import SpanExample, SpanReason
+from .parser import parse_responses, parse_responses_cot
+from .registry import make_label_check, make_label_check_cot
+from .task import SpanTask
+
+__all__ = [
+    "make_label_check",
+    "make_label_check_cot",
+    "parse_responses",
+    "parse_responses_cot",
+    "SpanExample",
+    "SpanReason",
+    "SpanTask",
+]
diff --git a/spacy_llm/tasks/span/examples.py b/spacy_llm/tasks/span/examples.py
new file mode 100644
index 00000000..7ed43882
--- /dev/null
+++ b/spacy_llm/tasks/span/examples.py
@@ -0,0 +1,89 @@
+import abc
+from typing import Dict, Iterable, List
+
+from spacy.tokens import Span
+
+from ...compat import BaseModel
+from ...ty import FewshotExample, Self
+
+
+class SpanExample(FewshotExample, abc.ABC):
+    text: str
+    entities: Dict[str, List[str]]
+
+
+class SpanReason(BaseModel):
+    text: str
+    is_entity: bool
+    label: str
+    reason: str
+
+    @classmethod
+    def from_str(cls, line: str, sep: str = "|") -> Self:
+        """Parse a single line of LLM output which identifies a span of text,
+        whether it's an entity, a label to assign to it, and a reason for that
+        assignment.
+
+        e.g. expected line would look like:
+        1. Golden State Warriors | True | BASKETBALL_TEAM | is a basketball team in the NBA
+
+        Handles an optional numbered list which we put in the prompt by default so the LLM
+        can better structure the order of output for the spans. This number isn't crucial for
+        the final parsing so we just strip it for now.
+
+        line (str): Line of LLM output to parse
+        sep (str): Optional separator to split on. Defaults to "|".
+
+        RETURNS (Self)
+        """
+        clean_str = line.strip()
+        if "." in clean_str:
+            clean_str = clean_str.split(".", maxsplit=1)[1]
+        components = [c.strip() for c in clean_str.split(sep)]
+        if len(components) != 4:
+            raise ValueError(
+                "Unable to parse line of LLM output into a SpanReason. ",
+                f"line: {line}",
+            )
+        return cls(
+            text=components[0],
+            is_entity=components[1].lower() == "true",
+            label=components[2],
+            reason=components[3],
+        )
+
+    def to_str(self, sep: str = "|") -> str:
+        """Output as a single line of text representing the expected LLM COT output
+        e.g.
+        'Golden State Warriors | True | BASKETBALL_TEAM | is a basketball team in the NBA'
+        """
+        return (
+            f"{self.text} {sep} {self.is_entity} {sep} {self.label} {sep} {self.reason}"
+        )
+
+    def __str__(self) -> str:
+        return self.to_str()
+
+
+class SpanCoTExample(FewshotExample, abc.ABC):
+    text: str
+    spans: List[SpanReason]
+
+    @staticmethod
+    def _extract_span_reasons(spans: Iterable[Span]) -> List[SpanReason]:
+        """Extracts SpanReasons from spans.
+        spans (Iterable[Span]): Spans to extract reasons from.
+        RETURNS (List[SpanReason]): SpanReason instances extracted from example.
+        """
+        span_reasons: List[SpanReason] = []
+        for span in spans:
+            span_reasons.append(
+                SpanReason(
+                    text=span.text,
+                    is_entity=True,
+                    label=span.label_,
+                    reason=f"is a {span.label_}",
+                )
+            )
+
+        return span_reasons
diff --git a/spacy_llm/tasks/span/parser.py b/spacy_llm/tasks/span/parser.py
new file mode 100644
index 00000000..467dcbbc
--- /dev/null
+++ b/spacy_llm/tasks/span/parser.py
@@ -0,0 +1,170 @@
+from typing import Callable, Dict, Iterable, List, Tuple
+
+from spacy.tokens import Doc, Span
+
+from ...tasks.util import find_substrings
+from .examples import SpanReason
+from .task import SpanTask
+
+
+def _format_response(
+    response: str, normalizer: Callable[[str], str], label_dict: Dict[str, str]
+) -> Iterable[Tuple[str, Iterable[str]]]:
+    """Parse raw string response into a structured format.
+    response (str): LLM response.
+    normalizer (Callable[[str], str]): normalizer function.
+    label_dict (Dict[str, str]): Mapping of normalized to non-normalized labels.
+    RETURNS (Iterable[Tuple[str, Iterable[str]]]): Formatted response.
+    """
+    output = []
+    assert normalizer is not None
+
+    for line in response.strip().split("\n"):
+        # Check if the formatting we want exists
+        # <entity label>: ent1, ent2
+        if line and ":" in line:
+            label, phrases = line.split(":", 1)
+            norm_label = normalizer(label)
+            if norm_label in label_dict:
+                # Get the phrases / spans for each entity
+                if phrases.strip():
+                    _phrases = [p.strip() for p in phrases.strip().split(",")]
+                    output.append((label_dict[norm_label], _phrases))
+
+    return output
+
+
+def parse_responses(
+    task: SpanTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[List[Span]]:
+    """Parses LLM responses for Span tasks.
+    task (SpanTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Iterable[Span]): Parsed spans per doc/response.
+    """
+    for doc, prompt_response in zip(docs, responses):
+        spans = []
+        for label, phrases in _format_response(
+            prompt_response, task._normalizer, task._label_dict
+        ):
+            # For each phrase, find the substrings in the text
+            # and create a Span
+            offsets = find_substrings(
+                doc.text,
+                phrases,
+                case_sensitive=task._case_sensitive_matching,
+                single_match=task._single_match,
+            )
+            for start, end in offsets:
+                span = doc.char_span(
+                    start, end, alignment_mode=task._alignment_mode, label=label
+                )
+                if span is not None:
+                    spans.append(span)
+
+        yield spans
+
+
+def _extract_span_reasons_cot(task: SpanTask, response: str) -> List[SpanReason]:
+    """Parse raw string response into a list of SpanReasons.
+    task (SpanTask): Task to extract span reasons for.
+    response (str): Raw string response from the LLM.
+    RETURNS (List[SpanReason]): List of SpanReasons parsed from the response.
+    """
+    span_reasons = []
+    for line in response.strip().split("\n"):
+        try:
+            span_reason = SpanReason.from_str(line)
+        except ValueError:
+            continue
+        if not span_reason.is_entity:
+            continue
+        norm_label = task.normalizer(span_reason.label)
+        if norm_label not in task.label_dict:
+            continue
+        label = task.label_dict[norm_label]
+        span_reason.label = label
+        span_reasons.append(span_reason)
+    return span_reasons
+
+
+def _find_spans_cot(
+    task: SpanTask, doc: Doc, span_reasons: List[SpanReason]
+) -> List[Span]:
+    """Find a list of spaCy Spans from a list of SpanReasons
+    for a single spaCy Doc
+    task (SpanTask): Task to extract span reasons for.
+    doc (Doc): Input doc to parse spans for
+    span_reasons (List[SpanReason]): List of SpanReasons to find in doc
+    RETURNS (List[Span]): List of spaCy Spans found in doc
+    """
+    find_after = 0
+    spans = []
+    prev_span = None
+    n_span_reasons = len(span_reasons)
+    idx = 0
+    while idx < n_span_reasons:
+        span_reason = span_reasons[idx]
+
+        # For each phrase, find the SpanReason substring in the text
+        # and create a Span
+        offsets = find_substrings(
+            doc.text,
+            [span_reason.text],
+            case_sensitive=task.case_sensitive_matching,
+            single_match=True,
+            find_after=find_after,
+        )
+        if not offsets:
+            idx += 1
+            continue
+
+        # Must have exactly one match because single_match=True
+        assert len(offsets) == 1
+        start, end = offsets[0]
+
+        span = doc.char_span(
+            start,
+            end,
+            alignment_mode=task.alignment_mode,
+            label=span_reason.label,
+        )
+
+        if span is None:
+            # If we couldn't resolve a span, just skip to the next
+            # span_reason
+            idx += 1
+            continue
+
+        if span == prev_span:
+            # If the span is the same as the previous span,
+            # re-run this span_reason but look farther into the text
+            find_after = span.end_char
+            continue
+
+        spans.append(span)
+        find_after = span.start_char if task.allow_overlap else span.end_char
+        prev_span = span
+        idx += 1
+
+    return sorted(set(spans))
+
+
+def parse_responses_cot(
+    task: SpanTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[List[Span]]:
+    """Since we provide entities in a numbered list, we expect the LLM to
+    output entities in the order they occur in the text. This parse
+    function now incrementally finds substrings in the text and tracks the
+    last found span's start character to ensure we don't overwrite
+    previously found spans.
+    task (SpanTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Iterable[List[Span]]): Spans to assign per doc.
+    """
+    for doc, llm_response in zip(docs, responses):
+        span_reasons = _extract_span_reasons_cot(task, llm_response)
+        spans = _find_spans_cot(task, doc, span_reasons)
+        yield spans
diff --git a/spacy_llm/tasks/span/registry.py b/spacy_llm/tasks/span/registry.py
new file mode 100644
index 00000000..86d4080d
--- /dev/null
+++ b/spacy_llm/tasks/span/registry.py
@@ -0,0 +1,16 @@
+from typing import Callable, Iterable
+
+from ...registry import registry
+from .examples import SpanCoTExample, SpanExample
+from .task import SpanTask
+from .util import check_label_consistency, check_label_consistency_cot
+
+
+@registry.llm_misc("spacy.LabelCheck.v1")
+def make_label_check() -> Callable[[SpanTask], Iterable[SpanExample]]:
+    return check_label_consistency
+
+
+@registry.llm_misc("spacy.LabelCheckCoT.v1")
+def make_label_check_cot() -> Callable[[SpanTask], Iterable[SpanCoTExample]]:
+    return check_label_consistency_cot
diff --git a/spacy_llm/tasks/span/task.py b/spacy_llm/tasks/span/task.py
new file mode 100644
index 00000000..a7495c19
--- /dev/null
+++ b/spacy_llm/tasks/span/task.py
@@ -0,0 +1,137 @@
+import abc
+import typing
+from typing import Callable, Dict, Iterable, List, Optional, Type, Union
+
+from spacy.tokens import Doc, Span
+
+from ...compat import Literal, Protocol, Self
+from ...ty import FewshotExample, TaskResponseParser
+from ..builtin_task import BuiltinTaskWithLabels
+from .examples import SpanCoTExample, SpanExample
+
+_SpanTaskContraT = typing.TypeVar(
+    "_SpanTaskContraT", bound="SpanTask", contravariant=True
+)
+
+
+class SpanTaskLabelCheck(Protocol[_SpanTaskContraT]):
+    """Generic protocol for checking label consistency of SpanTask."""
+
+    def __call__(self, task: _SpanTaskContraT) -> Iterable[FewshotExample]:
+        ...
+
+
+class SpanTask(BuiltinTaskWithLabels, abc.ABC):
+    """Base class for Span-related tasks, eg NER and SpanCat."""
+
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        labels: List[str],
+        template: str,
+        label_definitions: Optional[Dict[str, str]],
+        prompt_examples: Optional[List[FewshotExample]],
+        description: Optional[str],
+        normalizer: Optional[Callable[[str], str]],
+        alignment_mode: Literal["strict", "contract", "expand"],  # noqa: F821
+        case_sensitive_matching: bool,
+        allow_overlap: bool,
+        single_match: bool,
+        check_label_consistency: SpanTaskLabelCheck,
+    ):
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+            labels=labels,
+            label_definitions=label_definitions,
+            normalizer=normalizer,
+        )
+
+        self._prompt_example_type = typing.cast(
+            Type[SpanExample], self._prompt_example_type
+        )
+        self._validate_alignment(alignment_mode)
+        self._alignment_mode = alignment_mode
+        self._case_sensitive_matching = case_sensitive_matching
+        self._allow_overlap = allow_overlap
+        self._single_match = single_match
+        self._check_label_consistency = check_label_consistency
+        self._description = description
+
+        if self._prompt_examples:
+            self._prompt_examples = list(self._check_label_consistency(self))
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[str]:
+        return super().generate_prompts(
+            docs=docs,
+            description=self._description,
+            labels=list(self._label_dict.values()),
+            label_definitions=self._label_definitions,
+            examples=self._prompt_examples,
+            allow_overlap=self._allow_overlap,
+            **kwargs,
+        )
+
+    @staticmethod
+    def _validate_alignment(alignment_mode: str):
+        """Raises error if specified alignment_mode is not supported.
+        alignment_mode (str): Alignment mode to check.
+        """
+        # ideally, this list should be taken from spaCy, but it's not currently exposed from doc.pyx.
+        alignment_modes = ("strict", "contract", "expand")
+        if alignment_mode not in alignment_modes:
+            raise ValueError(
+                f"Unsupported alignment mode '{alignment_mode}'. Supported modes: {', '.join(alignment_modes)}"
+            )
+
+    def assign_spans(
+        self,
+        doc: Doc,
+        spans: List[Span],
+    ) -> None:
+        """Assign spans to the document."""
+        raise NotImplementedError()
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        for doc, spans in zip(docs, self._parse_responses(self, docs, responses)):
+            self.assign_spans(doc, spans)
+            yield doc
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return [
+            "_label_dict",
+            "_template",
+            "_label_definitions",
+            "_alignment_mode",
+            "_case_sensitive_matching",
+        ]
+
+    @property
+    def alignment_mode(self) -> Literal["strict", "contract", "expand"]:  # noqa: F821
+        return self._alignment_mode
+
+    @property
+    def case_sensitive_matching(self) -> bool:
+        return self._case_sensitive_matching
+
+    @property
+    def allow_overlap(self) -> bool:
+        return self._allow_overlap
+
+    @property
+    def prompt_examples(self) -> Optional[Iterable[FewshotExample]]:
+        return self._prompt_examples
+
+    @property
+    def prompt_example_type(self) -> Union[Type[SpanExample], Type[SpanCoTExample]]:
+        return self._prompt_example_type
+
+    @property
+    def single_match(self) -> bool:
+        return self._single_match
diff --git a/spacy_llm/tasks/span/util.py b/spacy_llm/tasks/span/util.py
new file mode 100644
index 00000000..54c25443
--- /dev/null
+++ b/spacy_llm/tasks/span/util.py
@@ -0,0 +1,108 @@
+import warnings
+from typing import List
+
+from .examples import SpanCoTExample, SpanExample
+from .task import SpanTask
+
+
+def check_label_consistency(task: SpanTask) -> List[SpanExample]:
+    """Checks consistency of labels between examples and defined labels for non-CoT SpanTask. Emits warning on
+    inconsistency.
+    RETURNS (List[SpanExample]): List of SpanExamples with valid labels.
+    """
+    assert task.prompt_examples
+    assert issubclass(task.prompt_example_type, SpanExample)
+
+    example_labels = {
+        task.normalizer(key): key
+        for example in task.prompt_examples
+        for key in example.entities
+    }
+    unspecified_labels = {
+        example_labels[key]
+        for key in (set(example_labels.keys()) - set(task.label_dict.keys()))
+    }
+    if not set(example_labels.keys()) <= set(task.label_dict.keys()):
+        warnings.warn(
+            f"Examples contain labels that are not specified in the task configuration. The latter contains the "
+            f"following labels: {sorted(list(set(task.label_dict.values())))}. Labels in examples missing from "
+            f"the task configuration: {sorted(list(unspecified_labels))}. Please ensure your label specification "
+            f"and example labels are consistent."
+        )
+
+    # Return examples without non-declared labels. If an example only has undeclared labels, it is discarded.
+    return [
+        example
+        for example in [
+            task.prompt_example_type(
+                text=example.text,
+                entities={
+                    label: entities
+                    for label, entities in example.entities.items()
+                    if task.normalizer(label) in task.label_dict
+                },
+            )
+            for example in task.prompt_examples
+        ]
+        if len(example.entities)
+    ]
+
+
+def check_label_consistency_cot(task: SpanTask) -> List[SpanCoTExample]:
+    """Checks consistency of labels between examples and defined labels for CoT version of SpanTask. Emits warning on
+    inconsistency.
+    RETURNS (List[SpanExampleCoT]): List of SpanExamples with valid labels.
+    """
+    assert task.prompt_examples
+    assert issubclass(task.prompt_example_type, SpanCoTExample)
+
+    null_labels = {
+        task.normalizer(entity.label): entity.label
+        for example in task.prompt_examples
+        for entity in example.spans
+        if not entity.is_entity
+    }
+
+    if len(null_labels) > 1:
+        warnings.warn(
+            f"Negative examples contain multiple negative labels: {', '.join(null_labels.keys())}."
+        )
+
+    example_labels = {
+        task.normalizer(entity.label): entity.label
+        for example in task.prompt_examples
+        for entity in example.spans
+        if entity.is_entity
+    }
+
+    unspecified_labels = {
+        example_labels[key]
+        for key in (set(example_labels.keys()) - set(task.label_dict.keys()))
+    }
+    if not set(example_labels.keys()) <= set(task.label_dict.keys()):
+        warnings.warn(
+            f"Examples contain labels that are not specified in the task configuration. The latter contains the "
+            f"following labels: {sorted(list(set(task.label_dict.values())))}. Labels in examples missing from "
+            f"the task configuration: {sorted(list(unspecified_labels))}. Please ensure your label specification "
+            f"and example labels are consistent."
+        )
+
+    # Return examples without non-declared labels. If an example only has undeclared labels, it is discarded.
+    include_labels = dict(task.label_dict)
+    include_labels.update(null_labels)
+
+    return [
+        example
+        for example in [
+            task.prompt_example_type(
+                text=example.text,
+                spans=[
+                    entity
+                    for entity in example.spans
+                    if task.normalizer(entity.label) in include_labels
+                ],
+            )
+            for example in task.prompt_examples
+        ]
+        if len(example.spans)
+    ]
diff --git a/spacy_llm/tasks/spancat.py b/spacy_llm/tasks/spancat.py
deleted file mode 100644
index a1cf74fe..00000000
--- a/spacy_llm/tasks/spancat.py
+++ /dev/null
@@ -1,227 +0,0 @@
-from collections import defaultdict
-from typing import Any, Callable, Dict, Iterable, List, Optional, Union
-
-from spacy.language import Language
-from spacy.pipeline.spancat import spancat_score
-from spacy.tokens import Doc, Span
-from spacy.training import Example
-
-from ..compat import Literal
-from ..registry import registry
-from ..ty import ExamplesConfigType
-from ..util import split_labels
-from .span import SpanExample, SpanTask
-from .templates import read_template
-
-_DEFAULT_SPANCAT_TEMPLATE_V1 = read_template("spancat")
-_DEFAULT_SPANCAT_TEMPLATE_V2 = read_template("spancat.v2")
-
-
-@registry.llm_tasks("spacy.SpanCat.v1")
-def make_spancat_task(
-    labels: str = "",
-    examples: Optional[Callable[[], Iterable[Any]]] = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-    case_sensitive_matching: bool = False,
-    single_match: bool = False,
-):
-    """SpanCat.v1 task factory.
-
-    labels (str): Comma-separated list of labels to pass to the template.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Map of label -> description
-        of the label to help the language model output the entities wanted.
-        It is usually easier to provide these definitions rather than
-        full examples, although both can be provided.
-    spans_key (str): Key of the `Doc.spans` dict to save under.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-    alignment_mode (str): "strict", "contract" or "expand".
-    case_sensitive: Whether to search without case sensitivity.
-    single_match (bool): If False, allow one substring to match multiple times in
-        the text. If True, returns the first hit.
-    """
-    labels_list = split_labels(labels)
-    span_examples = (
-        [SpanExample(**eg) for eg in examples()] if callable(examples) else examples
-    )
-    return SpanCatTask(
-        labels=labels_list,
-        template=_DEFAULT_SPANCAT_TEMPLATE_V1,
-        prompt_examples=span_examples,
-        normalizer=normalizer,
-        alignment_mode=alignment_mode,
-        case_sensitive_matching=case_sensitive_matching,
-        single_match=single_match,
-    )
-
-
-@registry.llm_tasks("spacy.SpanCat.v2")
-def make_spancat_task_v2(
-    labels: Union[List[str], str] = [],
-    template: str = _DEFAULT_SPANCAT_TEMPLATE_V2,
-    label_definitions: Optional[Dict[str, str]] = None,
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-    case_sensitive_matching: bool = False,
-    single_match: bool = False,
-):
-    """SpanCat.v2 task factory.
-
-    labels (Union[str, List[str]]): List of labels to pass to the template,
-        either an actual list or a comma-separated string.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Map of label -> description
-        of the label to help the language model output the entities wanted.
-        It is usually easier to provide these definitions rather than
-        full examples, although both can be provided.
-    spans_key (str): Key of the `Doc.spans` dict to save under.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-    alignment_mode (str): "strict", "contract" or "expand".
-    case_sensitive: Whether to search without case sensitivity.
-    single_match (bool): If False, allow one substring to match multiple times in
-        the text. If True, returns the first hit.
-    """
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    span_examples = [SpanExample(**eg) for eg in raw_examples] if raw_examples else None
-    return SpanCatTask(
-        labels=labels_list,
-        template=template,
-        label_definitions=label_definitions,
-        prompt_examples=span_examples,
-        normalizer=normalizer,
-        alignment_mode=alignment_mode,
-        case_sensitive_matching=case_sensitive_matching,
-        single_match=single_match,
-    )
-
-
-class SpanCatTask(SpanTask):
-    def __init__(
-        self,
-        labels: List[str] = [],
-        template: str = _DEFAULT_SPANCAT_TEMPLATE_V2,
-        label_definitions: Optional[Dict[str, str]] = None,
-        spans_key: str = "sc",
-        prompt_examples: Optional[List[SpanExample]] = None,
-        normalizer: Optional[Callable[[str], str]] = None,
-        alignment_mode: Literal["strict", "contract", "expand"] = "contract",
-        case_sensitive_matching: bool = False,
-        single_match: bool = False,
-    ):
-        """Default SpanCat task.
-
-        labels (List[str]): List of labels to pass to the template.
-            Leave empty to populate it at initialization time (only if examples are provided).
-        template (str): Prompt template passed to the model.
-        label_definitions (Optional[Dict[str, str]]): Map of label -> description
-            of the label to help the language model output the entities wanted.
-            It is usually easier to provide these definitions rather than
-            full examples, although both can be provided.
-        spans_key (str): Key of the `Doc.spans` dict to save under.
-        prompt_examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        normalizer (Optional[Callable[[str], str]]): optional normalizer function.
-        alignment_mode (str): "strict", "contract" or "expand".
-        case_sensitive: Whether to search without case sensitivity.
-        single_match (bool): If False, allow one substring to match multiple times in
-            the text. If True, returns the first hit.
-        """
-        super(SpanCatTask, self).__init__(
-            labels=labels,
-            template=template,
-            label_definitions=label_definitions,
-            prompt_examples=prompt_examples,
-            normalizer=normalizer,
-            alignment_mode=alignment_mode,
-            case_sensitive_matching=case_sensitive_matching,
-            single_match=single_match,
-        )
-        self._spans_key = spans_key
-
-    def assign_spans(
-        self,
-        doc: Doc,
-        spans: List[Span],
-    ) -> None:
-        """Assign spans to the document."""
-        doc.spans[self._spans_key] = sorted(spans)  # type: ignore [type-var]
-
-    def scorer(
-        self,
-        examples: Iterable[Example],
-    ) -> Dict[str, Any]:
-        return spancat_score(
-            examples,
-            spans_key=self._spans_key,
-            allow_overlap=True,
-        )
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        labels: List[str] = [],
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initialize the SpanCat task, by auto-discovering labels.
-
-        Labels can be set through, by order of precedence:
-
-        - the `[initialize]` section of the pipeline configuration
-        - the `labels` argument supplied to the task factory
-        - the labels found in the examples
-
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        labels (List[str]): Optional list of labels.
-        n_prompt_examples (int): How many prompt examples to infer from the Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        if not labels:
-            labels = list(self._label_dict.values())
-        infer_labels = not labels
-
-        for eg in get_examples():
-            if infer_labels:
-                for span in eg.reference.spans.get(self._spans_key, []):
-                    labels.append(span.label_)
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(self._create_prompt_example(eg))
-
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return [
-            "_spans_key",
-            "_label_dict",
-            "_template",
-            "_label_definitions",
-            "_alignment_mode",
-            "_case_sensitive_matching",
-            "_single_match",
-        ]
-
-    def _create_prompt_example(self, example: Example) -> SpanExample:
-        """Create a spancat prompt example from a spaCy example."""
-        entities = defaultdict(list)
-        for span in example.reference.spans[self._spans_key]:
-            entities[span.label_].append(span.text)
-
-        return SpanExample(text=example.reference.text, entities=entities)
diff --git a/spacy_llm/tasks/spancat/__init__.py b/spacy_llm/tasks/spancat/__init__.py
new file mode 100644
index 00000000..88c8744b
--- /dev/null
+++ b/spacy_llm/tasks/spancat/__init__.py
@@ -0,0 +1,11 @@
+from .registry import make_spancat_task, make_spancat_task_v2, make_spancat_task_v3
+from .task import SpanCatTask
+from .util import SpanCatExample
+
+__all__ = [
+    "make_spancat_task",
+    "make_spancat_task_v2",
+    "make_spancat_task_v3",
+    "SpanCatExample",
+    "SpanCatTask",
+]
diff --git a/spacy_llm/tasks/spancat/registry.py b/spacy_llm/tasks/spancat/registry.py
new file mode 100644
index 00000000..668b1f51
--- /dev/null
+++ b/spacy_llm/tasks/spancat/registry.py
@@ -0,0 +1,192 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type, Union
+
+from ...compat import Literal
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, Scorer, TaskResponseParser
+from ...util import split_labels
+from ..span import parse_responses as parse_span_responses
+from ..span import parse_responses_cot as parse_span_responses_cot
+from ..span.util import check_label_consistency as check_labels
+from ..span.util import check_label_consistency_cot as check_labels_cot
+from .task import DEFAULT_SPANCAT_TEMPLATE_V1, DEFAULT_SPANCAT_TEMPLATE_V2
+from .task import DEFAULT_SPANCAT_TEMPLATE_V3, SpanCatTask
+from .util import SpanCatCoTExample, SpanCatExample, score
+
+
+@registry.llm_tasks("spacy.SpanCat.v1")
+def make_spancat_task(
+    parse_responses: Optional[TaskResponseParser[SpanCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: str = "",
+    examples: Optional[Callable[[], Iterable[Any]]] = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    single_match: bool = False,
+    spans_key: str = "sc",
+    scorer: Optional[Scorer] = None,
+):
+    """SpanCat.v1 task factory.
+
+    parse_responses (Optional[TaskResponseParser[SpanCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (str): Comma-separated list of labels to pass to the template.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching (bool): Whether to search without case sensitivity.
+    single_match (bool): If False, allow one substring to match multiple times in
+        the text. If True, returns the first hit.
+    spans_key (str): Key of the `Doc.spans` dict to save under.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    example_type = prompt_example_type or SpanCatExample
+    span_examples = (
+        [example_type(**eg) for eg in examples()] if callable(examples) else examples
+    )
+
+    return SpanCatTask(
+        labels=labels_list,
+        parse_responses=parse_responses or parse_span_responses,
+        prompt_example_type=example_type,
+        template=DEFAULT_SPANCAT_TEMPLATE_V1,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=single_match,
+        label_definitions=None,
+        spans_key=spans_key,
+        scorer=scorer or score,
+        description=None,
+        check_label_consistency=check_labels,
+    )
+
+
+@registry.llm_tasks("spacy.SpanCat.v2")
+def make_spancat_task_v2(
+    parse_responses: Optional[TaskResponseParser[SpanCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_SPANCAT_TEMPLATE_V2,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    single_match: bool = False,
+    spans_key: str = "sc",
+    scorer: Optional[Scorer] = None,
+):
+    """SpanCat.v2 task factory.
+
+    parse_responses (Optional[TaskResponseParser[SpanCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[str, List[str]]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    label_definitions (Optional[Dict[str, str]]): Map of label -> description
+        of the label to help the language model output the entities wanted.
+        It is usually easier to provide these definitions rather than
+        full examples, although both can be provided.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching (bool): Whether to search without case sensitivity.
+    single_match (bool): If False, allow one substring to match multiple times in
+        the text. If True, returns the first hit.
+    spans_key (str): Key of the `Doc.spans` dict to save under.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or SpanCatExample
+    span_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return SpanCatTask(
+        parse_responses=parse_responses or parse_span_responses,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=single_match,
+        spans_key=spans_key,
+        scorer=scorer or score,
+        description=None,
+        check_label_consistency=check_labels,
+    )
+
+
+@registry.llm_tasks("spacy.SpanCat.v3")
+def make_spancat_task_v3(
+    parse_responses: Optional[TaskResponseParser[SpanCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_SPANCAT_TEMPLATE_V3,
+    description: Optional[str] = None,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    alignment_mode: Literal["strict", "contract", "expand"] = "contract",
+    case_sensitive_matching: bool = False,
+    spans_key: str = "sc",
+    scorer: Optional[Scorer] = None,
+):
+    """SpanCat.v3 task factory for SpanCat with chain-of-thought prompting.
+
+    parse_responses (Optional[TaskResponseParser[SpanCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[str, List[str]]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    description (str): A description of what to recognize or not recognize as entities.
+    label_definitions (Optional[Dict[str, str]]): Map of label -> description
+        of the label to help the language model output the entities wanted.
+        It is usually easier to provide these definitions rather than
+        full examples, although both can be provided.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+    alignment_mode (str): "strict", "contract" or "expand".
+    case_sensitive_matching (bool): Whether to search without case sensitivity.
+    spans_key (str): Key of the `Doc.spans` dict to save under.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or SpanCatCoTExample
+    span_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return SpanCatTask(
+        parse_responses=parse_responses or parse_span_responses_cot,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=span_examples,
+        normalizer=normalizer,
+        alignment_mode=alignment_mode,
+        case_sensitive_matching=case_sensitive_matching,
+        single_match=False,
+        spans_key=spans_key,
+        scorer=scorer or score,
+        description=description,
+        check_label_consistency=check_labels_cot,
+    )
diff --git a/spacy_llm/tasks/spancat/task.py b/spacy_llm/tasks/spancat/task.py
new file mode 100644
index 00000000..59bdad63
--- /dev/null
+++ b/spacy_llm/tasks/spancat/task.py
@@ -0,0 +1,113 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type
+
+from spacy.language import Language
+from spacy.tokens import Doc, Span
+from spacy.training import Example
+
+from ...compat import Literal, Self
+from ...ty import FewshotExample, Scorer, TaskResponseParser
+from ..span import SpanTask
+from ..span.task import SpanTaskLabelCheck
+from ..templates import read_template
+
+DEFAULT_SPANCAT_TEMPLATE_V1 = read_template("spancat.v1")
+DEFAULT_SPANCAT_TEMPLATE_V2 = read_template("spancat.v2")
+DEFAULT_SPANCAT_TEMPLATE_V3 = read_template("spancat.v3")
+
+
+class SpanCatTask(SpanTask):
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        labels: List[str],
+        template: str,
+        label_definitions: Optional[Dict[str, str]],
+        spans_key: str,
+        prompt_examples: Optional[List[FewshotExample]],
+        normalizer: Optional[Callable[[str], str]],
+        alignment_mode: Literal["strict", "contract", "expand"],
+        case_sensitive_matching: bool,
+        single_match: bool,
+        scorer: Scorer,
+        description: Optional[str],
+        check_label_consistency: SpanTaskLabelCheck[Self],
+    ):
+        """Default SpanCat task.
+
+        parse_responses (TaskResponseParser): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        labels (List[str]): List of labels to pass to the template.
+            Leave empty to populate it at initialization time (only if examples are provided).
+        template (str): Prompt template passed to the model.
+        label_definitions (Optional[Dict[str, str]]): Map of label -> description
+            of the label to help the language model output the entities wanted.
+            It is usually easier to provide these definitions rather than
+            full examples, although both can be provided.
+        spans_key (str): Key of the `Doc.spans` dict to save under.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        normalizer (Optional[Callable[[str], str]]): optional normalizer function.
+        alignment_mode (str): "strict", "contract" or "expand".
+        case_sensitive_matching (bool): Whether to search without case sensitivity.
+        single_match (bool): If False, allow one substring to match multiple times in
+            the text. If True, returns the first hit.
+        scorer (Scorer): Scorer function.
+        description (str): A description of what to recognize or not recognize as entities.
+        check_label_consistency (SpanTaskLabelCheck): Callable to check label consistency.
+        """
+        super(SpanCatTask, self).__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            labels=labels,
+            template=template,
+            label_definitions=label_definitions,
+            prompt_examples=prompt_examples,
+            normalizer=normalizer,
+            alignment_mode=alignment_mode,
+            case_sensitive_matching=case_sensitive_matching,
+            single_match=single_match,
+            description=description,
+            allow_overlap=True,
+            check_label_consistency=check_label_consistency,
+        )
+        self._spans_key = spans_key
+        self._scorer = scorer
+
+    def assign_spans(
+        self,
+        doc: Doc,
+        spans: List[Span],
+    ) -> None:
+        """Assign spans to the document."""
+        doc.spans[self._spans_key] = sorted(spans)  # type: ignore [type-var]
+
+    def scorer(self, examples: Iterable[Example]) -> Dict[str, Any]:
+        return self._scorer(examples, spans_key=self._spans_key, allow_overlap=True)
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        labels: List[str] = [],
+        n_prompt_examples: int = 0,
+    ) -> None:
+        super()._initialize(
+            get_examples=get_examples,
+            nlp=nlp,
+            labels=labels,
+            n_prompt_examples=n_prompt_examples,
+            spans_key=self._spans_key,
+        )
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return [*super()._cfg_keys, "_spans_key"]
+
+    def _extract_labels_from_example(self, example: Example) -> List[str]:
+        return [
+            span.label_ for span in example.reference.spans.get(self._spans_key, [])
+        ]
+
+    @property
+    def spans_key(self) -> str:
+        return self._spans_key
diff --git a/spacy_llm/tasks/spancat/util.py b/spacy_llm/tasks/spancat/util.py
new file mode 100644
index 00000000..80788394
--- /dev/null
+++ b/spacy_llm/tasks/spancat/util.py
@@ -0,0 +1,42 @@
+from collections import defaultdict
+from typing import Any, Dict, Iterable
+
+from spacy.pipeline.spancat import spancat_score
+from spacy.training import Example
+
+from ...compat import Self
+from ..span import SpanExample
+from ..span.examples import SpanCoTExample
+
+
+class SpanCatExample(SpanExample):
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        entities = defaultdict(list)
+        for span in example.reference.spans[kwargs["spans_key"]]:
+            entities[span.label_].append(span.text)
+
+        return cls(text=example.reference.text, entities=entities)
+
+
+class SpanCatCoTExample(SpanCoTExample):
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        return cls(
+            text=example.reference.text,
+            spans=SpanCoTExample._extract_span_reasons(
+                example.reference.spans[kwargs["spans_key"]]
+            ),
+        )
+
+
+def score(examples: Iterable[Example], **kwargs) -> Dict[str, Any]:
+    """Score spancat accuracy in examples.
+    examples (Iterable[Example]): Examples to score.
+    RETURNS (Dict[str, Any]): Dict with metric name -> score.
+    """
+    return spancat_score(
+        examples,
+        spans_key=kwargs["spans_key"],
+        allow_overlap=True,
+    )
diff --git a/spacy_llm/tasks/summarization.py b/spacy_llm/tasks/summarization.py
deleted file mode 100644
index 998b3f3b..00000000
--- a/spacy_llm/tasks/summarization.py
+++ /dev/null
@@ -1,147 +0,0 @@
-import warnings
-from typing import Any, Callable, Iterable, List, Optional, Type
-
-import jinja2
-from pydantic import BaseModel
-from spacy.language import Language
-from spacy.tokens import Doc
-from spacy.training import Example
-
-from ..registry import registry
-from ..ty import ExamplesConfigType
-from .templates import read_template
-from .util import SerializableTask
-from .util.serialization import ExampleType
-
-_DEFAULT_SUMMARIZATION_TEMPLATE_V1 = read_template("summarization")
-
-
-class SummarizationExample(BaseModel):
-    text: str
-    summary: str
-
-
-@registry.llm_tasks("spacy.Summarization.v1")
-def make_summarization_task(
-    template: str = _DEFAULT_SUMMARIZATION_TEMPLATE_V1,
-    examples: ExamplesConfigType = None,
-    max_n_words: Optional[int] = None,
-    field: str = "summary",
-):
-    """Summarization.v1 task factory.
-
-    template (str): Prompt template passed to the model.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    max_n_words (int): Max. number of words to use in summary.
-    field (str): The name of the doc extension in which to store the summary.
-    """
-    raw_examples = examples() if callable(examples) else examples
-    span_examples = (
-        [SummarizationExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-
-    return SummarizationTask(
-        template=template, examples=span_examples, max_n_words=max_n_words, field=field
-    )
-
-
-class SummarizationTask(SerializableTask[SummarizationExample]):
-    def __init__(
-        self,
-        template: str = _DEFAULT_SUMMARIZATION_TEMPLATE_V1,
-        examples: Optional[List[SummarizationExample]] = None,
-        max_n_words: Optional[int] = None,
-        field: str = "summary",
-    ):
-        """Default summarization task.
-
-        template (str): Prompt template passed to the model.
-        examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        max_n_words (int): Max. number of words to use in summary.
-        field (str): The name of the doc extension in which to store the summary.
-        """
-        self._template = template
-        self._max_n_words = max_n_words
-        self._field = field
-        self._prompt_examples = examples or []
-        self._check_example_summaries = True
-        if not Doc.has_extension(field):
-            Doc.set_extension(field, default=None)
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initializes prompt examples from Doc examples.
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        n_prompt_examples (int): How many prompt examples to infer from the provided Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        for eg in get_examples():
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(
-                    SummarizationExample(
-                        text=eg.reference.text,
-                        summary=getattr(eg.reference._, self._field),
-                    )
-                )
-
-    def _check_prompt_example_summary_len(self) -> None:
-        """Checks whether summaries of prompt examples are of expected lengths. Warns if they aren't."""
-        if self._max_n_words is None:
-            return
-
-        for pr_ex in self._prompt_examples:
-            len_summary = len(pr_ex.summary.split())
-            len_text = len(pr_ex.text.split())
-            if len_summary >= len_text * 1.2:
-                warnings.warn(
-                    f"The provided example '{pr_ex.text[:30]}...' has a summary of token length {len_summary} and a text "
-                    f"of token length {len_text}. Ensure that your examples' summaries are shorter than their original "
-                    f"texts."
-                )
-            if len_summary > self._max_n_words * 1.2:
-                warnings.warn(
-                    f"The provided example '{pr_ex.text[:20]}...' has a summary of length {len_summary}, but "
-                    f"`max_n_words` == {self._max_n_words}. If your examples are longer than they should be, the "
-                    f"LLM will likely produce responses that are too long."
-                )
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        if self._check_example_summaries:
-            self._check_prompt_example_summary_len()
-            self._check_example_summaries = False
-
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=doc.text,
-                examples=self._prompt_examples,
-                max_n_words=self._max_n_words,
-            )
-            yield prompt
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        for doc, prompt_response in zip(docs, responses):
-            setattr(doc._, self._field, prompt_response.replace("'''", "").strip())
-            yield doc
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return ["_template"]
-
-    @property
-    def _Example(self) -> Type[ExampleType]:
-        return SummarizationExample
diff --git a/spacy_llm/tasks/summarization/__init__.py b/spacy_llm/tasks/summarization/__init__.py
new file mode 100644
index 00000000..f45cf28b
--- /dev/null
+++ b/spacy_llm/tasks/summarization/__init__.py
@@ -0,0 +1,5 @@
+from .registry import make_summarization_task
+from .task import SummarizationTask
+from .util import SummarizationExample
+
+__all__ = ["make_summarization_task", "SummarizationExample", "SummarizationTask"]
diff --git a/spacy_llm/tasks/summarization/parser.py b/spacy_llm/tasks/summarization/parser.py
new file mode 100644
index 00000000..0af52f6e
--- /dev/null
+++ b/spacy_llm/tasks/summarization/parser.py
@@ -0,0 +1,18 @@
+from typing import Iterable
+
+from spacy.tokens import Doc
+
+from .task import SummarizationTask
+
+
+def parse_responses_v1(
+    task: SummarizationTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[str]:
+    """Parses LLM responses for spacy.Summarization.v1.
+    task (SummarizationTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Iterable[str]): Summary per doc/response.
+    """
+    for prompt_response in responses:
+        yield prompt_response.replace("'''", "").strip()
diff --git a/spacy_llm/tasks/summarization/registry.py b/spacy_llm/tasks/summarization/registry.py
new file mode 100644
index 00000000..6d3c6540
--- /dev/null
+++ b/spacy_llm/tasks/summarization/registry.py
@@ -0,0 +1,43 @@
+from typing import Optional, Type
+
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, TaskResponseParser
+from .parser import parse_responses_v1
+from .task import DEFAULT_SUMMARIZATION_TEMPLATE_V1, SummarizationTask
+from .util import SummarizationExample
+
+
+@registry.llm_tasks("spacy.Summarization.v1")
+def make_summarization_task(
+    template: str = DEFAULT_SUMMARIZATION_TEMPLATE_V1,
+    parse_responses: Optional[TaskResponseParser[SummarizationTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    examples: ExamplesConfigType = None,
+    max_n_words: Optional[int] = None,
+    field: str = "summary",
+):
+    """Summarization.v1 task factory.
+
+    template (str): Prompt template passed to the model.
+    parse_responses (Optional[TaskResponseParser[SummarizationTask]]): Callable for parsing LLM responses for
+        this task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    max_n_words (int): Max. number of words to use in summary.
+    field (str): The name of the doc extension in which to store the summary.
+    """
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or SummarizationExample
+    span_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return SummarizationTask(
+        template=template,
+        parse_responses=parse_responses or parse_responses_v1,
+        prompt_example_type=example_type,
+        prompt_examples=span_examples,
+        max_n_words=max_n_words,
+        field=field,
+    )
diff --git a/spacy_llm/tasks/summarization/task.py b/spacy_llm/tasks/summarization/task.py
new file mode 100644
index 00000000..d951e59e
--- /dev/null
+++ b/spacy_llm/tasks/summarization/task.py
@@ -0,0 +1,105 @@
+import warnings
+from typing import Callable, Iterable, List, Optional, Type
+
+from spacy.language import Language
+from spacy.tokens import Doc
+from spacy.training import Example
+
+from ...compat import Self
+from ...ty import FewshotExample, TaskResponseParser
+from ..builtin_task import BuiltinTask
+from ..templates import read_template
+
+DEFAULT_SUMMARIZATION_TEMPLATE_V1 = read_template("summarization.v1")
+
+
+class SummarizationTask(BuiltinTask):
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        template: str,
+        max_n_words: Optional[int],
+        field: str,
+        prompt_examples: Optional[List[FewshotExample]],
+    ):
+        """Default summarization task.
+
+        template (str): Prompt template passed to the model.
+        parse_responses (TaskResponseParser[Self]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        max_n_words (Optional[int]): Max. number of words to use in summary.
+        field (str): The name of the doc extension in which to store the summary.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+        )
+        self._max_n_words = max_n_words
+        self._field = field
+        self._check_example_summaries = True
+
+        if not Doc.has_extension(field):
+            Doc.set_extension(field, default=None)
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        n_prompt_examples: int = 0,
+    ) -> None:
+        super()._initialize(
+            get_examples=get_examples,
+            nlp=nlp,
+            n_prompt_examples=n_prompt_examples,
+        )
+
+    def _check_prompt_example_summary_len(self) -> None:
+        """Checks whether summaries of prompt examples are of expected lengths. Warns if they aren't."""
+        if self._max_n_words is None:
+            return
+
+        for pr_ex in self._prompt_examples:
+            len_summary = len(pr_ex.summary.split())
+            len_text = len(pr_ex.text.split())
+            if len_summary >= len_text * 1.2:
+                warnings.warn(
+                    f"The provided example '{pr_ex.text[:30]}...' has a summary of token length {len_summary} and a text "
+                    f"of token length {len_text}. Ensure that your examples' summaries are shorter than their original "
+                    f"texts."
+                )
+            if len_summary > self._max_n_words * 1.2:
+                warnings.warn(
+                    f"The provided example '{pr_ex.text[:20]}...' has a summary of length {len_summary}, but "
+                    f"`max_n_words` == {self._max_n_words}. If your examples are longer than they should be, the "
+                    f"LLM will likely produce responses that are too long."
+                )
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[str]:
+        if self._check_example_summaries:
+            self._check_prompt_example_summary_len()
+            self._check_example_summaries = False
+
+        return super().generate_prompts(docs=docs, max_n_words=self._max_n_words)
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        for doc, summary in zip(docs, self._parse_responses(self, docs, responses)):
+            setattr(doc._, self._field, summary)
+            yield doc
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return ["_template"]
+
+    @property
+    def field(self) -> str:
+        return self._field
+
+    @property
+    def max_n_words(self) -> Optional[int]:
+        return self._max_n_words
diff --git a/spacy_llm/tasks/summarization/util.py b/spacy_llm/tasks/summarization/util.py
new file mode 100644
index 00000000..b8c18752
--- /dev/null
+++ b/spacy_llm/tasks/summarization/util.py
@@ -0,0 +1,15 @@
+from spacy.training import Example
+
+from ...compat import BaseModel, Self
+
+
+class SummarizationExample(BaseModel):
+    text: str
+    summary: str
+
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        return cls(
+            text=example.reference.text,
+            summary=getattr(example.reference._, kwargs["field"]),
+        )
diff --git a/spacy_llm/tasks/templates/lemma.jinja b/spacy_llm/tasks/templates/lemma.v1.jinja
similarity index 93%
rename from spacy_llm/tasks/templates/lemma.jinja
rename to spacy_llm/tasks/templates/lemma.v1.jinja
index 181193db..0ee12699 100644
--- a/spacy_llm/tasks/templates/lemma.jinja
+++ b/spacy_llm/tasks/templates/lemma.v1.jinja
@@ -9,10 +9,10 @@ Include the final punctuation token in this list.
 Prefix with your output with "Lemmatized text".
 
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 Below are some examples (only use these as a guide):
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/ner.jinja b/spacy_llm/tasks/templates/ner.v1.jinja
similarity index 91%
rename from spacy_llm/tasks/templates/ner.jinja
rename to spacy_llm/tasks/templates/ner.v1.jinja
index c0492704..8b2fab5d 100644
--- a/spacy_llm/tasks/templates/ner.jinja
+++ b/spacy_llm/tasks/templates/ner.v1.jinja
@@ -5,12 +5,12 @@ From the text below, extract the following entities in the following format:
 {# whitespace #}
 {%- endfor -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/ner.v2.jinja b/spacy_llm/tasks/templates/ner.v2.jinja
index b0837792..2979de7b 100644
--- a/spacy_llm/tasks/templates/ner.v2.jinja
+++ b/spacy_llm/tasks/templates/ner.v2.jinja
@@ -22,11 +22,11 @@ Assume these definitions are written by an expert and follow them closely.
 {%- endif -%}
 {# whitespace #}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 Text:
 '''
 {{ example.text }}
diff --git a/spacy_llm/tasks/templates/ner.v3.jinja b/spacy_llm/tasks/templates/ner.v3.jinja
new file mode 100644
index 00000000..be77b7ae
--- /dev/null
+++ b/spacy_llm/tasks/templates/ner.v3.jinja
@@ -0,0 +1,58 @@
+You are an expert Named Entity Recognition (NER) system.
+Your task is to accept Text as input and extract named entities.
+Entities must have one of the following labels: {{ ', '.join(labels) }}.
+If a span is not an entity label it: `==NONE==`.
+{# whitespace #}
+{# whitespace #}
+{%- if description -%}
+{# whitespace #}
+{{ description }}
+{# whitespace #}
+{%- endif -%}
+{%- if label_definitions -%}
+Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
+Assume these definitions are written by an expert and follow them closely.
+{# whitespace #}
+{%- for label, definition in label_definitions.items() -%}
+{{ label }}: {{ definition }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endif -%}
+{%- if prompt_examples -%}
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+{# whitespace #}
+{# whitespace #}
+{%- for example in prompt_examples -%}
+Paragraph: {{ example.text }}
+Answer:
+{# whitespace #}
+{%- for span in example.spans -%}
+{{ loop.index }}. {{ span.to_str() }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endfor -%}
+{%- else -%}
+{# whitespace #}
+Here is an example of the output format for a paragraph using different labels than this task requires.
+Only use this output format but use the labels provided
+above instead of the ones defined in the example below.
+Do not output anything besides entities in this output format.
+Output entities in the order they occur in the input paragraph regardless of label.
+
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+
+Paragraph: Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok.
+Answer:
+1. Sriracha sauce | True | INGREDIENT | is an ingredient to add to a stir fry
+2. really well | False | ==NONE== | is a description of how well sriracha sauce goes with hoisin stir fry
+3. hoisin stir fry | True | DISH | is a dish with stir fry vegetables and hoisin sauce
+4. wok | True | EQUIPMENT | is a piece of cooking equipment used to stir fry ingredients
+{# whitespace #}
+{# whitespace #}
+{%- endif -%}
+Paragraph: {{ text }}
+Answer:
diff --git a/spacy_llm/tasks/templates/rel.jinja b/spacy_llm/tasks/templates/rel.v1.jinja
similarity index 95%
rename from spacy_llm/tasks/templates/rel.jinja
rename to spacy_llm/tasks/templates/rel.v1.jinja
index b09826e8..75f2a495 100644
--- a/spacy_llm/tasks/templates/rel.jinja
+++ b/spacy_llm/tasks/templates/rel.v1.jinja
@@ -28,11 +28,11 @@ Assume these definitions are written by an expert and follow them closely.
 {# whitespace #}
 {# whitespace #}
 {%- endif -%}
-{%- if examples -%}
+{%- if prompt_examples -%}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 Text:
 '''
 {{ preannotate(example) }}
diff --git a/spacy_llm/tasks/templates/sentiment.jinja b/spacy_llm/tasks/templates/sentiment.v1.jinja
similarity index 87%
rename from spacy_llm/tasks/templates/sentiment.jinja
rename to spacy_llm/tasks/templates/sentiment.v1.jinja
index 6763dda3..ff210d6a 100644
--- a/spacy_llm/tasks/templates/sentiment.jinja
+++ b/spacy_llm/tasks/templates/sentiment.v1.jinja
@@ -1,9 +1,9 @@
 Analyse whether the text surrounded by ''' is positive or negative. Respond with a float value between 0 and 1. 1 represents an exclusively positive sentiment, 0 an exclusively negative sentiment.
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 Below are some examples (only use these as a guide):
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/spancat.jinja b/spacy_llm/tasks/templates/spancat.v1.jinja
similarity index 91%
rename from spacy_llm/tasks/templates/spancat.jinja
rename to spacy_llm/tasks/templates/spancat.v1.jinja
index aa175c70..c3fd3c27 100644
--- a/spacy_llm/tasks/templates/spancat.jinja
+++ b/spacy_llm/tasks/templates/spancat.v1.jinja
@@ -5,12 +5,12 @@ From the text below, extract the following (possibly overlapping) entities in th
 {# whitespace #}
 {%- endfor -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/spancat.v2.jinja b/spacy_llm/tasks/templates/spancat.v2.jinja
index c086a68d..7d285d3c 100644
--- a/spacy_llm/tasks/templates/spancat.v2.jinja
+++ b/spacy_llm/tasks/templates/spancat.v2.jinja
@@ -23,11 +23,11 @@ Assume these definitions are written by an expert and follow them closely.
 {%- endif -%}
 {# whitespace #}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 Text:
 '''
 {{ example.text }}
diff --git a/spacy_llm/tasks/templates/spancat.v3.jinja b/spacy_llm/tasks/templates/spancat.v3.jinja
new file mode 100644
index 00000000..9339bdef
--- /dev/null
+++ b/spacy_llm/tasks/templates/spancat.v3.jinja
@@ -0,0 +1,60 @@
+You are an expert Entity Recognition system.
+Your task is to accept Text as input and extract named entities.
+The entities you extract can overlap with each other.
+{# whitespace #}
+Entities must have one of the following labels: {{ ', '.join(labels) }}.
+If a span is not an entity label it: `==NONE==`.
+{# whitespace #}
+{# whitespace #}
+{%- if description -%}
+{# whitespace #}
+{{ description }}
+{# whitespace #}
+{%- endif -%}
+{%- if label_definitions -%}
+Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
+Assume these definitions are written by an expert and follow them closely.
+{# whitespace #}
+{%- for label, definition in label_definitions.items() -%}
+{{ label }}: {{ definition }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endif -%}
+{%- if prompt_examples %}
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+{# whitespace #}
+{# whitespace #}
+{%- for example in prompt_examples -%}
+Paragraph: {{ example.text }}
+Answer:
+{# whitespace #}
+{%- for span in example.spans -%}
+{{ loop.index }}. {{ span.to_str() }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endfor -%}
+{%- else %}
+{# whitespace #}
+Here is an example of the output format for a paragraph using different labels than this task requires.
+Only use this output format but use the labels provided
+above instead of the ones defined in the example below.
+Do not output anything besides entities in this output format.
+Output entities in the order they occur in the input paragraph regardless of label.
+
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+
+Paragraph: Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok.
+Answer:
+1. Sriracha sauce | True | INGREDIENT | is an ingredient to add to a stir fry
+2. really well | False | ==NONE== | is a description of how well sriracha sauce goes with hoisin stir fry
+3. hoisin stir fry | True | DISH | is a dish with stir fry vegetables and hoisin sauce
+4. wok | True | EQUIPMENT | is a piece of cooking equipment used to stir fry ingredients
+{# whitespace #}
+{# whitespace #}
+{%- endif -%}
+Paragraph: {{ text }}
+Answer:
diff --git a/spacy_llm/tasks/templates/summarization.jinja b/spacy_llm/tasks/templates/summarization.v1.jinja
similarity index 90%
rename from spacy_llm/tasks/templates/summarization.jinja
rename to spacy_llm/tasks/templates/summarization.v1.jinja
index 8b60060c..f795f451 100644
--- a/spacy_llm/tasks/templates/summarization.jinja
+++ b/spacy_llm/tasks/templates/summarization.v1.jinja
@@ -4,11 +4,11 @@ You are an expert summarization system. Your task is to accept Text as input and
 The summary must not, under any circumstances, contain more than {{ max_n_words }} words.
 {%- endif -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/textcat.jinja b/spacy_llm/tasks/templates/textcat.v1.jinja
similarity index 94%
rename from spacy_llm/tasks/templates/textcat.jinja
rename to spacy_llm/tasks/templates/textcat.v1.jinja
index a22fa83d..0d53c719 100644
--- a/spacy_llm/tasks/templates/textcat.jinja
+++ b/spacy_llm/tasks/templates/textcat.v1.jinja
@@ -20,12 +20,12 @@ If the text cannot be classified into any of the provided labels, answer `==NONE
 {# whitespace #}
 {%- endif -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/textcat.v2.jinja b/spacy_llm/tasks/templates/textcat.v2.jinja
index 57dca562..092241a1 100644
--- a/spacy_llm/tasks/templates/textcat.v2.jinja
+++ b/spacy_llm/tasks/templates/textcat.v2.jinja
@@ -25,12 +25,12 @@ If the text cannot be classified into any of the provided labels, answer `==NONE
 {# whitespace #}
 {%- endif -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/templates/textcat.v3.jinja b/spacy_llm/tasks/templates/textcat.v3.jinja
index 4425e573..d4f42ff9 100644
--- a/spacy_llm/tasks/templates/textcat.v3.jinja
+++ b/spacy_llm/tasks/templates/textcat.v3.jinja
@@ -40,12 +40,12 @@ Assume these definitions are written by an expert and follow them closely.
 {# whitespace #}
 {%- endif -%}
 {# whitespace #}
-{%- if examples -%}
+{%- if prompt_examples -%}
 {# whitespace #}
 Below are some examples (only use these as a guide):
 {# whitespace #}
 {# whitespace #}
-{%- for example in examples -%}
+{%- for example in prompt_examples -%}
 {# whitespace #}
 Text:
 '''
diff --git a/spacy_llm/tasks/textcat.py b/spacy_llm/tasks/textcat.py
deleted file mode 100644
index 9fe35ae9..00000000
--- a/spacy_llm/tasks/textcat.py
+++ /dev/null
@@ -1,406 +0,0 @@
-from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Type, Union
-
-import jinja2
-from pydantic import BaseModel
-from spacy.language import Language
-from spacy.scorer import Scorer
-from spacy.tokens import Doc
-from spacy.training import Example
-from wasabi import msg
-
-from spacy_llm.tasks.util.serialization import SerializableTask
-
-from ..registry import lowercase_normalizer, registry
-from ..ty import ExamplesConfigType
-from ..util import split_labels
-from .templates import read_template
-
-_DEFAULT_TEXTCAT_TEMPLATE_V1 = read_template("textcat")
-_DEFAULT_TEXTCAT_TEMPLATE_V2 = read_template("textcat.v2")
-_DEFAULT_TEXTCAT_TEMPLATE_V3 = read_template("textcat.v3")
-
-
-class TextCatExample(BaseModel):
-    text: str
-    answer: str
-
-
-@registry.llm_tasks("spacy.TextCat.v1")
-def make_textcat_task(
-    labels: str = "",
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    exclusive_classes: bool = False,
-    allow_none: bool = True,
-    verbose: bool = False,
-) -> "TextCatTask":
-    """TextCat.v1 task factory.
-
-    You can use either binary or multilabel text classification based on the
-    labels you provide.
-
-    If a single label is provided, binary classification
-    will be used. The label will get a score of `0` or `1` in `doc.cats`.
-
-    Otherwise, multilabel classification will be used. The document labels
-    in `doc.cats` will be a dictionary of strings and their score.
-
-    Lastly, you can toggle between exclusive or no-exclusive text
-    categorization by passing a flag to the `exclusive_classes` parameter.
-
-    labels (str): Comma-separated list of labels to pass to the template.
-        This task assumes binary classification if a single label is provided.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-    exclusive_classes (bool): If True, require the language model to suggest only one
-        label per class. This is automatically set when using binary classification.
-    allow_none (bool): if True, there might be cases where no label is applicable.
-    verbose (bool): If True, show extra information.
-    """
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    textcat_examples = (
-        [TextCatExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-    return TextCatTask(
-        labels=labels_list,
-        template=_DEFAULT_TEXTCAT_TEMPLATE_V1,
-        prompt_examples=textcat_examples,
-        normalizer=normalizer,
-        exclusive_classes=exclusive_classes,
-        allow_none=allow_none,
-        verbose=verbose,
-    )
-
-
-@registry.llm_tasks("spacy.TextCat.v2")
-def make_textcat_task_v2(
-    labels: Union[List[str], str] = [],
-    template: str = _DEFAULT_TEXTCAT_TEMPLATE_V2,
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    exclusive_classes: bool = False,
-    allow_none: bool = True,
-    verbose: bool = False,
-) -> "TextCatTask":
-    """TextCat.v2 task factory.
-
-    You can use either binary or multilabel text classification based on the
-    labels you provide.
-
-    If a single label is provided, binary classification
-    will be used. The label will get a score of `0` or `1` in `doc.cats`.
-
-    Otherwise, multilabel classification will be used. The document labels
-    in `doc.cats` will be a dictionary of strings and their score.
-
-    Lastly, you can toggle between exclusive or no-exclusive text
-    categorization by passing a flag to the `exclusive_classes` parameter.
-
-    labels (Union[List[str], str]): List of labels to pass to the template,
-        either an actual list or a comma-separated string.
-        This task assumes binary classification if a single label is provided.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-    exclusive_classes (bool): If True, require the language model to suggest only one
-        label per class. This is automatically set when using binary classification.
-    allow_none (bool): if True, there might be cases where no label is applicable.
-    verbose (bool): If True, show extra information.
-    """
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    textcat_examples = (
-        [TextCatExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-    return TextCatTask(
-        labels=labels_list,
-        template=template,
-        prompt_examples=textcat_examples,
-        normalizer=normalizer,
-        exclusive_classes=exclusive_classes,
-        allow_none=allow_none,
-        verbose=verbose,
-    )
-
-
-@registry.llm_tasks("spacy.TextCat.v3")
-def make_textcat_task_v3(
-    labels: Union[List[str], str] = [],
-    template: str = _DEFAULT_TEXTCAT_TEMPLATE_V3,
-    label_definitions: Optional[Dict[str, str]] = None,
-    examples: ExamplesConfigType = None,
-    normalizer: Optional[Callable[[str], str]] = None,
-    exclusive_classes: bool = False,
-    allow_none: bool = True,
-    verbose: bool = False,
-) -> "TextCatTask":
-    """TextCat.v3 task factory.
-
-    You can use either binary or multilabel text classification based on the
-    labels you provide.
-
-    If a single label is provided, binary classification
-    will be used. The label will get a score of `0` or `1` in `doc.cats`.
-
-    Otherwise, multilabel classification will be used. The document labels
-    in `doc.cats` will be a dictionary of strings and their score.
-
-    Lastly, you can toggle between exclusive or no-exclusive text
-    categorization by passing a flag to the `exclusive_classes` parameter.
-
-    labels (Union[List[str], str]): List of labels to pass to the template,
-        either an actual list or a comma-separated string.
-        This task assumes binary classification if a single label is provided.
-        Leave empty to populate it at initialization time (only if examples are provided).
-    template (str): Prompt template passed to the model.
-    label_definitions (Optional[Dict[str, str]]): Optional dict mapping a label to a description of that label.
-        These descriptions are added to the prompt to help instruct the LLM on what to extract.
-    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-        reads a file containing task examples for few-shot learning. If None is
-        passed, then zero-shot learning will be used.
-    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-    exclusive_classes (bool): If True, require the language model to suggest only one
-        label per class. This is automatically set when using binary classification.
-    allow_none (bool): if True, there might be cases where no label is applicable.
-    verbose (bool): If True, show extra information.
-    """
-
-    labels_list = split_labels(labels)
-    raw_examples = examples() if callable(examples) else examples
-    textcat_examples = (
-        [TextCatExample(**eg) for eg in raw_examples] if raw_examples else None
-    )
-
-    return TextCatTask(
-        labels=labels_list,
-        template=template,
-        label_definitions=label_definitions,
-        prompt_examples=textcat_examples,
-        normalizer=normalizer,
-        exclusive_classes=exclusive_classes,
-        allow_none=allow_none,
-        verbose=verbose,
-    )
-
-
-class TextCatTask(SerializableTask[TextCatExample]):
-    def __init__(
-        self,
-        labels: List[str] = [],
-        template: str = _DEFAULT_TEXTCAT_TEMPLATE_V3,
-        label_definitions: Optional[Dict[str, str]] = None,
-        prompt_examples: Optional[List[TextCatExample]] = None,
-        normalizer: Optional[Callable[[str], str]] = None,
-        exclusive_classes: bool = False,
-        allow_none: bool = True,
-        verbose: bool = False,
-    ):
-        """Default TextCat task.
-
-        You can use either binary or multilabel text classification based on the
-        labels you provide.
-
-        If a single label is provided, binary classification
-        will be used. The label will get a score of `0` or `1` in `doc.cats`.
-
-        Otherwise, multilabel classification will be used. The document labels
-        in `doc.cats` will be a dictionary of strings and their score.
-
-        Lastly, you can toggle between exclusive or no-exclusive text
-        categorization by passing a flag to the `exclusive_classes` parameter.
-
-        labels (List[str]): List of labels to pass to the template. This task
-            assumes binary classification if a single label is provided.
-            Leave empty to populate it at initialization time (only if examples are provided).
-        template (str): Prompt template passed to the model.
-        label_definitions (Optional[Dict[str, str]]): Optional dict mapping a label to a description of that label.
-            These descriptions are added to the prompt to help instruct the LLM on what to extract.
-        prompt_examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that
-            reads a file containing task examples for few-shot learning. If None is
-            passed, then zero-shot learning will be used.
-        normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
-        exclusive_classes (bool): If True, require the language model to suggest only one
-            label per class. This is automatically set when using binary classification.
-        allow_none (bool): if True, there might be cases where no label is applicable.
-        verbose (bool): If True, show extra information.
-        """
-        self._template = template
-        self._normalizer = normalizer if normalizer else lowercase_normalizer()
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-        self._label_definitions = label_definitions
-        self._prompt_examples = prompt_examples or []
-        # Textcat configuration
-        self._use_binary = True if len(self._label_dict) == 1 else False
-        self._exclusive_classes = exclusive_classes
-        self._allow_none = allow_none
-        self._verbose = verbose
-
-        if self._use_binary and not self._exclusive_classes:
-            msg.warn(
-                "Binary classification should always be exclusive. Setting "
-                "the `exclusive_classes` parameter to True."
-            )
-            self._exclusive_classes = True
-
-    @property
-    def labels(self) -> Tuple[str, ...]:
-        return tuple(self._label_dict.values())
-
-    @property
-    def prompt_template(self) -> str:
-        return self._template
-
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
-        environment = jinja2.Environment()
-        _template = environment.from_string(self._template)
-        for doc in docs:
-            prompt = _template.render(
-                text=doc.text,
-                labels=list(self._label_dict.values()),
-                label_definitions=self._label_definitions,
-                examples=self._prompt_examples,
-                exclusive_classes=self._exclusive_classes,
-                allow_none=self._allow_none,
-            )
-            yield prompt
-
-    def _format_response(self, response: str) -> Dict[str, float]:
-        """Parse raw string response into a structured format
-
-        The returned dictionary contains the labels mapped to their score.
-        """
-        categories: Dict[str, float]
-        response = response.strip()
-        if self._use_binary:
-            # Binary classification: We only have one label
-            label: str = list(self._label_dict.values())[0]
-            score = 1.0 if response.upper() == "POS" else 0.0
-            categories = {label: score}
-        else:
-            # Multilabel classification
-            categories = {label: 0.0 for label in self._label_dict.values()}
-
-            pred_labels = response.split(",")
-            if self._exclusive_classes and len(pred_labels) > 1:
-                # Don't use anything but raise a debug message
-                # Don't raise an error. Let user abort if they want to.
-                msg.text(
-                    f"LLM returned multiple labels for this exclusive task: {pred_labels}.",
-                    " Will store an empty label instead.",
-                    show=self._verbose,
-                )
-                pred_labels = []
-
-            for pred in pred_labels:
-                if self._normalizer(pred.strip()) in self._label_dict:
-                    category = self._label_dict[self._normalizer(pred.strip())]
-                    categories[category] = 1.0
-        return categories
-
-    def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[str]
-    ) -> Iterable[Doc]:
-        for doc, prompt_response in zip(docs, responses):
-            cats = self._format_response(prompt_response)
-            doc.cats = cats
-            yield doc
-
-    def scorer(
-        self,
-        examples: Iterable[Example],
-    ) -> Dict[str, Any]:
-        return Scorer.score_cats(
-            examples,
-            attr="cats",
-            labels=self._label_dict.values(),
-            multi_label=not self._exclusive_classes,
-        )
-
-    def initialize(
-        self,
-        get_examples: Callable[[], Iterable["Example"]],
-        nlp: Language,
-        labels: List[str] = [],
-        n_prompt_examples: int = 0,
-        **kwargs: Any,
-    ) -> None:
-        """Initialize the TextCat task, by auto-discovering labels.
-
-        Labels can be set through, by order of precedence:
-
-        - the `[initialize]` section of the pipeline configuration
-        - the `labels` argument supplied to the task factory
-        - the labels found in the examples
-
-        get_examples (Callable[[], Iterable["Example"]]): Callable that provides examples
-            for initialization.
-        nlp (Language): Language instance.
-        labels (List[str]): Optional list of labels.
-        n_prompt_examples (int): How many prompt examples to infer from the Example objects.
-            0 by default. Takes all examples if set to -1.
-        """
-        if not labels:
-            labels = list(self._label_dict.values())
-        infer_labels = not labels
-
-        if infer_labels:
-            labels = []
-
-        for eg in get_examples():
-            if infer_labels:
-                for cat in eg.reference.cats.keys():
-                    labels.append(cat)
-            if n_prompt_examples < 0 or len(self._prompt_examples) < n_prompt_examples:
-                self._prompt_examples.append(self._create_prompt_example(eg))
-
-        self._label_dict = {
-            self._normalizer(label): label for label in sorted(set(labels))
-        }
-
-    @property
-    def _cfg_keys(self) -> List[str]:
-        return [
-            "_template",
-            "_label_dict",
-            "_label_definitions",
-            "_use_binary",
-            "_exclusive_classes",
-            "_allow_none",
-            "_verbose",
-        ]
-
-    @property
-    def _Example(self) -> Type[TextCatExample]:
-        return TextCatExample
-
-    def _create_prompt_example(self, example: Example) -> TextCatExample:
-        """Create a textcat prompt example from a spaCy example."""
-        if self._use_binary:
-            answer = (
-                "POS"
-                if example.reference.cats[list(self._label_dict.values())[0]] == 1.0
-                else "NEG"
-            )
-        else:
-            answer = ",".join(
-                [
-                    label
-                    for label, score in example.reference.cats.items()
-                    if score == 1.0
-                ]
-            )
-
-        textcat_example = TextCatExample(
-            text=example.reference.text,
-            answer=answer,
-        )
-        return textcat_example
diff --git a/spacy_llm/tasks/textcat/__init__.py b/spacy_llm/tasks/textcat/__init__.py
new file mode 100644
index 00000000..86117116
--- /dev/null
+++ b/spacy_llm/tasks/textcat/__init__.py
@@ -0,0 +1,11 @@
+from .registry import make_textcat_task, make_textcat_task_v2, make_textcat_task_v3
+from .task import TextCatTask
+from .util import TextCatExample
+
+__all__ = [
+    "make_textcat_task",
+    "make_textcat_task_v2",
+    "make_textcat_task_v3",
+    "TextCatExample",
+    "TextCatTask",
+]
diff --git a/spacy_llm/tasks/textcat/parser.py b/spacy_llm/tasks/textcat/parser.py
new file mode 100644
index 00000000..24228f7f
--- /dev/null
+++ b/spacy_llm/tasks/textcat/parser.py
@@ -0,0 +1,46 @@
+from typing import Dict, Iterable
+
+from spacy.tokens import Doc
+from wasabi import msg
+
+from .task import TextCatTask
+
+
+def parse_responses_v1_v2_v3(
+    task: TextCatTask, docs: Iterable[Doc], responses: Iterable[str]
+) -> Iterable[Dict[str, float]]:
+    """Parses LLM responses for spacy.TextCat.v1, v2 and v3
+    task (LemmaTask): Task instance.
+    docs (Iterable[Doc]): Corresponding Doc instances.
+    responses (Iterable[str]): LLM responses.
+    RETURNS (Dict[str, float]): TextCat scores per class.
+    """
+    for response in responses:
+        categories: Dict[str, float]
+        response = response.strip()
+        if task.use_binary:
+            # Binary classification: We only have one label
+            label: str = list(task.label_dict.values())[0]
+            score = 1.0 if response.upper() == "POS" else 0.0
+            categories = {label: score}
+        else:
+            # Multilabel classification
+            categories = {label: 0.0 for label in task.label_dict.values()}
+
+            pred_labels = response.split(",")
+            if task.exclusive_classes and len(pred_labels) > 1:
+                # Don't use anything but raise a debug message
+                # Don't raise an error. Let user abort if they want to.
+                msg.text(
+                    f"LLM returned multiple labels for this exclusive task: {pred_labels}.",
+                    " Will store an empty label instead.",
+                    show=task.verbose,
+                )
+                pred_labels = []
+
+            for pred in pred_labels:
+                if task.normalizer(pred.strip()) in task.label_dict:
+                    category = task.label_dict[task.normalizer(pred.strip())]
+                    categories[category] = 1.0
+
+        yield categories
diff --git a/spacy_llm/tasks/textcat/registry.py b/spacy_llm/tasks/textcat/registry.py
new file mode 100644
index 00000000..e92841b2
--- /dev/null
+++ b/spacy_llm/tasks/textcat/registry.py
@@ -0,0 +1,207 @@
+from typing import Callable, Dict, List, Optional, Type, Union
+
+from ...registry import registry
+from ...ty import ExamplesConfigType, FewshotExample, Scorer, TaskResponseParser
+from ...util import split_labels
+from .parser import parse_responses_v1_v2_v3
+from .task import DEFAULT_TEXTCAT_TEMPLATE_V1, DEFAULT_TEXTCAT_TEMPLATE_V2
+from .task import DEFAULT_TEXTCAT_TEMPLATE_V3, TextCatTask
+from .util import TextCatExample, score
+
+
+@registry.llm_tasks("spacy.TextCat.v1")
+def make_textcat_task(
+    parse_responses: Optional[TaskResponseParser[TextCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: str = "",
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    exclusive_classes: bool = False,
+    allow_none: bool = True,
+    verbose: bool = False,
+    scorer: Optional[Scorer] = None,
+) -> "TextCatTask":
+    """TextCat.v1 task factory.
+
+    You can use either binary or multilabel text classification based on the
+    labels you provide.
+
+    If a single label is provided, binary classification
+    will be used. The label will get a score of `0` or `1` in `doc.cats`.
+
+    Otherwise, multilabel classification will be used. The document labels
+    in `doc.cats` will be a dictionary of strings and their score.
+
+    Lastly, you can toggle between exclusive or no-exclusive text
+    categorization by passing a flag to the `exclusive_classes` parameter.
+
+    parse_responses (Optional[TaskResponseParser[TextCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (str): Comma-separated list of labels to pass to the template.
+        This task assumes binary classification if a single label is provided.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+    exclusive_classes (bool): If True, require the language model to suggest only one
+        label per class. This is automatically set when using binary classification.
+    allow_none (bool): if True, there might be cases where no label is applicable.
+    verbose (bool): If True, show extra information.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or TextCatExample
+    textcat_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+    return TextCatTask(
+        parse_responses=parse_responses or parse_responses_v1_v2_v3,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=DEFAULT_TEXTCAT_TEMPLATE_V1,
+        prompt_examples=textcat_examples,
+        normalizer=normalizer,
+        exclusive_classes=exclusive_classes,
+        allow_none=allow_none,
+        verbose=verbose,
+        label_definitions=None,
+        scorer=scorer or score,
+    )
+
+
+@registry.llm_tasks("spacy.TextCat.v2")
+def make_textcat_task_v2(
+    parse_responses: Optional[TaskResponseParser[TextCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_TEXTCAT_TEMPLATE_V2,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    exclusive_classes: bool = False,
+    allow_none: bool = True,
+    verbose: bool = False,
+    scorer: Optional[Scorer] = None,
+) -> "TextCatTask":
+    """TextCat.v2 task factory.
+
+    You can use either binary or multilabel text classification based on the
+    labels you provide.
+
+    If a single label is provided, binary classification
+    will be used. The label will get a score of `0` or `1` in `doc.cats`.
+
+    Otherwise, multilabel classification will be used. The document labels
+    in `doc.cats` will be a dictionary of strings and their score.
+
+    Lastly, you can toggle between exclusive or no-exclusive text
+    categorization by passing a flag to the `exclusive_classes` parameter.
+
+    parse_responses (Optional[TaskResponseParser[TextCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[List[str], str]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        This task assumes binary classification if a single label is provided.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+    exclusive_classes (bool): If True, require the language model to suggest only one
+        label per class. This is automatically set when using binary classification.
+    allow_none (bool): if True, there might be cases where no label is applicable.
+    verbose (bool): If True, show extra information.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or TextCatExample
+    textcat_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return TextCatTask(
+        parse_responses=parse_responses or parse_responses_v1_v2_v3,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        prompt_examples=textcat_examples,
+        normalizer=normalizer,
+        exclusive_classes=exclusive_classes,
+        allow_none=allow_none,
+        verbose=verbose,
+        label_definitions=None,
+        scorer=scorer or score,
+    )
+
+
+@registry.llm_tasks("spacy.TextCat.v3")
+def make_textcat_task_v3(
+    parse_responses: Optional[TaskResponseParser[TextCatTask]] = None,
+    prompt_example_type: Optional[Type[FewshotExample]] = None,
+    labels: Union[List[str], str] = [],
+    template: str = DEFAULT_TEXTCAT_TEMPLATE_V3,
+    label_definitions: Optional[Dict[str, str]] = None,
+    examples: ExamplesConfigType = None,
+    normalizer: Optional[Callable[[str], str]] = None,
+    exclusive_classes: bool = False,
+    allow_none: bool = True,
+    verbose: bool = False,
+    scorer: Optional[Scorer] = None,
+) -> "TextCatTask":
+    """TextCat.v3 task factory.
+
+    You can use either binary or multilabel text classification based on the
+    labels you provide.
+
+    If a single label is provided, binary classification
+    will be used. The label will get a score of `0` or `1` in `doc.cats`.
+
+    Otherwise, multilabel classification will be used. The document labels
+    in `doc.cats` will be a dictionary of strings and their score.
+
+    Lastly, you can toggle between exclusive or no-exclusive text
+    categorization by passing a flag to the `exclusive_classes` parameter.
+
+    parse_responses (Optional[TaskResponseParser[TextCatTask]]): Callable for parsing LLM responses for this
+        task.
+    prompt_example_type (Optional[Type[FewshotExample]]): Type to use for fewshot examples.
+    labels (Union[List[str], str]): List of labels to pass to the template,
+        either an actual list or a comma-separated string.
+        This task assumes binary classification if a single label is provided.
+        Leave empty to populate it at initialization time (only if examples are provided).
+    template (str): Prompt template passed to the model.
+    label_definitions (Optional[Dict[str, str]]): Optional dict mapping a label to a description of that label.
+        These descriptions are added to the prompt to help instruct the LLM on what to extract.
+    examples (Optional[Callable[[], Iterable[Any]]]): Optional callable that reads a file containing task examples for
+        few-shot learning. If None is passed, then zero-shot learning will be used.
+    normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+    exclusive_classes (bool): If True, require the language model to suggest only one
+        label per class. This is automatically set when using binary classification.
+    allow_none (bool): if True, there might be cases where no label is applicable.
+    verbose (bool): If True, show extra information.
+    scorer (Optional[Scorer]): Scorer function.
+    """
+
+    labels_list = split_labels(labels)
+    raw_examples = examples() if callable(examples) else examples
+    example_type = prompt_example_type or TextCatExample
+    textcat_examples = (
+        [example_type(**eg) for eg in raw_examples] if raw_examples else None
+    )
+
+    return TextCatTask(
+        parse_responses=parse_responses or parse_responses_v1_v2_v3,
+        prompt_example_type=example_type,
+        labels=labels_list,
+        template=template,
+        label_definitions=label_definitions,
+        prompt_examples=textcat_examples,
+        normalizer=normalizer,
+        exclusive_classes=exclusive_classes,
+        allow_none=allow_none,
+        verbose=verbose,
+        scorer=scorer or score,
+    )
diff --git a/spacy_llm/tasks/textcat/task.py b/spacy_llm/tasks/textcat/task.py
new file mode 100644
index 00000000..b638db17
--- /dev/null
+++ b/spacy_llm/tasks/textcat/task.py
@@ -0,0 +1,158 @@
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type
+
+from spacy.language import Language
+from spacy.tokens import Doc
+from spacy.training import Example
+from wasabi import msg
+
+from ...compat import Self
+from ...ty import FewshotExample, Scorer, TaskResponseParser
+from ..builtin_task import BuiltinTaskWithLabels
+from ..templates import read_template
+
+DEFAULT_TEXTCAT_TEMPLATE_V1 = read_template("textcat.v1")
+DEFAULT_TEXTCAT_TEMPLATE_V2 = read_template("textcat.v2")
+DEFAULT_TEXTCAT_TEMPLATE_V3 = read_template("textcat.v3")
+
+
+class TextCatTask(BuiltinTaskWithLabels):
+    def __init__(
+        self,
+        parse_responses: TaskResponseParser[Self],
+        prompt_example_type: Type[FewshotExample],
+        labels: List[str],
+        template: str,
+        label_definitions: Optional[Dict[str, str]],
+        prompt_examples: Optional[List[FewshotExample]],
+        normalizer: Optional[Callable[[str], str]],
+        exclusive_classes: bool,
+        allow_none: bool,
+        verbose: bool,
+        scorer: Scorer,
+    ):
+        """Default TextCat task.
+
+        You can use either binary or multilabel text classification based on the
+        labels you provide.
+
+        If a single label is provided, binary classification
+        will be used. The label will get a score of `0` or `1` in `doc.cats`.
+
+        Otherwise, multilabel classification will be used. The document labels
+        in `doc.cats` will be a dictionary of strings and their score.
+
+        Lastly, you can toggle between exclusive or no-exclusive text
+        categorization by passing a flag to the `exclusive_classes` parameter.
+
+        parse_responses (TaskResponseParser[Self]): Callable for parsing LLM responses for this task.
+        prompt_example_type (Type[FewshotExample]): Type to use for fewshot examples.
+        labels (List[str]): List of labels to pass to the template. This task
+            assumes binary classification if a single label is provided.
+            Leave empty to populate it at initialization time (only if examples are provided).
+        template (str): Prompt template passed to the model.
+        label_definitions (Optional[Dict[str, str]]): Optional dict mapping a label to a description of that label.
+            These descriptions are added to the prompt to help instruct the LLM on what to extract.
+        prompt_examples (Optional[List[FewshotExample]]): Optional list of few-shot examples to include in prompts.
+        normalizer (Optional[Callable[[str], str]]): Optional normalizer function.
+        exclusive_classes (bool): If True, require the language model to suggest only one
+            label per class. This is automatically set when using binary classification.
+        allow_none (bool): if True, there might be cases where no label is applicable.
+        verbose (bool): If True, show extra information.
+        scorer (Scorer): Scorer function.
+        """
+        super().__init__(
+            parse_responses=parse_responses,
+            prompt_example_type=prompt_example_type,
+            template=template,
+            prompt_examples=prompt_examples,
+            labels=labels,
+            label_definitions=label_definitions,
+            normalizer=normalizer,
+        )
+        # Textcat configuration
+        self._use_binary = True if len(self._label_dict) == 1 else False
+        self._exclusive_classes = exclusive_classes
+        self._allow_none = allow_none
+        self._verbose = verbose
+        self._scorer = scorer
+
+        if self._use_binary and not self._exclusive_classes:
+            msg.warn(
+                "Binary classification should always be exclusive. Setting "
+                "the `exclusive_classes` parameter to True."
+            )
+            self._exclusive_classes = True
+
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[str]:
+        return super().generate_prompts(
+            docs=docs,
+            labels=list(self._label_dict.values()),
+            label_definitions=self._label_definitions,
+            exclusive_classes=self._exclusive_classes,
+            allow_none=self._allow_none,
+        )
+
+    def parse_responses(
+        self, docs: Iterable[Doc], responses: Iterable[str]
+    ) -> Iterable[Doc]:
+        for doc, cats in zip(docs, self._parse_responses(self, docs, responses)):
+            doc.cats = cats
+            yield doc
+
+    def scorer(
+        self,
+        examples: Iterable[Example],
+    ) -> Dict[str, Any]:
+        return self._scorer(
+            examples,
+            attr="cats",
+            labels=self._label_dict.values(),
+            multi_label=not self._exclusive_classes,
+        )
+
+    def initialize(
+        self,
+        get_examples: Callable[[], Iterable["Example"]],
+        nlp: Language,
+        labels: List[str] = [],
+        n_prompt_examples: int = 0,
+    ) -> None:
+        super()._initialize(
+            get_examples=get_examples,
+            nlp=nlp,
+            labels=labels,
+            n_prompt_examples=n_prompt_examples,
+            use_binary=self._use_binary,
+            label_dict=self._label_dict,
+        )
+
+    @property
+    def _cfg_keys(self) -> List[str]:
+        return [
+            "_template",
+            "_label_dict",
+            "_label_definitions",
+            "_use_binary",
+            "_exclusive_classes",
+            "_allow_none",
+            "_verbose",
+        ]
+
+    def _extract_labels_from_example(self, example: Example) -> List[str]:
+        return list(example.reference.cats.keys())
+
+    @property
+    def use_binary(self) -> bool:
+        return self._use_binary
+
+    @property
+    def exclusive_classes(self) -> bool:
+        return self._exclusive_classes
+
+    @property
+    def allow_none(self) -> bool:
+        return self._allow_none
+
+    @property
+    def verbose(self) -> bool:
+        return self._verbose
diff --git a/spacy_llm/tasks/textcat/util.py b/spacy_llm/tasks/textcat/util.py
new file mode 100644
index 00000000..0b7d8689
--- /dev/null
+++ b/spacy_llm/tasks/textcat/util.py
@@ -0,0 +1,46 @@
+from typing import Any, Dict, Iterable
+
+from spacy.scorer import Scorer
+from spacy.training import Example
+
+from ...compat import BaseModel, Self
+
+
+class TextCatExample(BaseModel):
+    text: str
+    answer: str
+
+    @classmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        if kwargs["use_binary"]:
+            answer = (
+                "POS"
+                if example.reference.cats[list(kwargs["label_dict"].values())[0]] == 1.0
+                else "NEG"
+            )
+        else:
+            answer = ",".join(
+                [
+                    label
+                    for label, score in example.reference.cats.items()
+                    if score == 1.0
+                ]
+            )
+
+        return TextCatExample(
+            text=example.reference.text,
+            answer=answer,
+        )
+
+
+def score(examples: Iterable[Example], **kwargs) -> Dict[str, Any]:
+    """Score textcat accuracy in examples.
+    examples (Iterable[Example]): Examples to score.
+    RETURNS (Dict[str, Any]): Dict with metric name -> score.
+    """
+    return Scorer.score_cats(
+        examples,
+        attr=kwargs["attr"],
+        labels=kwargs["labels"],
+        multi_label=kwargs["multi_label"],
+    )
diff --git a/spacy_llm/tasks/util/__init__.py b/spacy_llm/tasks/util/__init__.py
index 924f6eaa..0e147ca0 100644
--- a/spacy_llm/tasks/util/__init__.py
+++ b/spacy_llm/tasks/util/__init__.py
@@ -1,4 +1,3 @@
 from .parsing import find_substrings
-from .serialization import SerializableTask
 
-__all__ = ["find_substrings", "SerializableTask"]
+__all__ = ["find_substrings"]
diff --git a/spacy_llm/tasks/util/parsing.py b/spacy_llm/tasks/util/parsing.py
index 4bd5f327..df73593a 100644
--- a/spacy_llm/tasks/util/parsing.py
+++ b/spacy_llm/tasks/util/parsing.py
@@ -1,4 +1,4 @@
-from typing import Iterable, Tuple
+from typing import Iterable, List, Tuple
 
 
 def _unique(items: Iterable[str]) -> Iterable[str]:
@@ -18,7 +18,8 @@ def find_substrings(
     *,
     case_sensitive: bool = False,
     single_match: bool = False,
-) -> Iterable[Tuple[int, int]]:
+    find_after: int = 0,
+) -> List[Tuple[int, int]]:
     """Given a list of substrings, find their character start and end positions
     in a text"""
 
@@ -30,7 +31,7 @@ def find_substrings(
     substrings = _unique(substrings)
     offsets = []
     for substring in substrings:
-        search_from = 0
+        search_from = find_after
         # Search until one hit is found. Continue only if single_match is False.
         while True:
             start = text.find(substring, search_from)
diff --git a/spacy_llm/tasks/util/serialization.py b/spacy_llm/tasks/util/serialization.py
deleted file mode 100644
index 75fdc642..00000000
--- a/spacy_llm/tasks/util/serialization.py
+++ /dev/null
@@ -1,136 +0,0 @@
-import abc
-from pathlib import Path
-from typing import Any, Dict, Generic, List, Tuple, Type, TypeVar, cast
-
-import srsly
-from pydantic import BaseModel
-from spacy import util
-
-ExampleType = TypeVar("ExampleType", bound=BaseModel)
-
-
-class SerializableTask(abc.ABC, Generic[ExampleType]):
-    """A task that can be serialized and deserialized."""
-
-    _prompt_examples: List[ExampleType]
-
-    @property
-    @abc.abstractmethod
-    def _cfg_keys(self) -> List[str]:
-        """A list of configuration attributes to serialize."""
-        pass
-
-    @property
-    @abc.abstractmethod
-    def _Example(self) -> Type[ExampleType]:
-        """The example type."""
-        pass
-
-    def get_cfg(self) -> Dict[str, Any]:
-        """Serialize the task's configuration attributes."""
-        cfg = {key: getattr(self, key) for key in self._cfg_keys}
-        return cfg
-
-    def set_cfg(self, cfg: Dict[str, Any]) -> None:
-        """Deserialize the task's configuration attributes.
-
-        cfg (Dict[str, Any]): dictionary containing configuration attributes.
-        """
-        for key, value in cfg.items():
-            setattr(self, key, value)
-
-    def _get_prompt_examples(self) -> List[Dict[str, Any]]:
-        """Serialize examples."""
-        examples = [eg.dict() for eg in self._prompt_examples]
-        return examples
-
-    def _set_prompt_examples(self, examples: List[Dict[str, Any]]) -> None:
-        """Deserialize examples from bytes.
-
-        examples (List[Dict[str, Any]]): serialized examples.
-        """
-        self._prompt_examples = [self._Example.parse_obj(eg) for eg in examples]
-
-    def to_bytes(
-        self,
-        *,
-        exclude: Tuple[str] = cast(Tuple[str], tuple()),
-    ) -> bytes:
-        """Serialize the LLMWrapper to a bytestring.
-
-        exclude (Tuple): Names of properties to exclude from serialization.
-        RETURNS (bytes): The serialized object.
-        """
-
-        serialize = {
-            "cfg": lambda: srsly.json_dumps(self.get_cfg()),
-            "prompt_examples": lambda: srsly.msgpack_dumps(self._get_prompt_examples()),
-        }
-
-        return util.to_bytes(serialize, exclude)
-
-    def from_bytes(
-        self,
-        bytes_data: bytes,
-        *,
-        exclude: Tuple[str] = cast(Tuple[str], tuple()),
-    ) -> "SerializableTask":
-        """Load the Task from a bytestring.
-
-        bytes_data (bytes): The data to load.
-        exclude (Tuple[str]): Names of properties to exclude from deserialization.
-        RETURNS (SpanTask): Modified SpanTask instance.
-        """
-
-        deserialize = {
-            "cfg": lambda b: self.set_cfg(srsly.json_loads(b)),
-            "prompt_examples": lambda b: self._set_prompt_examples(
-                srsly.msgpack_loads(b)
-            ),
-        }
-
-        util.from_bytes(bytes_data, deserialize, exclude)
-        return self
-
-    def to_disk(
-        self,
-        path: Path,
-        *,
-        exclude: Tuple[str] = cast(Tuple[str], tuple()),
-    ) -> None:
-        """Serialize the task to disk.
-
-        path (Path): A path (currently unused).
-        exclude (Tuple): Names of properties to exclude from serialization.
-        """
-
-        serialize = {
-            "cfg": lambda p: srsly.write_json(p, self.get_cfg()),
-            "prompt_examples": lambda p: srsly.write_msgpack(
-                p, self._get_prompt_examples()
-            ),
-        }
-
-        util.to_disk(path, serialize, exclude)
-
-    def from_disk(
-        self,
-        path: Path,
-        *,
-        exclude: Tuple[str] = cast(Tuple[str], tuple()),
-    ) -> "SerializableTask":
-        """Deserialize the task from disk.
-
-        path (Path): A path (currently unused).
-        exclude (Tuple): Names of properties to exclude from serialization.
-        """
-
-        deserialize = {
-            "cfg": lambda p: self.set_cfg(srsly.read_json(p)),
-            "prompt_examples": lambda p: self._set_prompt_examples(
-                srsly.read_msgpack(p)
-            ),
-        }
-
-        util.from_disk(path, deserialize, exclude)
-        return self
diff --git a/spacy_llm/ty.py b/spacy_llm/ty.py
index e31a272b..c984df9a 100644
--- a/spacy_llm/ty.py
+++ b/spacy_llm/ty.py
@@ -1,21 +1,23 @@
+import abc
 import inspect
 import typing
 import warnings
 from pathlib import Path
-from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Type, Union
-from typing import cast
+from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Type, TypeVar
+from typing import Union, cast
 
 from spacy.tokens import Doc
 from spacy.training.example import Example
 from spacy.vocab import Vocab
 
-from .compat import Protocol, runtime_checkable
+from .compat import BaseModel, Protocol, Self, runtime_checkable
 from .models import langchain
 
-_Prompt = Any
-_Response = Any
+_PromptType = Any
+_ResponseType = Any
+_ParsedResponseType = Any
 
-PromptExecutor = Callable[[Iterable[_Prompt]], Iterable[_Response]]
+PromptExecutorType = Callable[[Iterable[_PromptType]], Iterable[_ResponseType]]
 ExamplesConfigType = Union[
     Iterable[Dict[str, Any]], Callable[[], Iterable[Dict[str, Any]]], None
 ]
@@ -55,24 +57,69 @@ def from_disk(
         ...
 
 
+class FewshotExample(abc.ABC, BaseModel):
+    @classmethod
+    @abc.abstractmethod
+    def generate(cls, example: Example, **kwargs) -> Self:
+        """Create a fewshot example from a spaCy example.
+        example (Example): spaCy example.
+        """
+
+
 @runtime_checkable
-class Scorable(Protocol):
+class ScorableTask(Protocol):
+    """Differs from Scorable in that it describes an object with a scorer() function, i. e. a scorable
+    as checked for by the LLMWrapper component.
+    """
+
     def scorer(
         self,
         examples: Iterable[Example],
     ) -> Dict[str, Any]:
         """Scores performance on examples."""
-        ...
+
+
+@runtime_checkable
+class Scorer(Protocol):
+    """Differs from ScorableTask in that it describes a Callable with a call signature matching the scorer()
+    function + kwargs, i. e. a scorable as passed via the configuration.
+    """
+
+    def __call__(self, examples: Iterable[Example], **kwargs) -> Dict[str, Any]:
+        """Score performance on examples.
+        examples (Iterable[Example]): Examples to score.
+        RETURNS (Dict[str, Any]): Dict with metric name -> score.
+        """
 
 
 @runtime_checkable
 class LLMTask(Protocol):
-    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[_Prompt]:
-        ...
+    def generate_prompts(self, docs: Iterable[Doc], **kwargs) -> Iterable[_PromptType]:
+        """Generate prompts from docs.
+        docs (Iterable[Doc]): Docs to generate prompts from.
+        RETURNS (Iterable[_PromptType]): Iterable with one prompt per doc.
+        """
 
     def parse_responses(
-        self, docs: Iterable[Doc], responses: Iterable[_Response]
+        self, docs: Iterable[Doc], responses: Iterable[_ResponseType]
     ) -> Iterable[Doc]:
+        """
+        Parses LLM responses.
+        docs (Iterable[Doc]): Docs to map responses into.
+        responses ([Iterable[_ResponseType]]): LLM responses.
+        RETURNS (Iterable[Doc]]): Updated docs.
+        """
+
+
+_TaskContraT = TypeVar("_TaskContraT", bound=LLMTask, contravariant=True)
+
+
+class TaskResponseParser(Protocol[_TaskContraT]):
+    """Generic protocol for parsing functions with specific tasks."""
+
+    def __call__(
+        self, task: _TaskContraT, docs: Iterable[Doc], responses: Iterable[Any]
+    ) -> Iterable[Any]:
         ...
 
 
@@ -84,11 +131,14 @@ def prompt_template(self) -> str:
 
 
 @runtime_checkable
-class Labeled(Protocol):
+class LabeledTask(Protocol):
     @property
     def labels(self) -> Tuple[str, ...]:
         ...
 
+    def add_label(self, label: str) -> int:
+        ...
+
 
 @runtime_checkable
 class Cache(Protocol):
@@ -154,7 +204,7 @@ def _do_args_match(out_arg: Iterable, in_arg: Iterable) -> bool:
     )
 
 
-def _extract_model_call_signature(model: PromptExecutor) -> Dict[str, Any]:
+def _extract_model_call_signature(model: PromptExecutorType) -> Dict[str, Any]:
     """Extract call signature from model object.
     model (PromptExecutor): Model object to extract call signature from.
     RETURNS (Dict[str, Any]): Type per argument name.
@@ -190,7 +240,7 @@ def _extract_model_call_signature(model: PromptExecutor) -> Dict[str, Any]:
     return signature
 
 
-def validate_type_consistency(task: LLMTask, model: PromptExecutor) -> None:
+def validate_type_consistency(task: LLMTask, model: PromptExecutorType) -> None:
     """Check whether the types of the task and model signatures match.
     task (LLMTask): Specified task.
     backend (PromptExecutor): Specified model.
diff --git a/spacy_llm/util.py b/spacy_llm/util.py
index 93f3bea9..de079128 100644
--- a/spacy_llm/util.py
+++ b/spacy_llm/util.py
@@ -1,10 +1,9 @@
 from pathlib import Path
 from typing import Any, Dict, Iterable, List, Union
 
-from confection import Config
+from confection import Config, SimpleFrozenDict
 from spacy.language import Language
-from spacy.util import SimpleFrozenDict, get_sourced_components, load_config
-from spacy.util import load_model_from_config
+from spacy.util import get_sourced_components, load_config, load_model_from_config
 
 
 def split_labels(labels: Union[str, Iterable[str]]) -> List[str]:
diff --git a/usage_examples/multitask_openai/fewshot.cfg b/usage_examples/multitask_openai/fewshot.cfg
index 726fdd39..87e27733 100644
--- a/usage_examples/multitask_openai/fewshot.cfg
+++ b/usage_examples/multitask_openai/fewshot.cfg
@@ -19,9 +19,9 @@ labels = SIZE,TYPE,TOPPING,PRODUCT
 path = ${paths.examples}
 
 [components.llm_ner.model]
-@llm_models = "spacy.Text-Davinci.v1"
+@llm_models = "spacy.Text-Davinci.v2"
 name = "text-davinci-003"
-config = {"temperature": 0.3}
+config = {"temperature": 0.0}
 
 [components.llm_textcat]
 factory = "llm"
@@ -36,6 +36,6 @@ exclusive_classes=True
 path = ${paths.examples}
 
 [components.llm_textcat.model]
-@llm_models = "spacy.Text-Davinci.v1"
+@llm_models = "spacy.Text-Davinci.v2"
 name = "text-davinci-003"
-config = {"temperature": 0.3}
+config = {"temperature": 0.0}
diff --git a/usage_examples/multitask_openai/zeroshot.cfg b/usage_examples/multitask_openai/zeroshot.cfg
index fc91a507..cafd4b9c 100644
--- a/usage_examples/multitask_openai/zeroshot.cfg
+++ b/usage_examples/multitask_openai/zeroshot.cfg
@@ -12,19 +12,18 @@ factory = "llm"
 labels = SIZE,TYPE,TOPPING,PRODUCT
 
 [components.llm_ner.model]
-@llm_models = "spacy.Text-Davinci.v1"
+@llm_models = "spacy.Text-Davinci.v2"
 name = "text-davinci-003"
-config = {"temperature": 0.3}
+config = {"temperature": 0.0}
 
 [components.llm_textcat]
 factory = "llm"
 
 [components.llm_textcat.task]
-@llm_tasks = "spacy.TextCat.v2"
+@llm_tasks = "spacy.TextCat.v3"
 labels = INFORMATION,ORDER
 exclusive_classes=True
 
 [components.llm_textcat.model]
-@llm_models = "spacy.Text-Davinci.v1"
+@llm_models = "spacy.Text-Davinci.v2"
 name = "text-davinci-003"
-config = {"temperature": 0.3}
diff --git a/usage_examples/ner_dolly/examples.yml b/usage_examples/ner_dolly/examples.yml
index 7e653f7d..6d02fb1e 100644
--- a/usage_examples/ner_dolly/examples.yml
+++ b/usage_examples/ner_dolly/examples.yml
@@ -1,15 +1,15 @@
 - text: Jack and Jill went up the hill.
   entities:
-    PER:
+    PERSON:
       - Jack
       - Jill
-    LOC:
+    LOCATION:
       - hill
 - text: Jack fell down and broke his crown.
   entities:
-    PER:
+    PERSON:
       - Jack
 - text: Jill came tumbling after.
   entities:
-    PER:
+    PERSON:
       - Jill
diff --git a/usage_examples/ner_v3_openai/README.md b/usage_examples/ner_v3_openai/README.md
new file mode 100644
index 00000000..2102362e
--- /dev/null
+++ b/usage_examples/ner_v3_openai/README.md
@@ -0,0 +1,76 @@
+# Using GPT Models from OpenAI for Named Entity Recognition (NER)
+
+
+This example shows how you can use a model from OpenAI for Named Entity Recognition (NER).
+The NER prompt is based on the [PromptNER](https://arxiv.org/abs/2305.15444) paper and
+utilizes Chain-of-Thought reasoning to extract named entities.
+
+First, create a new API key from
+[openai.com](https://platform.openai.com/account/api-keys) or fetch an existing
+one. Record the secret key and make sure this is available as an environmental
+variable:
+
+```sh
+export OPENAI_API_KEY="sk-..."
+export OPENAI_API_ORG="org-..."
+```
+
+Then, you can run the pipeline on a sample text via:
+
+
+```sh
+python run_pipeline.py [TEXT] [PATH TO CONFIG] [PATH TO FILE WITH EXAMPLES]
+```
+
+For example:
+
+```sh
+python run_pipeline.py \
+    ""Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok." \
+    ./fewshot.cfg
+    ./examples.json
+```
+
+This example assings labels for DISH, INGREDIENT, and EQUIPMENT.
+
+You can change around the labels and examples for your use case.
+You can find the few-shot examples in the
+`examples.json` file. Feel free to change and update it to your liking.
+We also support other file formats, including `yml` and `jsonl` for these examples.
+
+
+### Negative examples
+
+While not required, The Chain-of-Thought reasoning for the `spacy.NER.v3` task
+works best in our experience when both positive and negative examples are provided.
+
+This prompts the Language model with concrete examples of what **is not** an entity
+for your use case.
+
+Here's an example that helps define the INGREDIENT label for the LLM.
+
+```json
+[
+    {
+        "text": "You can't get a great chocolate flavor with carob.",
+        "spans": [
+            {
+                "text": "chocolate",
+                "is_entity": false,
+                "label": "==NONE==",
+                "reason": "is a flavor in this context, not an ingredient"
+            },
+            {
+                "text": "carob",
+                "is_entity": true,
+                "label": "INGREDIENT",
+                "reason": "is an ingredient to add chocolate flavor"
+            }
+        ]
+    }
+    ...
+]
+```
+
+In this example, "chocolate" is not an ingredient even though it could be in other contexts.
+We explain that via the "reason" property of this example.
diff --git a/usage_examples/ner_v3_openai/__init__.py b/usage_examples/ner_v3_openai/__init__.py
new file mode 100644
index 00000000..06fab2f6
--- /dev/null
+++ b/usage_examples/ner_v3_openai/__init__.py
@@ -0,0 +1,3 @@
+from .run_pipeline import run_pipeline
+
+__all__ = ["run_pipeline"]
diff --git a/usage_examples/ner_v3_openai/examples.json b/usage_examples/ner_v3_openai/examples.json
new file mode 100644
index 00000000..f38f5e92
--- /dev/null
+++ b/usage_examples/ner_v3_openai/examples.json
@@ -0,0 +1,36 @@
+[
+  {
+    "text": "You can't get a great chocolate flavor with carob.",
+    "spans": [
+      {
+        "text": "chocolate",
+        "is_entity": false,
+        "label": "==NONE==",
+        "reason": "is a flavor in this context, not an ingredient"
+      },
+      {
+        "text": "carob",
+        "is_entity": true,
+        "label": "INGREDIENT",
+        "reason": "is an ingredient to add chocolate flavor"
+      }
+    ]
+  },
+  {
+    "text": "You can probably sand-blast it if it's an anodized aluminum pan",
+    "spans": [
+      {
+        "text": "sand-blast",
+        "is_entity": false,
+        "label": "==NONE==",
+        "reason": "is a cleaning technique, not some kind of equipment"
+      },
+      {
+        "text": "anodized aluminum pan",
+        "is_entity": true,
+        "label": "EQUIPMENT",
+        "reason": "is a piece of cooking equipment, anodized is included since it describes the type of pan"
+      }
+    ]
+  }
+]
diff --git a/usage_examples/ner_v3_openai/fewshot.cfg b/usage_examples/ner_v3_openai/fewshot.cfg
new file mode 100644
index 00000000..6d024875
--- /dev/null
+++ b/usage_examples/ner_v3_openai/fewshot.cfg
@@ -0,0 +1,31 @@
+[paths]
+examples = null
+
+[nlp]
+lang = "en"
+pipeline = ["llm"]
+
+[components]
+
+[components.llm]
+factory = "llm"
+
+[components.llm.task]
+@llm_tasks = "spacy.NER.v3"
+labels = ["DISH", "INGREDIENT", "EQUIPMENT"]
+description = Entities are the names food dishes,
+    ingredients, and any kind of cooking equipment.
+    Adjectives, verbs, adverbs are not entities.
+    Pronouns are not entities.
+
+[components.llm.task.label_definitions]
+DISH = "Known food dishes, e.g. Lobster Ravioli, garlic bread"
+INGREDIENT = "Individual parts of a food dish, including herbs and spices."
+EQUIPMENT = "Any kind of cooking equipment. e.g. oven, cooking pot, grill"
+
+[components.llm.task.examples]
+@misc = "spacy.FewShotReader.v1"
+path = "${paths.examples}"
+
+[components.llm.model]
+@llm_models = "spacy.GPT-3-5.v1"
diff --git a/usage_examples/ner_v3_openai/run_pipeline.py b/usage_examples/ner_v3_openai/run_pipeline.py
new file mode 100644
index 00000000..ac182bbc
--- /dev/null
+++ b/usage_examples/ner_v3_openai/run_pipeline.py
@@ -0,0 +1,29 @@
+from pathlib import Path
+
+import typer
+from wasabi import msg
+
+from spacy_llm.util import assemble
+
+Arg = typer.Argument
+Opt = typer.Option
+
+
+def run_pipeline(
+    # fmt: off
+    text: str = Arg("", help="Text to perform Named Entity Recognition on."),
+    config_path: Path = Arg(..., help="Path to the configuration file to use."),
+    examples_path: Path = Arg(..., help="Path to the examples file to use."),
+    verbose: bool = Opt(False, "--verbose", "-v", help="Show extra information."),
+    # fmt: on
+):
+    msg.text(f"Loading config from {config_path}", show=verbose)
+    nlp = assemble(config_path, overrides={"paths.examples": str(examples_path)})
+    doc = nlp(text)
+
+    msg.text(f"Text: {doc.text}")
+    msg.text(f"Entities: {[(ent.text, ent.label_) for ent in doc.ents]}")
+
+
+if __name__ == "__main__":
+    typer.run(run_pipeline)
diff --git a/usage_examples/rel_openai/fewshot.cfg b/usage_examples/rel_openai/fewshot.cfg
index 1995e94a..f65fe26f 100644
--- a/usage_examples/rel_openai/fewshot.cfg
+++ b/usage_examples/rel_openai/fewshot.cfg
@@ -22,7 +22,7 @@ labels = LivesIn,Visits
 path = ${paths.examples}
 
 [components.llm_rel.model]
-@llm_models = "spacy.GPT-3-5.v1"
+@llm_models = "spacy.GPT-3-5.v2"
 
 [initialize]
 vectors = "en_core_web_md"
diff --git a/usage_examples/rel_openai/zeroshot.cfg b/usage_examples/rel_openai/zeroshot.cfg
index ef0d3f94..3a38afc1 100644
--- a/usage_examples/rel_openai/zeroshot.cfg
+++ b/usage_examples/rel_openai/zeroshot.cfg
@@ -18,7 +18,7 @@ factory = "llm"
 labels = LivesIn,Visits
 
 [components.llm_rel.model]
-@llm_models = "spacy.GPT-3-5.v1"
+@llm_models = "spacy.GPT-3-5.v2"
 
 [initialize]
 vectors = "en_core_web_md"
diff --git a/usage_examples/streamlit/streamlit_app.py b/usage_examples/streamlit/streamlit_app.py
index 4cebf177..f08527be 100644
--- a/usage_examples/streamlit/streamlit_app.py
+++ b/usage_examples/streamlit/streamlit_app.py
@@ -22,7 +22,7 @@
 [components.llm.backend]
 @llm_models = "spacy.REST.v1"
 api = "OpenAI"
-config = {"model": "gpt-3.5-turbo", "temperature": 0.3}
+config = {"model": "gpt-3.5-turbo", "temperature": 0.0}
 
 [components.llm.task]
 @llm_tasks = "spacy.NER.v2"
@@ -47,7 +47,7 @@
 [components.llm.backend]
 @llm_models = "spacy.REST.v1"
 api = "OpenAI"
-config = {"model": "gpt-3.5-turbo", "temperature": 0.3}
+config = {"model": "gpt-3.5-turbo", "temperature": 0.0}
 
 [components.llm.task]
 @llm_tasks = "spacy.TextCat.v2"
diff --git a/usage_examples/textcat_dolly/README.md b/usage_examples/textcat_dolly/README.md
new file mode 100644
index 00000000..049191d0
--- /dev/null
+++ b/usage_examples/textcat_dolly/README.md
@@ -0,0 +1,33 @@
+# Using open-source Dolly models hosted on Huggingface
+
+This example shows how you can use the [open-source Dolly
+models](https://github.com/databrickslabs/dolly) hosted on Huggingface for categorizing texts in
+zero- or few-shot settings. Here, we perform binary text classification to
+determine if a given text is an `INSULT` or a `COMPLIMENT`.
+
+You can run the pipeline on a sample text via:
+
+```sh
+python run_pipeline.py [TEXT] [PATH TO CONFIG] [PATH TO FILE WITH EXAMPLES]
+```
+
+For example:
+
+```sh
+python run_pipeline.py "You look great today! Nice shirt!" ./zeroshot.cfg
+```
+or, for few-shot:
+```sh
+python run_pipeline.py "You look great today! Nice shirt!" ./fewshot.cfg ./examples.jsonl
+```
+
+You can also include examples to perform few-shot annotation. To do so, use the
+`fewshot.cfg` file instead. You can find the few-shot examples in
+the `examples.jsonl` file. Feel free to change and update it to your liking.
+We also support other file formats, including `.yml`, `.yaml` and `.json`.
+
+Finally, you can update the Dolly model in the configuration file. We're using
+[`dolly-v2-3b`](https://huggingface.co/databricks/dolly-v2-3b) by default, but
+you can change it to a larger model size like
+[`dolly-v2-7b`](https://huggingface.co/databricks/dolly-v2-7b) or
+[`dolly-v2-12b`](https://huggingface.co/databricks/dolly-v2-12b).
diff --git a/usage_examples/textcat_dolly/__init__.py b/usage_examples/textcat_dolly/__init__.py
new file mode 100644
index 00000000..06fab2f6
--- /dev/null
+++ b/usage_examples/textcat_dolly/__init__.py
@@ -0,0 +1,3 @@
+from .run_pipeline import run_pipeline
+
+__all__ = ["run_pipeline"]
diff --git a/usage_examples/textcat_dolly/examples.jsonl b/usage_examples/textcat_dolly/examples.jsonl
new file mode 100644
index 00000000..68ea1d4b
--- /dev/null
+++ b/usage_examples/textcat_dolly/examples.jsonl
@@ -0,0 +1,3 @@
+{"text":"Shall I compare thee to a summer's day? Thou art more lovely and more temperate","answer":"COMPLIMENT"}
+{"text":"That you have such a February face, so full of frost, of storm and cloudiness","answer":"INSULT"}
+{"text":"Thou art wise as thou art beautiful","answer":"COMPLIMENT"}
diff --git a/usage_examples/textcat_dolly/fewshot.cfg b/usage_examples/textcat_dolly/fewshot.cfg
new file mode 100644
index 00000000..25dda2a1
--- /dev/null
+++ b/usage_examples/textcat_dolly/fewshot.cfg
@@ -0,0 +1,28 @@
+[paths]
+examples = null
+
+[nlp]
+lang = "en"
+pipeline = ["llm"]
+batch_size = 128
+
+[components]
+
+[components.llm]
+factory = "llm"
+
+[components.llm.model]
+@llm_models = "spacy.Dolly.v1"
+name = "dolly-v2-3b"
+
+[components.llm.task]
+@llm_tasks = "spacy.TextCat.v2"
+labels = COMPLIMENT,INSULT
+exclusive_classes = true
+
+[components.llm.task.examples]
+@misc = "spacy.FewShotReader.v1"
+path = ${paths.examples}
+
+[components.llm.task.normalizer]
+@misc = "spacy.LowercaseNormalizer.v1"
diff --git a/usage_examples/textcat_dolly/run_pipeline.py b/usage_examples/textcat_dolly/run_pipeline.py
new file mode 100644
index 00000000..cd0c8baa
--- /dev/null
+++ b/usage_examples/textcat_dolly/run_pipeline.py
@@ -0,0 +1,43 @@
+import os
+from pathlib import Path
+from typing import Optional
+
+import typer
+from wasabi import msg
+
+from spacy_llm.util import assemble
+
+Arg = typer.Argument
+Opt = typer.Option
+
+
+def run_pipeline(
+    # fmt: off
+    text: str = Arg("", help="Text to perform text categorization on."),
+    config_path: Path = Arg(..., help="Path to the configuration file to use."),
+    examples_path: Optional[Path] = Arg(None, help="Path to the examples file to use (few-shot only)."),
+    verbose: bool = Opt(False, "--verbose", "-v", help="Show extra information."),
+    # fmt: on
+):
+    if not os.getenv("OPENAI_API_KEY", None):
+        msg.fail(
+            "OPENAI_API_KEY env variable was not found. "
+            "Set it by running 'export OPENAI_API_KEY=...' and try again.",
+            exits=1,
+        )
+
+    msg.text(f"Loading config from {config_path}", show=verbose)
+    nlp = assemble(
+        config_path,
+        overrides={}
+        if examples_path is None
+        else {"paths.examples": str(examples_path)},
+    )
+    doc = nlp(text)
+
+    msg.text(f"Text: {doc.text}")
+    msg.text(f"Categories: {doc.cats}")
+
+
+if __name__ == "__main__":
+    typer.run(run_pipeline)
diff --git a/usage_examples/textcat_dolly/zeroshot.cfg b/usage_examples/textcat_dolly/zeroshot.cfg
new file mode 100644
index 00000000..76e87114
--- /dev/null
+++ b/usage_examples/textcat_dolly/zeroshot.cfg
@@ -0,0 +1,22 @@
+[nlp]
+lang = "en"
+pipeline = ["llm"]
+batch_size = 128
+
+[components]
+
+[components.llm]
+factory = "llm"
+
+[components.llm.model]
+@llm_models = "spacy.Dolly.v1"
+name = "dolly-v2-3b"
+
+[components.llm.task]
+@llm_tasks = "spacy.TextCat.v2"
+labels = COMPLIMENT,INSULT
+examples = null
+exclusive_classes = true
+
+[components.llm.task.normalizer]
+@misc = "spacy.LowercaseNormalizer.v1"
diff --git a/usage_examples/textcat_openai/fewshot.cfg b/usage_examples/textcat_openai/fewshot.cfg
index e7bfd0f9..c23024aa 100644
--- a/usage_examples/textcat_openai/fewshot.cfg
+++ b/usage_examples/textcat_openai/fewshot.cfg
@@ -12,8 +12,8 @@ batch_size = 128
 factory = "llm"
 
 [components.llm.model]
-@llm_models = "spacy.GPT-3-5.v1"
-config = {"temperature": 0.3}
+@llm_models = "spacy.GPT-3-5.v2"
+config = {"temperature": 0.0}
 
 [components.llm.task]
 @llm_tasks = "spacy.TextCat.v2"
diff --git a/usage_examples/textcat_openai/zeroshot.cfg b/usage_examples/textcat_openai/zeroshot.cfg
index 41bb5c4d..85950c30 100644
--- a/usage_examples/textcat_openai/zeroshot.cfg
+++ b/usage_examples/textcat_openai/zeroshot.cfg
@@ -9,8 +9,8 @@ batch_size = 128
 factory = "llm"
 
 [components.llm.model]
-@llm_models = "spacy.GPT-3-5.v1"
-config = {"temperature": 0.3}
+@llm_models = "spacy.GPT-3-5.v2"
+config = {"temperature": 0.0}
 
 [components.llm.task]
 @llm_tasks = "spacy.TextCat.v2"


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 4a695b51..5368a66a 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -39,7 +39,7 @@ jobs:
 
     steps:
       - uses: actions/checkout@v3
-        
+
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
diff --git a/spacy_llm/tests/models/test_dolly.py b/spacy_llm/tests/models/test_dolly.py
index c142950f..615d6a0c 100644
--- a/spacy_llm/tests/models/test_dolly.py
+++ b/spacy_llm/tests/models/test_dolly.py
@@ -5,6 +5,8 @@
 from confection import Config  # type: ignore[import]
 from thinc.compat import has_torch_cuda_gpu
 
+from ...compat import torch
+
 _PIPE_CFG = {
     "model": {
         "@llm_models": "spacy.Dolly.v1",
@@ -44,6 +46,7 @@ def test_init():
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
     nlp.get_pipe("llm")._model.get_model_names()
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -52,6 +55,7 @@ def test_init_from_config():
     orig_config = Config().from_str(_NLP_CONFIG)
     nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     assert nlp.pipe_names == ["llm"]
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -62,3 +66,4 @@ def test_invalid_model():
     config["components"]["llm"]["model"]["name"] = "dolly-the-sheep"
     with pytest.raises(ValueError, match="unexpected value; permitted"):
         spacy.util.load_model_from_config(config, auto_fill=True)
+    torch.cuda.empty_cache()
diff --git a/spacy_llm/tests/models/test_falcon.py b/spacy_llm/tests/models/test_falcon.py
index fc6e9b22..eb886922 100644
--- a/spacy_llm/tests/models/test_falcon.py
+++ b/spacy_llm/tests/models/test_falcon.py
@@ -5,6 +5,8 @@
 from confection import Config  # type: ignore[import]
 from thinc.compat import has_torch_cuda_gpu
 
+from ...compat import torch
+
 _PIPE_CFG = {
     "model": {
         "@llm_models": "spacy.Falcon.v1",
@@ -42,6 +44,7 @@ def test_init():
     cfg = copy.deepcopy(_PIPE_CFG)
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -50,6 +53,7 @@ def test_init_from_config():
     orig_config = Config().from_str(_NLP_CONFIG)
     nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     assert nlp.pipe_names == ["llm"]
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -60,3 +64,4 @@ def test_invalid_model():
     config["components"]["llm"]["model"]["name"] = "x"
     with pytest.raises(ValueError, match="unexpected value; permitted"):
         spacy.util.load_model_from_config(config, auto_fill=True)
+    torch.cuda.empty_cache()
diff --git a/spacy_llm/tests/models/test_llama2.py b/spacy_llm/tests/models/test_llama2.py
index d8d5ea23..0dd4623d 100644
--- a/spacy_llm/tests/models/test_llama2.py
+++ b/spacy_llm/tests/models/test_llama2.py
@@ -5,6 +5,8 @@
 from confection import Config  # type: ignore[import]
 from thinc.compat import has_torch_cuda_gpu
 
+from ...compat import torch
+
 _PIPE_CFG = {
     "model": {
         "@llm_models": "spacy.Llama2.v1",
@@ -34,6 +36,7 @@
 """
 
 
+@pytest.mark.skip(reason="CI runner needs more GPU memory")
 @pytest.mark.gpu
 @pytest.mark.skipif(not has_torch_cuda_gpu, reason="needs GPU & CUDA")
 def test_init():
@@ -42,16 +45,20 @@ def test_init():
     cfg = copy.deepcopy(_PIPE_CFG)
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
+@pytest.mark.skip(reason="CI runner needs more GPU memory")
 @pytest.mark.gpu
 @pytest.mark.skipif(not has_torch_cuda_gpu, reason="needs GPU & CUDA")
 def test_init_from_config():
     orig_config = Config().from_str(_NLP_CONFIG)
     nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     assert nlp.pipe_names == ["llm"]
+    torch.cuda.empty_cache()
 
 
+@pytest.mark.skip(reason="CI runner needs more GPU memory")
 @pytest.mark.gpu
 @pytest.mark.skipif(not has_torch_cuda_gpu, reason="needs GPU & CUDA")
 def test_invalid_model():
@@ -60,3 +67,4 @@ def test_invalid_model():
     config["components"]["llm"]["model"]["name"] = "x"
     with pytest.raises(ValueError, match="unexpected value; permitted"):
         spacy.util.load_model_from_config(config, auto_fill=True)
+    torch.cuda.empty_cache()
diff --git a/spacy_llm/tests/models/test_openllama.py b/spacy_llm/tests/models/test_openllama.py
index 990328ca..5cb8a497 100644
--- a/spacy_llm/tests/models/test_openllama.py
+++ b/spacy_llm/tests/models/test_openllama.py
@@ -5,6 +5,8 @@
 from confection import Config  # type: ignore[import]
 from thinc.compat import has_torch_cuda_gpu
 
+from ...compat import torch
+
 _PIPE_CFG = {
     "model": {
         "@llm_models": "spacy.OpenLLaMA.v1",
@@ -40,6 +42,7 @@ def test_init():
     nlp = spacy.blank("en")
     nlp.add_pipe("llm", config=_PIPE_CFG)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -51,6 +54,7 @@ def test_init_with_set_config():
     cfg["model"]["config_run"] = {"max_new_tokens": 32}
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -59,6 +63,7 @@ def test_init_from_config():
     orig_config = Config().from_str(_NLP_CONFIG)
     nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     assert nlp.pipe_names == ["llm"]
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -69,3 +74,4 @@ def test_invalid_model():
     config["components"]["llm"]["model"]["name"] = "anything-else"
     with pytest.raises(ValueError, match="unexpected value; permitted"):
         spacy.util.load_model_from_config(config, auto_fill=True)
+    torch.cuda.empty_cache()
diff --git a/spacy_llm/tests/models/test_rest.py b/spacy_llm/tests/models/test_rest.py
index 87af07e8..5d5f9152 100644
--- a/spacy_llm/tests/models/test_rest.py
+++ b/spacy_llm/tests/models/test_rest.py
@@ -12,8 +12,7 @@
 
 PIPE_CFG = {
     "model": {
-        "@llm_models": "spacy.GPT-3-5.v1",
-        "config": {"temperature": 0.3},
+        "@llm_models": "spacy.GPT-3-5.v2",
     },
     "task": {"@llm_tasks": "spacy.TextCat.v1", "labels": "POSITIVE,NEGATIVE"},
 }
diff --git a/spacy_llm/tests/models/test_stablelm.py b/spacy_llm/tests/models/test_stablelm.py
index 35281e93..36ba1249 100644
--- a/spacy_llm/tests/models/test_stablelm.py
+++ b/spacy_llm/tests/models/test_stablelm.py
@@ -5,6 +5,8 @@
 from confection import Config  # type: ignore[import]
 from thinc.compat import has_torch_cuda_gpu
 
+from ...compat import torch
+
 _PIPE_CFG = {
     "model": {
         "@llm_models": "spacy.StableLM.v1",
@@ -45,6 +47,7 @@ def test_init(name: str):
     cfg["model"]["name"] = name
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -53,6 +56,7 @@ def test_init_from_config():
     orig_config = Config().from_str(_NLP_CONFIG)
     nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     assert nlp.pipe_names == ["llm"]
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
@@ -64,6 +68,7 @@ def test_init_with_set_config():
     cfg["model"]["config_run"] = {"temperature": 0.3}
     nlp.add_pipe("llm", config=cfg)
     nlp("This is a test.")
+    torch.cuda.empty_cache()
 
 
 @pytest.mark.gpu
diff --git a/spacy_llm/tests/pipeline/test_llm.py b/spacy_llm/tests/pipeline/test_llm.py
index fd8d75c2..6ef297b1 100644
--- a/spacy_llm/tests/pipeline/test_llm.py
+++ b/spacy_llm/tests/pipeline/test_llm.py
@@ -279,8 +279,7 @@ def test_pipe_labels():
     labels = ["COMPLIMENT", "INSULT"]
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {"temperature": 0.3}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
     config = Config().from_str(cfg_string)
@@ -304,6 +303,9 @@ def test_llm_task_factories():
 
         [components.llm]
         factory = "llm_{task_handle.split('.')[1].lower()}"
+
+        [components.llm.model]
+        @llm_models = "test.NoOpModel.v1"
         """
         config = Config().from_str(cfg_string)
         assemble_from_config(config)
diff --git a/spacy_llm/tests/tasks/examples/ner.json b/spacy_llm/tests/tasks/examples/ner.json
index 31899f00..38444f71 100644
--- a/spacy_llm/tests/tasks/examples/ner.json
+++ b/spacy_llm/tests/tasks/examples/ner.json
@@ -1,30 +1,31 @@
 [
   {
     "text": "Jack and Jill went up the hill.",
-    "entities": {
-      "PER": [
-        "Jack",
-        "Jill"
-      ],
-      "LOC": [
-        "hill"
-      ]
-    }
-  },
-  {
-    "text": "Jack fell down and broke his crown.",
-    "entities": {
-      "PER": [
-        "Jack"
-      ]
-    }
-  },
-  {
-    "text": "Jill came tumbling after.",
-    "entities": {
-      "PER": [
-        "Jill"
-      ]
-    }
+    "spans": [
+      {
+        "text": "Jack",
+        "is_entity": true,
+        "label": "PER",
+        "reason": "is the name of a person"
+      },
+      {
+        "text": "Jill",
+        "is_entity": true,
+        "label": "PER",
+        "reason": "is the name of a person"
+      },
+      {
+        "text": "went up",
+        "is_entity": false,
+        "label": "==NONE==",
+        "reason": "is a verb"
+      },
+      {
+        "text": "hill",
+        "is_entity": true,
+        "label": "LOC",
+        "reason": "is a location"
+      }
+    ]
   }
-]
\ No newline at end of file
+]
diff --git a/spacy_llm/tests/tasks/examples/ner.jsonl b/spacy_llm/tests/tasks/examples/ner.jsonl
index 7d54cca5..c23a1b42 100644
--- a/spacy_llm/tests/tasks/examples/ner.jsonl
+++ b/spacy_llm/tests/tasks/examples/ner.jsonl
@@ -1,3 +1,1 @@
-{"text":"Jack and Jill went up the hill.","entities":{"PER":["Jack","Jill"],"LOC":["hill"]}}
-{"text":"Jack fell down and broke his crown.","entities":{"PER":["Jack"]}}
-{"text":"Jill came tumbling after.","entities":{"PER":["Jill"]}}
+{"text":"Jack and Jill went up the hill.","spans":[{"text":"Jack","is_entity":true,"label":"PER","reason":"is the name of a person"},{"text":"Jill","is_entity":true,"label":"PER","reason":"is the name of a person"},{"text":"went up","is_entity":false,"label":"==NONE==","reason":"is a verb"},{"text":"hill","is_entity":true,"label":"LOC","reason":"is a location"}]}
diff --git a/spacy_llm/tests/tasks/examples/ner.yml b/spacy_llm/tests/tasks/examples/ner.yml
index 7e653f7d..2e519e3c 100644
--- a/spacy_llm/tests/tasks/examples/ner.yml
+++ b/spacy_llm/tests/tasks/examples/ner.yml
@@ -1,15 +1,18 @@
 - text: Jack and Jill went up the hill.
-  entities:
-    PER:
-      - Jack
-      - Jill
-    LOC:
-      - hill
-- text: Jack fell down and broke his crown.
-  entities:
-    PER:
-      - Jack
-- text: Jill came tumbling after.
-  entities:
-    PER:
-      - Jill
+  spans:
+    - text: Jack
+      is_entity: true
+      label: PER
+      reason: is the name of a person
+    - text: Jill
+      is_entity: true
+      label: PER
+      reason: is the name of a person
+    - text: went up
+      is_entity: false
+      label: ==NONE==
+      reason: is a verb
+    - text: hill
+      is_entity: true
+      label: LOC
+      reason: is a location
diff --git a/spacy_llm/tests/tasks/examples/ner_inconsistent.yml b/spacy_llm/tests/tasks/examples/ner_inconsistent.yml
index 1111e3de..e71336bd 100644
--- a/spacy_llm/tests/tasks/examples/ner_inconsistent.yml
+++ b/spacy_llm/tests/tasks/examples/ner_inconsistent.yml
@@ -1,20 +1,48 @@
 - text: Jack and Jill went up the hill.
-  entities:
-    PERSON:
-      - Jack
-      - Jill
-    LOCATION:
-      - hill
+  spans:
+    - text: Jack
+      is_entity: true
+      label: PERSON
+      reason: is the name of a person
+    - text: Jill
+      is_entity: true
+      label: PERSON
+      reason: is the name of a person
+    - text: went up
+      is_entity: false
+      label: ==NONE==
+      reason: is a verb
+    - text: hill
+      is_entity: true
+      label: LOCATION
+      reason: is a location
+
 - text: spaCy is a great tool
-  entities:
-    TECH:
-      - spaCy
+  spans:
+    - text: spaCy
+      is_entity: true
+      label: TECH
+      reason: is a software product
+
 - text: Jack and Jill went up the hill and spaCy is a great tool.
-  entities:
-    PERSON:
-      - Jack
-      - Jill
-    LOCATION:
-      - hill
-    TECH:
-      - spaCy
\ No newline at end of file
+  spans:
+    - text: Jack
+      is_entity: true
+      label: PERSON
+      reason: is the name of a person
+    - text: Jill
+      is_entity: true
+      label: PERSON
+      reason: is the name of a person
+    - text: went up
+      is_entity: false
+      label: ==NONE==
+      reason: is a verb
+    - text: hill
+      is_entity: true
+      label: LOCATION
+      reason: is a location
+    - text: spaCy
+      is_entity: true
+      label: TECH
+      reason: is a software product
diff --git a/spacy_llm/tests/tasks/examples/spancat.json b/spacy_llm/tests/tasks/examples/spancat.json
new file mode 100644
index 00000000..595efe69
--- /dev/null
+++ b/spacy_llm/tests/tasks/examples/spancat.json
@@ -0,0 +1,37 @@
+[
+  {
+    "text": "Jack and Jill went up the hill.",
+    "spans": [
+      {
+        "text": "Jack",
+        "is_entity": true,
+        "label": "PER",
+        "reason": "is the name of a person"
+      },
+      {
+        "text": "Jill",
+        "is_entity": true,
+        "label": "PER",
+        "reason": "is the name of a person"
+      },
+      {
+        "text": "went up",
+        "is_entity": false,
+        "label": "==NONE==",
+        "reason": "is a verb"
+      },
+      {
+        "text": "hill",
+        "is_entity": true,
+        "label": "LOC",
+        "reason": "is a location"
+      },
+      {
+        "text": "hill",
+        "is_entity": true,
+        "label": "DESTINATION",
+        "reason": "is a destination"
+      }
+    ]
+  }
+]
diff --git a/spacy_llm/tests/tasks/examples/spancat.jsonl b/spacy_llm/tests/tasks/examples/spancat.jsonl
new file mode 100644
index 00000000..168e37bd
--- /dev/null
+++ b/spacy_llm/tests/tasks/examples/spancat.jsonl
@@ -0,0 +1,1 @@
+{"text":"Jack and Jill went up the hill.","spans":[{"text":"Jack","is_entity":true,"label":"PER","reason":"is the name of a person"},{"text":"Jill","is_entity":true,"label":"PER","reason":"is the name of a person"},{"text":"went up","is_entity":false,"label":"==NONE==","reason":"is a verb"},{"text":"hill","is_entity":true,"label":"LOC","reason":"is a location"},{"text":"hill","is_entity":true,"label":"DESTINATION","reason":"is a destination"}]}
diff --git a/spacy_llm/tests/tasks/examples/spancat.yml b/spacy_llm/tests/tasks/examples/spancat.yml
new file mode 100644
index 00000000..d3765ff6
--- /dev/null
+++ b/spacy_llm/tests/tasks/examples/spancat.yml
@@ -0,0 +1,22 @@
+  - text: Jack and Jill went up the hill.
+    spans:
+      - text: Jack
+        is_entity: true
+        label: PER
+        reason: is the name of a person
+      - text: Jill
+        is_entity: true
+        label: PER
+        reason: is the name of a person
+      - text: went up
+        is_entity: false
+        label: ==NONE==
+        reason: is a verb
+      - text: hill
+        is_entity: true
+        label: LOC
+        reason: is a location
+      - text: hill
+        is_entity: true
+        label: DESTINATION
+        reason: is a destination
diff --git a/spacy_llm/tests/tasks/legacy/__init__.py b/spacy_llm/tests/tasks/legacy/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/spacy_llm/tests/tasks/legacy/examples/ner.json b/spacy_llm/tests/tasks/legacy/examples/ner.json
new file mode 100644
index 00000000..31899f00
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/examples/ner.json
@@ -0,0 +1,30 @@
+[
+  {
+    "text": "Jack and Jill went up the hill.",
+    "entities": {
+      "PER": [
+        "Jack",
+        "Jill"
+      ],
+      "LOC": [
+        "hill"
+      ]
+    }
+  },
+  {
+    "text": "Jack fell down and broke his crown.",
+    "entities": {
+      "PER": [
+        "Jack"
+      ]
+    }
+  },
+  {
+    "text": "Jill came tumbling after.",
+    "entities": {
+      "PER": [
+        "Jill"
+      ]
+    }
+  }
+]
\ No newline at end of file
diff --git a/spacy_llm/tests/tasks/legacy/examples/ner.jsonl b/spacy_llm/tests/tasks/legacy/examples/ner.jsonl
new file mode 100644
index 00000000..7d54cca5
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/examples/ner.jsonl
@@ -0,0 +1,3 @@
+{"text":"Jack and Jill went up the hill.","entities":{"PER":["Jack","Jill"],"LOC":["hill"]}}
+{"text":"Jack fell down and broke his crown.","entities":{"PER":["Jack"]}}
+{"text":"Jill came tumbling after.","entities":{"PER":["Jill"]}}
diff --git a/spacy_llm/tests/tasks/legacy/examples/ner.yml b/spacy_llm/tests/tasks/legacy/examples/ner.yml
new file mode 100644
index 00000000..7e653f7d
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/examples/ner.yml
@@ -0,0 +1,15 @@
+- text: Jack and Jill went up the hill.
+  entities:
+    PER:
+      - Jack
+      - Jill
+    LOC:
+      - hill
+- text: Jack fell down and broke his crown.
+  entities:
+    PER:
+      - Jack
+- text: Jill came tumbling after.
+  entities:
+    PER:
+      - Jill
diff --git a/spacy_llm/tests/tasks/legacy/examples/ner_inconsistent.yml b/spacy_llm/tests/tasks/legacy/examples/ner_inconsistent.yml
new file mode 100644
index 00000000..1111e3de
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/examples/ner_inconsistent.yml
@@ -0,0 +1,20 @@
+- text: Jack and Jill went up the hill.
+  entities:
+    PERSON:
+      - Jack
+      - Jill
+    LOCATION:
+      - hill
+- text: spaCy is a great tool
+  entities:
+    TECH:
+      - spaCy
+- text: Jack and Jill went up the hill and spaCy is a great tool.
+  entities:
+    PERSON:
+      - Jack
+      - Jill
+    LOCATION:
+      - hill
+    TECH:
+      - spaCy
\ No newline at end of file
diff --git a/spacy_llm/tests/tasks/legacy/templates/ner.jinja2 b/spacy_llm/tests/tasks/legacy/templates/ner.jinja2
new file mode 100644
index 00000000..edc6042c
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/templates/ner.jinja2
@@ -0,0 +1,8 @@
+This is a test NER template. Here are the labels
+{# whitespace #}
+{%- for label in labels -%}
+{{ label }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+Here is the text: {{ text }}
\ No newline at end of file
diff --git a/spacy_llm/tests/tasks/legacy/test_ner.py b/spacy_llm/tests/tasks/legacy/test_ner.py
new file mode 100644
index 00000000..6d8ff727
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/test_ner.py
@@ -0,0 +1,882 @@
+import json
+import re
+from pathlib import Path
+
+import pytest
+import spacy
+import srsly
+from confection import Config
+from spacy.tokens import Span
+from spacy.training import Example
+from spacy.util import make_tempdir
+
+from spacy_llm.pipeline import LLMWrapper
+from spacy_llm.registry import fewshot_reader, file_reader, lowercase_normalizer
+from spacy_llm.registry import strip_normalizer
+from spacy_llm.tasks.ner import NERTask, make_ner_task_v2
+from spacy_llm.tasks.util import find_substrings
+from spacy_llm.ty import LabeledTask, LLMTask
+from spacy_llm.util import assemble_from_config, split_labels
+
+from ...compat import has_openai_key
+
+EXAMPLES_DIR = Path(__file__).parent / "examples"
+TEMPLATES_DIR = Path(__file__).parent / "templates"
+
+
+@pytest.fixture
+def zeroshot_cfg_string():
+    return """
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v1"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.fixture
+def zeroshot_cfg_string_v2_lds():
+    return """
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v2"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.label_definitions]
+    PER = "Any named individual in the text"
+    ORG = "Any named organization in the text"
+    LOC = "The name of any politically or geographically defined location"
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.fixture
+def fewshot_cfg_string():
+    return f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v1"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.fixture
+def fewshot_cfg_string_v2():
+    return f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v2"
+    labels = ["PER", "ORG", "LOC"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.fixture
+def ext_template_cfg_string():
+    """Simple zero-shot config with an external template"""
+
+    return f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v2"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.template]
+    @misc = "spacy.FileReader.v1"
+    path = {str((Path(__file__).parent / "templates" / "ner.jinja2"))}
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+@pytest.mark.parametrize(
+    "cfg_string",
+    [
+        "zeroshot_cfg_string",
+        "zeroshot_cfg_string_v2_lds",
+        "fewshot_cfg_string",
+        "fewshot_cfg_string_v2",
+        "ext_template_cfg_string",
+    ],
+)
+def test_ner_config(cfg_string, request):
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    assert nlp.pipe_names == ["llm"]
+
+    # also test nlp config from a dict in add_pipe
+    component_cfg = dict(orig_config["components"]["llm"])
+    component_cfg.pop("factory")
+
+    nlp2 = spacy.blank("en")
+    nlp2.add_pipe("llm", config=component_cfg)
+    assert nlp2.pipe_names == ["llm"]
+
+    pipe = nlp.get_pipe("llm")
+    assert isinstance(pipe, LLMWrapper)
+    assert isinstance(pipe.task, LLMTask)
+
+    labels = orig_config["components"]["llm"]["task"]["labels"]
+    labels = split_labels(labels)
+    task = pipe.task
+    assert isinstance(task, LabeledTask)
+    assert sorted(task.labels) == sorted(tuple(labels))
+    assert pipe.labels == task.labels
+    assert nlp.pipe_labels["llm"] == list(task.labels)
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+@pytest.mark.parametrize(
+    "cfg_string",
+    [
+        "zeroshot_cfg_string",
+        "zeroshot_cfg_string_v2_lds",
+        "fewshot_cfg_string",
+        "fewshot_cfg_string_v2",
+        "ext_template_cfg_string",
+    ],
+)
+def test_ner_predict(cfg_string, request):
+    """Use OpenAI to get zero-shot NER results.
+    Note that this test may fail randomly, as the LLM's output is unguaranteed to be consistent/predictable
+    """
+    orig_cfg_string = cfg_string
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    text = "Marc and Bob both live in Ireland."
+    doc = nlp(text)
+
+    if orig_cfg_string != "ext_template_cfg_string":
+        assert len(doc.ents) > 0
+        for ent in doc.ents:
+            assert ent.label_ in ["PER", "ORG", "LOC"]
+
+
+@pytest.mark.external
+@pytest.mark.parametrize(
+    "cfg_string",
+    [
+        "zeroshot_cfg_string",
+        "zeroshot_cfg_string_v2_lds",
+        "fewshot_cfg_string",
+        "fewshot_cfg_string_v2",
+        "ext_template_cfg_string",
+    ],
+)
+def test_ner_io(cfg_string, request):
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    assert nlp.pipe_names == ["llm"]
+    # ensure you can save a pipeline to disk and run it after loading
+    with make_tempdir() as tmpdir:
+        nlp.to_disk(tmpdir)
+        nlp2 = spacy.load(tmpdir)
+    assert nlp2.pipe_names == ["llm"]
+    text = "Marc and Bob both live in Ireland."
+    doc = nlp2(text)
+    assert len(doc.ents) >= 0  # can be zero if template is too simple / test-like
+    for ent in doc.ents:
+        assert ent.label_ in ["PER", "ORG", "LOC"]
+
+
+@pytest.mark.parametrize(
+    "text,input_strings,result_strings,result_offsets",
+    [
+        (
+            "Felipe and Jaime went to the library.",
+            ["Felipe", "Jaime", "library"],
+            ["Felipe", "Jaime", "library"],
+            [(0, 6), (11, 16), (29, 36)],
+        ),  # simple
+        (
+            "The Manila Observatory was founded in 1865 in Manila.",
+            ["Manila", "The Manila Observatory"],
+            ["Manila", "Manila", "The Manila Observatory"],
+            [(4, 10), (46, 52), (0, 22)],
+        ),  # overlapping and duplicated
+        (
+            "Take the road from downtown and turn left at the public market.",
+            ["public market", "downtown"],
+            ["public market", "downtown"],
+            [(49, 62), (19, 27)]
+            # flipped
+        ),
+    ],
+)
+def test_ensure_offsets_correspond_to_substrings(
+    text, input_strings, result_strings, result_offsets
+):
+    offsets = find_substrings(text, input_strings)
+    # Compare strings instead of offsets, but we need to get
+    # those strings first from the text
+    assert result_offsets == offsets
+    found_substrings = [text[start:end] for start, end in offsets]
+    assert result_strings == found_substrings
+
+
+@pytest.mark.parametrize(
+    "text,response,gold_ents",
+    [
+        # simple
+        (
+            "Jean Jacques and Jaime went to the library.",
+            "PER: Jean Jacques, Jaime\nLOC: library",
+            [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "LOC")],
+        ),
+        # overlapping: should only return the longest span
+        (
+            "The Manila Observatory was founded in 1865.",
+            "LOC: The Manila Observatory, Manila, Manila Observatory",
+            [("The Manila Observatory", "LOC")],
+        ),
+        # flipped: order shouldn't matter
+        (
+            "Take the road from Downtown and turn left at the public market.",
+            "LOC: public market, Downtown",
+            [("Downtown", "LOC"), ("public market", "LOC")],
+        ),
+    ],
+)
+def test_ner_zero_shot_task(text, response, gold_ents):
+    labels = "PER,ORG,LOC"
+    llm_ner = make_ner_task_v2(labels=labels)
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list so we get what's inside
+    doc_out = list(llm_ner.parse_responses([doc_in], [response]))[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc_out.ents]
+    assert pred_ents == gold_ents
+
+
+@pytest.mark.parametrize(
+    "response,normalizer,gold_ents",
+    [
+        (
+            "PER: Jean Jacques, Jaime",
+            None,
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, Jaime",
+            strip_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, Jaime",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques, Jaime",
+            strip_normalizer(),
+            [],
+        ),
+        (
+            "per: Jean Jacques, Jaime",
+            None,
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques\nPER: Jaime",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques, Jaime\nOrg: library",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "ORG")],
+        ),
+        (
+            "per: Jean Jacques, Jaime\nRANDOM: library",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+    ],
+)
+def test_ner_labels(response, normalizer, gold_ents):
+    text = "Jean Jacques and Jaime went to the library."
+    labels = "PER,ORG,LOC"
+    llm_ner = make_ner_task_v2(labels=labels, normalizer=normalizer)
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_ner.parse_responses([doc_in], [response]))[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc_out.ents]
+    assert pred_ents == gold_ents
+
+
+@pytest.mark.parametrize(
+    "response,alignment_mode,gold_ents",
+    [
+        (
+            "PER: Jacq",
+            "strict",
+            [],
+        ),
+        (
+            "PER: Jacq",
+            "contract",
+            [],
+        ),
+        (
+            "PER: Jacq",
+            "expand",
+            [("Jacques", "PER")],
+        ),
+        (
+            "PER: Jean J",
+            "contract",
+            [("Jean", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, aim",
+            "strict",
+            [("Jean Jacques", "PER")],
+        ),
+        (
+            "PER: random",
+            "expand",
+            [],
+        ),
+    ],
+)
+def test_ner_alignment(response, alignment_mode, gold_ents):
+    text = "Jean Jacques and Jaime went to the library."
+    labels = "PER,ORG,LOC"
+    llm_ner = make_ner_task_v2(labels=labels, alignment_mode=alignment_mode)  # type: ignore
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_ner.parse_responses([doc_in], [response]))[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc_out.ents]
+    assert pred_ents == gold_ents
+
+
+def test_invalid_alignment_mode():
+    labels = "PER,ORG,LOC"
+    with pytest.raises(ValueError, match="Unsupported alignment mode 'invalid"):
+        make_ner_task_v2(labels=labels, alignment_mode="invalid")  # type: ignore
+
+
+@pytest.mark.parametrize(
+    "response,case_sensitive,single_match,gold_ents",
+    [
+        (
+            "PER: Jean",
+            False,
+            False,
+            [("jean", "PER"), ("Jean", "PER"), ("Jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            False,
+            True,
+            [("jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            True,
+            False,
+            [("Jean", "PER"), ("Jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            True,
+            True,
+            [("Jean", "PER")],
+        ),
+    ],
+)
+def test_ner_matching(response, case_sensitive, single_match, gold_ents):
+    text = "This guy jean (or Jean) is the president of the Jean Foundation."
+    labels = "PER,ORG,LOC"
+    llm_ner = make_ner_task_v2(
+        labels=labels, case_sensitive_matching=case_sensitive, single_match=single_match
+    )
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_ner.parse_responses([doc_in], [response]))[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc_out.ents]
+    assert pred_ents == gold_ents
+
+
+def test_jinja_template_rendering_without_examples():
+    """Test if jinja template renders as we expected
+
+    We apply the .strip() method for each prompt so that we don't have to deal
+    with annoying newlines and spaces at the edge of the text.
+    """
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+
+    llm_ner = make_ner_task_v2(labels=labels, examples=None)
+    prompt = list(llm_ner.generate_prompts([doc]))[0]
+
+    assert (
+        prompt.strip()
+        == """
+You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
+From the Text input provided, extract named entities for each label in the following format:
+
+LOC: <comma delimited list of strings>
+ORG: <comma delimited list of strings>
+PER: <comma delimited list of strings>
+
+
+Here is the text that needs labeling:
+
+Text:
+'''
+Alice and Bob went to the supermarket
+'''
+""".strip()
+    )
+
+
+@pytest.mark.parametrize(
+    "examples_path",
+    [
+        str(EXAMPLES_DIR / "ner.json"),
+        str(EXAMPLES_DIR / "ner.yml"),
+        str(EXAMPLES_DIR / "ner.jsonl"),
+    ],
+)
+def test_jinja_template_rendering_with_examples(examples_path):
+    """Test if jinja2 template renders as expected
+
+    We apply the .strip() method for each prompt so that we don't have to deal
+    with annoying newlines and spaces at the edge of the text.
+    """
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+
+    examples = fewshot_reader(examples_path)
+    llm_ner = make_ner_task_v2(labels=labels, examples=examples)
+    prompt = list(llm_ner.generate_prompts([doc]))[0]
+
+    assert (
+        prompt.strip()
+        == """
+You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
+From the Text input provided, extract named entities for each label in the following format:
+
+LOC: <comma delimited list of strings>
+ORG: <comma delimited list of strings>
+PER: <comma delimited list of strings>
+
+
+Below are some examples (only use these as a guide):
+
+Text:
+'''
+Jack and Jill went up the hill.
+'''
+
+PER: Jack, Jill
+LOC: hill
+
+Text:
+'''
+Jack fell down and broke his crown.
+'''
+
+PER: Jack
+
+Text:
+'''
+Jill came tumbling after.
+'''
+
+PER: Jill
+
+
+Here is the text that needs labeling:
+
+Text:
+'''
+Alice and Bob went to the supermarket
+'''""".strip()
+    )
+
+
+def test_jinja_template_rendering_with_label_definitions():
+    """Test if jinja2 template renders as expected
+
+    We apply the .strip() method for each prompt so that we don't have to deal
+    with annoying newlines and spaces at the edge of the text.
+    """
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+    llm_ner = make_ner_task_v2(
+        labels=labels,
+        label_definitions={
+            "PER": "Person definition",
+            "ORG": "Organization definition",
+            "LOC": "Location definition",
+        },
+    )
+    prompt = list(llm_ner.generate_prompts([doc]))[0]
+
+    assert (
+        prompt.strip()
+        == """
+You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
+From the Text input provided, extract named entities for each label in the following format:
+
+LOC: <comma delimited list of strings>
+ORG: <comma delimited list of strings>
+PER: <comma delimited list of strings>
+
+Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
+Assume these definitions are written by an expert and follow them closely.
+
+PER: Person definition
+ORG: Organization definition
+LOC: Location definition
+
+
+Here is the text that needs labeling:
+
+Text:
+'''
+Alice and Bob went to the supermarket
+'''""".strip()
+    )
+
+
+def test_example_not_following_basemodel():
+    wrong_example = [
+        {
+            "text": "I'm a wrong example. Entities should be a dict, not a list",
+            # Should be: {"PER": ["Entities"], "ORG": ["dict", "list"]}
+            "entities": [("PER", ("Entities")), ("ORG", ("dict", "list"))],
+        }
+    ]
+    with make_tempdir() as tmpdir:
+        tmp_path = tmpdir / "wrong_example.yml"
+        srsly.write_yaml(tmp_path, wrong_example)
+
+        with pytest.raises(ValueError):
+            make_ner_task_v2(labels="PER,ORG,LOC", examples=fewshot_reader(tmp_path))
+
+
+def test_external_template_actually_loads():
+    template_path = str(TEMPLATES_DIR / "ner.jinja2")
+    template = file_reader(template_path)
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+
+    llm_ner = make_ner_task_v2(labels=labels, template=template)
+    prompt = list(llm_ner.generate_prompts([doc]))[0]
+    assert (
+        prompt.strip()
+        == """
+This is a test NER template. Here are the labels
+LOC
+ORG
+PER
+
+Here is the text: Alice and Bob went to the supermarket
+""".strip()
+    )
+
+
+@pytest.fixture
+def noop_config():
+    return """
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v2"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "test.NoOpModel.v1"
+    output = "PER: Alice,Bob"
+    """
+
+
+@pytest.mark.parametrize("n_detections", [0, 1, 2])
+def test_ner_scoring(noop_config, n_detections):
+    config = Config().from_str(noop_config)
+    nlp = assemble_from_config(config)
+
+    examples = []
+
+    for text in ["Alice works with Bob.", "Bob lives with Alice."]:
+        predicted = nlp.make_doc(text)
+        reference = predicted.copy()
+
+        reference.ents = [
+            Span(reference, 0, 1, label="PER"),
+            Span(reference, 3, 4, label="PER"),
+        ][:n_detections]
+
+        examples.append(Example(predicted, reference))
+
+    scores = nlp.evaluate(examples)
+
+    assert scores["ents_p"] == n_detections / 2
+
+
+@pytest.mark.parametrize("n_prompt_examples", [-1, 0, 1, 2])
+def test_ner_init(noop_config, n_prompt_examples: int):
+    config = Config().from_str(noop_config)
+    del config["components"]["llm"]["task"]["labels"]
+
+    nlp = assemble_from_config(config)
+
+    examples = []
+
+    for text in [
+        "Alice works with Bob in London.",
+        "Bob lives with Alice in Manchester.",
+    ]:
+        predicted = nlp.make_doc(text)
+        reference = predicted.copy()
+
+        reference.ents = [
+            Span(reference, 0, 1, label="PER"),
+            Span(reference, 3, 4, label="PER"),
+            Span(reference, 5, 6, label="LOC"),
+        ]
+
+        examples.append(Example(predicted, reference))
+
+    _, llm = nlp.pipeline[0]
+    task: NERTask = llm._task
+
+    assert set(task._label_dict.values()) == set()
+    assert not task._prompt_examples
+
+    nlp.config["initialize"]["components"]["llm"] = {
+        "n_prompt_examples": n_prompt_examples
+    }
+    nlp.initialize(lambda: examples)
+
+    assert set(task._label_dict.values()) == {"PER", "LOC"}
+    if n_prompt_examples >= 0:
+        assert len(task._prompt_examples) == n_prompt_examples
+    else:
+        assert len(task._prompt_examples) == len(examples)
+
+    if n_prompt_examples > 0:
+        for eg in task._prompt_examples:
+            assert set(eg.entities.keys()) == {"PER", "LOC"}
+
+
+def test_ner_serde(noop_config):
+    config = Config().from_str(noop_config)
+    del config["components"]["llm"]["task"]["labels"]
+
+    nlp1 = assemble_from_config(config)
+    nlp2 = assemble_from_config(config)
+
+    labels = {"loc": "LOC", "per": "PER"}
+
+    task1: NERTask = nlp1.get_pipe("llm")._task
+    task2: NERTask = nlp2.get_pipe("llm")._task
+
+    # Artificially add labels to task1
+    task1._label_dict = labels
+
+    assert task1._label_dict == labels
+    assert task2._label_dict == dict()
+
+    b = nlp1.to_bytes()
+    nlp2.from_bytes(b)
+
+    assert task1._label_dict == task2._label_dict == labels
+
+
+def test_ner_to_disk(noop_config, tmp_path: Path):
+    config = Config().from_str(noop_config)
+    del config["components"]["llm"]["task"]["labels"]
+
+    nlp1 = assemble_from_config(config)
+    nlp2 = assemble_from_config(config)
+
+    labels = {"loc": "LOC", "per": "PER"}
+
+    task1: NERTask = nlp1.get_pipe("llm")._task
+    task2: NERTask = nlp2.get_pipe("llm")._task
+
+    # Artificially add labels to task1
+    task1._label_dict = labels
+
+    assert task1._label_dict == labels
+    assert task2._label_dict == dict()
+
+    path = tmp_path / "model"
+
+    nlp1.to_disk(path)
+
+    cfgs = list(path.rglob("cfg"))
+    assert len(cfgs) == 1
+
+    cfg = json.loads(cfgs[0].read_text())
+    assert cfg["_label_dict"] == labels
+
+    nlp2.from_disk(path)
+
+    assert task1._label_dict == task2._label_dict == labels
+
+
+def test_label_inconsistency():
+    """Test whether inconsistency between specified labels and labels in examples is detected."""
+    cfg = f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.NER.v2"
+    labels = ["PERSON", "LOCATION"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner_inconsistent.yml"))}
+
+    [components.llm.model]
+    @llm_models = "test.NoOpModel.v1"
+    """
+
+    config = Config().from_str(cfg)
+    with pytest.warns(
+        UserWarning,
+        match=re.escape(
+            "Examples contain labels that are not specified in the task configuration. The latter contains the "
+            "following labels: ['LOCATION', 'PERSON']. Labels in examples missing from the task configuration: "
+            "['TECH']. Please ensure your label specification and example labels are consistent."
+        ),
+    ):
+        nlp = assemble_from_config(config)
+
+    prompt_examples = nlp.get_pipe("llm")._task._prompt_examples
+    assert len(prompt_examples) == 2
+    assert prompt_examples[0].text == "Jack and Jill went up the hill."
+    assert prompt_examples[0].entities == {
+        "LOCATION": ["hill"],
+        "PERSON": ["Jack", "Jill"],
+    }
+    assert (
+        prompt_examples[1].text
+        == "Jack and Jill went up the hill and spaCy is a great tool."
+    )
+    assert prompt_examples[1].entities == {
+        "LOCATION": ["hill"],
+        "PERSON": ["Jack", "Jill"],
+    }
diff --git a/spacy_llm/tests/tasks/legacy/test_spancat.py b/spacy_llm/tests/tasks/legacy/test_spancat.py
new file mode 100644
index 00000000..124dd94d
--- /dev/null
+++ b/spacy_llm/tests/tasks/legacy/test_spancat.py
@@ -0,0 +1,603 @@
+from pathlib import Path
+
+import pytest
+import spacy
+import srsly
+from confection import Config
+from spacy.tokens import Span
+from spacy.training import Example
+from spacy.util import make_tempdir
+
+from spacy_llm.pipeline import LLMWrapper
+from spacy_llm.registry import fewshot_reader, lowercase_normalizer, strip_normalizer
+from spacy_llm.tasks.spancat import SpanCatTask, make_spancat_task_v2
+from spacy_llm.tasks.util import find_substrings
+from spacy_llm.ty import LabeledTask, LLMTask
+from spacy_llm.util import assemble_from_config, split_labels
+
+from ...compat import has_openai_key
+
+EXAMPLES_DIR = Path(__file__).parent / "examples"
+
+
+@pytest.fixture
+def zeroshot_cfg_string():
+    return """
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.SpanCat.v2"
+    labels = PER,ORG,LOC
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.fixture
+def fewshot_cfg_string():
+    return f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.SpanCat.v2"
+    labels = ["PER", "ORG", "LOC"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v1"
+    """
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+@pytest.mark.parametrize("cfg_string", ["fewshot_cfg_string", "zeroshot_cfg_string"])
+def test_spancat_config(cfg_string, request):
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    assert nlp.pipe_names == ["llm"]
+
+    pipe = nlp.get_pipe("llm")
+    assert isinstance(pipe, LLMWrapper)
+    assert isinstance(pipe.task, LLMTask)
+
+    labels = orig_config["components"]["llm"]["task"]["labels"]
+    labels = split_labels(labels)
+    task = pipe.task
+    assert isinstance(task, LabeledTask)
+    assert sorted(task.labels) == sorted(tuple(labels))
+    assert pipe.labels == task.labels
+    assert nlp.pipe_labels["llm"] == list(task.labels)
+
+
+@pytest.mark.external
+@pytest.mark.parametrize("cfg_string", ["zeroshot_cfg_string", "fewshot_cfg_string"])
+def test_spancat_predict(cfg_string, request):
+    """Use OpenAI to get zero-shot NER results.
+    Note that this test may fail randomly, as the LLM's output is unguaranteed to be consistent/predictable
+    """
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    text = "Marc and Bob both live in Ireland."
+    doc = nlp(text)
+    assert len(doc.spans["sc"]) > 0
+    for ent in doc.spans["sc"]:
+        assert ent.label_ in ["PER", "ORG", "LOC"]
+
+
+@pytest.mark.external
+@pytest.mark.parametrize("cfg_string", ["zeroshot_cfg_string", "fewshot_cfg_string"])
+def test_spancat_io(cfg_string, request):
+    cfg_string = request.getfixturevalue(cfg_string)
+    orig_config = Config().from_str(cfg_string)
+    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+    assert nlp.pipe_names == ["llm"]
+    # ensure you can save a pipeline to disk and run it after loading
+    with make_tempdir() as tmpdir:
+        nlp.to_disk(tmpdir)
+        nlp2 = spacy.load(tmpdir)
+    assert nlp2.pipe_names == ["llm"]
+    text = "Marc and Bob both live in Ireland."
+    doc = nlp2(text)
+    assert len(doc.spans["sc"]) > 0
+    for ent in doc.spans["sc"]:
+        assert ent.label_ in ["PER", "ORG", "LOC"]
+
+
+@pytest.mark.parametrize(
+    "text,input_strings,result_strings,result_offsets",
+    [
+        (
+            "Felipe and Jaime went to the library.",
+            ["Felipe", "Jaime", "library"],
+            ["Felipe", "Jaime", "library"],
+            [(0, 6), (11, 16), (29, 36)],
+        ),  # simple
+        (
+            "The Manila Observatory was founded in 1865 in Manila.",
+            ["Manila", "The Manila Observatory"],
+            ["Manila", "Manila", "The Manila Observatory"],
+            [(4, 10), (46, 52), (0, 22)],
+        ),  # overlapping and duplicated
+        (
+            "Take the road from downtown and turn left at the public market.",
+            ["public market", "downtown"],
+            ["public market", "downtown"],
+            [(49, 62), (19, 27)]
+            # flipped
+        ),
+    ],
+)
+def test_ensure_offsets_correspond_to_substrings(
+    text, input_strings, result_strings, result_offsets
+):
+    offsets = find_substrings(text, input_strings)
+    # Compare strings instead of offsets, but we need to get
+    # those strings first from the text
+    assert result_offsets == offsets
+    found_substrings = [text[start:end] for start, end in offsets]
+    assert result_strings == found_substrings
+
+
+@pytest.mark.parametrize(
+    "text,response,gold_spans",
+    [
+        # simple
+        (
+            "Jean Jacques and Jaime went to the library.",
+            "PER: Jean Jacques, Jaime\nLOC: library",
+            [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "LOC")],
+        ),
+        # overlapping: should only return all spans
+        (
+            "The Manila Observatory was founded in 1865.",
+            "LOC: The Manila Observatory, Manila, Manila Observatory",
+            [
+                ("The Manila Observatory", "LOC"),
+                ("Manila", "LOC"),
+                ("Manila Observatory", "LOC"),
+            ],
+        ),
+        # flipped: order shouldn't matter
+        (
+            "Take the road from Downtown and turn left at the public market.",
+            "LOC: public market, Downtown",
+            [("Downtown", "LOC"), ("public market", "LOC")],
+        ),
+    ],
+)
+def test_spancat_zero_shot_task(text, response, gold_spans):
+    labels = "PER,ORG,LOC"
+    llm_spancat = make_spancat_task_v2(labels=labels)
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list so we get what's inside
+    doc_out = list(llm_spancat.parse_responses([doc_in], [response]))[0]
+    pred_spans = [(span.text, span.label_) for span in doc_out.spans["sc"]]
+    assert pred_spans == gold_spans
+
+
+@pytest.mark.parametrize(
+    "response,normalizer,gold_spans",
+    [
+        (
+            "PER: Jean Jacques, Jaime",
+            None,
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, Jaime",
+            strip_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, Jaime",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques, Jaime",
+            strip_normalizer(),
+            [],
+        ),
+        (
+            "per: Jean Jacques, Jaime",
+            None,
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques\nPER: Jaime",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+        (
+            "per: Jean Jacques, Jaime\nOrg: library",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "ORG")],
+        ),
+        (
+            "per: Jean Jacques, Jaime\nRANDOM: library",
+            lowercase_normalizer(),
+            [("Jean Jacques", "PER"), ("Jaime", "PER")],
+        ),
+    ],
+)
+def test_spancat_labels(response, normalizer, gold_spans):
+    text = "Jean Jacques and Jaime went to the library."
+    labels = "PER,ORG,LOC"
+    llm_spancat = make_spancat_task_v2(labels=labels, normalizer=normalizer)
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_spancat.parse_responses([doc_in], [response]))[0]
+    pred_spans = [(span.text, span.label_) for span in doc_out.spans["sc"]]
+    assert pred_spans == gold_spans
+
+
+@pytest.mark.parametrize(
+    "response,alignment_mode,gold_spans",
+    [
+        (
+            "PER: Jacq",
+            "strict",
+            [],
+        ),
+        (
+            "PER: Jacq",
+            "contract",
+            [],
+        ),
+        (
+            "PER: Jacq",
+            "expand",
+            [("Jacques", "PER")],
+        ),
+        (
+            "PER: Jean J",
+            "contract",
+            [("Jean", "PER")],
+        ),
+        (
+            "PER: Jean Jacques, aim",
+            "strict",
+            [("Jean Jacques", "PER")],
+        ),
+        (
+            "PER: random",
+            "expand",
+            [],
+        ),
+    ],
+)
+def test_spancat_alignment(response, alignment_mode, gold_spans):
+    text = "Jean Jacques and Jaime went to the library."
+    labels = "PER,ORG,LOC"
+    llm_spancat = make_spancat_task_v2(labels=labels, alignment_mode=alignment_mode)  # type: ignore
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_spancat.parse_responses([doc_in], [response]))[0]
+    pred_spans = [(span.text, span.label_) for span in doc_out.spans["sc"]]
+    assert pred_spans == gold_spans
+
+
+def test_invalid_alignment_mode():
+    labels = "PER,ORG,LOC"
+    with pytest.raises(ValueError, match="Unsupported alignment mode 'invalid"):
+        make_spancat_task_v2(labels=labels, alignment_mode="invalid")  # type: ignore
+
+
+@pytest.mark.parametrize(
+    "response,case_sensitive,single_match,gold_spans",
+    [
+        (
+            "PER: Jean",
+            False,
+            False,
+            [("jean", "PER"), ("Jean", "PER"), ("Jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            False,
+            True,
+            [("jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            True,
+            False,
+            [("Jean", "PER"), ("Jean", "PER")],
+        ),
+        (
+            "PER: Jean",
+            True,
+            True,
+            [("Jean", "PER")],
+        ),
+    ],
+)
+def test_spancat_matching(response, case_sensitive, single_match, gold_spans):
+    text = "This guy jean (or Jean) is the president of the Jean Foundation."
+    labels = "PER,ORG,LOC"
+    llm_spancat = make_spancat_task_v2(
+        labels=labels, case_sensitive_matching=case_sensitive, single_match=single_match
+    )
+    # Prepare doc
+    nlp = spacy.blank("en")
+    doc_in = nlp.make_doc(text)
+    # Pass to the parser
+    # Note: parser() returns a list
+    doc_out = list(llm_spancat.parse_responses([doc_in], [response]))[0]
+    pred_spans = [(span.text, span.label_) for span in doc_out.spans["sc"]]
+    assert pred_spans == gold_spans
+
+
+def test_jinja_template_rendering_without_examples():
+    """Test if jinja template renders as we expected
+
+    We apply the .strip() method for each prompt so that we don't have to deal
+    with annoying newlines and spaces at the edge of the text.
+    """
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+
+    llm_spancat = make_spancat_task_v2(labels=labels, examples=None)
+    prompt = list(llm_spancat.generate_prompts([doc]))[0]
+
+    assert (
+        prompt.strip()
+        == """
+You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
+The entities you extract for each label can overlap with each other.
+From the Text input provided, extract named entities for each label in the following format:
+
+LOC: <comma delimited list of strings>
+ORG: <comma delimited list of strings>
+PER: <comma delimited list of strings>
+
+
+Here is the text that needs labeling:
+
+Text:
+'''
+Alice and Bob went to the supermarket
+'''
+""".strip()
+    )
+
+
+@pytest.mark.parametrize(
+    "examples_path",
+    [
+        str(EXAMPLES_DIR / "ner.json"),
+        str(EXAMPLES_DIR / "ner.yml"),
+        str(EXAMPLES_DIR / "ner.jsonl"),
+    ],
+)
+def test_jinja_template_rendering_with_examples(examples_path):
+    """Test if jinja2 template renders as expected
+
+    We apply the .strip() method for each prompt so that we don't have to deal
+    with annoying newlines and spaces at the edge of the text.
+    """
+    labels = "PER,ORG,LOC"
+    nlp = spacy.blank("en")
+    doc = nlp.make_doc("Alice and Bob went to the supermarket")
+
+    examples = fewshot_reader(examples_path)
+    llm_spancat = make_spancat_task_v2(labels=labels, examples=examples)
+    prompt = list(llm_spancat.generate_prompts([doc]))[0]
+
+    assert (
+        prompt.strip()
+        == """
+You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
+The entities you extract for each label can overlap with each other.
+From the Text input provided, extract named entities for each label in the following format:
+
+LOC: <comma delimited list of strings>
+ORG: <comma delimited list of strings>
+PER: <comma delimited list of strings>
+
+
+Below are some examples (only use these as a guide):
+
+Text:
+'''
+Jack and Jill went up the hill.
+'''
+
+PER: Jack, Jill
+LOC: hill
+
+Text:
+'''
+Jack fell down and broke his crown.
+'''
+
+PER: Jack
+
+Text:
+'''
+Jill came tumbling after.
+'''
+
+PER: Jill
+
+
+Here is the text that needs labeling:
+
+Text:
+'''
+Alice and Bob went to the supermarket
+'''""".strip()
+    )
+
+
+def test_example_not_following_basemodel():
+    wrong_example = [
+        {
+            "text": "I'm a wrong example. Entities should be a dict, not a list",
+            # Should be: {"PER": ["Entities"], "ORG": ["dict", "list"]}
+            "entities": [("PER", ("Entities")), ("ORG", ("dict", "list"))],
+        }
+    ]
+    with make_tempdir() as tmpdir:
+        tmp_path = tmpdir / "wrong_example.yml"
+        srsly.write_yaml(tmp_path, wrong_example)
+
+    with pytest.raises(ValueError):
+        make_spancat_task_v2(labels="PER,ORG,LOC", examples=fewshot_reader(tmp_path))
+
+
+@pytest.fixture
+def noop_config():
+    return """
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.SpanCat.v2"
+    labels = ["PER", "ORG", "LOC"]
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "test.NoOpModel.v1"
+    output = "PER: Bob,Alice"
+    """
+
+
+@pytest.mark.parametrize("n_detections", [0, 1, 2])
+def test_spancat_scoring(noop_config, n_detections):
+    config = Config().from_str(noop_config)
+    nlp = assemble_from_config(config)
+
+    examples = []
+
+    for text in ["Alice works with Bob.", "Bob lives with Alice."]:
+        predicted = nlp.make_doc(text)
+        reference = predicted.copy()
+
+        reference.spans["sc"] = [
+            Span(reference, 0, 1, label="PER"),
+            Span(reference, 3, 4, label="PER"),
+        ][:n_detections]
+
+        examples.append(Example(predicted, reference))
+
+    scores = nlp.evaluate(examples)
+
+    assert scores["spans_sc_p"] == n_detections / 2
+
+
+@pytest.mark.parametrize("n_prompt_examples", [-1, 0, 1, 2])
+def test_spancat_init(noop_config, n_prompt_examples: bool):
+    config = Config().from_str(noop_config)
+    del config["components"]["llm"]["task"]["labels"]
+    nlp = assemble_from_config(config)
+
+    examples = []
+
+    for text in [
+        "Alice works with Bob in London.",
+        "Bob lives with Alice in Manchester.",
+    ]:
+        predicted = nlp.make_doc(text)
+        reference = predicted.copy()
+
+        reference.spans["sc"] = [
+            Span(reference, 0, 1, label="PER"),
+            Span(reference, 3, 4, label="PER"),
+            Span(reference, 5, 6, label="LOC"),
+        ]
+
+        examples.append(Example(predicted, reference))
+
+    _, llm = nlp.pipeline[0]
+    task: SpanCatTask = llm._task  # type: ignore
+
+    assert set(task._label_dict.values()) == set()
+    assert not task._prompt_examples
+
+    nlp.config["initialize"]["components"]["llm"] = {
+        "n_prompt_examples": n_prompt_examples
+    }
+
+    nlp.initialize(lambda: examples)
+
+    assert set(task._label_dict.values()) == {"PER", "LOC"}
+    if n_prompt_examples >= 0:
+        assert len(task._prompt_examples) == n_prompt_examples
+    else:
+        assert len(task._prompt_examples) == len(examples)
+
+    if n_prompt_examples > 0:
+        for eg in task._prompt_examples:
+            assert set(eg.entities.keys()) == {"PER", "LOC"}
+
+
+def test_spancat_serde(noop_config):
+    config = Config().from_str(noop_config)
+    del config["components"]["llm"]["task"]["labels"]
+
+    nlp1 = assemble_from_config(config)
+    nlp2 = assemble_from_config(config)
+
+    labels = {"loc": "LOC", "per": "PER"}
+
+    task1: SpanCatTask = nlp1.get_pipe("llm")._task  # type: ignore
+    task2: SpanCatTask = nlp2.get_pipe("llm")._task  # type: ignore
+
+    # Artificially add labels to task1
+    task1._label_dict = labels
+
+    assert task1._label_dict == labels
+    assert task2._label_dict == dict()
+
+    b = nlp1.to_bytes()
+    nlp2.from_bytes(b)
+
+    assert task1._label_dict == task2._label_dict == labels
diff --git a/spacy_llm/tests/tasks/templates/ner.jinja2 b/spacy_llm/tests/tasks/templates/ner.jinja2
index edc6042c..55803184 100644
--- a/spacy_llm/tests/tasks/templates/ner.jinja2
+++ b/spacy_llm/tests/tasks/templates/ner.jinja2
@@ -1,8 +1,31 @@
-This is a test NER template. Here are the labels
+Here's the test template for the tests and stuff
 {# whitespace #}
-{%- for label in labels -%}
-{{ label }}
+{{ description }}
+{# whitespace #}
+{%- if label_definitions -%}
+Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
+Assume these definitions are written by an expert and follow them closely.
+{# whitespace #}
+{%- for label, definition in label_definitions.items() -%}
+{{ label }}: {{ definition }}
 {# whitespace #}
 {%- endfor -%}
 {# whitespace #}
-Here is the text: {{ text }}
\ No newline at end of file
+{# whitespace #}
+{%- endif -%}
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+{# whitespace #}
+{# whitespace #}
+{%- for example in prompt_examples -%}
+Paragraph: {{ example.text }}
+Answer:
+{# whitespace #}
+{%- for span in example.spans -%}
+{{ loop.index }}. {{ span.to_str() }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endfor -%}
+Paragraph: {{ text }}
+Answer:
diff --git a/spacy_llm/tests/tasks/templates/spancat.jinja2 b/spacy_llm/tests/tasks/templates/spancat.jinja2
new file mode 100644
index 00000000..88d775f4
--- /dev/null
+++ b/spacy_llm/tests/tasks/templates/spancat.jinja2
@@ -0,0 +1,32 @@
+Here's the test template for the tests and stuff
+The entities you extract can overlap with each other.
+{# whitespace #}
+{{ description }}
+{# whitespace #}
+{%- if label_definitions -%}
+Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
+Assume these definitions are written by an expert and follow them closely.
+{# whitespace #}
+{%- for label, definition in label_definitions.items() -%}
+{{ label }}: {{ definition }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endif -%}
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+{# whitespace #}
+{# whitespace #}
+{%- for example in prompt_examples -%}
+Paragraph: {{ example.text }}
+Answer:
+{# whitespace #}
+{%- for span in example.spans -%}
+{{ loop.index }}. {{ span.to_str() }}
+{# whitespace #}
+{%- endfor -%}
+{# whitespace #}
+{# whitespace #}
+{%- endfor -%}
+Paragraph: {{ text }}
+Answer:
diff --git a/spacy_llm/tests/tasks/test_lemma.py b/spacy_llm/tests/tasks/test_lemma.py
index dafd5ea2..ec30e3ce 100644
--- a/spacy_llm/tests/tasks/test_lemma.py
+++ b/spacy_llm/tests/tasks/test_lemma.py
@@ -56,8 +56,7 @@ def zeroshot_cfg_string():
     @llm_tasks = "spacy.Lemma.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {"temperature": 0}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -82,8 +81,7 @@ def fewshot_cfg_string():
     path = {str((Path(__file__).parent / "examples" / "lemma.yml"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -109,8 +107,7 @@ def ext_template_cfg_string():
     path = {str((Path(__file__).parent / "templates" / "lemma.jinja2"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -197,7 +194,7 @@ def test_jinja_template_rendering_without_examples():
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     text = "Alice and Bob went to the supermarket"
     doc = nlp.make_doc(text)
 
@@ -240,7 +237,7 @@ def test_jinja_template_rendering_with_examples(examples_path):
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     text = "Alice and Bob went to the supermarket."
     doc = nlp.make_doc(text)
 
@@ -333,7 +330,7 @@ def test_external_template_actually_loads():
     template_path = str(TEMPLATES_DIR / "lemma.jinja2")
     template = file_reader(template_path)
     text = "Alice and Bob went to the supermarket"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(text)
 
     lemma_task = make_lemma_task(template=template)
diff --git a/spacy_llm/tests/tasks/test_ner.py b/spacy_llm/tests/tasks/test_ner.py
index b1605c1c..7d7c8c82 100644
--- a/spacy_llm/tests/tasks/test_ner.py
+++ b/spacy_llm/tests/tasks/test_ner.py
@@ -1,21 +1,26 @@
 import json
 import re
 from pathlib import Path
+from typing import Callable, List, Tuple, cast
 
 import pytest
 import spacy
 import srsly
 from confection import Config
+from spacy.language import Language
 from spacy.tokens import Span
 from spacy.training import Example
 from spacy.util import make_tempdir
 
+from spacy_llm.compat import Literal, ValidationError
 from spacy_llm.pipeline import LLMWrapper
 from spacy_llm.registry import fewshot_reader, file_reader, lowercase_normalizer
 from spacy_llm.registry import strip_normalizer
-from spacy_llm.tasks.ner import NERTask, make_ner_task_v2
+from spacy_llm.tasks.ner import NERTask, make_ner_task_v3
+from spacy_llm.tasks.span import SpanReason
+from spacy_llm.tasks.span.parser import _extract_span_reasons_cot
 from spacy_llm.tasks.util import find_substrings
-from spacy_llm.ty import Labeled, LLMTask
+from spacy_llm.ty import LabeledTask, LLMTask
 from spacy_llm.util import assemble_from_config, split_labels
 
 from ..compat import has_openai_key
@@ -25,33 +30,18 @@
 
 
 @pytest.fixture
-def zeroshot_cfg_string():
-    return """
-    [nlp]
-    lang = "en"
-    pipeline = ["llm"]
-    batch_size = 128
-
-    [components]
-
-    [components.llm]
-    factory = "llm"
-
-    [components.llm.task]
-    @llm_tasks = "spacy.NER.v1"
-    labels = PER,ORG,LOC
+def examples_dir():
+    return EXAMPLES_DIR
 
-    [components.llm.task.normalizer]
-    @misc = "spacy.LowercaseNormalizer.v1"
 
-    [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    """
+@pytest.fixture
+def templates_dir():
+    return TEMPLATES_DIR
 
 
 @pytest.fixture
-def zeroshot_cfg_string_v2_lds():
-    return """
+def noop_config():
+    return f"""
     [nlp]
     lang = "en"
     pipeline = ["llm"]
@@ -63,24 +53,25 @@ def zeroshot_cfg_string_v2_lds():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.NER.v2"
+    @llm_tasks = "spacy.NER.v3"
     labels = PER,ORG,LOC
 
-    [components.llm.task.label_definitions]
-    PER = "Any named individual in the text"
-    ORG = "Any named organization in the text"
-    LOC = "The name of any politically or geographically defined location"
-
     [components.llm.task.normalizer]
     @misc = "spacy.LowercaseNormalizer.v1"
 
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner.json"))}
+
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "test.NoOpModel.v1"
+    output = 1. Bob | True | PER | is the name of a person
+        2. Alice | True | PER | is the name of a person
     """
 
 
 @pytest.fixture
-def fewshot_cfg_string():
+def fewshot_cfg_string_v3_lds():
     return f"""
     [nlp]
     lang = "en"
@@ -93,23 +84,29 @@ def fewshot_cfg_string():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.NER.v1"
+    @llm_tasks = "spacy.NER.v3"
+    description = "This is a description"
     labels = PER,ORG,LOC
 
     [components.llm.task.examples]
     @misc = "spacy.FewShotReader.v1"
-    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+    path = {str((Path(__file__).parent / "examples" / "ner.json"))}
+
+    [components.llm.task.label_definitions]
+    PER = "Any named individual in the text"
+    ORG = "Any named organization in the text"
+    LOC = "The name of any politically or geographically defined location"
 
     [components.llm.task.normalizer]
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
 @pytest.fixture
-def fewshot_cfg_string_v2():
+def fewshot_cfg_string_v3():
     return f"""
     [nlp]
     lang = "en"
@@ -122,18 +119,19 @@ def fewshot_cfg_string_v2():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.NER.v2"
+    @llm_tasks = "spacy.NER.v3"
+    description = "This is a description"
     labels = ["PER", "ORG", "LOC"]
 
     [components.llm.task.examples]
     @misc = "spacy.FewShotReader.v1"
-    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+    path = {str((Path(__file__).parent / "examples" / "ner.json"))}
 
     [components.llm.task.normalizer]
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -152,8 +150,13 @@ def ext_template_cfg_string():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.NER.v2"
-    labels = PER,ORG,LOC
+    @llm_tasks = "spacy.NER.v3"
+    description = "This is a description"
+    labels = ["PER", "ORG", "LOC"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "ner.json"))}
 
     [components.llm.task.template]
     @misc = "spacy.FileReader.v1"
@@ -163,30 +166,37 @@ def ext_template_cfg_string():
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
-@pytest.mark.external
-@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
-@pytest.mark.parametrize(
-    "cfg_string",
-    [
-        "zeroshot_cfg_string",
-        "zeroshot_cfg_string_v2_lds",
-        "fewshot_cfg_string",
-        "fewshot_cfg_string_v2",
+@pytest.fixture(
+    params=[
+        "fewshot_cfg_string_v3_lds",
+        "fewshot_cfg_string_v3",
         "ext_template_cfg_string",
-    ],
+    ]
 )
-def test_ner_config(cfg_string, request):
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+def config(request) -> Config:
+    cfg_str = request.getfixturevalue(request.param)
+    config = Config().from_str(cfg_str)
+    return config
+
+
+@pytest.fixture
+def nlp(config: Config) -> Language:
+    nlp = assemble_from_config(config)
+    return nlp
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_ner_config(config: Config):
+    nlp = assemble_from_config(config)
     assert nlp.pipe_names == ["llm"]
 
     # also test nlp config from a dict in add_pipe
-    component_cfg = dict(orig_config["components"]["llm"])
+    component_cfg = dict(config["components"]["llm"])
     component_cfg.pop("factory")
 
     nlp2 = spacy.blank("en")
@@ -197,10 +207,10 @@ def test_ner_config(cfg_string, request):
     assert isinstance(pipe, LLMWrapper)
     assert isinstance(pipe.task, LLMTask)
 
-    labels = orig_config["components"]["llm"]["task"]["labels"]
+    labels = config["components"]["llm"]["task"]["labels"]
     labels = split_labels(labels)
     task = pipe.task
-    assert isinstance(task, Labeled)
+    assert isinstance(task, LabeledTask)
     assert sorted(task.labels) == sorted(tuple(labels))
     assert pipe.labels == task.labels
     assert nlp.pipe_labels["llm"] == list(task.labels)
@@ -209,47 +219,67 @@ def test_ner_config(cfg_string, request):
 @pytest.mark.external
 @pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
 @pytest.mark.parametrize(
-    "cfg_string",
+    "cfg_str",
+    ["fewshot_cfg_string_v3_lds", "fewshot_cfg_string_v3"],
+)
+@pytest.mark.parametrize(
+    "text,gold_ents",
     [
-        "zeroshot_cfg_string",
-        "zeroshot_cfg_string_v2_lds",
-        "fewshot_cfg_string",
-        "fewshot_cfg_string_v2",
-        "ext_template_cfg_string",
+        (
+            "Marc and Bob both live in Ireland.",
+            [("Marc", "PER"), ("Bob", "PER"), ("Ireland", "LOC")],
+        ),
     ],
 )
-def test_ner_predict(cfg_string, request):
-    """Use OpenAI to get zero-shot NER results.
+def test_ner_predict(cfg_str, text, gold_ents, request):
+    """Use OpenAI to get NER results.
     Note that this test may fail randomly, as the LLM's output is unguaranteed to be consistent/predictable
     """
-    orig_cfg_string = cfg_string
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
-    text = "Marc and Bob both live in Ireland."
+    config = Config().from_str(request.getfixturevalue(cfg_str))
+
+    nlp = spacy.util.load_model_from_config(config, auto_fill=True)
     doc = nlp(text)
 
-    if orig_cfg_string != "ext_template_cfg_string":
-        assert len(doc.ents) > 0
-        for ent in doc.ents:
-            assert ent.label_ in ["PER", "ORG", "LOC"]
+    assert len(doc.ents) == len(gold_ents)
+    for pred_ent, gold_ent in zip(doc.ents, gold_ents):
+        assert (
+            gold_ent[0] in pred_ent.text
+        )  # occassionally, the LLM predicts "in Ireland" instead of just "Ireland"
+        assert pred_ent.label_ in gold_ent[1].split("|")
 
 
 @pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
 @pytest.mark.parametrize(
-    "cfg_string",
+    "text,gold_ents",
     [
-        "zeroshot_cfg_string",
-        "zeroshot_cfg_string_v2_lds",
-        "fewshot_cfg_string",
-        "fewshot_cfg_string_v2",
-        "ext_template_cfg_string",
+        (
+            "Marc and Bob both live in Ireland.",
+            [("Marc", "PER"), ("Bob", "PER"), ("Ireland", "LOC")],
+        ),
     ],
 )
-def test_ner_io(cfg_string, request):
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+def test_llm_ner_predict(text, gold_ents):
+    """Use llm_ner factory with default OpenAI model to get NER results.
+    Note that this test may fail randomly, as the LLM's output is unguaranteed to be consistent/predictable
+    """
+    nlp = spacy.blank("en")
+    llm = nlp.add_pipe("llm_ner")
+    for ent_str, ent_label in gold_ents:
+        llm.add_label(ent_label)
+    doc = nlp(text)
+
+    assert len(doc.ents) == len(gold_ents)
+    for pred_ent, gold_ent in zip(doc.ents, gold_ents):
+        assert (
+            gold_ent[0] in pred_ent.text
+        )  # occassionally, the LLM predicts "in Ireland" instead of just "Ireland"
+        assert pred_ent.label_ in gold_ent[1].split("|")
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_ner_io(nlp: Language):
     assert nlp.pipe_names == ["llm"]
     # ensure you can save a pipeline to disk and run it after loading
     with make_tempdir() as tmpdir:
@@ -298,93 +328,70 @@ def test_ensure_offsets_correspond_to_substrings(
     assert result_strings == found_substrings
 
 
-@pytest.mark.parametrize(
-    "text,response,gold_ents",
-    [
-        # simple
-        (
-            "Jean Jacques and Jaime went to the library.",
-            "PER: Jean Jacques, Jaime\nLOC: library",
-            [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "LOC")],
-        ),
-        # overlapping: should only return the longest span
-        (
-            "The Manila Observatory was founded in 1865.",
-            "LOC: The Manila Observatory, Manila, Manila Observatory",
-            [("The Manila Observatory", "LOC")],
-        ),
-        # flipped: order shouldn't matter
-        (
-            "Take the road from Downtown and turn left at the public market.",
-            "LOC: public market, Downtown",
-            [("Downtown", "LOC"), ("public market", "LOC")],
-        ),
-    ],
-)
-def test_ner_zero_shot_task(text, response, gold_ents):
-    labels = "PER,ORG,LOC"
-    llm_ner = make_ner_task_v2(labels=labels)
-    # Prepare doc
-    nlp = spacy.blank("xx")
-    doc_in = nlp.make_doc(text)
-    # Pass to the parser
-    # Note: parser() returns a list so we get what's inside
-    doc_out = list(llm_ner.parse_responses([doc_in], [response]))[0]
-    pred_ents = [(ent.text, ent.label_) for ent in doc_out.ents]
-    assert pred_ents == gold_ents
-
-
 @pytest.mark.parametrize(
     "response,normalizer,gold_ents",
     [
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is a person's name\n"
+            "2. Jaime | True | PER | is a person's name\n",
             None,
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is a person's name\n"
+            "2. Jaime | True | PER | is a person's name\n",
             strip_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is a person's name\n"
+            "2. Jaime | True | PER | is a person's name\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | per | is a person's name\n"
+            "2. Jaime | True | per | is a person's name\n",
             strip_normalizer(),
             [],
         ),
         (
-            "per: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | per | is a person's name\n"
+            "2. Jaime | True | per | is a person's name\n",
             None,
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques\nPER: Jaime",
+            "1. Jean Jacques | True | per | is a person's name\n"
+            "2. Jaime | True | PER | is a person's name\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques, Jaime\nOrg: library",
+            "1. Jean Jacques | True | per | is a person's name\n"
+            "2. Jaime | True | per | is a person's name\n"
+            "3. library | True | Org | is a organization\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "ORG")],
         ),
         (
-            "per: Jean Jacques, Jaime\nRANDOM: library",
+            "1. Jean Jacques | True | per | is a person's name\n"
+            "2. Jaime | True | per | is a person's name\n"
+            "3. Jaime | True | RANDOM | is an entity\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
     ],
 )
-def test_ner_labels(response, normalizer, gold_ents):
+def test_ner_labels(
+    response: str, normalizer: Callable[[str], str], gold_ents: List[Tuple[str, str]]
+):
     text = "Jean Jacques and Jaime went to the library."
     labels = "PER,ORG,LOC"
-    llm_ner = make_ner_task_v2(labels=labels, normalizer=normalizer)
+
+    llm_ner = make_ner_task_v3(examples=[], labels=labels, normalizer=normalizer)
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -397,43 +404,50 @@ def test_ner_labels(response, normalizer, gold_ents):
     "response,alignment_mode,gold_ents",
     [
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "strict",
             [],
         ),
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "contract",
             [],
         ),
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "expand",
             [("Jacques", "PER")],
         ),
         (
-            "PER: Jean J",
+            "1. Jean J | True | PER | is a person's name",
             "contract",
             [("Jean", "PER")],
         ),
         (
-            "PER: Jean Jacques, aim",
+            "1. Jean Jacques | True | PER | is a person's name",
             "strict",
             [("Jean Jacques", "PER")],
         ),
         (
-            "PER: random",
+            "1. random | True | PER | is a person's name",
             "expand",
             [],
         ),
     ],
+    ids=["strict_1", "contract_1", "expand_1", "strict_2", "contract_2", "expand_2"],
 )
-def test_ner_alignment(response, alignment_mode, gold_ents):
+def test_ner_alignment(
+    response: str,
+    alignment_mode: Literal["strict", "contract", "expand"],
+    gold_ents: List[Tuple[str, str]],
+):
     text = "Jean Jacques and Jaime went to the library."
     labels = "PER,ORG,LOC"
-    llm_ner = make_ner_task_v2(labels=labels, alignment_mode=alignment_mode)
+    llm_ner = make_ner_task_v3(
+        examples=[], labels=labels, alignment_mode=alignment_mode
+    )
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -445,46 +459,46 @@ def test_ner_alignment(response, alignment_mode, gold_ents):
 def test_invalid_alignment_mode():
     labels = "PER,ORG,LOC"
     with pytest.raises(ValueError, match="Unsupported alignment mode 'invalid"):
-        make_ner_task_v2(labels=labels, alignment_mode="invalid")  # type: ignore
+        make_ner_task_v3(examples=[], labels=labels, alignment_mode="invalid")  # type: ignore
 
 
 @pytest.mark.parametrize(
-    "response,case_sensitive,single_match,gold_ents",
+    "response, case_sensitive, gold_ents",
     [
         (
-            "PER: Jean",
-            False,
-            False,
-            [("jean", "PER"), ("Jean", "PER"), ("Jean", "PER")],
-        ),
-        (
-            "PER: Jean",
+            "1. Jean | True | PER | is a person's name",
             False,
-            True,
             [("jean", "PER")],
         ),
         (
-            "PER: Jean",
+            "1. Jean | True | PER | is a person's name",
             True,
-            False,
-            [("Jean", "PER"), ("Jean", "PER")],
+            [("Jean", "PER")],
         ),
         (
-            "PER: Jean",
-            True,
-            True,
-            [("Jean", "PER")],
+            "1. jean | True | PER | is a person's name\n"
+            "2. Jean | True | PER | is a person's name\n"
+            "3. Jean Foundation | True | ORG | is the name of an Organization name",
+            False,
+            [("jean", "PER"), ("Jean", "PER"), ("Jean Foundation", "ORG")],
         ),
     ],
+    ids=[
+        "single_ent_case_insensitive",
+        "single_ent_case_sensitive",
+        "multiple_ents_case_insensitive",
+    ],
 )
-def test_ner_matching(response, case_sensitive, single_match, gold_ents):
+def test_ner_matching(
+    response: str, case_sensitive: bool, gold_ents: List[Tuple[str, str]]
+):
     text = "This guy jean (or Jean) is the president of the Jean Foundation."
     labels = "PER,ORG,LOC"
-    llm_ner = make_ner_task_v2(
-        labels=labels, case_sensitive_matching=case_sensitive, single_match=single_match
+    llm_ner = make_ner_task_v3(
+        examples=[], labels=labels, case_sensitive_matching=case_sensitive
     )
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -494,116 +508,100 @@ def test_ner_matching(response, case_sensitive, single_match, gold_ents):
 
 
 def test_jinja_template_rendering_without_examples():
-    """Test if jinja template renders as we expected
+    """Test if jinja2 template renders as expected
 
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
     labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
-
-    llm_ner = make_ner_task_v2(labels=labels, examples=None)
+    llm_ner = make_ner_task_v3(labels=labels)
     prompt = list(llm_ner.generate_prompts([doc]))[0]
 
     assert (
         prompt.strip()
         == """
-You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
-From the Text input provided, extract named entities for each label in the following format:
+You are an expert Named Entity Recognition (NER) system.
+Your task is to accept Text as input and extract named entities.
+Entities must have one of the following labels: LOC, ORG, PER.
+If a span is not an entity label it: `==NONE==`.
 
-LOC: <comma delimited list of strings>
-ORG: <comma delimited list of strings>
-PER: <comma delimited list of strings>
 
+Here is an example of the output format for a paragraph using different labels than this task requires.
+Only use this output format but use the labels provided
+above instead of the ones defined in the example below.
+Do not output anything besides entities in this output format.
+Output entities in the order they occur in the input paragraph regardless of label.
 
-Here is the text that needs labeling:
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
 
-Text:
-'''
-Alice and Bob went to the supermarket
-'''
+Paragraph: Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok.
+Answer:
+1. Sriracha sauce | True | INGREDIENT | is an ingredient to add to a stir fry
+2. really well | False | ==NONE== | is a description of how well sriracha sauce goes with hoisin stir fry
+3. hoisin stir fry | True | DISH | is a dish with stir fry vegetables and hoisin sauce
+4. wok | True | EQUIPMENT | is a piece of cooking equipment used to stir fry ingredients
+
+Paragraph: Alice and Bob went to the supermarket
+Answer:
 """.strip()
     )
 
 
-@pytest.mark.parametrize(
-    "examples_path",
-    [
-        str(EXAMPLES_DIR / "ner.json"),
-        str(EXAMPLES_DIR / "ner.yml"),
-        str(EXAMPLES_DIR / "ner.jsonl"),
-    ],
-)
-def test_jinja_template_rendering_with_examples(examples_path):
+@pytest.mark.parametrize("examples_file", ["ner.json", "ner.yml", "ner.jsonl"])
+def test_jinja_template_rendering_with_examples(examples_dir: Path, examples_file: str):
     """Test if jinja2 template renders as expected
 
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
+
     labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
-
-    examples = fewshot_reader(examples_path)
-    llm_ner = make_ner_task_v2(labels=labels, examples=examples)
+    examples = fewshot_reader(examples_dir / examples_file)
+    llm_ner = make_ner_task_v3(examples=examples, labels=labels)
     prompt = list(llm_ner.generate_prompts([doc]))[0]
 
     assert (
         prompt.strip()
         == """
-You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
-From the Text input provided, extract named entities for each label in the following format:
-
-LOC: <comma delimited list of strings>
-ORG: <comma delimited list of strings>
-PER: <comma delimited list of strings>
-
-
-Below are some examples (only use these as a guide):
-
-Text:
-'''
-Jack and Jill went up the hill.
-'''
-
-PER: Jack, Jill
-LOC: hill
-
-Text:
-'''
-Jack fell down and broke his crown.
-'''
-
-PER: Jack
-
-Text:
-'''
-Jill came tumbling after.
-'''
-
-PER: Jill
-
-
-Here is the text that needs labeling:
-
-Text:
-'''
-Alice and Bob went to the supermarket
-'''""".strip()
+You are an expert Named Entity Recognition (NER) system.
+Your task is to accept Text as input and extract named entities.
+Entities must have one of the following labels: LOC, ORG, PER.
+If a span is not an entity label it: `==NONE==`.
+
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+
+Paragraph: Jack and Jill went up the hill.
+Answer:
+1. Jack | True | PER | is the name of a person
+2. Jill | True | PER | is the name of a person
+3. went up | False | ==NONE== | is a verb
+4. hill | True | LOC | is a location
+
+Paragraph: Alice and Bob went to the supermarket
+Answer:
+""".strip()
     )
 
 
-def test_jinja_template_rendering_with_label_definitions():
+@pytest.mark.parametrize("examples_file", ["ner.json", "ner.yml", "ner.jsonl"])
+def test_jinja_template_rendering_with_label_definitions(
+    examples_dir: Path, examples_file: str
+):
     """Test if jinja2 template renders as expected
 
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
     labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
-    llm_ner = make_ner_task_v2(
+    examples = fewshot_reader(examples_dir / examples_file)
+    llm_ner = make_ner_task_v3(
+        examples=examples,
         labels=labels,
         label_definitions={
             "PER": "Person definition",
@@ -616,126 +614,118 @@ def test_jinja_template_rendering_with_label_definitions():
     assert (
         prompt.strip()
         == """
-You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
-From the Text input provided, extract named entities for each label in the following format:
-
-LOC: <comma delimited list of strings>
-ORG: <comma delimited list of strings>
-PER: <comma delimited list of strings>
+You are an expert Named Entity Recognition (NER) system.
+Your task is to accept Text as input and extract named entities.
+Entities must have one of the following labels: LOC, ORG, PER.
+If a span is not an entity label it: `==NONE==`.
 
 Below are definitions of each label to help aid you in what kinds of named entities to extract for each label.
 Assume these definitions are written by an expert and follow them closely.
-
 PER: Person definition
 ORG: Organization definition
 LOC: Location definition
 
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
 
-Here is the text that needs labeling:
+Paragraph: Jack and Jill went up the hill.
+Answer:
+1. Jack | True | PER | is the name of a person
+2. Jill | True | PER | is the name of a person
+3. went up | False | ==NONE== | is a verb
+4. hill | True | LOC | is a location
 
-Text:
-'''
-Alice and Bob went to the supermarket
-'''""".strip()
+Paragraph: Alice and Bob went to the supermarket
+Answer:
+""".strip()
     )
 
 
-def test_example_not_following_basemodel():
-    wrong_example = [
-        {
-            "text": "I'm a wrong example. Entities should be a dict, not a list",
-            # Should be: {"PER": ["Entities"], "ORG": ["dict", "list"]}
-            "entities": [("PER", ("Entities")), ("ORG", ("dict", "list"))],
-        }
-    ]
+@pytest.mark.parametrize(
+    "value, expected_type",
+    [
+        (
+            {
+                "text": "I'm a wrong example. Entities should be a dict, not a list",
+                # Should be: {"PER": ["Entities"], "ORG": ["dict", "list"]}
+                "entities": [("PER", ("Entities")), ("ORG", ("dict", "list"))],
+            },
+            ValidationError,
+        ),
+        (
+            {
+                "text": "Jack is a name",
+                "spans": [
+                    {
+                        "text": "Jack",
+                        "is_entity": True,
+                        "label": "PER",
+                        "reason": "is a person's name",
+                    }
+                ],
+            },
+            NERTask,
+        ),
+    ],
+)
+def test_fewshot_example_data(value: dict, expected_type: type):
     with make_tempdir() as tmpdir:
         tmp_path = tmpdir / "wrong_example.yml"
-        srsly.write_yaml(tmp_path, wrong_example)
+        srsly.write_yaml(tmp_path, [value])
 
-        with pytest.raises(ValueError):
-            make_ner_task_v2(labels="PER,ORG,LOC", examples=fewshot_reader(tmp_path))
+        try:
+            task = make_ner_task_v3(
+                examples=fewshot_reader(tmp_path), labels=["PER", "ORG", "LOC"]
+            )
+        except (ValidationError, ValueError) as e:
+            assert type(e) == expected_type
+        else:
+            assert type(task) == expected_type
 
 
 def test_external_template_actually_loads():
     template_path = str(TEMPLATES_DIR / "ner.jinja2")
     template = file_reader(template_path)
     labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
 
-    llm_ner = make_ner_task_v2(labels=labels, template=template)
+    llm_ner = make_ner_task_v3(examples=[], labels=labels, template=template)
     prompt = list(llm_ner.generate_prompts([doc]))[0]
-    assert (
-        prompt.strip()
-        == """
-This is a test NER template. Here are the labels
-LOC
-ORG
-PER
-
-Here is the text: Alice and Bob went to the supermarket
-""".strip()
-    )
-
-
-@pytest.fixture
-def noop_config():
-    return """
-    [nlp]
-    lang = "en"
-    pipeline = ["llm"]
-    batch_size = 128
-
-    [components]
-
-    [components.llm]
-    factory = "llm"
-
-    [components.llm.task]
-    @llm_tasks = "spacy.NER.v2"
-    labels = PER,ORG,LOC
-
-    [components.llm.task.normalizer]
-    @misc = "spacy.LowercaseNormalizer.v1"
-
-    [components.llm.model]
-    @llm_models = "test.NoOpModel.v1"
-    output = "PER: Alice,Bob"
-    """
+    assert prompt.strip().startswith("Here's the test template for the tests and stuff")
 
 
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
 @pytest.mark.parametrize("n_detections", [0, 1, 2])
-def test_ner_scoring(noop_config, n_detections):
-    config = Config().from_str(noop_config)
+def test_ner_scoring(fewshot_cfg_string_v3: str, n_detections: int):
+    config = Config().from_str(fewshot_cfg_string_v3)
     nlp = assemble_from_config(config)
 
     examples = []
-
     for text in ["Alice works with Bob.", "Bob lives with Alice."]:
         predicted = nlp.make_doc(text)
-        reference = predicted.copy()
-
-        reference.ents = [
-            Span(reference, 0, 1, label="PER"),
-            Span(reference, 3, 4, label="PER"),
-        ][:n_detections]
-
+        reference = nlp.make_doc(text)
+        ent1 = Span(reference, 0, 1, label="PER")
+        ent2 = Span(reference, 3, 4, label="PER")
+        reference.set_ents([ent1, ent2][:n_detections])
         examples.append(Example(predicted, reference))
 
     scores = nlp.evaluate(examples)
-
     assert scores["ents_p"] == n_detections / 2
+    assert scores["ents_r"] == (1 if n_detections != 0 else 0)
+    assert scores["ents_f"] == (
+        pytest.approx(0.666666666) if n_detections == 1 else n_detections / 2
+    )
 
 
 @pytest.mark.parametrize("n_prompt_examples", [-1, 0, 1, 2])
-def test_ner_init(noop_config, n_prompt_examples: int):
+def test_ner_init(noop_config: str, n_prompt_examples: int):
     config = Config().from_str(noop_config)
-    del config["components"]["llm"]["task"]["labels"]
-
+    config["components"]["llm"]["task"]["labels"] = ["PER", "LOC"]
+    config["components"]["llm"]["task"]["examples"] = []
     nlp = assemble_from_config(config)
 
     examples = []
-
     for text in [
         "Alice works with Bob in London.",
         "Bob lives with Alice in Manchester.",
@@ -743,50 +733,50 @@ def test_ner_init(noop_config, n_prompt_examples: int):
         predicted = nlp.make_doc(text)
         reference = predicted.copy()
 
-        reference.ents = [
-            Span(reference, 0, 1, label="PER"),
-            Span(reference, 3, 4, label="PER"),
-            Span(reference, 5, 6, label="LOC"),
-        ]
-
+        reference.set_ents(
+            [
+                Span(reference, 0, 1, label="PER"),
+                Span(reference, 3, 4, label="PER"),
+                Span(reference, 5, 6, label="LOC"),
+            ]
+        )
         examples.append(Example(predicted, reference))
 
-    _, llm = nlp.pipeline[0]
-    task: NERTask = llm._task
-
-    assert set(task._label_dict.values()) == set()
-    assert not task._prompt_examples
-
+    task = cast(NERTask, nlp.get_pipe("llm").task)
     nlp.config["initialize"]["components"]["llm"] = {
         "n_prompt_examples": n_prompt_examples
     }
     nlp.initialize(lambda: examples)
 
     assert set(task._label_dict.values()) == {"PER", "LOC"}
-    if n_prompt_examples >= 0:
-        assert len(task._prompt_examples) == n_prompt_examples
-    else:
+
+    if n_prompt_examples == -1:
         assert len(task._prompt_examples) == len(examples)
+    else:
+        assert len(task._prompt_examples) == n_prompt_examples
 
     if n_prompt_examples > 0:
         for eg in task._prompt_examples:
-            assert set(eg.entities.keys()) == {"PER", "LOC"}
+            prompt_example_labels = {ent.label for ent in eg.spans}
+            if "==NONE==" not in prompt_example_labels:
+                prompt_example_labels.add("==NONE==")
+            assert prompt_example_labels == {"==NONE==", "PER", "LOC"}
 
 
-def test_ner_serde(noop_config):
+def test_ner_serde(noop_config: str):
     config = Config().from_str(noop_config)
-    del config["components"]["llm"]["task"]["labels"]
 
     nlp1 = assemble_from_config(config)
     nlp2 = assemble_from_config(config)
 
     labels = {"loc": "LOC", "per": "PER"}
 
-    task1: NERTask = nlp1.get_pipe("llm")._task
-    task2: NERTask = nlp2.get_pipe("llm")._task
+    task1 = cast(NERTask, nlp1.get_pipe("llm").task)
+    task2 = cast(NERTask, nlp2.get_pipe("llm").task)
 
     # Artificially add labels to task1
     task1._label_dict = labels
+    task2._label_dict = {}
 
     assert task1._label_dict == labels
     assert task2._label_dict == dict()
@@ -797,26 +787,25 @@ def test_ner_serde(noop_config):
     assert task1._label_dict == task2._label_dict == labels
 
 
-def test_ner_to_disk(noop_config, tmp_path: Path):
+def test_ner_to_disk(noop_config: str, tmp_path: Path):
     config = Config().from_str(noop_config)
-    del config["components"]["llm"]["task"]["labels"]
 
     nlp1 = assemble_from_config(config)
     nlp2 = assemble_from_config(config)
 
-    labels = {"loc": "LOC", "per": "PER"}
+    labels = {"loc": "LOC", "org": "ORG", "per": "PER"}
 
-    task1: NERTask = nlp1.get_pipe("llm")._task
-    task2: NERTask = nlp2.get_pipe("llm")._task
+    task1 = cast(NERTask, nlp1.get_pipe("llm").task)
+    task2 = cast(NERTask, nlp2.get_pipe("llm").task)
 
     # Artificially add labels to task1
     task1._label_dict = labels
+    task2._label_dict = {}
 
     assert task1._label_dict == labels
     assert task2._label_dict == dict()
 
     path = tmp_path / "model"
-
     nlp1.to_disk(path)
 
     cfgs = list(path.rglob("cfg"))
@@ -826,7 +815,6 @@ def test_ner_to_disk(noop_config, tmp_path: Path):
     assert cfg["_label_dict"] == labels
 
     nlp2.from_disk(path)
-
     assert task1._label_dict == task2._label_dict == labels
 
 
@@ -843,7 +831,7 @@ def test_label_inconsistency():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.NER.v2"
+    @llm_tasks = "spacy.NER.v3"
     labels = ["PERSON", "LOCATION"]
 
     [components.llm.task.examples]
@@ -868,15 +856,143 @@ def test_label_inconsistency():
     prompt_examples = nlp.get_pipe("llm")._task._prompt_examples
     assert len(prompt_examples) == 2
     assert prompt_examples[0].text == "Jack and Jill went up the hill."
-    assert prompt_examples[0].entities == {
-        "LOCATION": ["hill"],
-        "PERSON": ["Jack", "Jill"],
-    }
+    assert prompt_examples[0].spans == [
+        SpanReason(
+            text="Jack",
+            is_entity=True,
+            label="PERSON",
+            reason="is the name of a person",
+        ),
+        SpanReason(
+            text="Jill",
+            is_entity=True,
+            label="PERSON",
+            reason="is the name of a person",
+        ),
+        SpanReason(
+            text="went up", is_entity=False, label="==NONE==", reason="is a verb"
+        ),
+        SpanReason(
+            text="hill", is_entity=True, label="LOCATION", reason="is a location"
+        ),
+    ]
     assert (
         prompt_examples[1].text
         == "Jack and Jill went up the hill and spaCy is a great tool."
     )
-    assert prompt_examples[1].entities == {
-        "LOCATION": ["hill"],
-        "PERSON": ["Jack", "Jill"],
-    }
+    assert prompt_examples[1].spans == [
+        SpanReason(
+            text="Jack",
+            is_entity=True,
+            label="PERSON",
+            reason="is the name of a person",
+        ),
+        SpanReason(
+            text="Jill",
+            is_entity=True,
+            label="PERSON",
+            reason="is the name of a person",
+        ),
+        SpanReason(
+            text="went up", is_entity=False, label="==NONE==", reason="is a verb"
+        ),
+        SpanReason(
+            text="hill", is_entity=True, label="LOCATION", reason="is a location"
+        ),
+    ]
+
+
+@pytest.mark.parametrize(
+    "text, response, gold_ents",
+    [
+        (
+            "The woman Paris was walking around in Paris, talking to her friend Paris",
+            "1. Paris | True | PER | is the name of the woman\n"
+            "2. Paris | True | LOC | is a city in France\n"
+            "3. Paris | True | PER | is the name of the woman\n",
+            [("Paris", "PER"), ("Paris", "LOC"), ("Paris", "PER")],
+        ),
+        (
+            "Walking around Paris as a woman named Paris is quite a trip.",
+            "1. Paris | True | LOC | is a city in France\n"
+            "2. Paris | True | PER | is the name of the woman\n",
+            [("Paris", "LOC"), ("Paris", "PER")],
+        ),
+    ],
+    ids=["3_ents", "2_ents"],
+)
+def test_regression_span_task_response_parse(
+    text: str, response: str, gold_ents: List[Tuple[str, str]]
+):
+    """Test based on spaCy issue: https://github.com/explosion/spaCy/discussions/12812
+    where parsing wasn't working for NER when the same text could map to 2 labels.
+    In the user's case "Paris" could be a person's name or a location.
+    """
+
+    nlp = spacy.blank("en")
+    example_doc = nlp.make_doc(text)
+    ner_task = make_ner_task_v3(examples=[], labels=["PER", "LOC"])
+    span_reasons = _extract_span_reasons_cot(ner_task, response)
+    assert len(span_reasons) == len(gold_ents)
+
+    docs = list(ner_task.parse_responses([example_doc], [response]))
+    assert len(docs) == 1
+
+    doc = docs[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc.ents]
+    assert pred_ents == gold_ents
+
+
+@pytest.mark.parametrize(
+    "text, response, gold_ents",
+    [
+        (
+            "FooBar, Inc. is a large organization in the U.S.",
+            "1. FooBar, Inc. | True | ORG | is the name of an organization\n"
+            "2. U.S. | True | LOC | is a country\n",
+            [("FooBar, Inc.", "ORG"), ("U.S.", "LOC")],
+        ),
+    ],
+)
+def test_regression_span_task_comma(
+    text: str, response: str, gold_ents: List[Tuple[str, str]]
+):
+    """Test that spacy.NER.v3 can deal with comma's in entities"""
+
+    nlp = spacy.blank("en")
+    example_doc = nlp.make_doc(text)
+    ner_task = make_ner_task_v3(examples=[], labels=["ORG", "LOC"])
+    span_reasons = _extract_span_reasons_cot(ner_task, response)
+    assert len(span_reasons) == len(gold_ents)
+    docs = list(ner_task.parse_responses([example_doc], [response]))
+    assert len(docs) == 1
+    doc = docs[0]
+    pred_ents = [(ent.text, ent.label_) for ent in doc.ents]
+    assert pred_ents == gold_ents
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_add_label():
+    nlp = spacy.blank("en")
+    llm = nlp.add_pipe(
+        "llm",
+        config={
+            "task": {
+                "@llm_tasks": "spacy.NER.v3",
+            },
+            "model": {
+                "@llm_models": "spacy.GPT-3-5.v1",
+            },
+        },
+    )
+
+    nlp.initialize()
+    text = "Jack and Jill visited France."
+    doc = nlp(text)
+    assert len(doc.ents) == 0
+
+    for label in ["PERSON", "LOCATION"]:
+        llm.add_label(label)
+    doc = nlp(text)
+    assert len(doc.ents) == 3
diff --git a/spacy_llm/tests/tasks/test_rel.py b/spacy_llm/tests/tasks/test_rel.py
index 961d7790..eb685f2e 100644
--- a/spacy_llm/tests/tasks/test_rel.py
+++ b/spacy_llm/tests/tasks/test_rel.py
@@ -9,10 +9,11 @@
 from spacy.util import get_lang_class
 
 from spacy_llm.pipeline import LLMWrapper
-from spacy_llm.tasks.rel import _DEFAULT_REL_TEMPLATE, RelationItem, RELTask
-from spacy_llm.ty import Labeled, LLMTask
+from spacy_llm.tasks.rel import DEFAULT_REL_TEMPLATE, RelationItem, RELTask
+from spacy_llm.ty import LabeledTask, LLMTask
 from spacy_llm.util import assemble_from_config, split_labels
 
+from ...tasks import make_rel_task
 from ..compat import has_openai_key
 
 EXAMPLES_DIR = Path(__file__).parent / "examples"
@@ -39,7 +40,7 @@ def zeroshot_cfg_string():
     labels = "LivesIn,Visits"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
 
     [initialize]
     vectors = "en_core_web_md"
@@ -71,7 +72,7 @@ def fewshot_cfg_string():
     path = {str(EXAMPLES_DIR / "rel.jsonl")}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
 
     [initialize]
     vectors = "en_core_web_md"
@@ -126,7 +127,7 @@ def test_rel_config(cfg_string, request: FixtureRequest):
     task = pipe.task
     labels = orig_config["components"]["llm"]["task"]["labels"]
     labels = split_labels(labels)
-    assert isinstance(task, Labeled)
+    assert isinstance(task, LabeledTask)
     assert task.labels == tuple(labels)
     assert set(pipe.labels) == set(task.labels)
     assert nlp.pipe_labels["llm"] == list(task.labels)
@@ -152,7 +153,7 @@ def test_rel_predict(task, cfg_string, request):
 
 @pytest.mark.parametrize("n_prompt_examples", [-1, 0, 1, 2])
 def test_rel_init(noop_config, n_prompt_examples: int):
-    RELTask._check_rel_extension()
+    RELTask._check_extension("rel")
 
     config = Config().from_str(noop_config)
     del config["components"]["llm"]["task"]["labels"]
@@ -237,15 +238,31 @@ def test_incorrect_indexing():
     """Tests whether incorrect indexing is handled properly (i. e. when the LLM response indices non-existent
     entities).
     """
-    task = RELTask(labels=["LivesIn", "WorksIn"], template=_DEFAULT_REL_TEMPLATE)
+    task = make_rel_task(
+        labels=["LivesIn", "WorksIn"],
+        template=DEFAULT_REL_TEMPLATE,
+        verbose=False,
+    )
 
     doc = Doc(get_lang_class("en")().vocab, words=["This", "is", "a", "test"])
     doc.ents = [Span(doc, 0, 1, label="TEST")]
     assert (
-        len(task._format_response('{"dep": 0, "dest": 1, "relation": "LivesIn"}', doc))
-        == 0
+        len(
+            list(
+                task._parse_responses(
+                    task, [doc], ['{"dep": 0, "dest": 0, "relation": "LivesIn"}']
+                )
+            )[0]
+        )
+        == 1
     )
     assert (
-        len(task._format_response('{"dep": 0, "dest": 0, "relation": "LivesIn"}', doc))
-        == 1
+        len(
+            list(
+                task._parse_responses(
+                    task, [doc], ['{"dep": 0, "dest": 1, "relation": "LivesIn"}']
+                )
+            )[0]
+        )
+        == 0
     )
diff --git a/spacy_llm/tests/tasks/test_sentiment.py b/spacy_llm/tests/tasks/test_sentiment.py
index d5435ecd..4b2bd63b 100644
--- a/spacy_llm/tests/tasks/test_sentiment.py
+++ b/spacy_llm/tests/tasks/test_sentiment.py
@@ -31,8 +31,7 @@ def zeroshot_cfg_string():
     @llm_tasks = "spacy.Sentiment.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {"temperature": 0}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -57,8 +56,7 @@ def fewshot_cfg_string():
     path = {str((Path(__file__).parent / "examples" / "sentiment.yml"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -84,8 +82,7 @@ def ext_template_cfg_string():
     path = {str((Path(__file__).parent / "templates" / "sentiment.jinja2"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -171,7 +168,7 @@ def test_jinja_template_rendering_without_examples():
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     text = "They're indifferent."
     doc = nlp.make_doc(text)
 
@@ -205,7 +202,7 @@ def test_jinja_template_rendering_with_examples(examples_path):
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     text = "It was the happiest day of her life."
     doc = nlp.make_doc(text)
 
@@ -254,7 +251,7 @@ def test_external_template_actually_loads():
     template_path = str(TEMPLATES_DIR / "sentiment.jinja2")
     template = file_reader(template_path)
     text = "There is a silver lining."
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(text)
 
     sentiment_task = make_sentiment_task(template=template)
diff --git a/spacy_llm/tests/tasks/test_span_utils.py b/spacy_llm/tests/tasks/test_span_utils.py
new file mode 100644
index 00000000..6a68f833
--- /dev/null
+++ b/spacy_llm/tests/tasks/test_span_utils.py
@@ -0,0 +1,51 @@
+import pytest
+
+from spacy_llm.tasks.span import SpanReason
+
+SPAN_REASON_FROM_STR_TEST_CASES = {
+    "invalid_order_no_error": (
+        "1. Golden State Warriors | BASKETBALL_TEAM | is a basketball team in the NBA | True",
+        SpanReason(
+            text="Golden State Warriors",
+            is_entity=False,
+            label="is a basketball team in the NBA",
+            reason="True",
+        ),
+    ),
+    "invalid_number_of_components": (
+        "1. Golden State Warriors | BASKETBALL_TEAM | OTHER SECTION | MORE THINGS | is a basketball team in the NBA | True",
+        ValueError(),
+    ),
+    "valid_entity_numbered": (
+        "1. Golden State Warriors | True | BASKETBALL_TEAM | is a basketball team in the NBA",
+        SpanReason(
+            text="Golden State Warriors",
+            is_entity=True,
+            label="BASKETBALL_TEAM",
+            reason="is a basketball team in the NBA",
+        ),
+    ),
+    "valid_entity_unnumbered": (
+        "Golden State Warriors | True | BASKETBALL_TEAM | is a basketball team in the NBA",
+        SpanReason(
+            text="Golden State Warriors",
+            is_entity=True,
+            label="BASKETBALL_TEAM",
+            reason="is a basketball team in the NBA",
+        ),
+    ),
+}
+
+
+@pytest.mark.parametrize(
+    "response, expected",
+    SPAN_REASON_FROM_STR_TEST_CASES.values(),
+    ids=SPAN_REASON_FROM_STR_TEST_CASES.keys(),
+)
+def test_span_reason_from_str(response: str, expected: SpanReason):
+    try:
+        span_reason = SpanReason.from_str(response)
+    except ValueError:
+        assert isinstance(expected, ValueError)
+    else:
+        assert span_reason == expected
diff --git a/spacy_llm/tests/tasks/test_spancat.py b/spacy_llm/tests/tasks/test_spancat.py
index 0c6f6487..97e5bd88 100644
--- a/spacy_llm/tests/tasks/test_spancat.py
+++ b/spacy_llm/tests/tasks/test_spancat.py
@@ -1,19 +1,20 @@
 from pathlib import Path
+from typing import Callable, List, Tuple, cast
 
 import pytest
 import spacy
-import srsly
 from confection import Config
+from spacy.language import Language
 from spacy.tokens import Span
 from spacy.training import Example
 from spacy.util import make_tempdir
 
 from spacy_llm.pipeline import LLMWrapper
 from spacy_llm.registry import fewshot_reader, lowercase_normalizer, strip_normalizer
-from spacy_llm.tasks import make_spancat_task_v2
+from spacy_llm.tasks import make_spancat_task_v3
 from spacy_llm.tasks.spancat import SpanCatTask
 from spacy_llm.tasks.util import find_substrings
-from spacy_llm.ty import Labeled, LLMTask
+from spacy_llm.ty import LabeledTask, LLMTask
 from spacy_llm.util import assemble_from_config, split_labels
 
 from ..compat import has_openai_key
@@ -22,8 +23,13 @@
 
 
 @pytest.fixture
-def zeroshot_cfg_string():
-    return """
+def examples_dir():
+    return EXAMPLES_DIR
+
+
+@pytest.fixture
+def noop_config():
+    return f"""
     [nlp]
     lang = "en"
     pipeline = ["llm"]
@@ -35,14 +41,20 @@ def zeroshot_cfg_string():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.SpanCat.v2"
-    labels = PER,ORG,LOC
+    @llm_tasks = "spacy.SpanCat.v3"
+    labels = ["PER", "ORG", "LOC", "DESTINATION"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "spancat.yml"))}
 
     [components.llm.task.normalizer]
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "test.NoOpModel.v1"
+    output = 1. Bob | True | PER | is the name of a person
+        2. Alice | True | PER | is the name of a person
     """
 
 
@@ -60,65 +72,109 @@ def fewshot_cfg_string():
     factory = "llm"
 
     [components.llm.task]
-    @llm_tasks = "spacy.SpanCat.v2"
-    labels = ["PER", "ORG", "LOC"]
+    @llm_tasks = "spacy.SpanCat.v3"
+    labels = ["PER", "ORG", "LOC", "DESTINATION"]
+
+    [components.llm.task.examples]
+    @misc = "spacy.FewShotReader.v1"
+    path = {str((Path(__file__).parent / "examples" / "spancat.yml"))}
+
+    [components.llm.task.normalizer]
+    @misc = "spacy.LowercaseNormalizer.v1"
+
+    [components.llm.model]
+    @llm_models = "spacy.GPT-3-5.v2"
+    """
+
+
+@pytest.fixture
+def ext_template_cfg_string():
+    """Simple zero-shot config with an external template"""
+
+    return f"""
+    [nlp]
+    lang = "en"
+    pipeline = ["llm"]
+    batch_size = 128
+
+    [components]
+    [components.llm]
+    factory = "llm"
+
+    [components.llm.task]
+    @llm_tasks = "spacy.SpanCat.v3"
+    description = "This is a description"
+    labels = ["PER", "ORG", "LOC", "DESTINATION"]
 
     [components.llm.task.examples]
     @misc = "spacy.FewShotReader.v1"
-    path = {str((Path(__file__).parent / "examples" / "ner.yml"))}
+    path = {str((Path(__file__).parent / "examples" / "spancat.json"))}
+
+    [components.llm.task.template]
+    @misc = "spacy.FileReader.v1"
+    path = {str((Path(__file__).parent / "templates" / "spancat.jinja2"))}
 
     [components.llm.task.normalizer]
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
+@pytest.fixture(
+    params=[
+        "fewshot_cfg_string",
+        "ext_template_cfg_string",
+    ]
+)
+def config(request) -> Config:
+    cfg_str = request.getfixturevalue(request.param)
+    config = Config().from_str(cfg_str)
+    return config
+
+
+@pytest.fixture
+def nlp(config: Config) -> Language:
+    nlp = assemble_from_config(config)
+    return nlp
+
+
 @pytest.mark.external
 @pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
-@pytest.mark.parametrize("cfg_string", ["fewshot_cfg_string", "zeroshot_cfg_string"])
-def test_spancat_config(cfg_string, request):
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+def test_spancat_config(config: Config):
+    nlp = assemble_from_config(config)
     assert nlp.pipe_names == ["llm"]
 
     pipe = nlp.get_pipe("llm")
     assert isinstance(pipe, LLMWrapper)
     assert isinstance(pipe.task, LLMTask)
 
-    labels = orig_config["components"]["llm"]["task"]["labels"]
+    labels = config["components"]["llm"]["task"]["labels"]
     labels = split_labels(labels)
     task = pipe.task
-    assert isinstance(task, Labeled)
+    assert isinstance(task, LabeledTask)
     assert sorted(task.labels) == sorted(tuple(labels))
     assert pipe.labels == task.labels
     assert nlp.pipe_labels["llm"] == list(task.labels)
 
 
 @pytest.mark.external
-@pytest.mark.parametrize("cfg_string", ["zeroshot_cfg_string", "fewshot_cfg_string"])
-def test_spancat_predict(cfg_string, request):
-    """Use OpenAI to get zero-shot NER results.
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_spancat_predict(nlp: Language):
+    """Use OpenAI to get zero-shot spancat results.
     Note that this test may fail randomly, as the LLM's output is unguaranteed to be consistent/predictable
     """
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
     text = "Marc and Bob both live in Ireland."
     doc = nlp(text)
     assert len(doc.spans["sc"]) > 0
     for ent in doc.spans["sc"]:
-        assert ent.label_ in ["PER", "ORG", "LOC"]
+        assert ent.label_ in ["PER", "ORG", "LOC", "DESTINATION"]
 
 
 @pytest.mark.external
-@pytest.mark.parametrize("cfg_string", ["zeroshot_cfg_string", "fewshot_cfg_string"])
-def test_spancat_io(cfg_string, request):
-    cfg_string = request.getfixturevalue(cfg_string)
-    orig_config = Config().from_str(cfg_string)
-    nlp = spacy.util.load_model_from_config(orig_config, auto_fill=True)
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_spancat_io(nlp: Language):
     assert nlp.pipe_names == ["llm"]
     # ensure you can save a pipeline to disk and run it after loading
     with make_tempdir() as tmpdir:
@@ -129,7 +185,7 @@ def test_spancat_io(cfg_string, request):
     doc = nlp2(text)
     assert len(doc.spans["sc"]) > 0
     for ent in doc.spans["sc"]:
-        assert ent.label_ in ["PER", "ORG", "LOC"]
+        assert ent.label_ in ["PER", "ORG", "LOC", "DESTINATION"]
 
 
 @pytest.mark.parametrize(
@@ -173,32 +229,31 @@ def test_ensure_offsets_correspond_to_substrings(
         # simple
         (
             "Jean Jacques and Jaime went to the library.",
-            "PER: Jean Jacques, Jaime\nLOC: library",
+            "1. Jean Jacques | True | PER | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n"
+            "3. library | True | LOC | is a place you can go to with lots of books\n",
             [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "LOC")],
         ),
         # overlapping: should only return all spans
         (
             "The Manila Observatory was founded in 1865.",
-            "LOC: The Manila Observatory, Manila, Manila Observatory",
+            "1. The Manila Observatory | True | LOC | is a place\n"
+            "2. Manila | True | LOC | is a city\n"
+            "3. Manila Observatory | True | LOC | is a place\n",
             [
                 ("The Manila Observatory", "LOC"),
                 ("Manila", "LOC"),
                 ("Manila Observatory", "LOC"),
             ],
         ),
-        # flipped: order shouldn't matter
-        (
-            "Take the road from Downtown and turn left at the public market.",
-            "LOC: public market, Downtown",
-            [("Downtown", "LOC"), ("public market", "LOC")],
-        ),
     ],
+    ids=["simple", "overlapping"],
 )
-def test_spancat_zero_shot_task(text, response, gold_spans):
+def test_spancat_matching_shot_task(text: str, response: str, gold_spans):
     labels = "PER,ORG,LOC"
-    llm_spancat = make_spancat_task_v2(labels=labels)
+    llm_spancat = make_spancat_task_v3(examples=[], labels=labels)
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list so we get what's inside
@@ -211,53 +266,67 @@ def test_spancat_zero_shot_task(text, response, gold_spans):
     "response,normalizer,gold_spans",
     [
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n",
             None,
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n",
             strip_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "PER: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | per | is the name of a person\n"
+            "2. Jaime | True | per | is the name of a person\n",
             strip_normalizer(),
             [],
         ),
         (
-            "per: Jean Jacques, Jaime",
+            "1. Jean Jacques | True | PER | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n",
             None,
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques\nPER: Jaime",
+            "1. Jean Jacques | True | per | is the name of a person\n"
+            "2. Jaime | True | PER | is the name of a person\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
         (
-            "per: Jean Jacques, Jaime\nOrg: library",
+            "1. Jean Jacques | True | per | is the name of a person\n"
+            "2. Jaime | True | per | is the name of a person\n"
+            "3. library | True | Org | is an organization\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER"), ("library", "ORG")],
         ),
         (
-            "per: Jean Jacques, Jaime\nRANDOM: library",
+            "1. Jean Jacques | True | per | is the name of a person\n"
+            "2. Jaime | True | per | is the name of a person\n"
+            "3. library | True | RANDOM | is an organization\n",
             lowercase_normalizer(),
             [("Jean Jacques", "PER"), ("Jaime", "PER")],
         ),
     ],
 )
-def test_spancat_labels(response, normalizer, gold_spans):
+def test_spancat_labels(
+    response: str, normalizer: Callable[[str], str], gold_spans: List[Tuple[str, str]]
+):
     text = "Jean Jacques and Jaime went to the library."
     labels = "PER,ORG,LOC"
-    llm_spancat = make_spancat_task_v2(labels=labels, normalizer=normalizer)
+    llm_spancat = make_spancat_task_v3(
+        examples=[], labels=labels, normalizer=normalizer
+    )
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -270,43 +339,46 @@ def test_spancat_labels(response, normalizer, gold_spans):
     "response,alignment_mode,gold_spans",
     [
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "strict",
             [],
         ),
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "contract",
             [],
         ),
         (
-            "PER: Jacq",
+            "1. Jacq | True | PER | is a person's name",
             "expand",
             [("Jacques", "PER")],
         ),
         (
-            "PER: Jean J",
+            "1. Jean J | True | PER | is a person's name",
             "contract",
             [("Jean", "PER")],
         ),
         (
-            "PER: Jean Jacques, aim",
+            "1. Jean Jacques | True | PER | is a person's name",
             "strict",
             [("Jean Jacques", "PER")],
         ),
         (
-            "PER: random",
+            "1. random | True | PER | is a person's name",
             "expand",
             [],
         ),
     ],
+    ids=["strict_1", "contract_1", "expand_1", "strict_2", "contract_2", "expand_2"],
 )
 def test_spancat_alignment(response, alignment_mode, gold_spans):
     text = "Jean Jacques and Jaime went to the library."
     labels = "PER,ORG,LOC"
-    llm_spancat = make_spancat_task_v2(labels=labels, alignment_mode=alignment_mode)
+    llm_spancat = make_spancat_task_v3(
+        examples=[], labels=labels, alignment_mode=alignment_mode
+    )
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -318,46 +390,44 @@ def test_spancat_alignment(response, alignment_mode, gold_spans):
 def test_invalid_alignment_mode():
     labels = "PER,ORG,LOC"
     with pytest.raises(ValueError, match="Unsupported alignment mode 'invalid"):
-        make_spancat_task_v2(labels=labels, alignment_mode="invalid")
+        make_spancat_task_v3(examples=[], labels=labels, alignment_mode="invalid")
 
 
 @pytest.mark.parametrize(
-    "response,case_sensitive,single_match,gold_spans",
+    "response, case_sensitive, gold_spans",
     [
         (
-            "PER: Jean",
-            False,
+            "1. Jean | True | PER | is a person's name",
             False,
-            [("jean", "PER"), ("Jean", "PER"), ("Jean", "PER")],
-        ),
-        (
-            "PER: Jean",
-            False,
-            True,
             [("jean", "PER")],
         ),
         (
-            "PER: Jean",
+            "1. Jean | True | PER | is a person's name",
             True,
-            False,
-            [("Jean", "PER"), ("Jean", "PER")],
+            [("Jean", "PER")],
         ),
         (
-            "PER: Jean",
-            True,
-            True,
-            [("Jean", "PER")],
+            "1. jean | True | PER | is a person's name\n"
+            "2. Jean | True | PER | is a person's name\n"
+            "3. Jean Foundation | True | ORG | is the name of an Organization name",
+            False,
+            [("jean", "PER"), ("Jean", "PER"), ("Jean Foundation", "ORG")],
         ),
     ],
+    ids=[
+        "single_ent_case_insensitive",
+        "single_ent_case_sensitive",
+        "multiple_ents_case_insensitive",
+    ],
 )
-def test_spancat_matching(response, case_sensitive, single_match, gold_spans):
+def test_spancat_matching(response, case_sensitive, gold_spans):
     text = "This guy jean (or Jean) is the president of the Jean Foundation."
     labels = "PER,ORG,LOC"
-    llm_spancat = make_spancat_task_v2(
-        labels=labels, case_sensitive_matching=case_sensitive, single_match=single_match
+    llm_spancat = make_spancat_task_v3(
+        examples=[], labels=labels, case_sensitive_matching=case_sensitive
     )
     # Prepare doc
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc_in = nlp.make_doc(text)
     # Pass to the parser
     # Note: parser() returns a list
@@ -367,36 +437,46 @@ def test_spancat_matching(response, case_sensitive, single_match, gold_spans):
 
 
 def test_jinja_template_rendering_without_examples():
-    """Test if jinja template renders as we expected
+    """Test if jinja2 template renders as expected
 
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
     labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
-
-    llm_spancat = make_spancat_task_v2(labels=labels, examples=None)
+    llm_spancat = make_spancat_task_v3(labels=labels)
     prompt = list(llm_spancat.generate_prompts([doc]))[0]
 
     assert (
         prompt.strip()
         == """
-You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
-The entities you extract for each label can overlap with each other.
-From the Text input provided, extract named entities for each label in the following format:
+You are an expert Entity Recognition system.
+Your task is to accept Text as input and extract named entities.
+The entities you extract can overlap with each other.
 
-LOC: <comma delimited list of strings>
-ORG: <comma delimited list of strings>
-PER: <comma delimited list of strings>
+Entities must have one of the following labels: LOC, ORG, PER.
+If a span is not an entity label it: `==NONE==`.
 
 
-Here is the text that needs labeling:
 
-Text:
-'''
-Alice and Bob went to the supermarket
-'''
+Here is an example of the output format for a paragraph using different labels than this task requires.
+Only use this output format but use the labels provided
+above instead of the ones defined in the example below.
+Do not output anything besides entities in this output format.
+Output entities in the order they occur in the input paragraph regardless of label.
+
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
+
+Paragraph: Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok.
+Answer:
+1. Sriracha sauce | True | INGREDIENT | is an ingredient to add to a stir fry
+2. really well | False | ==NONE== | is a description of how well sriracha sauce goes with hoisin stir fry
+3. hoisin stir fry | True | DISH | is a dish with stir fry vegetables and hoisin sauce
+4. wok | True | EQUIPMENT | is a piece of cooking equipment used to stir fry ingredients
+
+Paragraph: Alice and Bob went to the supermarket
+Answer:
 """.strip()
     )
 
@@ -404,140 +484,81 @@ def test_jinja_template_rendering_without_examples():
 @pytest.mark.parametrize(
     "examples_path",
     [
-        str(EXAMPLES_DIR / "ner.json"),
-        str(EXAMPLES_DIR / "ner.yml"),
-        str(EXAMPLES_DIR / "ner.jsonl"),
+        str(EXAMPLES_DIR / "spancat.json"),
+        str(EXAMPLES_DIR / "spancat.yml"),
+        str(EXAMPLES_DIR / "spancat.jsonl"),
     ],
 )
-def test_jinja_template_rendering_with_examples(examples_path):
+def test_jinja_template_rendering_with_examples(examples_path: Path):
     """Test if jinja2 template renders as expected
 
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    labels = "PER,ORG,LOC"
-    nlp = spacy.blank("xx")
+    labels = "PER,ORG,LOC,DESTINATION"
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Alice and Bob went to the supermarket")
 
     examples = fewshot_reader(examples_path)
-    llm_spancat = make_spancat_task_v2(labels=labels, examples=examples)
+    llm_spancat = make_spancat_task_v3(labels=labels, examples=examples)
     prompt = list(llm_spancat.generate_prompts([doc]))[0]
 
     assert (
         prompt.strip()
         == """
-You are an expert Named Entity Recognition (NER) system. Your task is to accept Text as input and extract named entities for the set of predefined entity labels.
-The entities you extract for each label can overlap with each other.
-From the Text input provided, extract named entities for each label in the following format:
-
-LOC: <comma delimited list of strings>
-ORG: <comma delimited list of strings>
-PER: <comma delimited list of strings>
-
-
-Below are some examples (only use these as a guide):
-
-Text:
-'''
-Jack and Jill went up the hill.
-'''
-
-PER: Jack, Jill
-LOC: hill
-
-Text:
-'''
-Jack fell down and broke his crown.
-'''
-
-PER: Jack
+You are an expert Entity Recognition system.
+Your task is to accept Text as input and extract named entities.
+The entities you extract can overlap with each other.
 
-Text:
-'''
-Jill came tumbling after.
-'''
+Entities must have one of the following labels: DESTINATION, LOC, ORG, PER.
+If a span is not an entity label it: `==NONE==`.
 
-PER: Jill
 
+Q: Given the paragraph below, identify a list of entities, and for each entry explain why it is or is not an entity:
 
-Here is the text that needs labeling:
+Paragraph: Jack and Jill went up the hill.
+Answer:
+1. Jack | True | PER | is the name of a person
+2. Jill | True | PER | is the name of a person
+3. went up | False | ==NONE== | is a verb
+4. hill | True | LOC | is a location
+5. hill | True | DESTINATION | is a destination
 
-Text:
-'''
-Alice and Bob went to the supermarket
-'''""".strip()
+Paragraph: Alice and Bob went to the supermarket
+Answer:
+""".strip()
     )
 
 
-def test_example_not_following_basemodel():
-    wrong_example = [
-        {
-            "text": "I'm a wrong example. Entities should be a dict, not a list",
-            # Should be: {"PER": ["Entities"], "ORG": ["dict", "list"]}
-            "entities": [("PER", ("Entities")), ("ORG", ("dict", "list"))],
-        }
-    ]
-    with make_tempdir() as tmpdir:
-        tmp_path = tmpdir / "wrong_example.yml"
-        srsly.write_yaml(tmp_path, wrong_example)
-
-    with pytest.raises(ValueError):
-        make_spancat_task_v2(labels="PER,ORG,LOC", examples=fewshot_reader(tmp_path))
-
-
-@pytest.fixture
-def noop_config():
-    return """
-    [nlp]
-    lang = "en"
-    pipeline = ["llm"]
-    batch_size = 128
-
-    [components]
-
-    [components.llm]
-    factory = "llm"
-
-    [components.llm.task]
-    @llm_tasks = "spacy.SpanCat.v2"
-    labels = ["PER", "ORG", "LOC"]
-
-    [components.llm.task.normalizer]
-    @misc = "spacy.LowercaseNormalizer.v1"
-
-    [components.llm.model]
-    @llm_models = "test.NoOpModel.v1"
-    output = "PER: Bob,Alice"
-    """
-
-
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
 @pytest.mark.parametrize("n_detections", [0, 1, 2])
-def test_spancat_scoring(noop_config, n_detections):
-    config = Config().from_str(noop_config)
+def test_spancat_scoring(fewshot_cfg_string: str, n_detections: int):
+    config = Config().from_str(fewshot_cfg_string)
     nlp = assemble_from_config(config)
 
     examples = []
-
     for text in ["Alice works with Bob.", "Bob lives with Alice."]:
         predicted = nlp.make_doc(text)
-        reference = predicted.copy()
-
-        reference.spans["sc"] = [
-            Span(reference, 0, 1, label="PER"),
-            Span(reference, 3, 4, label="PER"),
-        ][:n_detections]
-
+        reference = nlp.make_doc(text)
+        ent1 = Span(reference, 0, 1, label="PER")
+        ent2 = Span(reference, 3, 4, label="PER")
+        reference.spans["sc"] = [ent1, ent2][:n_detections]
         examples.append(Example(predicted, reference))
 
     scores = nlp.evaluate(examples)
-
     assert scores["spans_sc_p"] == n_detections / 2
+    assert scores["spans_sc_r"] == (1 if n_detections != 0 else 0)
+    assert scores["spans_sc_f"] == (
+        pytest.approx(0.666666666) if n_detections == 1 else n_detections / 2
+    )
 
 
 @pytest.mark.parametrize("n_prompt_examples", [-1, 0, 1, 2])
-def test_spancat_init(noop_config, n_prompt_examples: bool):
+def test_spancat_init(noop_config: str, n_prompt_examples: bool):
     config = Config().from_str(noop_config)
-    del config["components"]["llm"]["task"]["labels"]
+    config["components"]["llm"]["task"]["labels"] = ["PER", "LOC", "DESTINATION"]
+    config["components"]["llm"]["task"]["examples"] = []
     nlp = assemble_from_config(config)
 
     examples = []
@@ -560,40 +581,40 @@ def test_spancat_init(noop_config, n_prompt_examples: bool):
     _, llm = nlp.pipeline[0]
     task: SpanCatTask = llm._task
 
-    assert set(task._label_dict.values()) == set()
-    assert not task._prompt_examples
-
     nlp.config["initialize"]["components"]["llm"] = {
         "n_prompt_examples": n_prompt_examples
     }
 
     nlp.initialize(lambda: examples)
 
-    assert set(task._label_dict.values()) == {"PER", "LOC"}
-    if n_prompt_examples >= 0:
-        assert len(task._prompt_examples) == n_prompt_examples
-    else:
+    assert set(task._label_dict.values()) == {"PER", "LOC", "DESTINATION"}
+    if n_prompt_examples == -1:
         assert len(task._prompt_examples) == len(examples)
+    else:
+        assert len(task._prompt_examples) == n_prompt_examples
 
     if n_prompt_examples > 0:
         for eg in task._prompt_examples:
-            assert set(eg.entities.keys()) == {"PER", "LOC"}
+            prompt_example_labels = {ent.label for ent in eg.spans}
+            if "==NONE==" not in prompt_example_labels:
+                prompt_example_labels.add("==NONE==")
+            assert prompt_example_labels == {"==NONE==", "PER", "LOC"}
 
 
 def test_spancat_serde(noop_config):
     config = Config().from_str(noop_config)
-    del config["components"]["llm"]["task"]["labels"]
 
     nlp1 = assemble_from_config(config)
     nlp2 = assemble_from_config(config)
 
     labels = {"loc": "LOC", "per": "PER"}
 
-    task1: SpanCatTask = nlp1.get_pipe("llm")._task
-    task2: SpanCatTask = nlp2.get_pipe("llm")._task
+    task1 = cast(SpanCatTask, nlp1.get_pipe("llm").task)
+    task2 = cast(SpanCatTask, nlp2.get_pipe("llm").task)
 
     # Artificially add labels to task1
     task1._label_dict = labels
+    task2._label_dict = dict()
 
     assert task1._label_dict == labels
     assert task2._label_dict == dict()
diff --git a/spacy_llm/tests/tasks/test_summarization.py b/spacy_llm/tests/tasks/test_summarization.py
index dc897cab..d8912b38 100644
--- a/spacy_llm/tests/tasks/test_summarization.py
+++ b/spacy_llm/tests/tasks/test_summarization.py
@@ -36,8 +36,7 @@ def zeroshot_cfg_string():
     max_n_words = 20
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {"temperature": 0}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -63,8 +62,7 @@ def fewshot_cfg_string():
     path = {str((Path(__file__).parent / "examples" / "summarization.yml"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -91,8 +89,7 @@ def ext_template_cfg_string():
     path = {str((Path(__file__).parent / "templates" / "summarization.jinja2"))}
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
-    config = {{"temperature": 0}}
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -199,7 +196,7 @@ def test_summarization_predict(cfg_string, example_text, request):
         )
 
 
-# @pytest.mark.external
+@pytest.mark.external
 @pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
 @pytest.mark.parametrize(
     "cfg_string_and_field",
@@ -252,7 +249,7 @@ def test_jinja_template_rendering_without_examples(example_text):
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(example_text)
 
     llm_ner = make_summarization_task(examples=None, max_n_words=10)
@@ -285,11 +282,11 @@ def test_jinja_template_rendering_with_examples(examples_path, example_text):
     We apply the .strip() method for each prompt so that we don't have to deal
     with annoying newlines and spaces at the edge of the text.
     """
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(example_text)
 
-    examples = fewshot_reader(examples_path)
-    llm_ner = make_summarization_task(examples=examples, max_n_words=20)
+    prompt_examples = fewshot_reader(examples_path)
+    llm_ner = make_summarization_task(examples=prompt_examples, max_n_words=20)
 
     with pytest.warns(
         UserWarning,
@@ -342,7 +339,7 @@ def test_jinja_template_rendering_with_examples(examples_path, example_text):
 def test_external_template_actually_loads(example_text):
     template_path = str(TEMPLATES_DIR / "summarization.jinja2")
     template = file_reader(template_path)
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(example_text)
 
     llm_ner = make_summarization_task(template=template)
diff --git a/spacy_llm/tests/tasks/test_textcat.py b/spacy_llm/tests/tasks/test_textcat.py
index 1d93fa70..b6b0d641 100644
--- a/spacy_llm/tests/tasks/test_textcat.py
+++ b/spacy_llm/tests/tasks/test_textcat.py
@@ -13,7 +13,7 @@
 from spacy_llm.registry import fewshot_reader, file_reader, lowercase_normalizer
 from spacy_llm.registry import registry
 from spacy_llm.tasks.textcat import TextCatTask, make_textcat_task_v3
-from spacy_llm.ty import Labeled, LLMTask
+from spacy_llm.ty import LabeledTask, LLMTask
 from spacy_llm.util import assemble_from_config, split_labels
 
 from ..compat import has_openai_key
@@ -44,7 +44,7 @@ def zeroshot_cfg_string():
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -74,7 +74,7 @@ def fewshot_cfg_string():
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -106,7 +106,7 @@ def ext_template_cfg_string():
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -135,7 +135,7 @@ def zeroshot_cfg_string_v3_lds():
     @misc = "spacy.LowercaseNormalizer.v1"
 
     [components.llm.model]
-    @llm_models = "spacy.GPT-3-5.v1"
+    @llm_models = "spacy.GPT-3-5.v2"
     """
 
 
@@ -208,7 +208,7 @@ def test_textcat_config(task, cfg_string, request):
 
     labels = split_labels(labels)
     task = pipe.task
-    assert isinstance(task, Labeled)
+    assert isinstance(task, LabeledTask)
     assert sorted(task.labels) == sorted(tuple(labels))
     assert pipe.labels == task.labels
     assert nlp.pipe_labels["llm"] == list(task.labels)
@@ -316,7 +316,7 @@ def test_textcat_binary_labels_are_correct(text, response, expected_score):
         labels=label, exclusive_classes=True, normalizer=lowercase_normalizer()
     )
 
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp(text)
     pred = list(llm_textcat.parse_responses([doc], [response]))[0]
     assert list(pred.cats.keys())[0] == label
@@ -348,7 +348,7 @@ def test_textcat_multilabel_labels_are_correct(
         exclusive_classes=exclusive_classes,
         normalizer=lowercase_normalizer(),
     )
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc(text)
     pred = list(llm_textcat.parse_responses([doc], [response]))[0]
     # Take only those that have scores
@@ -371,13 +371,13 @@ def test_jinja_template_rendering_with_examples_for_binary(examples_path, binary
     with annoying newlines and spaces at the edge of the text.
     """
     text, labels, _, exclusive_classes, _ = binary
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp(text)
 
-    examples = fewshot_reader(examples_path)
+    prompt_examples = fewshot_reader(examples_path)
     llm_textcat = make_textcat_task_v3(
         labels=labels,
-        examples=examples,
+        examples=prompt_examples,
         exclusive_classes=exclusive_classes,
     )
     prompt = list(llm_textcat.generate_prompts([doc]))[0]
@@ -437,13 +437,13 @@ def test_jinja_template_rendering_with_examples_for_multilabel_exclusive(
     examples_path, multilabel_excl
 ):
     text, labels, _, exclusive_classes, _ = multilabel_excl
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp(text)
 
-    examples = fewshot_reader(examples_path)
+    prompt_examples = fewshot_reader(examples_path)
     llm_textcat = make_textcat_task_v3(
         labels=labels,
-        examples=examples,
+        examples=prompt_examples,
         exclusive_classes=exclusive_classes,
     )
     prompt = list(llm_textcat.generate_prompts([doc]))[0]
@@ -504,13 +504,13 @@ def test_jinja_template_rendering_with_examples_for_multilabel_nonexclusive(
     examples_path, multilabel_nonexcl
 ):
     text, labels, _, exclusive_classes, _ = multilabel_nonexcl
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp(text)
 
-    examples = fewshot_reader(examples_path)
+    prompt_examples = fewshot_reader(examples_path)
     llm_textcat = make_textcat_task_v3(
         labels=labels,
-        examples=examples,
+        examples=prompt_examples,
         exclusive_classes=exclusive_classes,
     )
     prompt = list(llm_textcat.generate_prompts([doc]))[0]
@@ -588,7 +588,7 @@ def test_external_template_actually_loads():
     template_path = str(TEMPLATES_DIR / "textcat.jinja2")
     template = file_reader(template_path)
     labels = "Recipe"
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp.make_doc("Combine 2 cloves of garlic with soy sauce")
 
     llm_textcat = make_textcat_task_v3(labels=labels, template=template)
@@ -668,7 +668,7 @@ def b(prompts: Iterable[str]) -> Iterable[str]:
 
 def test_jinja_template_rendering_with_label_definitions(multilabel_excl):
     text, labels, _, exclusive_classes, _ = multilabel_excl
-    nlp = spacy.blank("xx")
+    nlp = spacy.blank("en")
     doc = nlp(text)
 
     llm_textcat = make_textcat_task_v3(
@@ -820,3 +820,31 @@ def test_textcat_serde(noop_config, tmp_path: Path):
     nlp3.from_bytes(nlp1.to_bytes())
 
     assert task1._label_dict == task2._label_dict == task3._label_dict == labels
+
+
+@pytest.mark.external
+@pytest.mark.skipif(has_openai_key is False, reason="OpenAI API key not available")
+def test_add_label():
+    nlp = spacy.blank("en")
+    llm = nlp.add_pipe(
+        "llm",
+        config={
+            "task": {
+                "@llm_tasks": "spacy.TextCat.v3",
+            },
+            "model": {
+                "@llm_models": "spacy.GPT-3-5.v1",
+            },
+        },
+    )
+
+    nlp.initialize()
+    text = "I am feeling great."
+    doc = nlp(text)
+    assert len(doc.cats) == 0
+
+    for label in ["HAPPY", "SAD"]:
+        llm.add_label(label)
+    doc = nlp(text)
+    assert len(doc.cats) == 2
+    assert set(doc.cats.keys()) == {"HAPPY", "SAD"}
diff --git a/spacy_llm/tests/test_combinations.py b/spacy_llm/tests/test_combinations.py
index 796cc87f..0cd986c5 100644
--- a/spacy_llm/tests/test_combinations.py
+++ b/spacy_llm/tests/test_combinations.py
@@ -12,13 +12,13 @@
 @pytest.mark.skipif(has_langchain is False, reason="LangChain is not installed")
 @pytest.mark.parametrize(
     "model",
-    ["langchain.OpenAI.v1", "spacy.GPT-3-5.v1"],
-    ids=["langchain", "rest-openai"],
+    ["langchain.OpenAI.v1", "spacy.GPT-3-5.v1", "spacy.GPT-3-5.v2"],
+    ids=["langchain", "rest-openai", "rest-openai"],
 )
 @pytest.mark.parametrize(
     "task",
-    ["spacy.NER.v1", "spacy.NER.v2", "spacy.TextCat.v1"],
-    ids=["ner.v1", "ner.v2", "textcat"],
+    ["spacy.NER.v1", "spacy.NER.v3", "spacy.TextCat.v1"],
+    ids=["ner.v1", "ner.v3", "textcat"],
 )
 @pytest.mark.parametrize("n_process", [1])  # , 2
 def test_combinations(model: str, task: str, n_process: int):
diff --git a/usage_examples/tests/test_readme_examples.py b/usage_examples/tests/test_readme_examples.py
index a2a368a3..ea386750 100644
--- a/usage_examples/tests/test_readme_examples.py
+++ b/usage_examples/tests/test_readme_examples.py
@@ -1,3 +1,4 @@
+from pathlib import Path
 from typing import Callable, Iterable
 
 import pytest
@@ -27,8 +28,7 @@ def test_example_1_classifier():
         labels = ["COMPLIMENT", "INSULT"]
 
         [components.llm.model]
-        @llm_models = "spacy.GPT-3-5.v1"
-        config = {"temperature": 0.3}
+        @llm_models = "spacy.GPT-3-5.v2"
         """
 
         with open(tmpdir / "cfg", "w") as text_file:
@@ -41,7 +41,7 @@ def test_example_1_classifier():
 
 @pytest.mark.gpu
 @pytest.mark.skipif(not has_torch_cuda_gpu, reason="needs GPU & CUDA")
-def test_example_2_ner_hf():
+def test_example_2_classifier_hf():
     with util.make_tempdir() as tmpdir:
         cfg_str = """
         [nlp]
@@ -54,8 +54,8 @@ def test_example_2_ner_hf():
         factory = "llm"
 
         [components.llm.task]
-        @llm_tasks = "spacy.NER.v2"
-        labels = ["PERSON", "ORGANISATION", "LOCATION"]
+        @llm_tasks = "spacy.TextCat.v2"
+        labels = ["COMPLIMENT", "INSULT"]
 
         [components.llm.model]
         @llm_models = "spacy.Dolly.v1"
@@ -67,31 +67,101 @@ def test_example_2_ner_hf():
             text_file.write(cfg_str)
 
         nlp = assemble(tmpdir / "cfg")
-        doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
+        doc = nlp("You look gorgeous!")
+        print(doc.cats)  # noqa: T201
+
+
+@pytest.mark.external
+def test_example_3_ner():
+    examples_path = Path(__file__).parent.parent / "ner_v3_openai" / "examples.json"
+
+    with util.make_tempdir() as tmpdir:
+
+        cfg_str = f"""
+        [nlp]
+        lang = "en"
+        pipeline = ["llm"]
+
+        [components]
+
+        [components.llm]
+        factory = "llm"
+
+        [components.llm.task]
+        @llm_tasks = "spacy.NER.v3"
+        labels = ["DISH", "INGREDIENT", "EQUIPMENT"]
+        description = Entities are the names food dishes,
+            ingredients, and any kind of cooking equipment.
+            Adjectives, verbs, adverbs are not entities.
+            Pronouns are not entities.
+
+        [components.llm.task.label_definitions]
+        DISH = "Known food dishes, e.g. Lobster Ravioli, garlic bread"
+        INGREDIENT = "Individual parts of a food dish, including herbs and spices."
+        EQUIPMENT = "Any kind of cooking equipment. e.g. oven, cooking pot, grill"
+
+        [components.llm.task.examples]
+        @misc = "spacy.FewShotReader.v1"
+        path = {str(examples_path)}
+
+        [components.llm.model]
+        @llm_models = "spacy.GPT-3-5.v1"
+        """
+
+        with open(tmpdir / "cfg", "w") as text_file:
+            text_file.write(cfg_str)
+
+        nlp = assemble(tmpdir / "cfg")
+        doc = nlp(
+            "Sriracha sauce goes really well with hoisin stir fry, "
+            "but you should add it after you use the wok."
+        )
         print([(ent.text, ent.label_) for ent in doc.ents])  # noqa: T201
 
 
 @pytest.mark.external
-def test_example_3_python():
+def test_example_4_python():
     nlp = spacy.blank("en")
     nlp.add_pipe(
         "llm",
         config={
             "task": {
-                "@llm_tasks": "spacy.NER.v2",
-                "labels": ["PERSON", "ORGANISATION", "LOCATION"],
+                "@llm_tasks": "spacy.NER.v3",
+                "labels": ["DISH", "INGREDIENT", "EQUIPMENT"],
+                "examples": [
+                    {
+                        "text": "You can't get a great chocolate flavor with carob.",
+                        "spans": [
+                            {
+                                "text": "chocolate",
+                                "is_entity": False,
+                                "label": "==NONE==",
+                                "reason": "is a flavor in this context, not an ingredient",
+                            },
+                            {
+                                "text": "carob",
+                                "is_entity": True,
+                                "label": "INGREDIENT",
+                                "reason": "is an ingredient to add chocolate flavor",
+                            },
+                        ],
+                    },
+                ],
             },
             "model": {
-                "@llm_models": "spacy.GPT-3-5.v1",
+                "@llm_models": "spacy.GPT-3-5.v2",
             },
         },
     )
     nlp.initialize()
-    doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
+    doc = nlp(
+        "Sriracha sauce goes really well with hoisin stir fry, "
+        "but you should add it after you use the wok."
+    )
     print([(ent.text, ent.label_) for ent in doc.ents])  # noqa: T201
 
 
-def test_example_4_custom_model():
+def test_example_5_custom_model():
     import random
 
     @registry.llm_models("RandomClassification.v1")
diff --git a/usage_examples/tests/test_usage_examples.py b/usage_examples/tests/test_usage_examples.py
index 114a078f..4185f4c2 100644
--- a/usage_examples/tests/test_usage_examples.py
+++ b/usage_examples/tests/test_usage_examples.py
@@ -5,8 +5,8 @@
 
 from spacy_llm import cache  # noqa: F401
 
-from .. import multitask_openai, ner_dolly, ner_langchain_openai, rel_openai
-from .. import textcat_openai
+from .. import multitask_openai, ner_dolly, ner_langchain_openai, ner_v3_openai
+from .. import rel_openai, textcat_openai
 
 _USAGE_EXAMPLE_PATH = Path(__file__).parent.parent
 
@@ -27,6 +27,24 @@ def test_ner_dolly(config_name: str):
     )
 
 
+@pytest.mark.gpu
+@pytest.mark.skipif(not has_torch_cuda_gpu, reason="needs GPU & CUDA")
+@pytest.mark.parametrize("config_name", ("fewshot.cfg", "zeroshot.cfg"))
+def test_textcat_dolly(config_name: str):
+    """Test NER Dolly usage example.
+    config_name (str): Name of config file to use.
+    """
+    path = _USAGE_EXAMPLE_PATH / "textcat_openai"
+    textcat_openai.run_pipeline(
+        text="text",
+        config_path=path / config_name,
+        examples_path=None
+        if config_name == "zeroshot.cfg"
+        else path / "examples.jsonl",
+        verbose=False,
+    )
+
+
 @pytest.mark.external
 @pytest.mark.parametrize("config_name", ("fewshot.cfg", "zeroshot.cfg"))
 def test_textcat_openai(config_name: str):
@@ -45,6 +63,19 @@ def test_textcat_openai(config_name: str):
 
 
 @pytest.mark.external
+def test_ner_v3_openai():
+    """Test NER v3 OpenAI usage example."""
+    path = _USAGE_EXAMPLE_PATH / "ner_v3_openai"
+    ner_v3_openai.run_pipeline(
+        text="text",
+        config_path=path / "fewshot.cfg",
+        examples_path=path / "examples.json",
+        verbose=False,
+    )
+
+
+@pytest.mark.external
+@pytest.mark.filterwarnings("ignore::DeprecationWarning")
 def test_ner_langchain_openai():
     """Test NER LangChain OpenAI usage example."""
     ner_langchain_openai.run_pipeline(
@@ -53,6 +84,7 @@ def test_ner_langchain_openai():
 
 
 @pytest.mark.external
+@pytest.mark.filterwarnings("ignore::DeprecationWarning")
 @pytest.mark.parametrize("config_name", ("fewshot.cfg", "zeroshot.cfg"))
 def test_multitask_openai(config_name: str):
     """Test multitask OpenAI example.