Repository: home-assistant/core
Pull request #142445: Update ollama to allow selecting mutiple LLM APIs
Author: login: allenporter
Labels: name: core
name: cla-signed
name: integration: conversation
name: new-feature
name: has-tests
name: Quality Scale: No score
name: Quality Scale: internal
name: integration: ollama
Created: 2025-04-07T03:35:33Z | Merged: 2025-04-13T22:37:46Z

<!--
  You are amazing! Thanks for contributing to our project!
  Please, DO NOT DELETE ANY TEXT from this template! (unless instructed).
-->

## Proposed change
<!--
  Describe the big picture of your changes here to communicate to the
  maintainers why we should accept this pull request. If it fixes a bug
  or resolves a feature request, be sure to link to that issue in the
  additional information section.
-->
Update ollama to allow selecting mutiple LLM APIs. This is accomplished by providing a shim API Instance that combines prompts and tools from multiple other APIs and handles dispatching the original APIs.

## Type of change
<!--
  What type of change does your PR introduce to Home Assistant?
  NOTE: Please, check only 1! box!
  If your PR requires multiple boxes to be checked, you'll most likely need to
  split it into multiple PRs. This makes things easier and faster to code review.
-->

- [ ] Dependency upgrade
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New integration (thank you!)
- [x] New feature (which adds functionality to an existing integration)
- [ ] Deprecation (breaking change to happen in the future)
- [ ] Breaking change (fix/feature causing existing functionality to break)
- [ ] Code quality improvements to existing code or addition of tests

## Additional information
<!--
  Details are important, and help maintainers processing your PR.
  Please be sure to fill out additional details, if applicable.
-->

- This PR fixes or closes issue: fixes #
- This PR is related to issue: 
- Link to documentation pull request: 
- Link to developer documentation pull request: https://github.com/home-assistant/developers.home-assistant/pull/2629
- Link to frontend pull request: 

## Checklist
<!--
  Put an `x` in the boxes that apply. You can also fill these out after
  creating the PR. If you're unsure about any of them, don't hesitate to ask.
  We're here to help! This is simply a reminder of what we are going to look
  for before merging your code.
-->

- [ ] The code change is tested and works locally.
- [ ] Local tests pass. **Your PR cannot be merged unless tests pass**
- [ ] There is no commented out code in this PR.
- [ ] I have followed the [development checklist][dev-checklist]
- [ ] I have followed the [perfect PR recommendations][perfect-pr]
- [ ] The code has been formatted using Ruff (`ruff format homeassistant tests`)
- [ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

- [ ] Documentation added/updated for [www.home-assistant.io][docs-repository]

If the code communicates with devices, web services, or third-party tools:

- [ ] The [manifest file][manifest-docs] has all fields filled out correctly.  
      Updated and included derived files by running: `python3 -m script.hassfest`.
- [ ] New or updated dependencies have been added to `requirements_all.txt`.  
      Updated by running `python3 -m script.gen_requirements_all`.
- [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

<!--
  This project is very active and we have a high turnover of pull requests.

  Unfortunately, the number of incoming pull requests is higher than what our
  reviewers can review and merge so there is a long backlog of pull requests
  waiting for review. You can help here!
  
  By reviewing another pull request, you will help raise the code quality of
  that pull request and the final review will be faster. This way the general
  pace of pull request reviews will go up and your wait time will go down.
  
  When picking a pull request to review, try to choose one that hasn't yet
  been reviewed.

  Thanks for helping out!
-->

To help with the load of incoming pull requests:

- [ ] I have reviewed two other [open pull requests][prs] in this repository.

[prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure

<!--
  Thank you for contributing <3

  Below, some useful links you could explore:
-->
[dev-checklist]: https://developers.home-assistant.io/docs/development_checklist/
[manifest-docs]: https://developers.home-assistant.io/docs/creating_integration_manifest/
[quality-scale]: https://developers.home-assistant.io/docs/integration_quality_scale_index/
[docs-repository]: https://github.com/home-assistant/home-assistant.io
[perfect-pr]: https://developers.home-assistant.io/docs/review-process/#creating-the-perfect-pr



Diff:
diff --git a/homeassistant/components/conversation/chat_log.py b/homeassistant/components/conversation/chat_log.py
index 9ffcc7fc0d595d..8d8a17a5259d40 100644
--- a/homeassistant/components/conversation/chat_log.py
+++ b/homeassistant/components/conversation/chat_log.py
@@ -387,7 +387,7 @@ async def async_update_llm_data(
         self,
         conversing_domain: str,
         user_input: ConversationInput,
-        user_llm_hass_api: str | None = None,
+        user_llm_hass_api: str | list[str] | None = None,
         user_llm_prompt: str | None = None,
     ) -> None:
         """Set the LLM system prompt."""
diff --git a/homeassistant/components/ollama/config_flow.py b/homeassistant/components/ollama/config_flow.py
index 1024a824c25bf6..7379ea17ba6107 100644
--- a/homeassistant/components/ollama/config_flow.py
+++ b/homeassistant/components/ollama/config_flow.py
@@ -215,8 +215,6 @@ async def async_step_init(
     ) -> ConfigFlowResult:
         """Manage the options."""
         if user_input is not None:
-            if user_input[CONF_LLM_HASS_API] == "none":
-                user_input.pop(CONF_LLM_HASS_API)
             return self.async_create_entry(
                 title=_get_title(self.model), data=user_input
             )
@@ -234,18 +232,12 @@ def ollama_config_option_schema(
 ) -> dict:
     """Ollama options schema."""
     hass_apis: list[SelectOptionDict] = [
-        SelectOptionDict(
-            label="No control",
-            value="none",
-        )
-    ]
-    hass_apis.extend(
         SelectOptionDict(
             label=api.name,
             value=api.id,
         )
         for api in llm.async_get_apis(hass)
-    )
+    ]
 
     return {
         vol.Optional(
@@ -259,8 +251,7 @@ def ollama_config_option_schema(
         vol.Optional(
             CONF_LLM_HASS_API,
             description={"suggested_value": options.get(CONF_LLM_HASS_API)},
-            default="none",
-        ): SelectSelector(SelectSelectorConfig(options=hass_apis)),
+        ): SelectSelector(SelectSelectorConfig(options=hass_apis, multiple=True)),
         vol.Optional(
             CONF_NUM_CTX,
             description={"suggested_value": options.get(CONF_NUM_CTX, DEFAULT_NUM_CTX)},
diff --git a/homeassistant/helpers/llm.py b/homeassistant/helpers/llm.py
index aa6b3dc2cbff34..24062ba152152f 100644
--- a/homeassistant/helpers/llm.py
+++ b/homeassistant/helpers/llm.py
@@ -110,15 +110,29 @@ def unregister() -> None:
 
 
 async def async_get_api(
-    hass: HomeAssistant, api_id: str, llm_context: LLMContext
+    hass: HomeAssistant, api_id: str | list[str], llm_context: LLMContext
 ) -> APIInstance:
-    """Get an API."""
+    """Get an API.
+
+    This returns a single APIInstance for one or more API ids, merging into
+    a single instance of necessary.
+    """
     apis = _async_get_apis(hass)
 
-    if api_id not in apis:
-        raise HomeAssistantError(f"API {api_id} not found")
+    if isinstance(api_id, str):
+        api_id = [api_id]
+
+    for key in api_id:
+        if key not in apis:
+            raise HomeAssistantError(f"API {key} not found")
 
-    return await apis[api_id].async_get_api_instance(llm_context)
+    api: API
+    if len(api_id) == 1:
+        api = apis[api_id[0]]
+    else:
+        api = MergedAPI([apis[key] for key in api_id])
+
+    return await api.async_get_api_instance(llm_context)
 
 
 @callback
@@ -286,6 +300,102 @@ async def async_call(
         return response
 
 
+class NamespacedTool(Tool):
+    """A tool that wraps another tool, prepending a namespace.
+
+    This is used to support tools from multiple API. This tool dispatches
+    the original tool with the original non-namespaced name.
+    """
+
+    def __init__(self, namespace: str, tool: Tool) -> None:
+        """Init the class."""
+        self.namespace = namespace
+        self.name = f"{namespace}.{tool.name}"
+        self.description = tool.description
+        self.parameters = tool.parameters
+        self.tool = tool
+
+    async def async_call(
+        self, hass: HomeAssistant, tool_input: ToolInput, llm_context: LLMContext
+    ) -> JsonObjectType:
+        """Handle the intent."""
+        return await self.tool.async_call(
+            hass,
+            ToolInput(
+                tool_name=self.tool.name,
+                tool_args=tool_input.tool_args,
+                id=tool_input.id,
+            ),
+            llm_context,
+        )
+
+
+class MergedAPI(API):
+    """An API that represents a merged view of multiple APIs."""
+
+    def __init__(self, llm_apis: list[API]) -> None:
+        """Init the class."""
+        if not llm_apis:
+            raise ValueError("No APIs provided")
+        hass = llm_apis[0].hass
+        api_ids = [unicode_slug.slugify(api.id) for api in llm_apis]
+        if len(set(api_ids)) != len(api_ids):
+            raise ValueError("API IDs must be unique")
+        super().__init__(
+            hass=hass,
+            id="|".join(unicode_slug.slugify(api.id) for api in llm_apis),
+            name="Merged LLM API",
+        )
+        self.llm_apis = llm_apis
+
+    async def async_get_api_instance(self, llm_context: LLMContext) -> APIInstance:
+        """Return the instance of the API."""
+        # These usually don't do I/O and execute right away
+        llm_apis = [
+            await llm_api.async_get_api_instance(llm_context)
+            for llm_api in self.llm_apis
+        ]
+        prompt_parts = []
+        tools: list[Tool] = []
+        for api_instance in llm_apis:
+            namespace = unicode_slug.slugify(api_instance.api.name)
+            prompt_parts.append(
+                f'Follow these instructions for tools from "{namespace}":\n'
+            )
+            prompt_parts.append(api_instance.api_prompt)
+            prompt_parts.append("\n\n")
+            tools.extend(
+                [NamespacedTool(namespace, tool) for tool in api_instance.tools]
+            )
+
+        return APIInstance(
+            api=self,
+            api_prompt="".join(prompt_parts),
+            llm_context=llm_context,
+            tools=tools,
+            custom_serializer=self._custom_serializer(llm_apis),
+        )
+
+    def _custom_serializer(
+        self, llm_apis: list[APIInstance]
+    ) -> Callable[[Any], Any] | None:
+        serializers = [
+            api_instance.custom_serializer
+            for api_instance in llm_apis
+            if api_instance.custom_serializer is not None
+        ]
+        if not serializers:
+            return None
+
+        def merged(x: Any) -> Any:
+            for serializer in serializers:
+                if (result := serializer(x)) is not None:
+                    return result
+            return x
+
+        return merged
+
+
 class AssistAPI(API):
     """API exposing Assist API to LLMs."""
 
diff --git a/tests/components/conversation/test_chat_log.py b/tests/components/conversation/test_chat_log.py
index d7b3531c65873e..c9e72ae5a032dc 100644
--- a/tests/components/conversation/test_chat_log.py
+++ b/tests/components/conversation/test_chat_log.py
@@ -139,6 +139,48 @@ async def test_unknown_llm_api(
     assert exc_info.value.as_conversation_result().as_dict() == snapshot
 
 
+async def test_multiple_llm_apis(
+    hass: HomeAssistant,
+    mock_conversation_input: ConversationInput,
+) -> None:
+    """Test when we reference an LLM API."""
+
+    class MyTool(llm.Tool):
+        """Test tool."""
+
+        name = "test_tool"
+        description = "Test function"
+        parameters = vol.Schema(
+            {vol.Optional("param1", description="Test parameters"): str}
+        )
+
+    class MyAPI(llm.API):
+        """Test API."""
+
+        async def async_get_api_instance(
+            self, llm_context: llm.LLMContext
+        ) -> llm.APIInstance:
+            """Return a list of tools."""
+            return llm.APIInstance(self, "My API Prompt", llm_context, [MyTool()])
+
+    api = MyAPI(hass=hass, id="my-api", name="Test")
+    llm.async_register_api(hass, api)
+
+    with (
+        chat_session.async_get_chat_session(hass) as session,
+        async_get_chat_log(hass, session, mock_conversation_input) as chat_log,
+    ):
+        await chat_log.async_update_llm_data(
+            conversing_domain="test",
+            user_input=mock_conversation_input,
+            user_llm_hass_api=["assist", "my-api"],
+            user_llm_prompt=None,
+        )
+
+    assert chat_log.llm_api
+    assert chat_log.llm_api.api.id == "assist|my-api"
+
+
 async def test_template_error(
     hass: HomeAssistant,
     mock_conversation_input: ConversationInput,
diff --git a/tests/helpers/test_llm.py b/tests/helpers/test_llm.py
index 26c357c4b0a5f8..23c2eef1765421 100644
--- a/tests/helpers/test_llm.py
+++ b/tests/helpers/test_llm.py
@@ -25,6 +25,7 @@
 )
 from homeassistant.setup import async_setup_component
 from homeassistant.util import dt as dt_util
+from homeassistant.util.json import JsonObjectType
 
 from tests.common import MockConfigEntry, async_mock_service
 
@@ -45,9 +46,12 @@ def llm_context() -> llm.LLMContext:
 class MyAPI(llm.API):
     """Test API."""
 
+    prompt: str = ""
+    tools: list[llm.Tool] = []
+
     async def async_get_api_instance(self, _: llm.ToolInput) -> llm.APIInstance:
         """Return a list of tools."""
-        return llm.APIInstance(self, "", [], llm_context)
+        return llm.APIInstance(self, self.prompt, llm_context, self.tools)
 
 
 async def test_get_api_no_existing(
@@ -1326,3 +1330,57 @@ async def test_no_tools_exposed(hass: HomeAssistant) -> None:
     )
     api = await llm.async_get_api(hass, "assist", llm_context)
     assert api.tools == []
+
+
+async def test_merged_api(hass: HomeAssistant, llm_context: llm.LLMContext) -> None:
+    """Test an API instance that merges multiple llm apis."""
+
+    class MyTool(llm.Tool):
+        def __init__(self, name: str, description: str) -> None:
+            self.name = name
+            self.description = description
+
+        async def async_call(
+            self, hass: HomeAssistant, tool_input: llm.ToolInput, _: llm.LLMContext
+        ) -> JsonObjectType:
+            return {"result": {tool_input.tool_name: tool_input.tool_args}}
+
+    api1 = MyAPI(hass=hass, id="api-1", name="API 1")
+    api1.prompt = "This is prompt 1"
+    api1.tools = [MyTool(name="Tool_1", description="Description 1")]
+    llm.async_register_api(hass, api1)
+
+    api2 = MyAPI(hass=hass, id="api-2", name="API 2")
+    api2.prompt = "This is prompt 2"
+    api2.tools = [MyTool(name="Tool_2", description="Description 2")]
+    llm.async_register_api(hass, api2)
+
+    instance = await llm.async_get_api(hass, ["api-1", "api-2"], llm_context)
+    assert instance.api.id == "api-1|api-2"
+
+    assert (
+        instance.api_prompt
+        == """Follow these instructions for tools from "api-1":
+This is prompt 1
+
+Follow these instructions for tools from "api-2":
+This is prompt 2
+
+"""
+    )
+    assert [(tool.name, tool.description) for tool in instance.tools] == [
+        ("api-1.Tool_1", "Description 1"),
+        ("api-2.Tool_2", "Description 2"),
+    ]
+
+    # The test tool returns back the provided arguments so we can verify
+    # the original tool is invoked with the correct tool name and args.
+    result = await instance.async_call_tool(
+        llm.ToolInput(tool_name="api-1.Tool_1", tool_args={"arg1": "value1"})
+    )
+    assert result == {"result": {"Tool_1": {"arg1": "value1"}}}
+
+    result = await instance.async_call_tool(
+        llm.ToolInput(tool_name="api-2.Tool_2", tool_args={"arg2": "value2"})
+    )
+    assert result == {"result": {"Tool_2": {"arg2": "value2"}}}


Reviews:
user: login: balloob
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T00:37:11Z
user: login: balloob
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T00:40:32Z
user: login: balloob
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-08T03:17:16Z

Review comments:
diff_hunk: @@ -286,6 +301,100 @@ async def async_call(
         return response
 
 
+class NamespacedTool(Tool):
+    """A tool that wraps another tool, prepending a namespace.
+
+    This is used to support tools from multiple API. This tool dispatches
+    the original took with the original non-namespaced name.
path: homeassistant/helpers/llm.py
user: login: balloob
body: ```suggestion
    the original tool with the original non-namespaced name.
```
created_at: 2025-04-08T00:37:11Z
author_association: MEMBER
diff_hunk: @@ -286,6 +301,100 @@ async def async_call(
         return response
 
 
+class NamespacedTool(Tool):
+    """A tool that wraps another tool, prepending a namespace.
+
+    This is used to support tools from multiple API. This tool dispatches
+    the original took with the original non-namespaced name.
+    """
+
+    def __init__(self, namespace: str, tool: Tool) -> None:
+        """Init the class."""
+        self.namespace = namespace
+        self.name = f"{namespace}.{tool.name}"
+        self.description = tool.description
+        self.parameters = tool.parameters
+        self.tool = tool
+
+    async def async_call(
+        self, hass: HomeAssistant, tool_input: ToolInput, llm_context: LLMContext
+    ) -> JsonObjectType:
+        """Handle the intent."""
+        return await self.tool.async_call(
+            hass,
+            ToolInput(
+                tool_name=self.tool.name,
+                tool_args=tool_input.tool_args,
+                id=tool_input.id,
+            ),
+            llm_context,
+        )
+
+
+class MergedAPI(API):
+    """An API that represents a merged view of multiple APIs."""
+
+    def __init__(self, llm_apis: list[API]) -> None:
+        """Init the class."""
+        if not llm_apis:
+            raise ValueError("No APIs provided")
+        hass = llm_apis[0].hass
+        api_ids = [unicode_slug.slugify(api.id) for api in llm_apis]
+        if len(set(api_ids)) != len(api_ids):
+            raise ValueError("API IDs must be unique")
+        super().__init__(
+            hass=hass,
+            id="|".join(unicode_slug.slugify(api.id) for api in llm_apis),
+            name="Merged LLM API",
+        )
+        self.llm_apis = llm_apis
+
+    async def async_get_api_instance(self, llm_context: LLMContext) -> APIInstance:
+        """Return the instance of the API."""
+        llm_apis = await asyncio.gather(
path: homeassistant/helpers/llm.py
user: login: balloob
body: Since none of these do I/O, if you avoid `gather`, it will not yield the event loop and execute everything right away. 
created_at: 2025-04-08T00:40:32Z
author_association: MEMBER

PR comments:
user: login: home-assistant[bot]
created_at: 2025-04-07T03:35:40Z
body: 
Hey there @home-assistant/core, @synesthesiam, mind taking a look at this pull request as it has been labeled with an integration (`conversation`) you are listed as a [code owner](https://github.com/home-assistant/core/blob/dev/CODEOWNERS#L289) for? Thanks!

<details>
<summary>Code owner commands</summary>


Code owners of `conversation` can trigger bot actions by commenting:

- `@home-assistant close` Closes the pull request.
- `@home-assistant rename Awesome new title` Renames the pull request.
- `@home-assistant reopen` Reopen the pull request.
- `@home-assistant unassign conversation` Removes the current integration label and assignees on the pull request, add the integration domain after the command.
- `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
- `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

</details>


author_association: CONTRIBUTOR
user: login: home-assistant[bot]
created_at: 2025-04-07T03:35:40Z
body: 
Hey there @synesthesiam, mind taking a look at this pull request as it has been labeled with an integration (`ollama`) you are listed as a [code owner](https://github.com/home-assistant/core/blob/dev/CODEOWNERS#L1073) for? Thanks!

<details>
<summary>Code owner commands</summary>


Code owners of `ollama` can trigger bot actions by commenting:

- `@home-assistant close` Closes the pull request.
- `@home-assistant rename Awesome new title` Renames the pull request.
- `@home-assistant reopen` Reopen the pull request.
- `@home-assistant unassign ollama` Removes the current integration label and assignees on the pull request, add the integration domain after the command.
- `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
- `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

</details>


author_association: CONTRIBUTOR