Repository: deepset-ai/haystack
Pull request #8981: feat: async support for the `HuggingFaceLocalChatGenerator`
Author: login: davidsbatista
Labels: name: type:documentation
description: Improvements on the docs
name: topic:tests
name: topic:build/distribution
name: topic:CI
name: topic:core
Created: 2025-03-05T14:04:17Z | Merged: 2025-03-06T14:57:12Z

### Related Issues

- fixes #8954 

### Proposed Changes:

- Added `run_async` method to `HuggingFaceLocalChatGenerator`method
- async support for both streaming and non-streaming generation
- Added ThreadPoolExecutor management for handling async operations

### How did you test it?

- local unit tests and CI tests

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:` and added `!` in case the PR includes breaking changes.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Linked issue #8954: add run_async for HuggingFaceLocalChatGenerator

Diff:
diff --git a/haystack/components/generators/chat/hugging_face_local.py b/haystack/components/generators/chat/hugging_face_local.py
index f54c50181c..e4ad4ea80d 100644
--- a/haystack/components/generators/chat/hugging_face_local.py
+++ b/haystack/components/generators/chat/hugging_face_local.py
@@ -2,13 +2,15 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
+import asyncio
 import json
 import re
 import sys
+from concurrent.futures import ThreadPoolExecutor
 from typing import Any, Callable, Dict, List, Literal, Optional, Union
 
 from haystack import component, default_from_dict, default_to_dict, logging
-from haystack.dataclasses import ChatMessage, StreamingChunk, ToolCall
+from haystack.dataclasses import ChatMessage, StreamingChunk, ToolCall, select_streaming_callback
 from haystack.lazy_imports import LazyImport
 from haystack.tools import Tool, _check_duplicate_tool_names, deserialize_tools_inplace
 from haystack.utils import (
@@ -123,6 +125,7 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
         tools: Optional[List[Tool]] = None,
         tool_parsing_function: Optional[Callable[[str], Optional[List[ToolCall]]]] = None,
+        async_executor: Optional[ThreadPoolExecutor] = None,
     ):
         """
         Initializes the HuggingFaceLocalChatGenerator component.
@@ -165,6 +168,9 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         :param tool_parsing_function:
             A callable that takes a string and returns a list of ToolCall objects or None.
             If None, the default_tool_parser will be used which extracts tool calls using a predefined pattern.
+        :param async_executor:
+            Optional ThreadPoolExecutor to use for async calls. If not provided, a single-threaded executor will be
+            initialized and used
         """
         torch_and_transformers_import.check()
 
@@ -223,6 +229,27 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         self.pipeline = None
         self.tools = tools
 
+        self._owns_executor = async_executor is None
+        self.executor = (
+            ThreadPoolExecutor(thread_name_prefix=f"async-HFLocalChatGenerator-executor-{id(self)}", max_workers=1)
+            if async_executor is None
+            else async_executor
+        )
+
+    def __del__(self):
+        """
+        Cleanup when the instance is being destroyed.
+        """
+        if hasattr(self, "_owns_executor") and self._owns_executor and hasattr(self, "executor"):
+            self.executor.shutdown(wait=True)
+
+    def shutdown(self):
+        """
+        Explicitly shutdown the executor if we own it.
+        """
+        if self._owns_executor:
+            self.executor.shutdown(wait=True)
+
     def _get_telemetry_data(self) -> Dict[str, Any]:
         """
         Data that is sent to Posthog for usage analytics.
@@ -332,7 +359,9 @@ def run(
         if stop_words_criteria:
             generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
 
-        streaming_callback = streaming_callback or self.streaming_callback
+        streaming_callback = select_streaming_callback(
+            self.streaming_callback, streaming_callback, requires_async=False
+        )
         if streaming_callback:
             num_responses = generation_kwargs.get("num_return_sequences", 1)
             if num_responses > 1:
@@ -427,7 +456,8 @@ def create_message(  # pylint: disable=too-many-positional-arguments
         # If tool calls are detected, don't include the text content since it contains the raw tool call format
         return ChatMessage.from_assistant(tool_calls=tool_calls, text=None if tool_calls else text, meta=meta)
 
-    def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List[str]]:
+    @staticmethod
+    def _validate_stop_words(stop_words: Optional[List[str]]) -> Optional[List[str]]:
         """
         Validates the provided stop words.
 
@@ -443,3 +473,146 @@ def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List
             return None
 
         return list(set(stop_words or []))
+
+    @component.output_types(replies=List[ChatMessage])
+    async def run_async(
+        self,
+        messages: List[ChatMessage],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Asynchronously invokes text generation inference based on the provided messages and generation parameters.
+
+        This is the asynchronous version of the `run` method. It has the same parameters
+        and return values but can be used with `await` in an async code.
+
+        :param messages: A list of ChatMessage objects representing the input messages.
+        :param generation_kwargs: Additional keyword arguments for text generation.
+        :param streaming_callback: An optional callable for handling streaming responses.
+        :param tools: A list of tools for which the model can prepare calls.
+        :returns: A dictionary with the following keys:
+            - `replies`: A list containing the generated responses as ChatMessage instances.
+        """
+        if self.pipeline is None:
+            raise RuntimeError("The generation model has not been loaded. Please call warm_up() before running.")
+
+        tools = tools or self.tools
+        if tools and streaming_callback is not None:
+            raise ValueError("Using tools and streaming at the same time is not supported. Please choose one.")
+        _check_duplicate_tool_names(tools)
+
+        tokenizer = self.pipeline.tokenizer
+
+        # Check and update generation parameters
+        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}
+
+        stop_words = generation_kwargs.pop("stop_words", []) + generation_kwargs.pop("stop_sequences", [])
+        stop_words = self._validate_stop_words(stop_words)
+
+        # Set up stop words criteria if stop words exist
+        stop_words_criteria = StopWordsCriteria(tokenizer, stop_words, self.pipeline.device) if stop_words else None
+        if stop_words_criteria:
+            generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
+
+        # validate and select the streaming callback
+        streaming_callback = select_streaming_callback(self.streaming_callback, streaming_callback, requires_async=True)
+
+        if streaming_callback:
+            return await self._run_streaming_async(
+                messages, tokenizer, generation_kwargs, stop_words, streaming_callback
+            )
+
+        return await self._run_non_streaming_async(messages, tokenizer, generation_kwargs, stop_words, tools)
+
+    async def _run_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        streaming_callback: Callable[[StreamingChunk], None],
+    ):
+        """
+        Handles async streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages, tokenize=False, chat_template=self.chat_template, add_generation_prompt=True
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Set up streaming handler
+        generation_kwargs["streamer"] = HFTokenStreamingHandler(tokenizer, streaming_callback, stop_words)
+
+        # Generate responses asynchronously
+        output = await asyncio.get_running_loop().run_in_executor(
+            self.executor,
+            lambda: self.pipeline(prepared_prompt, **generation_kwargs),  # type: ignore # if self.executor was not passed it was initialized with max_workers=1 in init
+        )
+
+        replies = [o.get("generated_text", "") for o in output]
+
+        # Remove stop words from replies if present
+        for stop_word in stop_words or []:
+            replies = [reply.replace(stop_word, "").rstrip() for reply in replies]
+
+        chat_messages = [
+            self.create_message(reply, r_index, tokenizer, prepared_prompt, generation_kwargs, parse_tool_calls=False)
+            for r_index, reply in enumerate(replies)
+        ]
+
+        return {"replies": chat_messages}
+
+    async def _run_non_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Handles async non-streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages,
+            tokenize=False,
+            chat_template=self.chat_template,
+            add_generation_prompt=True,
+            tools=[tc.tool_spec for tc in tools] if tools else None,
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Generate responses asynchronously
+        output = await asyncio.get_running_loop().run_in_executor(
+            self.executor,
+            lambda: self.pipeline(prepared_prompt, **generation_kwargs),  # type: ignore # if self.executor was not passed it was initialized with max_workers=1 in init
+        )
+
+        replies = [o.get("generated_text", "") for o in output]
+
+        # Remove stop words from replies if present
+        for stop_word in stop_words or []:
+            replies = [reply.replace(stop_word, "").rstrip() for reply in replies]
+
+        chat_messages = [
+            self.create_message(
+                reply, r_index, tokenizer, prepared_prompt, generation_kwargs, parse_tool_calls=bool(tools)
+            )
+            for r_index, reply in enumerate(replies)
+        ]
+
+        return {"replies": chat_messages}
diff --git a/releasenotes/notes/adding-async-huggingface-local-chat-generator-962512f52282d12d.yaml b/releasenotes/notes/adding-async-huggingface-local-chat-generator-962512f52282d12d.yaml
new file mode 100644
index 0000000000..c2323a512b
--- /dev/null
+++ b/releasenotes/notes/adding-async-huggingface-local-chat-generator-962512f52282d12d.yaml
@@ -0,0 +1,5 @@
+---
+features:
+  - |
+    Add `run_async` method to HuggingFaceLocalChatGenerator. This method internally uses ThreadPoolExecutor to return coroutines
+    that can be awaited.
diff --git a/test/components/generators/chat/test_hugging_face_local.py b/test/components/generators/chat/test_hugging_face_local.py
index 38d25ec91a..6a173a4abd 100644
--- a/test/components/generators/chat/test_hugging_face_local.py
+++ b/test/components/generators/chat/test_hugging_face_local.py
@@ -1,18 +1,21 @@
 # SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
 #
 # SPDX-License-Identifier: Apache-2.0
-from unittest.mock import Mock, patch
+
+import asyncio
+import gc
 from typing import Optional, List
+from unittest.mock import Mock, patch
 
-from haystack.dataclasses.streaming_chunk import StreamingChunk
 import pytest
 from transformers import PreTrainedTokenizer
 
 from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
 from haystack.dataclasses import ChatMessage, ChatRole, ToolCall
+from haystack.dataclasses.streaming_chunk import StreamingChunk
+from haystack.tools import Tool
 from haystack.utils import ComponentDevice
 from haystack.utils.auth import Secret
-from haystack.tools import Tool
 
 
 # used to test serialization of streaming_callback
@@ -474,3 +477,76 @@ def test_default_tool_parser(self, model_info_mock, tools):
         assert len(results["replies"][0].tool_calls) == 1
         assert results["replies"][0].tool_calls[0].tool_name == "weather"
         assert results["replies"][0].tool_calls[0].arguments == {"city": "Berlin"}
+
+    # Async tests
+
+    async def test_run_async(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test basic async functionality"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        results = await generator.run_async(messages=chat_messages)
+
+        assert "replies" in results
+        assert isinstance(results["replies"][0], ChatMessage)
+        chat_message = results["replies"][0]
+        assert chat_message.is_from(ChatRole.ASSISTANT)
+        assert chat_message.text == "Berlin is cool"
+
+    async def test_run_async_with_tools(self, model_info_mock, mock_pipeline_tokenizer, tools):
+        """Test async functionality with tools"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model", tools=tools)
+        generator.pipeline = mock_pipeline_tokenizer
+        # Mock the pipeline to return a tool call format
+        generator.pipeline.return_value = [{"generated_text": '{"name": "weather", "arguments": {"city": "Berlin"}}'}]
+
+        messages = [ChatMessage.from_user("What's the weather in Berlin?")]
+        results = await generator.run_async(messages=messages)
+
+        assert len(results["replies"]) == 1
+        message = results["replies"][0]
+        assert message.tool_calls
+        tool_call = message.tool_calls[0]
+        assert isinstance(tool_call, ToolCall)
+        assert tool_call.tool_name == "weather"
+        assert tool_call.arguments == {"city": "Berlin"}
+
+    async def test_concurrent_async_requests(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test handling of multiple concurrent async requests"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        # Create multiple concurrent requests
+        tasks = [generator.run_async(messages=chat_messages) for _ in range(5)]
+        results = await asyncio.gather(*tasks)
+
+        for result in results:
+            assert "replies" in result
+            assert isinstance(result["replies"][0], ChatMessage)
+            assert result["replies"][0].text == "Berlin is cool"
+
+    async def test_async_error_handling(self, model_info_mock, mock_pipeline_tokenizer):
+        """Test error handling in async context"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+
+        # Test without warm_up
+        with pytest.raises(RuntimeError, match="The generation model has not been loaded"):
+            await generator.run_async(messages=[ChatMessage.from_user("test")])
+
+        # Test with invalid streaming callback
+        generator.pipeline = mock_pipeline_tokenizer
+        with pytest.raises(ValueError, match="Using tools and streaming at the same time is not supported"):
+            await generator.run_async(
+                messages=[ChatMessage.from_user("test")],
+                streaming_callback=lambda x: None,
+                tools=[Tool(name="test", description="test", parameters={}, function=lambda: None)],
+            )
+
+    def test_executor_shutdown(self, model_info_mock, mock_pipeline_tokenizer):
+        with patch("haystack.components.generators.chat.hugging_face_local.pipeline") as mock_pipeline:
+            generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+            executor = generator.executor
+            with patch.object(executor, "shutdown", wraps=executor.shutdown) as mock_shutdown:
+                del generator
+                gc.collect()
+                mock_shutdown.assert_called_once_with(wait=True)


Reviews:
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T11:19:12Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T11:37:41Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T12:23:14Z
user: login: Amnah199
body: Left some comments.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T12:39:47Z
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T13:28:27Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T13:49:26Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T13:53:33Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T13:54:00Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T13:56:56Z
user: login: Amnah199
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:05:48Z
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:12:12Z
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:13:47Z
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:24:55Z
user: login: Amnah199
body: LG! Well done 👍 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:55:52Z

Review comments:
diff_hunk: @@ -223,6 +229,12 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         self.pipeline = None
         self.tools = tools
 
+        self.executor = (
+            ThreadPoolExecutor(thread_name_prefix=f"async-HFLocalChatGenerator-executor-{id(self)}", max_workers=1)
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: It could be helpful to properly shutdown the executor to prevent any unfinished jobs or leaks. We can use [`shutdown()` ](https://docs.python.org/3/library/concurrent.futures.html)for this. Wdyt?
created_at: 2025-03-06T11:19:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -443,3 +456,146 @@ def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List
             return None
 
         return list(set(stop_words or []))
+
+    @component.output_types(replies=List[ChatMessage])
+    async def run_async(
+        self,
+        messages: List[ChatMessage],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Asynchronously invokes text generation inference based on the provided messages and generation parameters.
+
+        This is the asynchronous version of the `run` method. It has the same parameters
+        and return values but can be used with `await` in an async code.
+
+        :param messages: A list of ChatMessage objects representing the input messages.
+        :param generation_kwargs: Additional keyword arguments for text generation.
+        :param streaming_callback: An optional callable for handling streaming responses.
+        :param tools: A list of tools for which the model can prepare calls.
+        :returns: A dictionary with the following keys:
+            - `replies`: A list containing the generated responses as ChatMessage instances.
+        """
+        if self.pipeline is None:
+            raise RuntimeError("The generation model has not been loaded. Please call warm_up() before running.")
+
+        tools = tools or self.tools
+        if tools and streaming_callback is not None:
+            raise ValueError("Using tools and streaming at the same time is not supported. Please choose one.")
+        _check_duplicate_tool_names(tools)
+
+        tokenizer = self.pipeline.tokenizer
+
+        # Check and update generation parameters
+        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}
+
+        stop_words = generation_kwargs.pop("stop_words", []) + generation_kwargs.pop("stop_sequences", [])
+        stop_words = self._validate_stop_words(stop_words)
+
+        # Set up stop words criteria if stop words exist
+        stop_words_criteria = StopWordsCriteria(tokenizer, stop_words, self.pipeline.device) if stop_words else None
+        if stop_words_criteria:
+            generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
+
+        # validate and select the streaming callback
+        streaming_callback = select_streaming_callback(self.streaming_callback, streaming_callback, requires_async=True)
+
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: Based on [context](https://github.com/deepset-ai/haystack/pull/8943#discussion_r1977468867), it would be nice to add `select_streaming_callback` in `run` method as well.
created_at: 2025-03-06T11:37:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,4 @@
+---
+enhancements:
+  - |
+    The `HuggingFaceLocalChatGenerator` now also supports the async run method.
path: releasenotes/notes/adding-async-huggingface-local-chat-generator-962512f52282d12d.yaml
user: login: Amnah199
body: For the sake of consistency with other notes, we can make it something like:
```suggestion
---
features:
  - |
    Add `run_async` method to HuggingFaceLocalChatGenerator. This method internally uses ThreadPoolExecutor to return coroutines
    that can be awaited.
```
created_at: 2025-03-06T12:23:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,6 +229,12 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         self.pipeline = None
         self.tools = tools
 
+        self.executor = (
+            ThreadPoolExecutor(thread_name_prefix=f"async-HFLocalChatGenerator-executor-{id(self)}", max_workers=1)
path: haystack/components/generators/chat/hugging_face_local.py
user: login: davidsbatista
body: I've tracked when the ThreadPoolExecutor is created internally at the init() and not passed and in those cases when the class is destroyed I called I triggered the shutdown()
created_at: 2025-03-06T13:28:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,6 +229,12 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         self.pipeline = None
         self.tools = tools
 
+        self.executor = (
+            ThreadPoolExecutor(thread_name_prefix=f"async-HFLocalChatGenerator-executor-{id(self)}", max_workers=1)
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: Another quick question here, you added single worker to the executor by default. This won't allow any concurrent calls. Is this intended?
created_at: 2025-03-06T13:49:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -443,3 +473,146 @@ def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List
             return None
 
         return list(set(stop_words or []))
+
+    @component.output_types(replies=List[ChatMessage])
+    async def run_async(
+        self,
+        messages: List[ChatMessage],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Asynchronously invokes text generation inference based on the provided messages and generation parameters.
+
+        This is the asynchronous version of the `run` method. It has the same parameters
+        and return values but can be used with `await` in an async code.
+
+        :param messages: A list of ChatMessage objects representing the input messages.
+        :param generation_kwargs: Additional keyword arguments for text generation.
+        :param streaming_callback: An optional callable for handling streaming responses.
+        :param tools: A list of tools for which the model can prepare calls.
+        :returns: A dictionary with the following keys:
+            - `replies`: A list containing the generated responses as ChatMessage instances.
+        """
+        if self.pipeline is None:
+            raise RuntimeError("The generation model has not been loaded. Please call warm_up() before running.")
+
+        tools = tools or self.tools
+        if tools and streaming_callback is not None:
+            raise ValueError("Using tools and streaming at the same time is not supported. Please choose one.")
+        _check_duplicate_tool_names(tools)
+
+        tokenizer = self.pipeline.tokenizer
+
+        # Check and update generation parameters
+        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}
+
+        stop_words = generation_kwargs.pop("stop_words", []) + generation_kwargs.pop("stop_sequences", [])
+        stop_words = self._validate_stop_words(stop_words)
+
+        # Set up stop words criteria if stop words exist
+        stop_words_criteria = StopWordsCriteria(tokenizer, stop_words, self.pipeline.device) if stop_words else None
+        if stop_words_criteria:
+            generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
+
+        # validate and select the streaming callback
+        streaming_callback = select_streaming_callback(self.streaming_callback, streaming_callback, requires_async=True)
+
+        if streaming_callback:
+            return await self._run_streaming_async(
+                messages, tokenizer, generation_kwargs, stop_words, streaming_callback
+            )
+
+        return await self._run_non_streaming_async(messages, tokenizer, generation_kwargs, stop_words, tools)
+
+    async def _run_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        streaming_callback: Callable[[StreamingChunk], None],
+    ):
+        """
+        Handles async streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages, tokenize=False, chat_template=self.chat_template, add_generation_prompt=True
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Set up streaming handler
+        generation_kwargs["streamer"] = HFTokenStreamingHandler(tokenizer, streaming_callback, stop_words)
+
+        # Generate responses asynchronously
+        output = await asyncio.get_event_loop().run_in_executor(
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: Its usually preferred to use `asyncio.get_running_loop` instead. ([reference](https://docs.python.org/3/library/asyncio-eventloop.html))
created_at: 2025-03-06T13:53:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -443,3 +473,146 @@ def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List
             return None
 
         return list(set(stop_words or []))
+
+    @component.output_types(replies=List[ChatMessage])
+    async def run_async(
+        self,
+        messages: List[ChatMessage],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Asynchronously invokes text generation inference based on the provided messages and generation parameters.
+
+        This is the asynchronous version of the `run` method. It has the same parameters
+        and return values but can be used with `await` in an async code.
+
+        :param messages: A list of ChatMessage objects representing the input messages.
+        :param generation_kwargs: Additional keyword arguments for text generation.
+        :param streaming_callback: An optional callable for handling streaming responses.
+        :param tools: A list of tools for which the model can prepare calls.
+        :returns: A dictionary with the following keys:
+            - `replies`: A list containing the generated responses as ChatMessage instances.
+        """
+        if self.pipeline is None:
+            raise RuntimeError("The generation model has not been loaded. Please call warm_up() before running.")
+
+        tools = tools or self.tools
+        if tools and streaming_callback is not None:
+            raise ValueError("Using tools and streaming at the same time is not supported. Please choose one.")
+        _check_duplicate_tool_names(tools)
+
+        tokenizer = self.pipeline.tokenizer
+
+        # Check and update generation parameters
+        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}
+
+        stop_words = generation_kwargs.pop("stop_words", []) + generation_kwargs.pop("stop_sequences", [])
+        stop_words = self._validate_stop_words(stop_words)
+
+        # Set up stop words criteria if stop words exist
+        stop_words_criteria = StopWordsCriteria(tokenizer, stop_words, self.pipeline.device) if stop_words else None
+        if stop_words_criteria:
+            generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
+
+        # validate and select the streaming callback
+        streaming_callback = select_streaming_callback(self.streaming_callback, streaming_callback, requires_async=True)
+
+        if streaming_callback:
+            return await self._run_streaming_async(
+                messages, tokenizer, generation_kwargs, stop_words, streaming_callback
+            )
+
+        return await self._run_non_streaming_async(messages, tokenizer, generation_kwargs, stop_words, tools)
+
+    async def _run_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        streaming_callback: Callable[[StreamingChunk], None],
+    ):
+        """
+        Handles async streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages, tokenize=False, chat_template=self.chat_template, add_generation_prompt=True
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Set up streaming handler
+        generation_kwargs["streamer"] = HFTokenStreamingHandler(tokenizer, streaming_callback, stop_words)
+
+        # Generate responses asynchronously
+        output = await asyncio.get_event_loop().run_in_executor(
+            self.executor,
+            lambda: self.pipeline(prepared_prompt, **generation_kwargs),  # type: ignore # if self.executor was not passed it was initialized with max_workers=1 in init
+        )
+
+        replies = [o.get("generated_text", "") for o in output]
+
+        # Remove stop words from replies if present
+        for stop_word in stop_words or []:
+            replies = [reply.replace(stop_word, "").rstrip() for reply in replies]
+
+        chat_messages = [
+            self.create_message(reply, r_index, tokenizer, prepared_prompt, generation_kwargs, parse_tool_calls=False)
+            for r_index, reply in enumerate(replies)
+        ]
+
+        return {"replies": chat_messages}
+
+    async def _run_non_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Handles async non-streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages,
+            tokenize=False,
+            chat_template=self.chat_template,
+            add_generation_prompt=True,
+            tools=[tc.tool_spec for tc in tools] if tools else None,
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Generate responses asynchronously
+        output = await asyncio.get_event_loop().run_in_executor(
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: As above, replace with `get_running_loop.`
created_at: 2025-03-06T13:54:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,6 +168,9 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         :param tool_parsing_function:
             A callable that takes a string and returns a list of ToolCall objects or None.
             If None, the default_tool_parser will be used which extracts tool calls using a predefined pattern.
+        :param async_executor:
+            Optional ThreadPoolExecutor to use for async calls. If not provided, a single-threaded executor will be
+            initialized and used
path: haystack/components/generators/chat/hugging_face_local.py
user: login: Amnah199
body: It would be nice to have a brief note about why you choose to run blocking calls in a thread pool.
created_at: 2025-03-06T13:56:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -474,3 +476,67 @@ def test_default_tool_parser(self, model_info_mock, tools):
         assert len(results["replies"][0].tool_calls) == 1
         assert results["replies"][0].tool_calls[0].tool_name == "weather"
         assert results["replies"][0].tool_calls[0].arguments == {"city": "Berlin"}
+
+    # Async tests
+
+    async def test_run_async(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test basic async functionality"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        results = await generator.run_async(messages=chat_messages)
+
+        assert "replies" in results
+        assert isinstance(results["replies"][0], ChatMessage)
+        chat_message = results["replies"][0]
+        assert chat_message.is_from(ChatRole.ASSISTANT)
+        assert chat_message.text == "Berlin is cool"
+
+    async def test_run_async_with_tools(self, model_info_mock, mock_pipeline_tokenizer, tools):
+        """Test async functionality with tools"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model", tools=tools)
+        generator.pipeline = mock_pipeline_tokenizer
+        # Mock the pipeline to return a tool call format
+        generator.pipeline.return_value = [{"generated_text": '{"name": "weather", "arguments": {"city": "Berlin"}}'}]
+
+        messages = [ChatMessage.from_user("What's the weather in Berlin?")]
+        results = await generator.run_async(messages=messages)
+
+        assert len(results["replies"]) == 1
+        message = results["replies"][0]
+        assert message.tool_calls
+        tool_call = message.tool_calls[0]
+        assert isinstance(tool_call, ToolCall)
+        assert tool_call.tool_name == "weather"
+        assert tool_call.arguments == {"city": "Berlin"}
+
+    async def test_concurrent_async_requests(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test handling of multiple concurrent async requests"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        # Create multiple concurrent requests
+        tasks = [generator.run_async(messages=chat_messages) for _ in range(5)]
+        results = await asyncio.gather(*tasks)
+
+        for result in results:
+            assert "replies" in result
+            assert isinstance(result["replies"][0], ChatMessage)
+            assert result["replies"][0].text == "Berlin is cool"
+
+    async def test_async_error_handling(self, model_info_mock, mock_pipeline_tokenizer):
+        """Test error handling in async context"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+
+        # Test without warm_up
+        with pytest.raises(RuntimeError, match="The generation model has not been loaded"):
+            await generator.run_async(messages=[ChatMessage.from_user("test")])
+
+        # Test with invalid streaming callback
+        generator.pipeline = mock_pipeline_tokenizer
+        with pytest.raises(ValueError, match="Using tools and streaming at the same time is not supported"):
+            await generator.run_async(
+                messages=[ChatMessage.from_user("test")],
+                streaming_callback=lambda x: None,
+                tools=[Tool(name="test", description="test", parameters={}, function=lambda: None)],
+            )
path: test/components/generators/chat/test_hugging_face_local.py
user: login: Amnah199
body: Maybe we can add a test to check if the executor has shutdown properly.

```
def test_executor_explicit_shutdown(self):
        """Test that an explicit shutdown() properly stops the executor."""
        generator = HuggingFaceLocalChatGenerator(model="mock-model")
        # Capture the executor instance
        executor = generator.executor
        generator.shutdown()
        with pytest.raises(RuntimeError):
            # Attempting to submit a new task after shutdown should raise an error.
            executor.submit(lambda: 42)
```
created_at: 2025-03-06T14:05:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,6 +229,12 @@ def __init__(  # pylint: disable=too-many-positional-arguments
         self.pipeline = None
         self.tools = tools
 
+        self.executor = (
+            ThreadPoolExecutor(thread_name_prefix=f"async-HFLocalChatGenerator-executor-{id(self)}", max_workers=1)
path: haystack/components/generators/chat/hugging_face_local.py
user: login: davidsbatista
body: yeah this means the ThreadPoolExecutor will only have a single worker and tasks will be executed sequentially.

It's intended, the model inference uses the Transformers library itself and is not inherently asynchronous. Still, if a user wnats concurrent execution, they can still pass their own executor with more workers through the async_executor parameter. As discussed before the async support here is mostly for coherence among all generators.

created_at: 2025-03-06T14:12:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -443,3 +473,146 @@ def _validate_stop_words(self, stop_words: Optional[List[str]]) -> Optional[List
             return None
 
         return list(set(stop_words or []))
+
+    @component.output_types(replies=List[ChatMessage])
+    async def run_async(
+        self,
+        messages: List[ChatMessage],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
+        tools: Optional[List[Tool]] = None,
+    ):
+        """
+        Asynchronously invokes text generation inference based on the provided messages and generation parameters.
+
+        This is the asynchronous version of the `run` method. It has the same parameters
+        and return values but can be used with `await` in an async code.
+
+        :param messages: A list of ChatMessage objects representing the input messages.
+        :param generation_kwargs: Additional keyword arguments for text generation.
+        :param streaming_callback: An optional callable for handling streaming responses.
+        :param tools: A list of tools for which the model can prepare calls.
+        :returns: A dictionary with the following keys:
+            - `replies`: A list containing the generated responses as ChatMessage instances.
+        """
+        if self.pipeline is None:
+            raise RuntimeError("The generation model has not been loaded. Please call warm_up() before running.")
+
+        tools = tools or self.tools
+        if tools and streaming_callback is not None:
+            raise ValueError("Using tools and streaming at the same time is not supported. Please choose one.")
+        _check_duplicate_tool_names(tools)
+
+        tokenizer = self.pipeline.tokenizer
+
+        # Check and update generation parameters
+        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}
+
+        stop_words = generation_kwargs.pop("stop_words", []) + generation_kwargs.pop("stop_sequences", [])
+        stop_words = self._validate_stop_words(stop_words)
+
+        # Set up stop words criteria if stop words exist
+        stop_words_criteria = StopWordsCriteria(tokenizer, stop_words, self.pipeline.device) if stop_words else None
+        if stop_words_criteria:
+            generation_kwargs["stopping_criteria"] = StoppingCriteriaList([stop_words_criteria])
+
+        # validate and select the streaming callback
+        streaming_callback = select_streaming_callback(self.streaming_callback, streaming_callback, requires_async=True)
+
+        if streaming_callback:
+            return await self._run_streaming_async(
+                messages, tokenizer, generation_kwargs, stop_words, streaming_callback
+            )
+
+        return await self._run_non_streaming_async(messages, tokenizer, generation_kwargs, stop_words, tools)
+
+    async def _run_streaming_async(  # pylint: disable=too-many-positional-arguments
+        self,
+        messages: List[ChatMessage],
+        tokenizer: Union["PreTrainedTokenizer", "PreTrainedTokenizerFast"],
+        generation_kwargs: Dict[str, Any],
+        stop_words: Optional[List[str]],
+        streaming_callback: Callable[[StreamingChunk], None],
+    ):
+        """
+        Handles async streaming generation of responses.
+        """
+        # convert messages to HF format
+        hf_messages = [convert_message_to_hf_format(message) for message in messages]
+        prepared_prompt = tokenizer.apply_chat_template(
+            hf_messages, tokenize=False, chat_template=self.chat_template, add_generation_prompt=True
+        )
+
+        # Avoid some unnecessary warnings in the generation pipeline call
+        generation_kwargs["pad_token_id"] = (
+            generation_kwargs.get("pad_token_id", tokenizer.pad_token_id) or tokenizer.eos_token_id
+        )
+
+        # Set up streaming handler
+        generation_kwargs["streamer"] = HFTokenStreamingHandler(tokenizer, streaming_callback, stop_words)
+
+        # Generate responses asynchronously
+        output = await asyncio.get_event_loop().run_in_executor(
path: haystack/components/generators/chat/hugging_face_local.py
user: login: davidsbatista
body: ah got it, it got deprecated 👍🏽 
created_at: 2025-03-06T14:13:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -474,3 +476,67 @@ def test_default_tool_parser(self, model_info_mock, tools):
         assert len(results["replies"][0].tool_calls) == 1
         assert results["replies"][0].tool_calls[0].tool_name == "weather"
         assert results["replies"][0].tool_calls[0].arguments == {"city": "Berlin"}
+
+    # Async tests
+
+    async def test_run_async(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test basic async functionality"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        results = await generator.run_async(messages=chat_messages)
+
+        assert "replies" in results
+        assert isinstance(results["replies"][0], ChatMessage)
+        chat_message = results["replies"][0]
+        assert chat_message.is_from(ChatRole.ASSISTANT)
+        assert chat_message.text == "Berlin is cool"
+
+    async def test_run_async_with_tools(self, model_info_mock, mock_pipeline_tokenizer, tools):
+        """Test async functionality with tools"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model", tools=tools)
+        generator.pipeline = mock_pipeline_tokenizer
+        # Mock the pipeline to return a tool call format
+        generator.pipeline.return_value = [{"generated_text": '{"name": "weather", "arguments": {"city": "Berlin"}}'}]
+
+        messages = [ChatMessage.from_user("What's the weather in Berlin?")]
+        results = await generator.run_async(messages=messages)
+
+        assert len(results["replies"]) == 1
+        message = results["replies"][0]
+        assert message.tool_calls
+        tool_call = message.tool_calls[0]
+        assert isinstance(tool_call, ToolCall)
+        assert tool_call.tool_name == "weather"
+        assert tool_call.arguments == {"city": "Berlin"}
+
+    async def test_concurrent_async_requests(self, model_info_mock, mock_pipeline_tokenizer, chat_messages):
+        """Test handling of multiple concurrent async requests"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+        generator.pipeline = mock_pipeline_tokenizer
+
+        # Create multiple concurrent requests
+        tasks = [generator.run_async(messages=chat_messages) for _ in range(5)]
+        results = await asyncio.gather(*tasks)
+
+        for result in results:
+            assert "replies" in result
+            assert isinstance(result["replies"][0], ChatMessage)
+            assert result["replies"][0].text == "Berlin is cool"
+
+    async def test_async_error_handling(self, model_info_mock, mock_pipeline_tokenizer):
+        """Test error handling in async context"""
+        generator = HuggingFaceLocalChatGenerator(model="mocked-model")
+
+        # Test without warm_up
+        with pytest.raises(RuntimeError, match="The generation model has not been loaded"):
+            await generator.run_async(messages=[ChatMessage.from_user("test")])
+
+        # Test with invalid streaming callback
+        generator.pipeline = mock_pipeline_tokenizer
+        with pytest.raises(ValueError, match="Using tools and streaming at the same time is not supported"):
+            await generator.run_async(
+                messages=[ChatMessage.from_user("test")],
+                streaming_callback=lambda x: None,
+                tools=[Tool(name="test", description="test", parameters={}, function=lambda: None)],
+            )
path: test/components/generators/chat/test_hugging_face_local.py
user: login: davidsbatista
body: I think a better  test would rather be to create the class externally without passing an executor and destroy it, asserting that the def shutdown(self) was called.
created_at: 2025-03-06T14:24:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: coveralls
created_at: 2025-03-05T14:15:36Z
body: ## Pull Request Test Coverage Report for [Build 13701183556](https://coveralls.io/builds/72578329)

### Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

- For more information on this, see <a target="_blank" href="https://docs.coveralls.io/build-types#tracking-coverage-changes-with-pull_request-builds">Tracking coverage changes with pull request builds</a>.
- To avoid this issue with future PRs, see these <a target="_blank" href="https://docs.coveralls.io/build-types#recommended-ci-configurations">Recommended CI Configurations</a>.
- For a quick fix, <a target="_blank" href="https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/#update-your-pull-request-branch-by-rebasing">rebase this PR at GitHub</a>. Your next report should be accurate.

### Details

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **35** unchanged lines in **1** file lost coverage.
* Overall coverage decreased (**-0.09%**) to **89.95%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [components/generators/chat/hugging_face_local.py](https://coveralls.io/builds/72578329/source?filename=components%2Fgenerators%2Fchat%2Fhugging_face_local.py#L60) | 35 | 80.9% |
<!-- | **Total:** | **35** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/72578329/badge)](https://coveralls.io/builds/72578329) |
| :-- | --: |
| Change from base [Build 13698305508](https://coveralls.io/builds/72575749): |  -0.09% |
| Covered Lines: | 9666 |
| Relevant Lines: | 10746 |

---
##### 💛  - [Coveralls](https://coveralls.io)

author_association: COLLABORATOR
user: login: Amnah199
created_at: 2025-03-06T12:18:52Z
body: We can add additional helper methods, such as `prepare_gen_args`, `run_streaming`, and `run_non_streaming`,  to enhance the code's structure and reduce redundancy. However, since this is not a priority for this PR, the decision is up to you. 
author_association: CONTRIBUTOR
user: login: davidsbatista
created_at: 2025-03-06T13:35:48Z
body: > However, since this is not a priority for this PR, the decision is up to you.

I would rather for now leave it as it.
author_association: CONTRIBUTOR