Repository: deepset-ai/haystack
Pull request: refactor: HF API Embedders - use `InferenceClient.feature_extraction` instead of `InferenceClient.post`
### Related Issues

- fixes #8791

Our HF API Embedders support 3 different HF APIs using the `huggingface_hub` client: Serverless Inference API, Text Embeddings Inference (deployed locally or elsewhere), and HF Paid Inference Endpoints.

For several reasons, in the past we have opted to use `InferenceClient.post` method, but this is fragile, subject to changes (as has happened recently) and will soon be removed (in 0.31).

This is why I'm migrating to the `InferenceClient.feature_extraction` method. This solution works but may not be ideal, as explained in the comments.

### Proposed Changes:
- Use `feature_extraction` instead of `post`.
- Add some checks on the shape of the embeddings obtained from the API.

### How did you test it?
CI; new tests

In the CI, we are only testing the Serverless Inference API.

For these reasons, I also tested the current solution with:
- a Text Embeddings Inference container deployed locally
- a Paid Inference Endpoint (deployed on HF)

### Notes for the reviewer
Since the solution identified is not ideal, I also plan to contact a `huggingface_hub` maintainer and ask for advice.

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:` and added `!` in case the PR includes breaking changes.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
`HuggingFaceAPIDocumentEmbedder` is not compatible with `huggingface_hub>=0.28.0`
**Describe the bug**
`huggingface_hub>=0.28.0` (recently released) does not work well with our `HuggingFaceAPIDocumentEmbedder`: the final document does not contain an embedding with the expected type and size.

**To Reproduce**
```python
from haystack.components.embedders import HuggingFaceAPIDocumentEmbedder
from haystack import Document

docs = [Document(content="first document"), Document(content="second document"),
        Document(content="third document")]

embedder = HuggingFaceAPIDocumentEmbedder(api_type="serverless_inference_api",
                                                  api_params={"model": "BAAI/bge-small-en-v1.5"},)

docs_w_embeddings = embedder.run(docs)["documents"]

print(docs_w_embeddings[0])
# Document(id=70de273c10444421cfd2407811e7d30ef1ceef28bd47fa64587f4dcba49b779c, content: 'first document', embedding: vector of size 1)

print(docs_w_embeddings[0].embedding)
# [[[2.8279178142547607, 2.6550655364990234, ...]]]
```

**Expected behavior**
The embedding should be a vector of size 384 and not a nested structure.


**System:**
 - Haystack version (commit or version number): main (also previous versions are affected)

Hints from issue discussion:

Commit message: rm unused import

Patch to review:
diff --git a/haystack/components/embedders/hugging_face_api_document_embedder.py b/haystack/components/embedders/hugging_face_api_document_embedder.py
index 459e386976..d6d4153a19 100644
--- a/haystack/components/embedders/hugging_face_api_document_embedder.py
+++ b/haystack/components/embedders/hugging_face_api_document_embedder.py
@@ -2,7 +2,6 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-import json
 from typing import Any, Dict, List, Optional, Union
 
 from tqdm import tqdm
@@ -124,13 +123,11 @@ def __init__(
             Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
             if the backend uses Text Embeddings Inference.
             If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
-            It is always set to `True` and cannot be changed.
         :param normalize:
             Normalizes the embeddings to unit length.
             Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
             if the backend uses Text Embeddings Inference.
             If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
-            It is always set to `False` and cannot be changed.
         :param batch_size:
             Number of documents to process at once.
         :param progress_bar:
@@ -239,18 +236,24 @@ def _embed_batch(self, texts_to_embed: List[str], batch_size: int) -> List[List[
         """
         Embed a list of texts in batches.
         """
-
         all_embeddings = []
         for i in tqdm(
             range(0, len(texts_to_embed), batch_size), disable=not self.progress_bar, desc="Calculating embeddings"
         ):
             batch = texts_to_embed[i : i + batch_size]
-            response = self._client.post(
-                json={"inputs": batch, "truncate": self.truncate, "normalize": self.normalize},
-                task="feature-extraction",
+
+            np_embeddings = self._client.feature_extraction(
+                # this method does not officially support list of strings, but works as expected
+                text=batch,  # type: ignore[arg-type]
+                # Serverless Inference API does not support truncate and normalize, so we pass None in the request
+                truncate=self.truncate if self.api_type != HFEmbeddingAPIType.SERVERLESS_INFERENCE_API else None,
+                normalize=self.normalize if self.api_type != HFEmbeddingAPIType.SERVERLESS_INFERENCE_API else None,
             )
-            embeddings = json.loads(response.decode())
-            all_embeddings.extend(embeddings)
+
+            if np_embeddings.ndim != 2 or np_embeddings.shape[0] != len(batch):
+                raise ValueError(f"Expected embedding shape ({batch_size}, embedding_dim), got {np_embeddings.shape}")
+
+            all_embeddings.extend(np_embeddings.tolist())
 
         return all_embeddings
 
diff --git a/haystack/components/embedders/hugging_face_api_text_embedder.py b/haystack/components/embedders/hugging_face_api_text_embedder.py
index 2cd68d34da..c100f5c42e 100644
--- a/haystack/components/embedders/hugging_face_api_text_embedder.py
+++ b/haystack/components/embedders/hugging_face_api_text_embedder.py
@@ -2,7 +2,6 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-import json
 from typing import Any, Dict, List, Optional, Union
 
 from haystack import component, default_from_dict, default_to_dict, logging
@@ -104,13 +103,11 @@ def __init__(
             Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
             if the backend uses Text Embeddings Inference.
             If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
-            It is always set to `True` and cannot be changed.
         :param normalize:
             Normalizes the embeddings to unit length.
             Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
             if the backend uses Text Embeddings Inference.
             If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
-            It is always set to `False` and cannot be changed.
         """
         huggingface_hub_import.check()
 
@@ -200,10 +197,19 @@ def run(self, text: str):
 
         text_to_embed = self.prefix + text + self.suffix
 
-        response = self._client.post(
-            json={"inputs": [text_to_embed], "truncate": self.truncate, "normalize": self.normalize},
-            task="feature-extraction",
+        np_embedding = self._client.feature_extraction(
+            text=text_to_embed,
+            # Serverless Inference API does not support truncate and normalize, so we pass None in the request
+            truncate=self.truncate if self.api_type != HFEmbeddingAPIType.SERVERLESS_INFERENCE_API else None,
+            normalize=self.normalize if self.api_type != HFEmbeddingAPIType.SERVERLESS_INFERENCE_API else None,
         )
-        embedding = json.loads(response.decode())[0]
+
+        error_msg = f"Expected embedding shape (1, embedding_dim) or (embedding_dim,), got {np_embedding.shape}"
+        if np_embedding.ndim > 2:
+            raise ValueError(error_msg)
+        if np_embedding.ndim == 2 and np_embedding.shape[0] != 1:
+            raise ValueError(error_msg)
+
+        embedding = np_embedding.flatten().tolist()
 
         return {"embedding": embedding}
diff --git a/pyproject.toml b/pyproject.toml
index 9a5f15070b..eda943c19a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -87,7 +87,7 @@ extra-dependencies = [
   "numba>=0.54.0", # This pin helps uv resolve the dependency tree. See https://github.com/astral-sh/uv/issues/7881
 
   "transformers[torch,sentencepiece]==4.47.1", # ExtractiveReader, TransformersSimilarityRanker, LocalWhisperTranscriber, HFGenerators...
-  "huggingface_hub>=0.27.0, <0.28.0",                   # Hugging Face API Generators and Embedders
+  "huggingface_hub>=0.27.0",                   # Hugging Face API Generators and Embedders
   "sentence-transformers>=3.0.0",              # SentenceTransformersTextEmbedder and SentenceTransformersDocumentEmbedder
   "langdetect",                                # TextLanguageRouter and DocumentLanguageClassifier
   "openai-whisper>=20231106",                  # LocalWhisperTranscriber
diff --git a/test/components/embedders/test_hugging_face_api_document_embedder.py b/test/components/embedders/test_hugging_face_api_document_embedder.py
index b9332d5363..dc44639131 100644
--- a/test/components/embedders/test_hugging_face_api_document_embedder.py
+++ b/test/components/embedders/test_hugging_face_api_document_embedder.py
@@ -8,6 +8,8 @@
 import pytest
 from huggingface_hub.utils import RepositoryNotFoundError
 
+from numpy import array
+
 from haystack.components.embedders import HuggingFaceAPIDocumentEmbedder
 from haystack.dataclasses import Document
 from haystack.utils.auth import Secret
@@ -23,8 +25,8 @@ def mock_check_valid_model():
         yield mock
 
 
-def mock_embedding_generation(json, **kwargs):
-    response = str([[random.random() for _ in range(384)] for _ in range(len(json["inputs"]))]).encode()
+def mock_embedding_generation(text, **kwargs):
+    response = array([[random.random() for _ in range(384)] for _ in range(len(text))])
     return response
 
 
@@ -204,7 +206,7 @@ def test_prepare_texts_to_embed_w_suffix(self, mock_check_valid_model):
     def test_embed_batch(self, mock_check_valid_model):
         texts = ["text 1", "text 2", "text 3", "text 4", "text 5"]
 
-        with patch("huggingface_hub.InferenceClient.post") as mock_embedding_patch:
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
             mock_embedding_patch.side_effect = mock_embedding_generation
 
             embedder = HuggingFaceAPIDocumentEmbedder(
@@ -223,6 +225,35 @@ def test_embed_batch(self, mock_check_valid_model):
             assert len(embedding) == 384
             assert all(isinstance(x, float) for x in embedding)
 
+    def test_embed_batch_wrong_embedding_shape(self, mock_check_valid_model):
+        texts = ["text 1", "text 2", "text 3", "text 4", "text 5"]
+
+        # embedding ndim != 2
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
+            mock_embedding_patch.return_value = array([0.1, 0.2, 0.3])
+
+            embedder = HuggingFaceAPIDocumentEmbedder(
+                api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API,
+                api_params={"model": "BAAI/bge-small-en-v1.5"},
+                token=Secret.from_token("fake-api-token"),
+            )
+
+            with pytest.raises(ValueError):
+                embedder._embed_batch(texts_to_embed=texts, batch_size=2)
+
+        # embedding ndim == 2 but shape[0] != len(batch)
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
+            mock_embedding_patch.return_value = array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
+
+            embedder = HuggingFaceAPIDocumentEmbedder(
+                api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API,
+                api_params={"model": "BAAI/bge-small-en-v1.5"},
+                token=Secret.from_token("fake-api-token"),
+            )
+
+            with pytest.raises(ValueError):
+                embedder._embed_batch(texts_to_embed=texts, batch_size=2)
+
     def test_run_wrong_input_format(self, mock_check_valid_model):
         embedder = HuggingFaceAPIDocumentEmbedder(
             api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API, api_params={"model": "BAAI/bge-small-en-v1.5"}
@@ -252,7 +283,7 @@ def test_run(self, mock_check_valid_model):
             Document(content="A transformer is a deep learning architecture", meta={"topic": "ML"}),
         ]
 
-        with patch("huggingface_hub.InferenceClient.post") as mock_embedding_patch:
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
             mock_embedding_patch.side_effect = mock_embedding_generation
 
             embedder = HuggingFaceAPIDocumentEmbedder(
@@ -268,16 +299,14 @@ def test_run(self, mock_check_valid_model):
             result = embedder.run(documents=docs)
 
             mock_embedding_patch.assert_called_once_with(
-                json={
-                    "inputs": [
-                        "prefix Cuisine | I love cheese suffix",
-                        "prefix ML | A transformer is a deep learning architecture suffix",
-                    ],
-                    "truncate": True,
-                    "normalize": False,
-                },
-                task="feature-extraction",
+                text=[
+                    "prefix Cuisine | I love cheese suffix",
+                    "prefix ML | A transformer is a deep learning architecture suffix",
+                ],
+                truncate=None,
+                normalize=None,
             )
+
         documents_with_embeddings = result["documents"]
 
         assert isinstance(documents_with_embeddings, list)
@@ -294,7 +323,7 @@ def test_run_custom_batch_size(self, mock_check_valid_model):
             Document(content="A transformer is a deep learning architecture", meta={"topic": "ML"}),
         ]
 
-        with patch("huggingface_hub.InferenceClient.post") as mock_embedding_patch:
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
             mock_embedding_patch.side_effect = mock_embedding_generation
 
             embedder = HuggingFaceAPIDocumentEmbedder(
diff --git a/test/components/embedders/test_hugging_face_api_text_embedder.py b/test/components/embedders/test_hugging_face_api_text_embedder.py
index 6e699fca25..11557088be 100644
--- a/test/components/embedders/test_hugging_face_api_text_embedder.py
+++ b/test/components/embedders/test_hugging_face_api_text_embedder.py
@@ -7,7 +7,7 @@
 import random
 import pytest
 from huggingface_hub.utils import RepositoryNotFoundError
-
+from numpy import array
 from haystack.components.embedders import HuggingFaceAPITextEmbedder
 from haystack.utils.auth import Secret
 from haystack.utils.hf import HFEmbeddingAPIType
@@ -21,11 +21,6 @@ def mock_check_valid_model():
         yield mock
 
 
-def mock_embedding_generation(json, **kwargs):
-    response = str([[random.random() for _ in range(384)] for _ in range(len(json["inputs"]))]).encode()
-    return response
-
-
 class TestHuggingFaceAPITextEmbedder:
     def test_init_invalid_api_type(self):
         with pytest.raises(ValueError):
@@ -142,8 +137,8 @@ def test_run_wrong_input_format(self, mock_check_valid_model):
             embedder.run(text=list_integers_input)
 
     def test_run(self, mock_check_valid_model):
-        with patch("huggingface_hub.InferenceClient.post") as mock_embedding_patch:
-            mock_embedding_patch.side_effect = mock_embedding_generation
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
+            mock_embedding_patch.return_value = array([[random.random() for _ in range(384)]])
 
             embedder = HuggingFaceAPITextEmbedder(
                 api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API,
@@ -156,13 +151,35 @@ def test_run(self, mock_check_valid_model):
             result = embedder.run(text="The food was delicious")
 
             mock_embedding_patch.assert_called_once_with(
-                json={"inputs": ["prefix The food was delicious suffix"], "truncate": True, "normalize": False},
-                task="feature-extraction",
+                text="prefix The food was delicious suffix", truncate=None, normalize=None
             )
 
         assert len(result["embedding"]) == 384
         assert all(isinstance(x, float) for x in result["embedding"])
 
+    def test_run_wrong_embedding_shape(self, mock_check_valid_model):
+        # embedding ndim > 2
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
+            mock_embedding_patch.return_value = array([[[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]]])
+
+            embedder = HuggingFaceAPITextEmbedder(
+                api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API, api_params={"model": "BAAI/bge-small-en-v1.5"}
+            )
+
+            with pytest.raises(ValueError):
+                embedder.run(text="The food was delicious")
+
+        # embedding ndim == 2 but shape[0] != 1
+        with patch("huggingface_hub.InferenceClient.feature_extraction") as mock_embedding_patch:
+            mock_embedding_patch.return_value = array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
+
+            embedder = HuggingFaceAPITextEmbedder(
+                api_type=HFEmbeddingAPIType.SERVERLESS_INFERENCE_API, api_params={"model": "BAAI/bge-small-en-v1.5"}
+            )
+
+            with pytest.raises(ValueError):
+                embedder.run(text="The food was delicious")
+
     @pytest.mark.flaky(reruns=5, reruns_delay=5)
     @pytest.mark.integration
     @pytest.mark.skipif(


Code review comments:
diff_hunk: @@ -239,18 +236,24 @@ def _embed_batch(self, texts_to_embed: List[str], batch_size: int) -> List[List[
         """
         Embed a list of texts in batches.
         """
-
         all_embeddings = []
         for i in tqdm(
             range(0, len(texts_to_embed), batch_size), disable=not self.progress_bar, desc="Calculating embeddings"
         ):
             batch = texts_to_embed[i : i + batch_size]
-            response = self._client.post(
-                json={"inputs": batch, "truncate": self.truncate, "normalize": self.normalize},
-                task="feature-extraction",
+
+            np_embeddings = self._client.feature_extraction(
+                # this method does not officially support list of strings, but works as expected
+                text=batch,  # type: ignore[arg-type]
path: haystack/components/embedders/hugging_face_api_document_embedder.py
text: @author:
as explained in the comment, the method does what we need but this usage is not officially supported

@user1:
For my own understanding, I looked into the API. From this [discussion](https://github.com/huggingface/huggingface_hub/issues/1647), am I correct to deduce that both types `str` and `List[str] `are supported for `text`. We are unsure because the docs don't mention` List[str]` officially but the underlying models do expect lists and return correct results.
In that case, it would make sense to introduce this change.

@author:
I think you are basically right.
I reached out to the `huggingface_hub` maintainers here: https://github.com/huggingface/huggingface_hub/issues/2824
You can read this message to get a better understanding.

diff_hunk: @@ -239,18 +236,24 @@ def _embed_batch(self, texts_to_embed: List[str], batch_size: int) -> List[List[
         """
         Embed a list of texts in batches.
         """
-
         all_embeddings = []
         for i in tqdm(
             range(0, len(texts_to_embed), batch_size), disable=not self.progress_bar, desc="Calculating embeddings"
         ):
             batch = texts_to_embed[i : i + batch_size]
-            response = self._client.post(
-                json={"inputs": batch, "truncate": self.truncate, "normalize": self.normalize},
-                task="feature-extraction",
+
+            np_embeddings = self._client.feature_extraction(
+                # this method does not officially support list of strings, but works as expected
+                text=batch,  # type: ignore[arg-type]
+                # Serverless Inference API does not support truncate and normalize, so we pass None in the request
+                truncate=self.truncate if self.api_type != HFEmbeddingAPIType.SERVERLESS_INFERENCE_API else None,
path: haystack/components/embedders/hugging_face_api_document_embedder.py
text: @author:
`truncate` and `normalize` are not supported in the Serverless Inference API.

With `post`, these parameters are ignored if the server does not support them.
Using `feature_extraction`, we need to pass `None`, otherwise we get an error.