Repository: deepset-ai/haystack

Issue:
Add `dimensions` parameter to OpenAI Embedders to fully support the new models
**Is your feature request related to a problem? Please describe.**
There's no full support for the new OpenAI embedding models. 

**Describe the solution you'd like**
[OpenAI Docs](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings):

> By default, the length of the embedding vector will be 1536 for text-embedding-3-small or 3072 for text-embedding-3-large. You can reduce the dimensions of the embedding by passing in the [dimensions parameter](https://platform.openai.com/docs/api-reference/embeddings/create#embeddings-create-dimensions) without the embedding losing its concept-representing properties.

We need to add `dimensions` parameter to OpenAI Embedders to fully support new `text-embedding-3-small` and `text-embedding-3-large` models

**Describe alternatives you've considered**
N/A

**Additional context**
N/A



Discussion:
@bilgeyucel, suppose someone is using dimension parameter with old models which do not support the it do you want me to throw an error or give a warning?

```python
from openai import OpenAI
client = OpenAI(api_key="openai-api-key")

client.embeddings.create(
  model="text-embedding-ada-002",
  input="The food was delicious and the waiter...",
  encoding_format="float",
  dimensions=3072
```
Hi @sahusiddharth, thank you for working on this issue 🙏🏼 My opinion is throwing a warning instead of an error here but let's take opinion from the core team as well. Wdyt @anakin87? 
Hey!

1. To simplify things on the Haystack side, I would like to mimic the behavior of the OpenAI client: if it fails, we also fail with an expressive message. I would prefer not to check in Haystack whether or not the model supports the dimensions parameter (otherwise this requires changes every time a new model is released).
2. Since we also use OpenAI Embedders also with other model providers that have compatible APIs (but don't support dimensions), I would like the changes to be backward-compatible and fully functional when this parameter is missing.
hello @anakin87, I understood the first point. I would appreciate some further clarification on the second point. Specifically, I am considering making `dimensions` an optional parameter. In my assessment, I believe this adjustment should not pose any issues concerning backward compatibility or affect functionality if the parameter is omitted. What are your suggestions?

I would make `dimensions` optional and make sure to include it in the API call only if it is not a null value (`None`).

Resolving patch:
diff --git a/haystack/components/embedders/openai_document_embedder.py b/haystack/components/embedders/openai_document_embedder.py
--- a/haystack/components/embedders/openai_document_embedder.py
+++ b/haystack/components/embedders/openai_document_embedder.py
@@ -33,6 +33,7 @@ def __init__(
         self,
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
         model: str = "text-embedding-ada-002",
+        dimensions: Optional[int] = None,
         api_base_url: Optional[str] = None,
         organization: Optional[str] = None,
         prefix: str = "",
@@ -46,6 +47,7 @@ def __init__(
         Create a OpenAIDocumentEmbedder component.
         :param api_key: The OpenAI API key.
         :param model: The name of the model to use.
+        :param dimensions: The number of dimensions the resulting output embeddings should have. Only supported in text-embedding-3 and later models.
         :param api_base_url: The OpenAI API Base url, defaults to None. For more details, see OpenAI [docs](https://platform.openai.com/docs/api-reference/audio).
         :param organization: The Organization ID, defaults to `None`. See
         [production best practices](https://platform.openai.com/docs/guides/production-best-practices/setting-up-your-organization).
@@ -59,6 +61,7 @@ def __init__(
         """
         self.api_key = api_key
         self.model = model
+        self.dimensions = dimensions
         self.api_base_url = api_base_url
         self.organization = organization
         self.prefix = prefix
@@ -84,6 +87,7 @@ def to_dict(self) -> Dict[str, Any]:
         return default_to_dict(
             self,
             model=self.model,
+            dimensions=self.dimensions,
             organization=self.organization,
             api_base_url=self.api_base_url,
             prefix=self.prefix,
@@ -131,7 +135,10 @@ def _embed_batch(self, texts_to_embed: List[str], batch_size: int) -> Tuple[List
             range(0, len(texts_to_embed), batch_size), disable=not self.progress_bar, desc="Calculating embeddings"
         ):
             batch = texts_to_embed[i : i + batch_size]
-            response = self.client.embeddings.create(model=self.model, input=batch)
+            if self.dimensions is not None:
+                response = self.client.embeddings.create(model=self.model, dimensions=self.dimensions, input=batch)
+            else:
+                response = self.client.embeddings.create(model=self.model, input=batch)
             embeddings = [el.embedding for el in response.data]
             all_embeddings.extend(embeddings)
 
diff --git a/haystack/components/embedders/openai_text_embedder.py b/haystack/components/embedders/openai_text_embedder.py
--- a/haystack/components/embedders/openai_text_embedder.py
+++ b/haystack/components/embedders/openai_text_embedder.py
@@ -1,8 +1,8 @@
-from typing import List, Optional, Dict, Any
+from typing import Any, Dict, List, Optional
 
 from openai import OpenAI
 
-from haystack import component, default_to_dict, default_from_dict
+from haystack import component, default_from_dict, default_to_dict
 from haystack.utils import Secret, deserialize_secrets_inplace
 
 
@@ -31,6 +31,7 @@ def __init__(
         self,
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
         model: str = "text-embedding-ada-002",
+        dimensions: Optional[int] = None,
         api_base_url: Optional[str] = None,
         organization: Optional[str] = None,
         prefix: str = "",
@@ -42,6 +43,7 @@ def __init__(
         :param api_key: The OpenAI API key.
         :param model: The name of the OpenAI model to use. For more details on the available models,
             see [OpenAI documentation](https://platform.openai.com/docs/guides/embeddings/embedding-models).
+        :param dimensions: The number of dimensions the resulting output embeddings should have. Only supported in text-embedding-3 and later models.
         :param organization: The Organization ID, defaults to `None`. See
         [production best practices](https://platform.openai.com/docs/guides/production-best-practices/setting-up-your-organization).
         :param api_base_url: The OpenAI API Base url, defaults to None. For more details, see OpenAI [docs](https://platform.openai.com/docs/api-reference/audio).
@@ -49,6 +51,7 @@ def __init__(
         :param suffix: A string to add to the end of each text.
         """
         self.model = model
+        self.dimensions = dimensions
         self.organization = organization
         self.prefix = prefix
         self.suffix = suffix
@@ -69,6 +72,7 @@ def to_dict(self) -> Dict[str, Any]:
             organization=self.organization,
             prefix=self.prefix,
             suffix=self.suffix,
+            dimensions=self.dimensions,
             api_key=self.api_key.to_dict(),
         )
 
@@ -92,7 +96,11 @@ def run(self, text: str):
         # replace newlines, which can negatively affect performance.
         text_to_embed = text_to_embed.replace("\n", " ")
 
-        response = self.client.embeddings.create(model=self.model, input=text_to_embed)
+        if self.dimensions is not None:
+            response = self.client.embeddings.create(model=self.model, dimensions=self.dimensions, input=text_to_embed)
+        else:
+            response = self.client.embeddings.create(model=self.model, input=text_to_embed)
+
         meta = {"model": response.model, "usage": dict(response.usage)}
 
         return {"embedding": response.data[0].embedding, "meta": meta}


Test patch:
diff --git a/test/components/embedders/test_openai_document_embedder.py b/test/components/embedders/test_openai_document_embedder.py
--- a/test/components/embedders/test_openai_document_embedder.py
+++ b/test/components/embedders/test_openai_document_embedder.py
@@ -71,6 +71,7 @@ def test_to_dict(self, monkeypatch):
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api_base_url": None,
                 "model": "text-embedding-ada-002",
+                "dimensions": None,
                 "organization": None,
                 "prefix": "",
                 "suffix": "",
@@ -101,6 +102,7 @@ def test_to_dict_with_custom_init_parameters(self, monkeypatch):
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": False, "type": "env_var"},
                 "api_base_url": None,
                 "model": "model",
+                "dimensions": None,
                 "organization": "my-org",
                 "prefix": "prefix",
                 "suffix": "suffix",
diff --git a/test/components/embedders/test_openai_text_embedder.py b/test/components/embedders/test_openai_text_embedder.py
--- a/test/components/embedders/test_openai_text_embedder.py
+++ b/test/components/embedders/test_openai_text_embedder.py
@@ -1,9 +1,9 @@
 import os
-from haystack.utils.auth import Secret
 
 import pytest
 
 from haystack.components.embedders.openai_text_embedder import OpenAITextEmbedder
+from haystack.utils.auth import Secret
 
 
 class TestOpenAITextEmbedder:
@@ -44,6 +44,7 @@ def test_to_dict(self, monkeypatch):
             "type": "haystack.components.embedders.openai_text_embedder.OpenAITextEmbedder",
             "init_parameters": {
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
+                "dimensions": None,
                 "model": "text-embedding-ada-002",
                 "organization": None,
                 "prefix": "",
@@ -66,6 +67,7 @@ def test_to_dict_with_custom_init_parameters(self, monkeypatch):
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": False, "type": "env_var"},
                 "model": "model",
+                "dimensions": None,
                 "organization": "fake-organization",
                 "prefix": "prefix",
                 "suffix": "suffix",