Repository: deepset-ai/haystack

Issue:
Add `dimensions` parameter to OpenAI Embedders to fully support the new models
**Is your feature request related to a problem? Please describe.**
There's no full support for the new OpenAI embedding models. 

**Describe the solution you'd like**
[OpenAI Docs](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings):

> By default, the length of the embedding vector will be 1536 for text-embedding-3-small or 3072 for text-embedding-3-large. You can reduce the dimensions of the embedding by passing in the [dimensions parameter](https://platform.openai.com/docs/api-reference/embeddings/create#embeddings-create-dimensions) without the embedding losing its concept-representing properties.

We need to add `dimensions` parameter to OpenAI Embedders to fully support new `text-embedding-3-small` and `text-embedding-3-large` models

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Discussion:
coveralls: ## Pull Request Test Coverage Report for [Build 7695260836](https://coveralls.io/builds/65351566)


* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **18** unchanged lines in **2** files lost coverage.
* Overall coverage decreased (**-0.07%**) to **88.852%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [components/embedders/openai_text_embedder.py](https://coveralls.io/builds/65351566/source?filename=components%2Fembedders%2Fopenai_text_embedder.py#L65) | 8 | 70.37% |
| [components/embedders/openai_document_embedder.py](https://coveralls.io/builds/65351566/source?filename=components%2Fembedders%2Fopenai_document_embedder.py#L79) | 10 | 75.0% |
<!-- | **Total:** | **18** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/65351566/badge)](https://coveralls.io/builds/65351566) |
| :-- | --: |
| Change from base [Build 7669690020](https://coveralls.io/builds/65321155): |  -0.07% |
| Covered Lines: | 4551 |
| Relevant Lines: | 5122 |

---
##### 💛  - [Coveralls](https://coveralls.io)

sahusiddharth: @silvanocerza and @dfokina can you help me understand why linting is failing. before making this PR I remember doing linting. 
silvanocerza: > @silvanocerza and @dfokina can you help me understand why linting is failing. before making this PR I remember doing linting.

I'll handle it, no worries. :)
silvanocerza: @sahusiddharth MyPy can't infer the correct type of `self.dimensions` if you use `is not None` to verify it's set.
Should be alright now.
silvanocerza: Rebased to fix conflicts.

Patch:
diff --git a/haystack/components/embedders/openai_document_embedder.py b/haystack/components/embedders/openai_document_embedder.py
index e4d3ad13..7f409178 100644
--- a/haystack/components/embedders/openai_document_embedder.py
+++ b/haystack/components/embedders/openai_document_embedder.py
@@ -33,6 +33,7 @@ class OpenAIDocumentEmbedder:
         self,
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
         model: str = "text-embedding-ada-002",
+        dimensions: Optional[int] = None,
         api_base_url: Optional[str] = None,
         organization: Optional[str] = None,
         prefix: str = "",
@@ -46,6 +47,7 @@ class OpenAIDocumentEmbedder:
         Create a OpenAIDocumentEmbedder component.
         :param api_key: The OpenAI API key.
         :param model: The name of the model to use.
+        :param dimensions: The number of dimensions the resulting output embeddings should have. Only supported in text-embedding-3 and later models.
         :param api_base_url: The OpenAI API Base url, defaults to None. For more details, see OpenAI [docs](https://platform.openai.com/docs/api-reference/audio).
         :param organization: The Organization ID, defaults to `None`. See
         [production best practices](https://platform.openai.com/docs/guides/production-best-practices/setting-up-your-organization).
@@ -59,6 +61,7 @@ class OpenAIDocumentEmbedder:
         """
         self.api_key = api_key
         self.model = model
+        self.dimensions = dimensions
         self.api_base_url = api_base_url
         self.organization = organization
         self.prefix = prefix
@@ -84,6 +87,7 @@ class OpenAIDocumentEmbedder:
         return default_to_dict(
             self,
             model=self.model,
+            dimensions=self.dimensions,
             organization=self.organization,
             api_base_url=self.api_base_url,
             prefix=self.prefix,
@@ -131,7 +135,10 @@ class OpenAIDocumentEmbedder:
             range(0, len(texts_to_embed), batch_size), disable=not self.progress_bar, desc="Calculating embeddings"
         ):
             batch = texts_to_embed[i : i + batch_size]
-            response = self.client.embeddings.create(model=self.model, input=batch)
+            if self.dimensions is not None:
+                response = self.client.embeddings.create(model=self.model, dimensions=self.dimensions, input=batch)
+            else:
+                response = self.client.embeddings.create(model=self.model, input=batch)
             embeddings = [el.embedding for el in response.data]
             all_embeddings.extend(embeddings)
 
diff --git a/haystack/components/embedders/openai_text_embedder.py b/haystack/components/embedders/openai_text_embedder.py
index 02b7d3ed..5d30bccf 100644
--- a/haystack/components/embedders/openai_text_embedder.py
+++ b/haystack/components/embedders/openai_text_embedder.py
@@ -1,8 +1,8 @@
-from typing import List, Optional, Dict, Any
+from typing import Any, Dict, List, Optional
 
 from openai import OpenAI
 
-from haystack import component, default_to_dict, default_from_dict
+from haystack import component, default_from_dict, default_to_dict
 from haystack.utils import Secret, deserialize_secrets_inplace
 
 
@@ -31,6 +31,7 @@ class OpenAITextEmbedder:
         self,
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
         model: str = "text-embedding-ada-002",
+        dimensions: Optional[int] = None,
         api_base_url: Optional[str] = None,
         organization: Optional[str] = None,
         prefix: str = "",
@@ -42,6 +43,7 @@ class OpenAITextEmbedder:
         :param api_key: The OpenAI API key.
         :param model: The name of the OpenAI model to use. For more details on the available models,
             see [OpenAI documentation](https://platform.openai.com/docs/guides/embeddings/embedding-models).
+        :param dimensions: The number of dimensions the resulting output embeddings should have. Only supported in text-embedding-3 and later models.
         :param organization: The Organization ID, defaults to `None`. See
         [production best practices](https://platform.openai.com/docs/guides/production-best-practices/setting-up-your-organization).
         :param api_base_url: The OpenAI API Base url, defaults to None. For more details, see OpenAI [docs](https://platform.openai.com/docs/api-reference/audio).
@@ -49,6 +51,7 @@ class OpenAITextEmbedder:
         :param suffix: A string to add to the end of each text.
         """
         self.model = model
+        self.dimensions = dimensions
         self.organization = organization
         self.prefix = prefix
         self.suffix = suffix
@@ -69,6 +72,7 @@ class OpenAITextEmbedder:
             organization=self.organization,
             prefix=self.prefix,
             suffix=self.suffix,
+            dimensions=self.dimensions,
             api_key=self.api_key.to_dict(),
         )
 
@@ -92,7 +96,11 @@ class OpenAITextEmbedder:
         # replace newlines, which can negatively affect performance.
         text_to_embed = text_to_embed.replace("\n", " ")
 
-        response = self.client.embeddings.create(model=self.model, input=text_to_embed)
+        if self.dimensions is not None:
+            response = self.client.embeddings.create(model=self.model, dimensions=self.dimensions, input=text_to_embed)
+        else:
+            response = self.client.embeddings.create(model=self.model, input=text_to_embed)
+
         meta = {"model": response.model, "usage": dict(response.usage)}
 
         return {"embedding": response.data[0].embedding, "meta": meta}
diff --git a/releasenotes/notes/add-dimensions-parameter-to-OpenAI-Embedders-to-fully-support-the-new-models-1393cc235e457733.yaml b/releasenotes/notes/add-dimensions-parameter-to-OpenAI-Embedders-to-fully-support-the-new-models-1393cc235e457733.yaml
new file mode 100644
index 00000000..777047af
--- /dev/null
+++ b/releasenotes/notes/add-dimensions-parameter-to-OpenAI-Embedders-to-fully-support-the-new-models-1393cc235e457733.yaml
@@ -0,0 +1,4 @@
+---
+enhancements:
+  - |
+    add dimensions parameter to OpenAI Embedders to fully support new embedding models like text-embedding-3-small, text-embedding-3-large and upcoming ones


Test patch:
diff --git a/test/components/embedders/test_openai_document_embedder.py b/test/components/embedders/test_openai_document_embedder.py
index 833a485b..3e0a55f4 100644
--- a/test/components/embedders/test_openai_document_embedder.py
+++ b/test/components/embedders/test_openai_document_embedder.py
@@ -71,6 +71,7 @@ class TestOpenAIDocumentEmbedder:
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api_base_url": None,
                 "model": "text-embedding-ada-002",
+                "dimensions": None,
                 "organization": None,
                 "prefix": "",
                 "suffix": "",
@@ -101,6 +102,7 @@ class TestOpenAIDocumentEmbedder:
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": False, "type": "env_var"},
                 "api_base_url": None,
                 "model": "model",
+                "dimensions": None,
                 "organization": "my-org",
                 "prefix": "prefix",
                 "suffix": "suffix",
diff --git a/test/components/embedders/test_openai_text_embedder.py b/test/components/embedders/test_openai_text_embedder.py
index 5cd10e94..3a1e747f 100644
--- a/test/components/embedders/test_openai_text_embedder.py
+++ b/test/components/embedders/test_openai_text_embedder.py
@@ -1,9 +1,9 @@
 import os
-from haystack.utils.auth import Secret
 
 import pytest
 
 from haystack.components.embedders.openai_text_embedder import OpenAITextEmbedder
+from haystack.utils.auth import Secret
 
 
 class TestOpenAITextEmbedder:
@@ -44,6 +44,7 @@ class TestOpenAITextEmbedder:
             "type": "haystack.components.embedders.openai_text_embedder.OpenAITextEmbedder",
             "init_parameters": {
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
+                "dimensions": None,
                 "model": "text-embedding-ada-002",
                 "organization": None,
                 "prefix": "",
@@ -66,6 +67,7 @@ class TestOpenAITextEmbedder:
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": False, "type": "env_var"},
                 "model": "model",
+                "dimensions": None,
                 "organization": "fake-organization",
                 "prefix": "prefix",
                 "suffix": "suffix",