Repository: deepset-ai/haystack

Issue:
Weaviate: Simplify Auth
**Is your feature request related to a problem? Please describe.**
Simplify auth mechanism when using weaviate

**Describe the solution you'd like**
Only support the following auth mechanism:

1. [docker compose](https://weaviate.io/developers/weaviate/installation/docker-compose): optional username and password
2. [WCS](https://weaviate.io/developers/weaviate/installation/weaviate-cloud-services): using API key
3. [embedded](https://weaviate.io/developers/weaviate/installation/embedded): none

**Describe alternatives you've considered**
NA

**Additional context**
NA



Discussion:
Hi @hsm207 

this would mean we deprecate `AuthBearerToken` authentication and we add `AuthApiKey`, correct?
> Hi @hsm207
> 
> this would mean we deprecate `AuthBearerToken` authentication and we add `AuthApiKey`, correct?

@masci yes. And also add a new parameter e.g. use_embedded to used embedded weaviate.
Cool, so the `WeaviateDocumentStore` constructor could look like this:
```py
    def __init__(
        self,
        host: Union[str, List[str]] = "http://localhost",
        port: Union[int, List[int]] = 8080,
        ...
        username: Optional[str] = None,
        password: Optional[str] = None,
        api_key: Optional[str] = None,
        ...
        embedded_options: Optional[dict] = None,  # this dict would contain the same data as EmbeddedOptions
        ...
    ):
```

Can you confirm we would then deprecated the following?
```py
    def __init__(
        ...
        client_secret: Optional[str] = None,
        scope: Optional[str] = "offline_access",
        access_token: Optional[str] = None,
        expires_in: Optional[int] = 60,
        refresh_token: Optional[str] = None,
        ...
    ):
```
Also what would be the precedence order? Say you pass both `embedded_options` and `api_key`, it would be nice to raise an warning and default to one of the two, or maybe just raise an error?
> Can you confirm we would then deprecated the following?

we deprecate everything except `scope`. Since we already have this, we can maintain this in case the user has configured their own on-prem weaviate to use the OIDC Resource Owner Password flow.

> Also what would be the precedence order? Say you pass both embedded_options and api_key, it would be nice to raise an warning and default to one of the two, or maybe just raise an error?

I was thinking the following order is the easiest to code:

1.  Check `embedded_options`. If it is defined, then assume user wants to use embedded weaviate, otherwise, ...
2.  Check `api_key`. If it is defined, then assume user wants to connect using WCS, otherwise, ...
3.  Assume user wants to connect using docker compose (defaults to localhost:8080).

each method will have a INFO log to notify user the method that is used to connect to weaviate.

and we document this precedence in the docs.

So, if the user defines all the params, embedded_options will be used and the rest silently ignored and the user will see an INFO log message saying something like "trying to connect to embedded weaviate".

what do you think?


@hsm207 that sounds good to me, feel free to go ahead with the implementation and let me know how I can help!

Resolving patch:
diff --git a/haystack/document_stores/weaviate.py b/haystack/document_stores/weaviate.py
--- a/haystack/document_stores/weaviate.py
+++ b/haystack/document_stores/weaviate.py
@@ -1,30 +1,28 @@
-from typing import Any, Dict, Generator, List, Optional, Union
-
-import re
-import uuid
-import json
 import hashlib
+import json
 import logging
+import re
+import uuid
+from typing import Any, Dict, Generator, List, Optional, Union
 
 import numpy as np
 from tqdm.auto import tqdm
 
 try:
     import weaviate
-    from weaviate import client, AuthClientPassword, gql, AuthClientCredentials, AuthBearerToken
+    from weaviate import AuthApiKey, AuthBearerToken, AuthClientCredentials, AuthClientPassword, client, gql
 except (ImportError, ModuleNotFoundError) as ie:
     from haystack.utils.import_utils import _optional_component_not_installed
 
     _optional_component_not_installed(__name__, "weaviate", ie)
 
-from haystack.schema import Document, FilterType, Label
 from haystack.document_stores import KeywordDocumentStore
 from haystack.document_stores.base import get_batches_from_generator
 from haystack.document_stores.filter_utils import LogicalFilterClause
 from haystack.document_stores.utils import convert_date_to_rfc3339
 from haystack.errors import DocumentStoreError, HaystackError
 from haystack.nodes.retriever import DenseRetriever
-
+from haystack.schema import Document, FilterType, Label
 
 logger = logging.getLogger(__name__)
 UUID_PATTERN = re.compile(r"^[\da-f]{8}-([\da-f]{4}-){3}[\da-f]{12}$", re.IGNORECASE)
@@ -67,6 +65,13 @@ class WeaviateDocumentStore(KeywordDocumentStore):
     1. Start a Weaviate server (see https://weaviate.io/developers/weaviate/current/getting-started/installation.html)
     2. Init a WeaviateDocumentStore in Haystack
 
+    Connection Parameters Precedence:
+    The selection and priority of connection parameters are as follows:
+    1. If `use_embedded` is set to True, an embedded Weaviate instance will be used, and all other connection parameters will be ignored.
+    2. If `use_embedded` is False or not provided and an `api_key` is provided, the `api_key` will be used to authenticate through AuthApiKey, assuming a connection to a Weaviate Cloud Service (WCS) instance.
+    3. If neither `use_embedded` nor `api_key` is provided, but a `username` and `password` are provided, they will be used to authenticate through AuthClientPassword, assuming an OIDC Resource Owner Password flow.
+    4. If none of the above conditions are met, no authentication method will be used and a connection will be attempted with the provided `host` and `port` values without any authentication.
+
     Limitations:
     The current implementation is not supporting the storage of labels, so you cannot run any evaluation workflows.
     """
@@ -78,11 +83,10 @@ def __init__(
         timeout_config: tuple = (5, 15),
         username: Optional[str] = None,
         password: Optional[str] = None,
-        client_secret: Optional[str] = None,
         scope: Optional[str] = "offline_access",
-        access_token: Optional[str] = None,
-        expires_in: Optional[int] = 60,
-        refresh_token: Optional[str] = None,
+        api_key: Optional[str] = None,
+        use_embedded: bool = False,
+        embedded_options: Optional[dict] = None,
         additional_headers: Optional[Dict[str, Any]] = None,
         index: str = "Document",
         embedding_dim: int = 768,
@@ -106,11 +110,10 @@ def __init__(
         :param timeout_config: The Weaviate timeout config as a tuple of (retries, time out seconds).
         :param username: The Weaviate username (standard authentication using http_auth).
         :param password: Weaviate password (standard authentication using http_auth).
-        :param client_secret: The client secret to use when using the OIDC Client Credentials authentication flow.
         :param scope: The scope of the credentials when using the OIDC Resource Owner Password or Client Credentials authentication flow.
-        :param access_token: Access token to use when using OIDC and bearer tokens to authenticate.
-        :param expires_in: The time in seconds after which the access token expires.
-        :param refresh_token: The refresh token to use when using OIDC and bearer tokens to authenticate.
+        :param api_key: The Weaviate Cloud Services (WCS) API key (for WCS authentication).
+        :param use_embedded: Whether to use an embedded Weaviate instance. Default: False.
+        :param embedded_options: Custom options for the embedded Weaviate instance. Default: None.
         :param additional_headers: Additional headers to be included in the requests sent to Weaviate, for example the bearer token.
         :param index: Index name for document text, embedding, and metadata (in Weaviate terminology, this is a "Class" in the Weaviate schema).
         :param embedding_dim: The embedding vector size. Default: 768.
@@ -144,16 +147,22 @@ def __init__(
         super().__init__()
 
         # Connect to Weaviate server using python binding
-        weaviate_url = f"{host}:{port}"
-        secret = self._get_auth_secret(
-            username, password, client_secret, access_token, expires_in, refresh_token, scope
-        )
+        if not use_embedded:
+            weaviate_url = f"{host}:{port}"
+            auth_client_secret = self._get_auth_secret(username, password, api_key, scope)
+            embedded_options = None
+        else:
+            weaviate_url = None
+            auth_client_secret = None
+            embedded_options = self._get_embedded_options(embedded_options)
+
         # Timeout config can only be defined as a list in YAML, but Weaviate expects a tuple
         if isinstance(timeout_config, list):
             timeout_config = tuple(timeout_config)
         self.weaviate_client = client.Client(
             url=weaviate_url,
-            auth_client_secret=secret,
+            embedded_options=embedded_options,
+            auth_client_secret=auth_client_secret,
             timeout_config=timeout_config,
             additional_headers=additional_headers,
         )
@@ -200,21 +209,20 @@ def __init__(
     def _get_auth_secret(
         username: Optional[str] = None,
         password: Optional[str] = None,
-        client_secret: Optional[str] = None,
-        access_token: Optional[str] = None,
-        expires_in: Optional[int] = 60,
-        refresh_token: Optional[str] = None,
+        api_key: Optional[str] = None,
         scope: Optional[str] = "offline_access",
     ) -> Optional[Union["AuthClientPassword", "AuthClientCredentials", "AuthBearerToken"]]:
-        if username and password:
+        if api_key:
+            return AuthApiKey(api_key=api_key)
+        elif username and password:
             return AuthClientPassword(username, password, scope=scope)
-        elif client_secret:
-            return AuthClientCredentials(client_secret, scope=scope)
-        elif access_token:
-            return AuthBearerToken(access_token, expires_in=expires_in, refresh_token=refresh_token)
-
         return None
 
+    @staticmethod
+    def _get_embedded_options(embedded_options: Optional[Dict[str, Any]] = None) -> weaviate.EmbeddedOptions:
+        embedded_options = embedded_options or {}
+        return weaviate.EmbeddedOptions(**embedded_options)
+
     def _sanitize_index_name(self, index: Optional[str]) -> Optional[str]:
         if index is None:
             return None


Test patch:
diff --git a/test/document_stores/test_weaviate.py b/test/document_stores/test_weaviate.py
--- a/test/document_stores/test_weaviate.py
+++ b/test/document_stores/test_weaviate.py
@@ -1,9 +1,9 @@
-import uuid
 import json
+import uuid
 from unittest import mock
 
-import pytest
 import numpy as np
+import pytest
 import weaviate
 
 from haystack.document_stores.weaviate import WeaviateDocumentStore
@@ -273,20 +273,29 @@ def test__get_auth_secret(self):
         secret = WeaviateDocumentStore._get_auth_secret("user", "pass", scope="some_scope")
         assert isinstance(secret, weaviate.AuthClientPassword)
 
-        # Test with client_secret
-        secret = WeaviateDocumentStore._get_auth_secret(client_secret="client_secret_value", scope="some_scope")
-        assert isinstance(secret, weaviate.AuthClientCredentials)
-
-        # Test with access_token
-        secret = WeaviateDocumentStore._get_auth_secret(
-            access_token="access_token_value", expires_in=3600, refresh_token="refresh_token_value"
-        )
-        assert isinstance(secret, weaviate.AuthBearerToken)
+        # Test with api key
+        secret = WeaviateDocumentStore._get_auth_secret(api_key="wcs_api_key")
+        assert isinstance(secret, weaviate.AuthApiKey)
 
         # Test with no authentication method
         secret = WeaviateDocumentStore._get_auth_secret()
         assert secret is None
 
+    @pytest.mark.unit
+    @pytest.mark.parametrize(
+        "embedded_options, expected_options",
+        [
+            (None, weaviate.EmbeddedOptions()),
+            (
+                {"hostname": "http://localhost", "port": "8080"},
+                weaviate.EmbeddedOptions(hostname="http://localhost", port="8080"),
+            ),
+        ],
+    )
+    def test__get_embedded_options(self, embedded_options, expected_options):
+        options = WeaviateDocumentStore._get_embedded_options(embedded_options)
+        assert options == expected_options
+
     @pytest.mark.unit
     def test__get_current_properties(self, mocked_ds):
         mocked_ds.weaviate_client.schema.get.return_value = json.loads(