Repository: weaviate/weaviate (Go)

Issue:
Feature request: Authentication scheme for systems long-term credentials
I'm trying to integrate Firestore with Weaviate, similarly to how it's done with [Elastic Search](https://firebase.google.com/docs/firestore/solutions/search?provider=elastic). However, Weaviate seems to only support authenticating with a short-term JWT token. 

I couldn't find an easy programmatic way of generating JWT tokens that Weaviate would accept. In my use case I was using WCS which is using SeMI Auth OpenID provider.

A simple mechanism could be supporting a static username/password to authenticate to Weaviate or to utilize mTLS for even higher levels of security. It was unclear whether this is already supported in Weaviate and the client libraries. For Firebase functions we have to use Javascript so that's what I tried using.

The core requirement here is to have some long-term credential that can be used to authenticate to a public Weaviate instance.




Discussion: Looks like the Python Client does have a mechanism for refreshing the Access Token automatically for you if you supply it with your username/password: https://github.com/semi-technologies/weaviate-python-client/blob/5f6c93903dbc242d2df03a7ba9fd4fce0ceaf25b/weaviate/connect/connection.py#L166

and https://github.com/semi-technologies/weaviate-python-client/blob/5f6c93903dbc242d2df03a7ba9fd4fce0ceaf25b/weaviate/wcs/crud_wcs.py#L78

So that might be workable for usage with Javascript client as well, however doesn't look like the Javascript client has similar logic yet. Sadly, firebase functions only support javascript or typescript.
Added the backlog label to be able to vote for this on the roadmap. A vote for this issue is not a vote for a specific solution (we'll figure out what the right solution is when the time comes). But rather a vote is a feature that satisfies this:

> The core requirement here is to have some long-term credential that can be used to authenticate to a public Weaviate instance.
@samos123 do I understand correctly that the Elasticsearch example is just using basic auth? That would be quite cheap to add to Weaviate as well...
It's just using no auth right now, which isn't something you should use in a production system that's exposed over the internet. The firebase example requires an internet exposed weaviate instance.
Would firebase work with a self-hosted Weaviate instance as well or would it have to be a Weaviate-Cloud-Service provisioned one? The latter would make this a bit more complex because then we'd also have to take the auth for the WCS into account. But if self-hosted works, then we could build the following:

* Weaviate will support basic auth, user/pwd combinations can be specified using env vars 
* User self-hosts Weaviate with the above auth configured and exposes it on the internet
* The firebase connector uses basic auth to authenticate against the instance with the credentials set in the first step.

Probably not the prettiest solution, but one that would be minimal effort to build. 
Here are the requirements for Firebase integration:
* Weaviate exposed on the internet so it's accessible from Firebase functions
* Utilize the Javascript client so the JS client needs to be able to support the authentication mechanism

So yes self-hosted would work because this is just an example integration. That being said, the blog I wrote did use WCS but with authentication disabled, which was fine for demo/example usage.

I do see value in starting with basic auth as a simple option to expose a self-hosted Weaviate on the internet. I'm sure there are people who expose Weaviate publicly without really understanding that anyone can access their database. This option should make it very easy for self-hosted WCS users to at least prevent anonymous users to access the DB without requiring a full fledged OIDC server. I don't like dealing with OIDC unless I really have to.

Some customers hyper focused on security have very specific requirements on what kind of credentials are acceptable and what not. Maybe a guide that shows how to setup mTLS would be helpful later. Would mTLS work already? I think you could deploy a TLS proxy in front of Weaviate that does mTLS? What do you think? Each client would get it's own unique cert that could get revoked and traffic would be encrypted as well.
+1, need  "basic auth, user/pwd combinations can be specified using env vars"
Thanks for the feedback, please upvote the first post in this thread so your "+1" is counted :-)
Hi @samos123 and @wyzhangyuhan 

The JS client supports authentication with username/password with automatic refresh [since the latest version](https://github.com/semi-technologies/weaviate-javascript-client/releases/tag/v2.14.2), examples how to use are [here](https://github.com/semi-technologies/weaviate-javascript-client/pull/100). Any feedback would be appreciated, documentation will be added in the next days/weeks!

Java and go clients should also follow in the next weeks.

We do not support using env variables, is that something that people would prefer?

While testing the clients, I played around with [Okta](https://developer.okta.com/) as an identity provider which worked qute well. They also offer [dev accounts](https://developer.okta.com/) to test things out.

[Here is an example of docker compose file](https://github.com/semi-technologies/weaviate-python-client/blob/main/ci/docker-compose-okta-users.yml) that works with okta and a step-by-step guide on how to setup okta (replace the client_id, issuer_url and user in the docker compose file with your values):
<img width="817" alt="image" src="https://user-images.githubusercontent.com/5353192/210620927-52ddf95f-5cde-4358-962c-cdea1a202e62.png">

Nice! I think supporting env vars isn't important since you could easily do that yourself while calling the client. It does seem common to support env vars though. E.g. AWS and gcloud all support providing credentials through env vars directly.

OIDC is painful to setup and requires yet another component that can break. Some users just want a simple password or username/password to protect their public weaviate instance. Especially at larger enterprises having to integrate with corporate OIDC provider can be a very tedious and painful process. I've setup Dex and used it as an OIDC provider for k8s in the past.
Totally understand, just want to highlight what we do support right now.

Another option that does not involve configuring an OIDC provider/server yourself would be to use our auth server with your own weaviate instance, like in this example [docker-compose file](https://github.com/semi-technologies/weaviate-python-client/blob/main/ci/docker-compose-wcs.yml). Then you can authenticate with your WCS username/pw (client credentials will hopefully be added to WCS in a bit).

Obviously that is nothing that works in a large enterprise, but for a small setup it might be an easy start. 
having a user/password or a build-in key mechanism would really helpful!



Fix (patch):
diff --git a/adapters/handlers/rest/configure_api.go b/adapters/handlers/rest/configure_api.go
index e7e74c3b8b7..2110f6331a7 100644
--- a/adapters/handlers/rest/configure_api.go
+++ b/adapters/handlers/rest/configure_api.go
@@ -39,7 +39,6 @@ import (
 	modulestorage "github.com/weaviate/weaviate/adapters/repos/modules"
 	schemarepo "github.com/weaviate/weaviate/adapters/repos/schema"
 	"github.com/weaviate/weaviate/entities/dto"
-	"github.com/weaviate/weaviate/entities/models"
 	"github.com/weaviate/weaviate/entities/moduletools"
 	"github.com/weaviate/weaviate/entities/search"
 	enthnsw "github.com/weaviate/weaviate/entities/vectorindex/hnsw"
@@ -139,9 +138,9 @@ func configureAPI(api *operations.WeaviateAPI) http.Handler {
 
 	api.JSONConsumer = runtime.JSONConsumer()
 
-	api.OidcAuth = func(token string, scopes []string) (*models.Principal, error) {
-		return appState.OIDC.ValidateAndExtract(token, scopes)
-	}
+	api.OidcAuth = NewTokenAuthComposer(
+		appState.ServerConfig.Config.Authentication,
+		appState.APIKey, appState.OIDC)
 
 	api.Logger = func(msg string, args ...interface{}) {
 		appState.Logger.WithField("action", "restapi_management").Infof(msg, args...)
@@ -363,6 +362,7 @@ func startupRoutine(ctx context.Context) *state.State {
 		Debug("config loaded")
 
 	appState.OIDC = configureOIDC(appState)
+	appState.APIKey = configureAPIKey(appState)
 	appState.AnonymousAccess = configureAnonymousAccess(appState)
 	appState.Authorizer = configureAuthorizer(appState)
 
diff --git a/adapters/handlers/rest/configure_server.go b/adapters/handlers/rest/configure_server.go
index a1f06428cc8..00f3ada1fa3 100644
--- a/adapters/handlers/rest/configure_server.go
+++ b/adapters/handlers/rest/configure_server.go
@@ -23,6 +23,7 @@ import (
 	"github.com/weaviate/weaviate/adapters/handlers/rest/state"
 	"github.com/weaviate/weaviate/entities/schema"
 	"github.com/weaviate/weaviate/usecases/auth/authentication/anonymous"
+	"github.com/weaviate/weaviate/usecases/auth/authentication/apikey"
 	"github.com/weaviate/weaviate/usecases/auth/authentication/oidc"
 	"github.com/weaviate/weaviate/usecases/auth/authorization"
 	"github.com/weaviate/weaviate/usecases/config"
@@ -84,6 +85,16 @@ func configureOIDC(appState *state.State) *oidc.Client {
 	return c
 }
 
+func configureAPIKey(appState *state.State) *apikey.Client {
+	c, err := apikey.New(appState.ServerConfig.Config)
+	if err != nil {
+		appState.Logger.WithField("action", "oidc_init").WithError(err).Fatal("oidc client could not start up")
+		os.Exit(1)
+	}
+
+	return c
+}
+
 // configureAnonymousAccess will always be called, even if anonymous access is
 // disabled. In this case the middleware provided by this client will block
 // anonymous requests
diff --git a/adapters/handlers/rest/state/state.go b/adapters/handlers/rest/state/state.go
index 3d3b792e9dd..1f673e63ef1 100644
--- a/adapters/handlers/rest/state/state.go
+++ b/adapters/handlers/rest/state/state.go
@@ -17,6 +17,7 @@ import (
 	"github.com/weaviate/weaviate/adapters/repos/classifications"
 	"github.com/weaviate/weaviate/adapters/repos/db"
 	"github.com/weaviate/weaviate/usecases/auth/authentication/anonymous"
+	"github.com/weaviate/weaviate/usecases/auth/authentication/apikey"
 	"github.com/weaviate/weaviate/usecases/auth/authentication/oidc"
 	"github.com/weaviate/weaviate/usecases/auth/authorization"
 	"github.com/weaviate/weaviate/usecases/backup"
@@ -37,6 +38,7 @@ import (
 type State struct {
 	OIDC                  *oidc.Client
 	AnonymousAccess       *anonymous.Client
+	APIKey                *apikey.Client
 	Authorizer            authorization.Authorizer
 	ServerConfig          *config.WeaviateConfig
 	Locks                 locks.ConnectorSchemaLock
diff --git a/adapters/handlers/rest/token_validation.go b/adapters/handlers/rest/token_validation.go
new file mode 100644
index 00000000000..b19a8dd334a
--- /dev/null
+++ b/adapters/handlers/rest/token_validation.go
@@ -0,0 +1,65 @@
+//                           _       _
+// __      _____  __ ___   ___  __ _| |_ ___
+// \ \ /\ / / _ \/ _` \ \ / / |/ _` | __/ _ \
+//  \ V  V /  __/ (_| |\ V /| | (_| | ||  __/
+//   \_/\_/ \___|\__,_| \_/ |_|\__,_|\__\___|
+//
+//  Copyright © 2016 - 2023 Weaviate B.V. All rights reserved.
+//
+//  CONTACT: hello@weaviate.io
+//
+
+package rest
+
+import (
+	"github.com/golang-jwt/jwt/v4"
+	"github.com/pkg/errors"
+	"github.com/weaviate/weaviate/entities/models"
+	"github.com/weaviate/weaviate/usecases/config"
+)
+
+type openAPITokenFunc func(token string, scopes []string) (*models.Principal, error)
+
+// NewTokenAuthComposer provides an OpenAPI compatible token validation
+// function that validates the token either as OIDC or as an APIKey token
+// depending on which is configured. If both are configured, the scheme is
+// figured out at runtime.
+func NewTokenAuthComposer(config config.Authentication,
+	apikey apiKeyValidator, oidc oidcValidator,
+) openAPITokenFunc {
+	if config.APIKey.Enabled && config.OIDC.Enabled {
+		return pickAuthSchemeDynamically(apikey, oidc)
+	}
+
+	if config.APIKey.Enabled {
+		return apikey.ValidateAndExtract
+	}
+
+	// default to OIDC, even if no scheme is enabled, then it can deal with this
+	// scenario itself. This is the backward-compatible scenario.
+	return oidc.ValidateAndExtract
+}
+
+func pickAuthSchemeDynamically(
+	apiKey apiKeyValidator, oidc oidcValidator,
+) openAPITokenFunc {
+	return func(token string, scopes []string) (*models.Principal, error) {
+		_, err := jwt.Parse(token, func(t *jwt.Token) (interface{}, error) {
+			return nil, nil
+		})
+
+		if err != nil && errors.Is(err, jwt.ErrTokenMalformed) {
+			return apiKey.ValidateAndExtract(token, scopes)
+		}
+
+		return oidc.ValidateAndExtract(token, scopes)
+	}
+}
+
+type oidcValidator interface {
+	ValidateAndExtract(token string, scopes []string) (*models.Principal, error)
+}
+
+type apiKeyValidator interface {
+	ValidateAndExtract(token string, scopes []string) (*models.Principal, error)
+}
diff --git a/go.mod b/go.mod
index e3d2b1b1064..702d48a7ca1 100644
--- a/go.mod
+++ b/go.mod
@@ -15,7 +15,7 @@ require (
 	github.com/go-openapi/strfmt v0.21.3
 	github.com/go-openapi/swag v0.22.3
 	github.com/go-openapi/validate v0.21.0
-	github.com/golang-jwt/jwt/v4 v4.0.0
+	github.com/golang-jwt/jwt/v4 v4.5.0
 	github.com/google/uuid v1.3.0
 	github.com/hashicorp/memberlist v0.5.0
 	github.com/jessevdk/go-flags v1.4.0
diff --git a/go.sum b/go.sum
index 05bb72c9f80..f28da62bb4d 100644
--- a/go.sum
+++ b/go.sum
@@ -455,6 +455,8 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
 github.com/golang-jwt/jwt/v4 v4.0.0 h1:RAqyYixv1p7uEnocuy8P1nru5wprCh/MH2BIlW5z5/o=
 github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
+github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
+github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
diff --git a/tools/dev/run_dev_server.sh b/tools/dev/run_dev_server.sh
index de9ebf44fd2..d0d15f500d1 100755
--- a/tools/dev/run_dev_server.sh
+++ b/tools/dev/run_dev_server.sh
@@ -232,6 +232,37 @@ case $CONFIG in
         --port 8080
     ;;
 
+  local-apikey)
+      AUTHENTICATION_APIKEY_ENABLED=true \
+      AUTHENTICATION_APIKEY_ALLOWED_KEYS=my-secret-key,your-secret-key \
+      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=false \
+      DEFAULT_VECTORIZER_MODULE=none \
+      go_run ./cmd/weaviate-server \
+        --scheme http \
+        --host "127.0.0.1" \
+        --port 8080
+    ;;
+
+  local-wcs-oidc-and-apikey)
+      AUTHENTICATION_APIKEY_ENABLED=true \
+      AUTHENTICATION_APIKEY_ALLOWED_KEYS=my-secret-key,my-secret-read-only-key \
+      AUTHENTICATION_APIKEY_USERS=etienne@semi.technology,etienne+read-only@semi.technology \
+      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=false \
+      AUTHENTICATION_OIDC_ENABLED=true \
+      AUTHENTICATION_OIDC_ISSUER=https://auth.wcs.api.weaviate.io/auth/realms/SeMI\
+      AUTHENTICATION_OIDC_USERNAME_CLAIM=email \
+      AUTHENTICATION_OIDC_GROUPS_CLAIM=groups \
+      AUTHENTICATION_OIDC_CLIENT_ID=wcs \
+      AUTHORIZATION_ADMINLIST_ENABLED=true \
+      AUTHORIZATION_ADMINLIST_USERS=etienne@semi.technology \
+      AUTHORIZATION_ADMINLIST_READONLY_USERS=etienne+read-only@semi.technology \
+      DEFAULT_VECTORIZER_MODULE=none \
+      go_run ./cmd/weaviate-server \
+        --scheme http \
+        --host "127.0.0.1" \
+        --port 8080
+    ;;
+
   local-multi-text)
       CONTEXTIONARY_URL=localhost:9999 \
       AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
diff --git a/usecases/auth/authentication/apikey/client.go b/usecases/auth/authentication/apikey/client.go
new file mode 100644
index 00000000000..66cb811aae0
--- /dev/null
+++ b/usecases/auth/authentication/apikey/client.go
@@ -0,0 +1,117 @@
+//                           _       _
+// __      _____  __ ___   ___  __ _| |_ ___
+// \ \ /\ / / _ \/ _` \ \ / / |/ _` | __/ _ \
+//  \ V  V /  __/ (_| |\ V /| | (_| | ||  __/
+//   \_/\_/ \___|\__,_| \_/ |_|\__,_|\__\___|
+//
+//  Copyright © 2016 - 2023 Weaviate B.V. All rights reserved.
+//
+//  CONTACT: hello@weaviate.io
+//
+
+package apikey
+
+import (
+	"crypto/sha256"
+	"crypto/subtle"
+	"fmt"
+
+	errors "github.com/go-openapi/errors"
+	"github.com/weaviate/weaviate/entities/models"
+	"github.com/weaviate/weaviate/usecases/config"
+)
+
+type Client struct {
+	config     config.APIKey
+	keystorage [][sha256.Size]byte
+}
+
+func New(cfg config.Config) (*Client, error) {
+	c := &Client{
+		config: cfg.Authentication.APIKey,
+	}
+
+	if err := c.validateConfig(); err != nil {
+		return nil, fmt.Errorf("invalid apikey config: %w", err)
+	}
+
+	c.parseKeys()
+
+	return c, nil
+}
+
+func (c *Client) parseKeys() {
+	c.keystorage = make([][sha256.Size]byte, len(c.config.AllowedKeys))
+	for i, rawKey := range c.config.AllowedKeys {
+		c.keystorage[i] = sha256.Sum256([]byte(rawKey))
+	}
+}
+
+func (c *Client) validateConfig() error {
+	if !c.config.Enabled {
+		// don't validate if this scheme isn't used
+		return nil
+	}
+
+	if len(c.config.AllowedKeys) < 1 {
+		return fmt.Errorf("need at least one valid allowed key")
+	}
+
+	for _, key := range c.config.AllowedKeys {
+		if len(key) == 0 {
+			return fmt.Errorf("keys cannot have length 0")
+		}
+	}
+
+	if len(c.config.Users) < 1 {
+		return fmt.Errorf("need at least one user")
+	}
+
+	for _, key := range c.config.Users {
+		if len(key) == 0 {
+			return fmt.Errorf("users cannot have length 0")
+		}
+	}
+
+	if len(c.config.Users) > 1 && len(c.config.Users) != len(c.config.AllowedKeys) {
+		return fmt.Errorf("length of users and keys must match, alternatively provide single user for all keys")
+	}
+
+	return nil
+}
+
+func (c *Client) ValidateAndExtract(token string, scopes []string) (*models.Principal, error) {
+	if !c.config.Enabled {
+		return nil, errors.New(401, "apikey auth is not configured, please try another auth scheme or set up weaviate with apikey configured")
+	}
+
+	tokenPos, ok := c.isTokenAllowed(token)
+	if !ok {
+		return nil, errors.New(401, "invalid api key, please provide a valid api key")
+	}
+
+	return &models.Principal{
+		Username: c.getUser(tokenPos),
+	}, nil
+}
+
+func (c *Client) isTokenAllowed(token string) (int, bool) {
+	tokenHash := sha256.Sum256([]byte(token))
+
+	for i, allowed := range c.keystorage {
+		if subtle.ConstantTimeCompare(tokenHash[:], allowed[:]) == 1 {
+			return i, true
+		}
+	}
+
+	return -1, false
+}
+
+func (c *Client) getUser(pos int) string {
+	// passed validation guarantees that one of those options will work
+	if pos >= len(c.config.Users) {
+		return c.config.Users[0]
+	}
+
+	return c.config.Users[pos]
+}
diff --git a/usecases/config/authentication.go b/usecases/config/authentication.go
index 07438eb2e6c..2e605f29a8e 100644
--- a/usecases/config/authentication.go
+++ b/usecases/config/authentication.go
@@ -17,6 +17,7 @@ import "fmt"
 type Authentication struct {
 	OIDC            OIDC            `json:"oidc" yaml:"oidc"`
 	AnonymousAccess AnonymousAccess `json:"anonymous_access" yaml:"anonymous_access"`
+	APIKey          APIKey
 }
 
 // Validate the Authentication configuration. This only validates at a general
@@ -31,7 +32,7 @@ func (a Authentication) Validate() error {
 }
 
 func (a Authentication) anyAuthMethodSelected() bool {
-	return a.AnonymousAccess.Enabled || a.OIDC.Enabled
+	return a.AnonymousAccess.Enabled || a.OIDC.Enabled || a.APIKey.Enabled
 }
 
 // AnonymousAccess considers users without any auth information as
@@ -52,3 +53,9 @@ type OIDC struct {
 	GroupsClaim       string   `yaml:"groups_claim" json:"groups_claim"`
 	Scopes            []string `yaml:"scopes" json:"scopes"`
 }
+
+type APIKey struct {
+	Enabled     bool     `json:"enabled" yaml:"enabled"`
+	Users       []string `json:"users" yaml:"users"`
+	AllowedKeys []string `json:"allowed_keys" yaml:"allowed_keys"`
+}
diff --git a/usecases/config/environment.go b/usecases/config/environment.go
index 26b8f10b5cf..03c5cbae6c7 100644
--- a/usecases/config/environment.go
+++ b/usecases/config/environment.go
@@ -82,15 +82,32 @@ func FromEnv(config *Config) error {
 		}
 	}
 
+	if enabled(os.Getenv("AUTHENTICATION_APIKEY_ENABLED")) {
+		config.Authentication.APIKey.Enabled = true
+
+		if keysString, ok := os.LookupEnv("AUTHENTICATION_APIKEY_ALLOWED_KEYS"); ok {
+			keys := strings.Split(keysString, ",")
+			config.Authentication.APIKey.AllowedKeys = keys
+		}
+
+		if keysString, ok := os.LookupEnv("AUTHENTICATION_APIKEY_USERS"); ok {
+			keys := strings.Split(keysString, ",")
+			config.Authentication.APIKey.Users = keys
+		}
+	}
+
 	if enabled(os.Getenv("AUTHORIZATION_ADMINLIST_ENABLED")) {
 		config.Authorization.AdminList.Enabled = true
 
-		users := strings.Split(os.Getenv("AUTHORIZATION_ADMINLIST_USERS"), ",")
-		roUsers := strings.Split(os.Getenv("AUTHORIZATION_ADMINLIST_READONLY_USERS"),
-			",")
+		usersString, ok := os.LookupEnv("AUTHORIZATION_ADMINLIST_USERS")
+		if ok {
+			config.Authorization.AdminList.Users = strings.Split(usersString, ",")
+		}
 
-		config.Authorization.AdminList.ReadOnlyUsers = roUsers
-		config.Authorization.AdminList.Users = users
+		roUsersString, ok := os.LookupEnv("AUTHORIZATION_ADMINLIST_READONLY_USERS")
+		if ok {
+			config.Authorization.AdminList.ReadOnlyUsers = strings.Split(roUsersString, ",")
+		}
 	}
 
 	clusterCfg, err := parseClusterConfig()