Repository: TecharoHQ/anubis

Issue:
State layer for Anubis
I've been thinking about how to introduce state into Anubis for a while and the core problem here is that I want to have a very strong balance between these two big factors:

* Administrator experience: EG - Anubis should use tools that are as boring as possible and network-transparent as possible yet allow for piggybacking on existing infrastructure as much as possible
* Developer experience: EG - I don't want this to be a pain to implement, understand, and maintain

In light of that, I think the best way to have some kind of distributed state for Anubis is to use [Valkey](https://valkey.io/) (née Redis). This will enable stateful operations like [rate](https://github.com/TecharoHQ/anubis/issues/102) [limiting](https://github.com/TecharoHQ/anubis/issues/26). This could also enable clever cross-domain trickery like checking a client's challenge value against the Valkey database and "just waving them through" because they've passed a challenge elsewhere.

This could also allow Anubis to keep track of user agent strings it has seen and apply IP based rate limits based on the request load from each IP address and user agent vendor pair.

What do you think? I'm thinking Valkey because it is a very common service that is already deployed to the public Internet and a common dependency in many web frameworks. Maybe there would be a fallback using something like [bbolt](https://github.com/etcd-io/bbolt) for users that don't choose to use Valkey.

Hints from issue comments:
@JasonLovesDoggo: I think letting the end-user choose is fine. 

Imo we should support RESP (redis, valkey, other compatible servers) and use bbolt as a fallback if nothing is provided. 

The only constraint here is the only way to do the cross domain verification is if you share a kv instance between them (and bbolt would not support this).
@gmemstr: Bonus points if the Valkey implementation can support Sentinel for high availability. Should be as easy as a config option passed to the Valkey client library.@JasonLovesDoggo: > Bonus points if the Valkey implementation can support Sentinel for high availability. Should be as easy as a config option passed to the Valkey client library.

[go-redis](https://redis.uptrace.dev/guide/go-redis-sentinel.html#redis-server-client) seems to support it quite easily. No reason to not support it as an option imo. However we should make it strictly optional as some other redis alternatives like garnet don't support it https://github.com/microsoft/garnet/issues/91@daenney: Instead of bbolt as a fallback, you could consider something like https://github.com/nalgeon/redka. That can run with https://github.com/ncruces/go-sqlite3/, which gives you a WASM-based SQLite library, or https://gitlab.com/cznic/sqlite, allowing for CGO-free builds.@daegalus: https://github.com/buraksezer/olric

I don't know why, but ever since I discovered Hazelcast a bazillion years ago, and implemented it at EA for password reset, I've been enamored with in-memory distributed collections and DBs 

The one above is pure go and uses the Redis protocol to sync.

An alternative is to use a IPFS db or something like that with optin for people to sync with the community.


Solution patch:
diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt
index 8267488..7f72277 100644
--- a/.github/actions/spelling/allow.txt
+++ b/.github/actions/spelling/allow.txt
@@ -8,4 +8,5 @@ msgbox
 xeact
 ABee
 tencent
-maintnotifications
\ No newline at end of file
+maintnotifications
+azurediamond
diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt
index d23ed5c..145e53c 100644
--- a/.github/actions/spelling/expect.txt
+++ b/.github/actions/spelling/expect.txt
@@ -200,6 +200,7 @@ licstart
 lightpanda
 limsa
 Linting
+listor
 LLU
 loadbalancer
 lol
@@ -217,6 +218,10 @@ mnt
 Mojeek
 mojeekbot
 mozilla
+myclient
+mymaster
+mypass
+myuser
 nbf
 nepeat
 netsurf
@@ -267,7 +272,6 @@ qwantbot
 rac
 rawler
 rcvar
-rdb
 redhat
 redir
 redirectscheme
diff --git a/docs/docs/admin/policies.mdx b/docs/docs/admin/policies.mdx
index f95821d..2065f49 100644
--- a/docs/docs/admin/policies.mdx
+++ b/docs/docs/admin/policies.mdx
@@ -225,10 +225,10 @@ Using this backend will cause a lot of S3 operations, at least one for creating
 
 The `s3api` backend takes the following configuration options:
 
-| Name         | Type    | Example                                                              | Description                                                                                                                                 |
-| :----------- | :------ | :------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ |
-| `bucketName` | string  | The name of the dedicated bucket for Anubis to store information in. |
-| `pathStyle`  | boolean | `false`                                                              | If true, use path-style S3 API operations. Please consult your storage provider's documentation if you don't know what you should put here. |
+| Name         | Type    | Example       | Description                                                                                                                                 |
+| :----------- | :------ | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------ |
+| `bucketName` | string  | `anubis-data` | (Required) The name of the dedicated bucket for Anubis to store information in.                                                             |
+| `pathStyle`  | boolean | `false`       | If true, use path-style S3 API operations. Please consult your storage provider's documentation if you don't know what you should put here. |
 
 :::note
 
@@ -279,7 +279,7 @@ store:
 
 :::note
 
-You can also use [Redis](http://redis.io/) with Anubis.
+You can also use [Redis™](http://redis.io/) with Anubis.
 
 :::
 
@@ -291,15 +291,17 @@ This backend is ideal if you are running multiple instances of Anubis in a worke
 | Does your service get a lot of traffic?                       | ✅ Yes |
 | Do you want to store data persistently when Anubis restarts?  | ✅ Yes |
 | Do you run Anubis without mutable filesystem storage?         | ✅ Yes |
-| Do you have Redis or Valkey installed?                        | ✅ Yes |
+| Do you have Redis™ or Valkey installed?                       | ✅ Yes |
 
 #### Configuration
 
 The `valkey` backend takes the following configuration options:
 
-| Name  | Type   | Example                 | Description                                                                                                                                      |
-| :---- | :----- | :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
-| `url` | string | `redis://valkey:6379/0` | The URL for the instance of Redis or Valkey that Anubis should store data in. This is in the same format as `REDIS_URL` in many cloud providers. |
+| Name       | Type   | Example                 | Description                                                                                                                                       |
+| :--------- | :----- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `cluster`  | bool   | `false`                 | If true, use [Redis™ Clustering](https://redis.io/topics/cluster-spec) for storing Anubis data.                                                   |
+| `sentinel` | object | `{}`                    | See [Redis™ Sentinel docs](#redis-sentinel) for more detail and examples                                                                          |
+| `url`      | string | `redis://valkey:6379/0` | The URL for the instance of Redis™ or Valkey that Anubis should store data in. This is in the same format as `REDIS_URL` in many cloud providers. |
 
 Example:
 
@@ -314,6 +316,18 @@ store:
 
 This would have the Valkey client connect to host `valkey.int.techaro.lol` on port `6379` with database `0` (the default database).
 
+#### Redis™ Sentinel
+
+If you are using [Redis™ Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) for a high availability setup, you need to configure the `sentinel` object. This object takes the following configuration options:
+
+| Name         | Type                     | Example               | Description                                                                                                                                               |
+| :----------- | :----------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `addr`       | string or list of string | `10.43.208.130:26379` | (Required) The host and port of the Redis™ Sentinel server. When possible, use DNS names for this. If you have multiple addresses, supply a list of them. |
+| `clientName` | string                   | `Anubis`              | The client name reported to Redis™ Sentinel. Set this if you want to track Anubis connections to your Redis™ Sentinel.                                    |
+| `masterName` | string                   | `mymaster`            | (Required) The name of the master in the Redis™ Sentinel configuration. This is used to discover where to find client connection hosts/ports.             |
+| `username`   | string                   | `azurediamond`        | The username used to authenticate against the Redis™ Sentinel and Redis™ servers.                                                                         |
+| `password`   | string                   | `hunter2`             | The password used to authenticate against the Redis™ Sentinel and Redis™ servers.                                                                         |
+
 ## Risk calculation for downstream services
 
 In case your service needs it for risk calculation reasons, Anubis exposes information about the rules that any requests match using a few headers:
diff --git a/internal/listor.go b/internal/listor.go
new file mode 100644
index 0000000..b6ba57f
--- /dev/null
+++ b/internal/listor.go
@@ -0,0 +1,39 @@
+package internal
+
+import (
+	"encoding/json"
+)
+
+// ListOr[T any] is a slice that can contain either a single T or multiple T values.
+// During JSON unmarshaling, it checks if the first character is '[' to determine
+// whether to treat the JSON as an array or a single value.
+type ListOr[T any] []T
+
+func (lo *ListOr[T]) UnmarshalJSON(data []byte) error {
+	if len(data) == 0 {
+		return nil
+	}
+
+	// Check if first non-whitespace character is '['
+	firstChar := data[0]
+	for i := 0; i < len(data); i++ {
+		if data[i] != ' ' && data[i] != '\t' && data[i] != '\n' && data[i] != '\r' {
+			firstChar = data[i]
+			break
+		}
+	}
+
+	if firstChar == '[' {
+		// It's an array, unmarshal directly
+		return json.Unmarshal(data, (*[]T)(lo))
+	} else {
+		// It's a single value, unmarshal as a single item in a slice
+		var single T
+		if err := json.Unmarshal(data, &single); err != nil {
+			return err
+		}
+		*lo = ListOr[T]{single}
+	}
+
+	return nil
+}
\ No newline at end of file
diff --git a/lib/store/valkey/factory.go b/lib/store/valkey/factory.go
index c36b86d..309bf65 100644
--- a/lib/store/valkey/factory.go
+++ b/lib/store/valkey/factory.go
@@ -7,6 +7,7 @@ import (
 	"fmt"
 	"time"
 
+	"github.com/TecharoHQ/anubis/internal"
 	"github.com/TecharoHQ/anubis/lib/store"
 	valkey "github.com/redis/go-redis/v9"
 	"github.com/redis/go-redis/v9/maintnotifications"
@@ -16,26 +17,84 @@ func init() {
 	store.Register("valkey", Factory{})
 }
 
-// Errors kept as-is so other code/tests still pass.
 var (
 	ErrNoURL  = errors.New("valkey.Config: no URL defined")
 	ErrBadURL = errors.New("valkey.Config: URL is invalid")
+
+	// Sentinel validation errors
+	ErrSentinelMasterNameRequired = errors.New("valkey.Sentinel: masterName is required")
+	ErrSentinelAddrRequired       = errors.New("valkey.Sentinel: addr is required")
+	ErrSentinelAddrEmpty          = errors.New("valkey.Sentinel: addr cannot be empty")
 )
 
 // Config is what Anubis unmarshals from the "parameters" JSON.
 type Config struct {
 	URL     string `json:"url"`
 	Cluster bool   `json:"cluster,omitempty"`
+
+	Sentinel *Sentinel `json:"sentinel,omitempty"`
 }
 
 func (c Config) Valid() error {
-	if c.URL == "" {
-		return ErrNoURL
+	var errs []error
+
+	if c.URL == "" && c.Sentinel == nil {
+		errs = append(errs, ErrNoURL)
+	}
+
+	// Validate URL only if provided
+	if c.URL != "" {
+		if _, err := valkey.ParseURL(c.URL); err != nil {
+			errs = append(errs, fmt.Errorf("%w: %v", ErrBadURL, err))
+		}
+	}
+
+	if c.Sentinel != nil {
+		if err := c.Sentinel.Valid(); err != nil {
+			errs = append(errs, err)
+		}
+	}
+
+	if len(errs) > 0 {
+		return errors.Join(errs...)
+	}
+
+	return nil
+}
+
+type Sentinel struct {
+	MasterName string                  `json:"masterName"`
+	Addr       internal.ListOr[string] `json:"addr"`
+	ClientName string                  `json:"clientName,omitempty"`
+	Username   string                  `json:"username,omitempty"`
+	Password   string                  `json:"password,omitempty"`
+}
+
+func (s Sentinel) Valid() error {
+	var errs []error
+
+	if s.MasterName == "" {
+		errs = append(errs, ErrSentinelMasterNameRequired)
+	}
+
+	if len(s.Addr) == 0 {
+		errs = append(errs, ErrSentinelAddrRequired)
+	} else {
+		// Check if all addresses in the list are empty
+		allEmpty := true
+		for _, addr := range s.Addr {
+			if addr != "" {
+				allEmpty = false
+				break
+			}
+		}
+		if allEmpty {
+			errs = append(errs, ErrSentinelAddrEmpty)
+		}
 	}
 
-	// Just validate that it's a valid Redis URL.
-	if _, err := valkey.ParseURL(c.URL); err != nil {
-		return fmt.Errorf("%w: %v", ErrBadURL, err)
+	if len(errs) > 0 {
+		return errors.Join(errs...)
 	}
 
 	return nil
@@ -68,14 +127,15 @@ func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface
 		return nil, err
 	}
 
-	opts, err := valkey.ParseURL(cfg.URL)
-	if err != nil {
-		return nil, fmt.Errorf("valkey.Factory: %w", err)
-	}
-
 	var client redisClient
 
-	if cfg.Cluster {
+	switch {
+	case cfg.Cluster:
+		opts, err := valkey.ParseURL(cfg.URL)
+		if err != nil {
+			return nil, fmt.Errorf("valkey.Factory: %w", err)
+		}
+
 		// Cluster mode: use the parsed Addr as the seed node.
 		clusterOpts := &valkey.ClusterOptions{
 			Addrs: []string{opts.Addr},
@@ -86,7 +146,23 @@ func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface
 			},
 		}
 		client = valkey.NewClusterClient(clusterOpts)
-	} else {
+	case cfg.Sentinel != nil:
+		opts := &valkey.FailoverOptions{
+			MasterName:       cfg.Sentinel.MasterName,
+			SentinelAddrs:    cfg.Sentinel.Addr,
+			SentinelUsername: cfg.Sentinel.Username,
+			SentinelPassword: cfg.Sentinel.Password,
+			Username:         cfg.Sentinel.Username,
+			Password:         cfg.Sentinel.Password,
+			ClientName:       cfg.Sentinel.ClientName,
+		}
+		client = valkey.NewFailoverClusterClient(opts)
+	default:
+		opts, err := valkey.ParseURL(cfg.URL)
+		if err != nil {
+			return nil, fmt.Errorf("valkey.Factory: %w", err)
+		}
+
 		opts.MaintNotificationsConfig = &maintnotifications.Config{
 			Mode: maintnotifications.ModeDisabled,
 		}



Test patch:
diff --git a/internal/listor_test.go b/internal/listor_test.go
new file mode 100644
index 0000000..31fffbf
--- /dev/null
+++ b/internal/listor_test.go
@@ -0,0 +1,79 @@
+package internal
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func TestListOr_UnmarshalJSON(t *testing.T) {
+	t.Run("single value should be unmarshaled as single item", func(t *testing.T) {
+		var lo ListOr[string]
+
+		err := json.Unmarshal([]byte(`"hello"`), &lo)
+		if err != nil {
+			t.Fatalf("Failed to unmarshal single string: %v", err)
+		}
+
+		if len(lo) != 1 {
+			t.Fatalf("Expected 1 item, got %d", len(lo))
+		}
+
+		if lo[0] != "hello" {
+			t.Errorf("Expected 'hello', got %q", lo[0])
+		}
+	})
+
+	t.Run("array should be unmarshaled as multiple items", func(t *testing.T) {
+		var lo ListOr[string]
+
+		err := json.Unmarshal([]byte(`["hello", "world"]`), &lo)
+		if err != nil {
+			t.Fatalf("Failed to unmarshal array: %v", err)
+		}
+
+		if len(lo) != 2 {
+			t.Fatalf("Expected 2 items, got %d", len(lo))
+		}
+
+		if lo[0] != "hello" {
+			t.Errorf("Expected 'hello', got %q", lo[0])
+		}
+		if lo[1] != "world" {
+			t.Errorf("Expected 'world', got %q", lo[1])
+		}
+	})
+
+	t.Run("single number should be unmarshaled as single item", func(t *testing.T) {
+		var lo ListOr[int]
+
+		err := json.Unmarshal([]byte(`42`), &lo)
+		if err != nil {
+			t.Fatalf("Failed to unmarshal single number: %v", err)
+		}
+
+		if len(lo) != 1 {
+			t.Fatalf("Expected 1 item, got %d", len(lo))
+		}
+
+		if lo[0] != 42 {
+			t.Errorf("Expected 42, got %d", lo[0])
+		}
+	})
+
+	t.Run("array of numbers should be unmarshaled as multiple items", func(t *testing.T) {
+		var lo ListOr[int]
+
+		err := json.Unmarshal([]byte(`[1, 2, 3]`), &lo)
+		if err != nil {
+			t.Fatalf("Failed to unmarshal number array: %v", err)
+		}
+
+		if len(lo) != 3 {
+			t.Fatalf("Expected 3 items, got %d", len(lo))
+		}
+
+		if lo[0] != 1 || lo[1] != 2 || lo[2] != 3 {
+			t.Errorf("Expected [1, 2, 3], got %v", lo)
+		}
+	})
+}
\ No newline at end of file
diff --git a/lib/store/valkey/valkey_test.go b/lib/store/valkey/valkey_test.go
index 0ed8eb6..a509e1c 100644
--- a/lib/store/valkey/valkey_test.go
+++ b/lib/store/valkey/valkey_test.go
@@ -2,6 +2,7 @@ package valkey
 
 import (
 	"encoding/json"
+	"errors"
 	"os"
 	"testing"
 
@@ -45,3 +46,86 @@ func TestImpl(t *testing.T) {
 
 	storetest.Common(t, Factory{}, json.RawMessage(data))
 }
+
+func TestFactoryValid(t *testing.T) {
+	tests := []struct {
+		name        string
+		jsonData    string
+		expectError error
+	}{
+		{
+			name:        "empty config",
+			jsonData:    `{}`,
+			expectError: ErrNoURL,
+		},
+		{
+			name:        "valid URL only",
+			jsonData:    `{"url": "redis://localhost:6379"}`,
+			expectError: nil,
+		},
+		{
+			name:        "invalid URL",
+			jsonData:    `{"url": "invalid-url"}`,
+			expectError: ErrBadURL,
+		},
+		{
+			name:        "valid sentinel config",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": ["localhost:26379"], "password": "mypass"}}`,
+			expectError: nil,
+		},
+		{
+			name:        "sentinel missing masterName",
+			jsonData:    `{"sentinel": {"addr": ["localhost:26379"], "password": "mypass"}}`,
+			expectError: ErrSentinelMasterNameRequired,
+		},
+		{
+			name:        "sentinel missing addr",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "password": "mypass"}}`,
+			expectError: ErrSentinelAddrRequired,
+		},
+		{
+			name:        "sentinel empty addr",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": [""], "password": "mypass"}}`,
+			expectError: ErrSentinelAddrEmpty,
+		},
+		{
+			name:        "sentinel missing password",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": ["localhost:26379"]}}`,
+			expectError: nil,
+		},
+		{
+			name:        "sentinel with optional fields",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": ["localhost:26379"], "password": "mypass", "clientName": "myclient", "username": "myuser"}}`,
+			expectError: nil,
+		},
+		{
+			name:        "sentinel single address (not array)",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": "localhost:26379", "password": "mypass"}}`,
+			expectError: nil,
+		},
+		{
+			name:        "sentinel mixed empty and valid addresses",
+			jsonData:    `{"sentinel": {"masterName": "mymaster", "addr": ["", "localhost:26379", ""], "password": "mypass"}}`,
+			expectError: nil,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			factory := Factory{}
+			err := factory.Valid(json.RawMessage(tt.jsonData))
+
+			if tt.expectError == nil {
+				if err != nil {
+					t.Errorf("expected no error, got: %v", err)
+				}
+			} else {
+				if err == nil {
+					t.Errorf("expected error %v, got nil", tt.expectError)
+				} else if !errors.Is(err, tt.expectError) {
+					t.Errorf("expected error %v, got: %v", tt.expectError, err)
+				}
+			}
+		})
+	}
+}