Repository: cortexproject/cortex (Go)

Issue:
Allow use of non-dualstack endpoints for S3 blocks storage
**Describe the bug**
Cannot configure the S3 blocks storage endpoint to point to a non-dualstack endpoint. In our config we have the `blocks_storage.s3.endpoint` set to `s3.us-east-1.amazonaws.com`. However, we observe the Cortex pods only reaching out to `<bucket name>.s3.dualstack.us-east-1.amazonaws.com`.

**To Reproduce**
Steps to reproduce the behavior:
1. Start Cortex (v1.18.1)
2. In the config, set the `blocks_storage.s3.endpoint` to a non-dualstack endpoint such as `s3.us-east-1.amazonaws.com`
3. Send data to Cortex
4. Observe that the endpoint the pods are connecting to is still `<bucket name>.s3.dualstack.us-east-1.amazonaws.com`

**Expected behavior**
We would expect Cortex to support hitting this non-dualstack endpoint. If we configure the endpoint as `s3.us-east-1.amazonaws.com`, Cortex should hit `<bucket name>.s3.us-east-1.amazonaws.com`.

**Environment:**
 - Infrastructure: Kubernetes
 - Deployment tool: helm

**Additional Context**
Minio does support this configuration (https://github.com/minio/minio-go/pull/1945) as of v7.0.69 which is available in Cortex 1.17.1 and up. The value [defaults to true](https://github.com/minio/minio-go/blob/v7.0.69/api.go#L166) for backwards compatibility. [This function](https://github.com/minio/minio-go/blob/5cf2ee77b94a1946bc7d649f9de90e4bd8e3ebdb/api.go#L351) should allow the configuration of this option.


Discussion:
To add some context, we are seeing performance issues on our K8s clusters especially with cilium and its cache.
Saw this [comment](https://github.com/cilium/cilium/issues/27156#issuecomment-1663509574) and wanted to try this and not use dualstack s3 endpoints.

That's a configuration already exists in the S3 client we use. https://github.com/thanos-io/objstore/blob/a54d0f04b42aea5ef4e0fd0b9bb4ad75629ea070/providers/s3/s3.go#L120

Just need to expose this configuration in Cortex. Help wanted
Am working on a PR!



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e4820d90b11..0e79090a7b9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -61,6 +61,7 @@
 * [ENHANCEMENT] Distributor: Added `cortex_distributor_received_samples_per_labelset_total` metric to calculate ingestion rate per label set. #6443
 * [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
 * [ENHANCEMENT] StoreGateway: Added `cortex_bucket_store_indexheader_load_duration_seconds` and `cortex_bucket_store_indexheader_download_duration_seconds` metrics for time of downloading and loading index header files. #6445
+* [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via `-blocks-storage.s3.disable-dualstack`. #6522
 * [BUGFIX] Runtime-config: Handle absolute file paths when working directory is not / #6224
 * [BUGFIX] Ruler: Allow rule evaluation to complete during shutdown. #6326
 * [BUGFIX] Ring: update ring with new ip address when instance is lost, rejoins, but heartbeat is disabled.  #6271
diff --git a/docs/blocks-storage/querier.md b/docs/blocks-storage/querier.md
index 05a5bbdd6da..19317be05ff 100644
--- a/docs/blocks-storage/querier.md
+++ b/docs/blocks-storage/querier.md
@@ -286,6 +286,10 @@ blocks_storage:
     # CLI flag: -blocks-storage.s3.bucket-name
     [bucket_name: <string> | default = ""]
 
+    # If enabled, S3 endpoint will use the non-dualstack variant.
+    # CLI flag: -blocks-storage.s3.disable-dualstack
+    [disable_dualstack: <boolean> | default = false]
+
     # S3 secret access key
     # CLI flag: -blocks-storage.s3.secret-access-key
     [secret_access_key: <string> | default = ""]
diff --git a/docs/blocks-storage/store-gateway.md b/docs/blocks-storage/store-gateway.md
index e40abbb31c0..e7a65dd58c1 100644
--- a/docs/blocks-storage/store-gateway.md
+++ b/docs/blocks-storage/store-gateway.md
@@ -390,6 +390,10 @@ blocks_storage:
     # CLI flag: -blocks-storage.s3.bucket-name
     [bucket_name: <string> | default = ""]
 
+    # If enabled, S3 endpoint will use the non-dualstack variant.
+    # CLI flag: -blocks-storage.s3.disable-dualstack
+    [disable_dualstack: <boolean> | default = false]
+
     # S3 secret access key
     # CLI flag: -blocks-storage.s3.secret-access-key
     [secret_access_key: <string> | default = ""]
diff --git a/docs/configuration/config-file-reference.md b/docs/configuration/config-file-reference.md
index 28154b7b180..0b977b3aff7 100644
--- a/docs/configuration/config-file-reference.md
+++ b/docs/configuration/config-file-reference.md
@@ -541,6 +541,10 @@ s3:
   # CLI flag: -alertmanager-storage.s3.bucket-name
   [bucket_name: <string> | default = ""]
 
+  # If enabled, S3 endpoint will use the non-dualstack variant.
+  # CLI flag: -alertmanager-storage.s3.disable-dualstack
+  [disable_dualstack: <boolean> | default = false]
+
   # S3 secret access key
   # CLI flag: -alertmanager-storage.s3.secret-access-key
   [secret_access_key: <string> | default = ""]
@@ -836,6 +840,10 @@ s3:
   # CLI flag: -blocks-storage.s3.bucket-name
   [bucket_name: <string> | default = ""]
 
+  # If enabled, S3 endpoint will use the non-dualstack variant.
+  # CLI flag: -blocks-storage.s3.disable-dualstack
+  [disable_dualstack: <boolean> | default = false]
+
   # S3 secret access key
   # CLI flag: -blocks-storage.s3.secret-access-key
   [secret_access_key: <string> | default = ""]
@@ -4771,6 +4779,10 @@ s3:
   # CLI flag: -ruler-storage.s3.bucket-name
   [bucket_name: <string> | default = ""]
 
+  # If enabled, S3 endpoint will use the non-dualstack variant.
+  # CLI flag: -ruler-storage.s3.disable-dualstack
+  [disable_dualstack: <boolean> | default = false]
+
   # S3 secret access key
   # CLI flag: -ruler-storage.s3.secret-access-key
   [secret_access_key: <string> | default = ""]
@@ -5074,6 +5086,10 @@ s3:
   # CLI flag: -runtime-config.s3.bucket-name
   [bucket_name: <string> | default = ""]
 
+  # If enabled, S3 endpoint will use the non-dualstack variant.
+  # CLI flag: -runtime-config.s3.disable-dualstack
+  [disable_dualstack: <boolean> | default = false]
+
   # S3 secret access key
   # CLI flag: -runtime-config.s3.secret-access-key
   [secret_access_key: <string> | default = ""]
diff --git a/pkg/storage/bucket/s3/bucket_client.go b/pkg/storage/bucket/s3/bucket_client.go
index 53a0f4f5882..220afb90256 100644
--- a/pkg/storage/bucket/s3/bucket_client.go
+++ b/pkg/storage/bucket/s3/bucket_client.go
@@ -83,14 +83,15 @@ func newS3Config(cfg Config) (s3.Config, error) {
 	}
 
 	return s3.Config{
-		Bucket:         cfg.BucketName,
-		Endpoint:       cfg.Endpoint,
-		Region:         cfg.Region,
-		AccessKey:      cfg.AccessKeyID,
-		SecretKey:      cfg.SecretAccessKey.Value,
-		Insecure:       cfg.Insecure,
-		SSEConfig:      sseCfg,
-		SendContentMd5: cfg.SendContentMd5,
+		Bucket:           cfg.BucketName,
+		Endpoint:         cfg.Endpoint,
+		Region:           cfg.Region,
+		DisableDualstack: cfg.DisableDualstack,
+		AccessKey:        cfg.AccessKeyID,
+		SecretKey:        cfg.SecretAccessKey.Value,
+		Insecure:         cfg.Insecure,
+		SSEConfig:        sseCfg,
+		SendContentMd5:   cfg.SendContentMd5,
 		HTTPConfig: s3.HTTPConfig{
 			IdleConnTimeout:       model.Duration(cfg.HTTP.IdleConnTimeout),
 			ResponseHeaderTimeout: model.Duration(cfg.HTTP.ResponseHeaderTimeout),
diff --git a/pkg/storage/bucket/s3/config.go b/pkg/storage/bucket/s3/config.go
index bb7bb9f9f86..df5bd33ab29 100644
--- a/pkg/storage/bucket/s3/config.go
+++ b/pkg/storage/bucket/s3/config.go
@@ -66,6 +66,7 @@ type Config struct {
 	Endpoint           string         `yaml:"endpoint"`
 	Region             string         `yaml:"region"`
 	BucketName         string         `yaml:"bucket_name"`
+	DisableDualstack   bool           `yaml:"disable_dualstack"`
 	SecretAccessKey    flagext.Secret `yaml:"secret_access_key"`
 	AccessKeyID        string         `yaml:"access_key_id"`
 	Insecure           bool           `yaml:"insecure"`
@@ -89,6 +90,7 @@ func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
 	f.Var(&cfg.SecretAccessKey, prefix+"s3.secret-access-key", "S3 secret access key")
 	f.StringVar(&cfg.BucketName, prefix+"s3.bucket-name", "", "S3 bucket name")
 	f.StringVar(&cfg.Region, prefix+"s3.region", "", "S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it.")
+	f.BoolVar(&cfg.DisableDualstack, prefix+"s3.disable-dualstack", false, "If enabled, S3 endpoint will use the non-dualstack variant.")
 	f.StringVar(&cfg.Endpoint, prefix+"s3.endpoint", "", "The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format.")
 	f.BoolVar(&cfg.Insecure, prefix+"s3.insecure", false, "If enabled, use http:// for the S3 endpoint instead of https://. This could be useful in local dev/test environments while using an S3-compatible backend storage, like Minio.")
 	f.StringVar(&cfg.SignatureVersion, prefix+"s3.signature-version", SignatureVersionV4, fmt.Sprintf("The signature version to use for authenticating against S3. Supported values are: %s.", strings.Join(supportedSignatureVersions, ", ")))


Test patch:
diff --git a/pkg/storage/bucket/s3/config_test.go b/pkg/storage/bucket/s3/config_test.go
index b1f38ce6f46..a01a8a07b7e 100644
--- a/pkg/storage/bucket/s3/config_test.go
+++ b/pkg/storage/bucket/s3/config_test.go
@@ -51,6 +51,7 @@ func TestConfig(t *testing.T) {
 endpoint: test-endpoint
 region: test-region
 bucket_name: test-bucket-name
+disable_dualstack: true
 secret_access_key: test-secret-access-key
 access_key_id: test-access-key-id
 insecure: true
@@ -74,6 +75,7 @@ http:
 				Endpoint:         "test-endpoint",
 				Region:           "test-region",
 				BucketName:       "test-bucket-name",
+				DisableDualstack: true,
 				SecretAccessKey:  flagext.Secret{Value: "test-secret-access-key"},
 				AccessKeyID:      "test-access-key-id",
 				Insecure:         true,