Repository: cortexproject/cortex (Go)

Issue:
Support other block storage options as a storage backend for runtime configuration file
**Is your feature request related to a problem? Please describe.**
Currently Cortex reads the [runtime configuration file](https://cortexmetrics.io/docs/configuration/arguments/#runtime-configuration-file) from filesystem. Cortex should be able to support other storage backends that Cortex can read the configuration file from (e.g. S3) in order to better keep track of versioning and easier access.

This is already supported today for other Cortex configurations like
[blocks storage config](https://github.com/cortexproject/cortex/blob/909a090592b64c4dda5213681a6a7386316a5489/docs/blocks-storage/querier.md#blocks_storage_config).

**Describe the solution you'd like**
Inline the [`storage bucket config`](https://github.com/cortexproject/cortex/blob/master/pkg/storage/bucket/client.go) with the runtime configuration Config as done for other components like [rule store](https://github.com/cortexproject/cortex/blob/1ad8b03edbab2e4f305eee0b4e037fee52aceff4/pkg/ruler/rulestore/config.go#L16).

By default the storage backend will be `Filesystem`, the same as it is today.


Discussion: This can be useful for k8s cortex installations with many per-user limits overrides which are unable to use a ConfigMap as the storage solution for runtime configs given its [1 MiB limit](https://kubernetes.io/docs/concepts/configuration/configmap/).
@anna-tran The idea is to allow something like this, right? 

```
runtime_config:
  # existent field
  [period: <duration> | default = 10s]

  # existent field
  [file: <string> | default = ""]

  [backend: <string> | default = "filesystem"]

  s3: 
      [endpoint: <string> | default = ""]

      [region: <string> | default = ""]

      [bucket_name: <string> | default = ""]
       ...

  filesystem:
      [dir: <string> | default = ""]

   gcs:
       [bucket_name: <string> | default = ""]
       
       [service_account: <string> | default = ""]
```


@marianafranco Yes, for now we can start with only s3 and filesystem as options for the backend and extend to other storage/cloud services. We can keep the `runtime_config.file` as a legacy field and copy over the configuration to the `runtime_config.filesystem.dir` if it's not configured.
Don't we need the `runtime_config.file` since the S3 config don't have a way to specify the file name? 

https://github.com/cortexproject/cortex/blob/3d1315a553672faf6e0a883b443d58dca8b2db36/pkg/storage/bucket/s3/config.go#L60-L72
> Don't we need the runtime_config.file since the S3 config don't have a way to specify the file name?

Yeah I think we can reuse that field for the filename for S3. Just wanna double check, do we have any use case of using dynamic file name? If not, then this should be good enough.
Should we start with a proposal PR? 
> do we have any use case of using dynamic file name

I think to keep it simple we’ll require the file name to be specified like we do today.

@alvinlin123 I will start the proposal PR for it today 



Fix (patch):
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 39ce61d8f69..bf16ecb94dc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
 * [CHANGE] Alertmanager: Validating new fields on the PagerDuty AM config. #5290
 * [CHANGE] Ingester: Creating label `native-histogram-sample` on the `cortex_discarded_samples_total` to keep track of discarded native histogram samples. #5289
 * [FEATURE] Store Gateway: Add `max_downloaded_bytes_per_request` to limit max bytes to download per store gateway request.
+* [ENHANCEMENT] Support object storage backends for runtime configuration file. #5292
 * [BUGFIX] Ruler: Validate if rule group can be safely converted back to rule group yaml from protobuf message #5265
 * [BUGFIX] Querier: Convert gRPC `ResourceExhausted` status code from store gateway to 422 limit error. #5286
 * [BUGFIX] Alertmanager: Route web-ui requests to the alertmanager distributor when sharding is enabled. #5293
diff --git a/docs/configuration/config-file-reference.md b/docs/configuration/config-file-reference.md
index 5b6fa996a92..7802b3ffe4d 100644
--- a/docs/configuration/config-file-reference.md
+++ b/docs/configuration/config-file-reference.md
@@ -159,14 +159,9 @@ tenant_federation:
 # backend.
 [alertmanager_storage: <alertmanager_storage_config>]
 
-runtime_config:
-  # How often to check runtime config file.
-  # CLI flag: -runtime-config.reload-period
-  [period: <duration> | default = 10s]
-
-  # File with the configuration that can be updated in runtime.
-  # CLI flag: -runtime-config.file
-  [file: <string> | default = ""]
+# The runtime_configuration_storage_config configures the storage backend for
+# the runtime configuration file.
+[runtime_config: <runtime_configuration_storage_config>]
 
 # The memberlist_config configures the Gossip memberlist.
 [memberlist: <memberlist_config>]
@@ -4183,6 +4178,269 @@ local:
   [directory: <string> | default = ""]
 ```
 
+### `runtime_configuration_storage_config`
+
+The `runtime_configuration_storage_config` configures the storage backend for the runtime configuration file.
+
+```yaml
+# How often to check runtime config file.
+# CLI flag: -runtime-config.reload-period
+[period: <duration> | default = 10s]
+
+# File with the configuration that can be updated in runtime.
+# CLI flag: -runtime-config.file
+[file: <string> | default = ""]
+
+# Backend storage to use. Supported backends are: s3, gcs, azure, swift,
+# filesystem.
+# CLI flag: -runtime-config.backend
+[backend: <string> | default = "filesystem"]
+
+s3:
+  # The S3 bucket endpoint. It could be an AWS S3 endpoint listed at
+  # https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an
+  # S3-compatible service in hostname:port format.
+  # CLI flag: -runtime-config.s3.endpoint
+  [endpoint: <string> | default = ""]
+
+  # S3 region. If unset, the client will issue a S3 GetBucketLocation API call
+  # to autodetect it.
+  # CLI flag: -runtime-config.s3.region
+  [region: <string> | default = ""]
+
+  # S3 bucket name
+  # CLI flag: -runtime-config.s3.bucket-name
+  [bucket_name: <string> | default = ""]
+
+  # S3 secret access key
+  # CLI flag: -runtime-config.s3.secret-access-key
+  [secret_access_key: <string> | default = ""]
+
+  # S3 access key ID
+  # CLI flag: -runtime-config.s3.access-key-id
+  [access_key_id: <string> | default = ""]
+
+  # If enabled, use http:// for the S3 endpoint instead of https://. This could
+  # be useful in local dev/test environments while using an S3-compatible
+  # backend storage, like Minio.
+  # CLI flag: -runtime-config.s3.insecure
+  [insecure: <boolean> | default = false]
+
+  # The signature version to use for authenticating against S3. Supported values
+  # are: v4, v2.
+  # CLI flag: -runtime-config.s3.signature-version
+  [signature_version: <string> | default = "v4"]
+
+  # The s3 bucket lookup style. Supported values are: auto, virtual-hosted,
+  # path.
+  # CLI flag: -runtime-config.s3.bucket-lookup-type
+  [bucket_lookup_type: <string> | default = "auto"]
+
+  # The s3_sse_config configures the S3 server-side encryption.
+  # The CLI flags prefix for this block config is: runtime-config
+  [sse: <s3_sse_config>]
+
+  http:
+    # The time an idle connection will remain idle before closing.
+    # CLI flag: -runtime-config.s3.http.idle-conn-timeout
+    [idle_conn_timeout: <duration> | default = 1m30s]
+
+    # The amount of time the client will wait for a servers response headers.
+    # CLI flag: -runtime-config.s3.http.response-header-timeout
+    [response_header_timeout: <duration> | default = 2m]
+
+    # If the client connects via HTTPS and this option is enabled, the client
+    # will accept any certificate and hostname.
+    # CLI flag: -runtime-config.s3.http.insecure-skip-verify
+    [insecure_skip_verify: <boolean> | default = false]
+
+    # Maximum time to wait for a TLS handshake. 0 means no limit.
+    # CLI flag: -runtime-config.s3.tls-handshake-timeout
+    [tls_handshake_timeout: <duration> | default = 10s]
+
+    # The time to wait for a server's first response headers after fully writing
+    # the request headers if the request has an Expect header. 0 to send the
+    # request body immediately.
+    # CLI flag: -runtime-config.s3.expect-continue-timeout
+    [expect_continue_timeout: <duration> | default = 1s]
+
+    # Maximum number of idle (keep-alive) connections across all hosts. 0 means
+    # no limit.
+    # CLI flag: -runtime-config.s3.max-idle-connections
+    [max_idle_connections: <int> | default = 100]
+
+    # Maximum number of idle (keep-alive) connections to keep per-host. If 0, a
+    # built-in default value is used.
+    # CLI flag: -runtime-config.s3.max-idle-connections-per-host
+    [max_idle_connections_per_host: <int> | default = 100]
+
+    # Maximum number of connections per host. 0 means no limit.
+    # CLI flag: -runtime-config.s3.max-connections-per-host
+    [max_connections_per_host: <int> | default = 0]
+
+gcs:
+  # GCS bucket name
+  # CLI flag: -runtime-config.gcs.bucket-name
+  [bucket_name: <string> | default = ""]
+
+  # JSON representing either a Google Developers Console client_credentials.json
+  # file or a Google Developers service account key file. If empty, fallback to
+  # Google default logic.
+  # CLI flag: -runtime-config.gcs.service-account
+  [service_account: <string> | default = ""]
+
+azure:
+  # Azure storage account name
+  # CLI flag: -runtime-config.azure.account-name
+  [account_name: <string> | default = ""]
+
+  # Azure storage account key
+  # CLI flag: -runtime-config.azure.account-key
+  [account_key: <string> | default = ""]
+
+  # Azure storage container name
+  # CLI flag: -runtime-config.azure.container-name
+  [container_name: <string> | default = ""]
+
+  # Azure storage endpoint suffix without schema. The account name will be
+  # prefixed to this value to create the FQDN
+  # CLI flag: -runtime-config.azure.endpoint-suffix
+  [endpoint_suffix: <string> | default = ""]
+
+  # Number of retries for recoverable errors
+  # CLI flag: -runtime-config.azure.max-retries
+  [max_retries: <int> | default = 20]
+
+  # Azure storage MSI resource. Either this or account key must be set.
+  # CLI flag: -runtime-config.azure.msi-resource
+  [msi_resource: <string> | default = ""]
+
+  # Azure storage MSI resource managed identity client Id. If not supplied
+  # system assigned identity is used
+  # CLI flag: -runtime-config.azure.user-assigned-id
+  [user_assigned_id: <string> | default = ""]
+
+  http:
+    # The time an idle connection will remain idle before closing.
+    # CLI flag: -runtime-config.azure.http.idle-conn-timeout
+    [idle_conn_timeout: <duration> | default = 1m30s]
+
+    # The amount of time the client will wait for a servers response headers.
+    # CLI flag: -runtime-config.azure.http.response-header-timeout
+    [response_header_timeout: <duration> | default = 2m]
+
+    # If the client connects via HTTPS and this option is enabled, the client
+    # will accept any certificate and hostname.
+    # CLI flag: -runtime-config.azure.http.insecure-skip-verify
+    [insecure_skip_verify: <boolean> | default = false]
+
+    # Maximum time to wait for a TLS handshake. 0 means no limit.
+    # CLI flag: -runtime-config.azure.tls-handshake-timeout
+    [tls_handshake_timeout: <duration> | default = 10s]
+
+    # The time to wait for a server's first response headers after fully writing
+    # the request headers if the request has an Expect header. 0 to send the
+    # request body immediately.
+    # CLI flag: -runtime-config.azure.expect-continue-timeout
+    [expect_continue_timeout: <duration> | default = 1s]
+
+    # Maximum number of idle (keep-alive) connections across all hosts. 0 means
+    # no limit.
+    # CLI flag: -runtime-config.azure.max-idle-connections
+    [max_idle_connections: <int> | default = 100]
+
+    # Maximum number of idle (keep-alive) connections to keep per-host. If 0, a
+    # built-in default value is used.
+    # CLI flag: -runtime-config.azure.max-idle-connections-per-host
+    [max_idle_connections_per_host: <int> | default = 100]
+
+    # Maximum number of connections per host. 0 means no limit.
+    # CLI flag: -runtime-config.azure.max-connections-per-host
+    [max_connections_per_host: <int> | default = 0]
+
+swift:
+  # OpenStack Swift authentication API version. 0 to autodetect.
+  # CLI flag: -runtime-config.swift.auth-version
+  [auth_version: <int> | default = 0]
+
+  # OpenStack Swift authentication URL
+  # CLI flag: -runtime-config.swift.auth-url
+  [auth_url: <string> | default = ""]
+
+  # OpenStack Swift username.
+  # CLI flag: -runtime-config.swift.username
+  [username: <string> | default = ""]
+
+  # OpenStack Swift user's domain name.
+  # CLI flag: -runtime-config.swift.user-domain-name
+  [user_domain_name: <string> | default = ""]
+
+  # OpenStack Swift user's domain ID.
+  # CLI flag: -runtime-config.swift.user-domain-id
+  [user_domain_id: <string> | default = ""]
+
+  # OpenStack Swift user ID.
+  # CLI flag: -runtime-config.swift.user-id
+  [user_id: <string> | default = ""]
+
+  # OpenStack Swift API key.
+  # CLI flag: -runtime-config.swift.password
+  [password: <string> | default = ""]
+
+  # OpenStack Swift user's domain ID.
+  # CLI flag: -runtime-config.swift.domain-id
+  [domain_id: <string> | default = ""]
+
+  # OpenStack Swift user's domain name.
+  # CLI flag: -runtime-config.swift.domain-name
+  [domain_name: <string> | default = ""]
+
+  # OpenStack Swift project ID (v2,v3 auth only).
+  # CLI flag: -runtime-config.swift.project-id
+  [project_id: <string> | default = ""]
+
+  # OpenStack Swift project name (v2,v3 auth only).
+  # CLI flag: -runtime-config.swift.project-name
+  [project_name: <string> | default = ""]
+
+  # ID of the OpenStack Swift project's domain (v3 auth only), only needed if it
+  # differs the from user domain.
+  # CLI flag: -runtime-config.swift.project-domain-id
+  [project_domain_id: <string> | default = ""]
+
+  # Name of the OpenStack Swift project's domain (v3 auth only), only needed if
+  # it differs from the user domain.
+  # CLI flag: -runtime-config.swift.project-domain-name
+  [project_domain_name: <string> | default = ""]
+
+  # OpenStack Swift Region to use (v2,v3 auth only).
+  # CLI flag: -runtime-config.swift.region-name
+  [region_name: <string> | default = ""]
+
+  # Name of the OpenStack Swift container to put chunks in.
+  # CLI flag: -runtime-config.swift.container-name
+  [container_name: <string> | default = ""]
+
+  # Max retries on requests error.
+  # CLI flag: -runtime-config.swift.max-retries
+  [max_retries: <int> | default = 3]
+
+  # Time after which a connection attempt is aborted.
+  # CLI flag: -runtime-config.swift.connect-timeout
+  [connect_timeout: <duration> | default = 10s]
+
+  # Time after which an idle request is aborted. The timeout watchdog is reset
+  # each time some data is received, so the timeout triggers after X time no
+  # data is received on a request.
+  # CLI flag: -runtime-config.swift.request-timeout
+  [request_timeout: <duration> | default = 5s]
+
+filesystem:
+  # Local filesystem storage directory.
+  # CLI flag: -runtime-config.filesystem.dir
+  [dir: <string> | default = ""]
+```
+
 ### `s3_sse_config`
 
 The `s3_sse_config` configures the S3 server-side encryption. The supported CLI flags `<prefix>` used to reference this config block are:
@@ -4190,6 +4448,7 @@ The `s3_sse_config` configures the S3 server-side encryption. The supported CLI
 - `alertmanager-storage`
 - `blocks-storage`
 - `ruler-storage`
+- `runtime-config`
 
 &nbsp;
 
diff --git a/integration/configs.go b/integration/configs.go
index cffea19a5f5..07d9e78b152 100644
--- a/integration/configs.go
+++ b/integration/configs.go
@@ -22,6 +22,7 @@ const (
 	rulestoreBucketName = "cortex-rules"
 	alertsBucketName    = "cortex-alerts"
 	cortexConfigFile    = "config.yaml"
+	runtimeConfigFile   = "runtime-config.yaml"
 	blocksStorageEngine = "blocks"
 	clientCertFile      = "certs/client.crt"
 	clientKeyFile       = "certs/client.key"
diff --git a/pkg/cortex/modules.go b/pkg/cortex/modules.go
index aef97651480..ab03b711439 100644
--- a/pkg/cortex/modules.go
+++ b/pkg/cortex/modules.go
@@ -18,6 +18,7 @@ import (
 	v1 "github.com/prometheus/prometheus/web/api/v1"
 	"github.com/thanos-community/promql-engine/engine"
 	"github.com/thanos-community/promql-engine/logicalplan"
+	"github.com/thanos-io/objstore"
 	"github.com/thanos-io/thanos/pkg/discovery/dns"
 	"github.com/thanos-io/thanos/pkg/querysharding"
 	httpgrpc_server "github.com/weaveworks/common/httpgrpc/server"
@@ -46,6 +47,7 @@ import (
 	"github.com/cortexproject/cortex/pkg/ring/kv/memberlist"
 	"github.com/cortexproject/cortex/pkg/ruler"
 	"github.com/cortexproject/cortex/pkg/scheduler"
+	"github.com/cortexproject/cortex/pkg/storage/bucket"
 	"github.com/cortexproject/cortex/pkg/storegateway"
 	util_log "github.com/cortexproject/cortex/pkg/util/log"
 	"github.com/cortexproject/cortex/pkg/util/modules"
@@ -157,7 +159,12 @@ func (t *Cortex) initRuntimeConfig() (services.Service, error) {
 	// make sure to set default limits before we start loading configuration into memory
 	validation.SetDefaultLimitsForYAMLUnmarshalling(t.Cfg.LimitsConfig)
 
-	serv, err := runtimeconfig.New(t.Cfg.RuntimeConfig, prometheus.WrapRegistererWithPrefix("cortex_", prometheus.DefaultRegisterer), util_log.Logger)
+	registerer := prometheus.WrapRegistererWithPrefix("cortex_", prometheus.DefaultRegisterer)
+	logger := util_log.Logger
+	bucketClientFactory := func(ctx context.Context) (objstore.Bucket, error) {
+		return bucket.NewClient(ctx, t.Cfg.RuntimeConfig.StorageConfig, "runtime-config", logger, registerer)
+	}
+	serv, err := runtimeconfig.New(t.Cfg.RuntimeConfig, registerer, logger, bucketClientFactory)
 	if err == nil {
 		// TenantLimits just delegates to RuntimeConfig and doesn't have any state or need to do
 		// anything in the start/stopping phase. Thus we can create it as part of runtime config
diff --git a/pkg/cortex/runtime_config.go b/pkg/cortex/runtime_config.go
index 67bfd928127..3d612c7cc88 100644
--- a/pkg/cortex/runtime_config.go
+++ b/pkg/cortex/runtime_config.go
@@ -18,10 +18,10 @@ var (
 	errMultipleDocuments = errors.New("the provided runtime configuration contains multiple documents")
 )
 
-// runtimeConfigValues are values that can be reloaded from configuration file while Cortex is running.
+// RuntimeConfigValues are values that can be reloaded from configuration file while Cortex is running.
 // Reloading is done by runtime_config.Manager, which also keeps the currently loaded config.
 // These values are then pushed to the components that are interested in them.
-type runtimeConfigValues struct {
+type RuntimeConfigValues struct {
 	TenantLimits map[string]*validation.Limits `yaml:"overrides"`
 
 	Multi kv.MultiRuntimeConfig `yaml:"multi_kv_config"`
@@ -50,7 +50,7 @@ func (l *runtimeConfigTenantLimits) ByUserID(userID string) *validation.Limits {
 }
 
 func (l *runtimeConfigTenantLimits) AllByUserID() map[string]*validation.Limits {
-	cfg, ok := l.manager.GetConfig().(*runtimeConfigValues)
+	cfg, ok := l.manager.GetConfig().(*RuntimeConfigValues)
 	if cfg != nil && ok {
 		return cfg.TenantLimits
 	}
@@ -59,7 +59,7 @@ func (l *runtimeConfigTenantLimits) AllByUserID() map[string]*validation.Limits
 }
 
 func loadRuntimeConfig(r io.Reader) (interface{}, error) {
-	var overrides = &runtimeConfigValues{}
+	var overrides = &RuntimeConfigValues{}
 
 	decoder := yaml.NewDecoder(r)
 	decoder.SetStrict(true)
@@ -70,7 +70,7 @@ func loadRuntimeConfig(r io.Reader) (interface{}, error) {
 	}
 
 	// Ensure the provided YAML config is not composed of multiple documents,
-	if err := decoder.Decode(&runtimeConfigValues{}); !errors.Is(err, io.EOF) {
+	if err := decoder.Decode(&RuntimeConfigValues{}); !errors.Is(err, io.EOF) {
 		return nil, errMultipleDocuments
 	}
 
@@ -87,14 +87,14 @@ func multiClientRuntimeConfigChannel(manager *runtimeconfig.Manager) func() <-ch
 
 		// push initial config to the channel
 		val := manager.GetConfig()
-		if cfg, ok := val.(*runtimeConfigValues); ok && cfg != nil {
+		if cfg, ok := val.(*RuntimeConfigValues); ok && cfg != nil {
 			outCh <- cfg.Multi
 		}
 
 		ch := manager.CreateListenerChannel(1)
 		go func() {
 			for val := range ch {
-				if cfg, ok := val.(*runtimeConfigValues); ok && cfg != nil {
+				if cfg, ok := val.(*RuntimeConfigValues); ok && cfg != nil {
 					outCh <- cfg.Multi
 				}
 			}
@@ -111,7 +111,7 @@ func ingesterInstanceLimits(manager *runtimeconfig.Manager) func() *ingester.Ins
 
 	return func() *ingester.InstanceLimits {
 		val := manager.GetConfig()
-		if cfg, ok := val.(*runtimeConfigValues); ok && cfg != nil {
+		if cfg, ok := val.(*RuntimeConfigValues); ok && cfg != nil {
 			return cfg.IngesterLimits
 		}
 		return nil
@@ -120,7 +120,7 @@ func ingesterInstanceLimits(manager *runtimeconfig.Manager) func() *ingester.Ins
 
 func runtimeConfigHandler(runtimeCfgManager *runtimeconfig.Manager, defaultLimits validation.Limits) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		cfg, ok := runtimeCfgManager.GetConfig().(*runtimeConfigValues)
+		cfg, ok := runtimeCfgManager.GetConfig().(*RuntimeConfigValues)
 		if !ok || cfg == nil {
 			util.WriteTextResponse(w, "runtime config file doesn't exist")
 			return
@@ -131,7 +131,7 @@ func runtimeConfigHandler(runtimeCfgManager *runtimeconfig.Manager, defaultLimit
 		case "diff":
 			// Default runtime config is just empty struct, but to make diff work,
 			// we set defaultLimits for every tenant that exists in runtime config.
-			defaultCfg := runtimeConfigValues{}
+			defaultCfg := RuntimeConfigValues{}
 			defaultCfg.TenantLimits = map[string]*validation.Limits{}
 			for k, v := range cfg.TenantLimits {
 				if v != nil {
diff --git a/pkg/storage/bucket/client.go b/pkg/storage/bucket/client.go
index e560c44f9ad..f8bfa3e14b5 100644
--- a/pkg/storage/bucket/client.go
+++ b/pkg/storage/bucket/client.go
@@ -72,13 +72,17 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet) {
 }
 
 func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
+	cfg.RegisterFlagsWithPrefixAndBackend(prefix, f, S3)
+}
+
+func (cfg *Config) RegisterFlagsWithPrefixAndBackend(prefix string, f *flag.FlagSet, defaultBackend string) {
 	cfg.S3.RegisterFlagsWithPrefix(prefix, f)
 	cfg.GCS.RegisterFlagsWithPrefix(prefix, f)
 	cfg.Azure.RegisterFlagsWithPrefix(prefix, f)
 	cfg.Swift.RegisterFlagsWithPrefix(prefix, f)
 	cfg.Filesystem.RegisterFlagsWithPrefix(prefix, f)
 
-	f.StringVar(&cfg.Backend, prefix+"backend", S3, fmt.Sprintf("Backend storage to use. Supported backends are: %s.", strings.Join(cfg.supportedBackends(), ", ")))
+	f.StringVar(&cfg.Backend, prefix+"backend", defaultBackend, fmt.Sprintf("Backend storage to use. Supported backends are: %s.", strings.Join(cfg.supportedBackends(), ", ")))
 }
 
 func (cfg *Config) Validate() error {
diff --git a/pkg/util/runtimeconfig/manager.go b/pkg/util/runtimeconfig/manager.go
index 674dee288e8..f4bff920ec7 100644
--- a/pkg/util/runtimeconfig/manager.go
+++ b/pkg/util/runtimeconfig/manager.go
@@ -7,7 +7,6 @@ import (
 	"flag"
 	"fmt"
 	"io"
-	"os"
 	"sync"
 	"time"
 
@@ -16,10 +15,14 @@ import (
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
+	"github.com/thanos-io/objstore"
 
+	"github.com/cortexproject/cortex/pkg/storage/bucket"
 	"github.com/cortexproject/cortex/pkg/util/services"
 )
 
+type BucketClientFactory func(ctx context.Context) (objstore.Bucket, error)
+
 // Loader loads the configuration from file.
 type Loader func(r io.Reader) (interface{}, error)
 
@@ -31,12 +34,16 @@ type Config struct {
 	// non-empty value
 	LoadPath string `yaml:"file"`
 	Loader   Loader `yaml:"-"`
+
+	StorageConfig bucket.Config `yaml:",inline"`
 }
 
 // RegisterFlags registers flags.
 func (mc *Config) RegisterFlags(f *flag.FlagSet) {
 	f.StringVar(&mc.LoadPath, "runtime-config.file", "", "File with the configuration that can be updated in runtime.")
 	f.DurationVar(&mc.ReloadPeriod, "runtime-config.reload-period", 10*time.Second, "How often to check runtime config file.")
+
+	mc.StorageConfig.RegisterFlagsWithPrefixAndBackend("runtime-config.", f, bucket.Filesystem)
 }
 
 // Manager periodically reloads the configuration from a file, and keeps this
@@ -55,14 +62,21 @@ type Manager struct {
 
 	configLoadSuccess prometheus.Gauge
 	configHash        *prometheus.GaugeVec
+
+	bucketClient        objstore.Bucket
+	bucketClientFactory BucketClientFactory
 }
 
 // New creates an instance of Manager and starts reload config loop based on config
-func New(cfg Config, registerer prometheus.Registerer, logger log.Logger) (*Manager, error) {
+func New(cfg Config, registerer prometheus.Registerer, logger log.Logger, factory BucketClientFactory) (*Manager, error) {
 	if cfg.LoadPath == "" {
 		return nil, errors.New("LoadPath is empty")
 	}
 
+	if cfg.StorageConfig.Backend == "" {
+		return nil, errors.New("Backend should not be explicitly empty")
+	}
+
 	mgr := Manager{
 		cfg: cfg,
 		configLoadSuccess: promauto.With(registerer).NewGauge(prometheus.GaugeOpts{
@@ -73,19 +87,26 @@ func New(cfg Config, registerer prometheus.Registerer, logger log.Logger) (*Mana
 			Name: "runtime_config_hash",
 			Help: "Hash of the currently active runtime config file.",
 		}, []string{"sha256"}),
-		logger: logger,
+		logger:              logger,
+		bucketClientFactory: factory,
 	}
 
 	mgr.Service = services.NewBasicService(mgr.starting, mgr.loop, mgr.stopping)
 	return &mgr, nil
 }
 
-func (om *Manager) starting(_ context.Context) error {
+func (om *Manager) starting(ctx context.Context) error {
 	if om.cfg.LoadPath == "" {
 		return nil
 	}
 
-	return errors.Wrap(om.loadConfig(), "failed to load runtime config")
+	var err error
+	om.bucketClient, err = om.bucketClientFactory(ctx)
+	if err != nil {
+		return err
+	}
+
+	return errors.Wrap(om.loadConfig(ctx), "failed to load runtime config")
 }
 
 // CreateListenerChannel creates new channel that can be used to receive new config values.
@@ -131,7 +152,7 @@ func (om *Manager) loop(ctx context.Context) error {
 	for {
 		select {
 		case <-ticker.C:
-			err := om.loadConfig()
+			err := om.loadConfig(ctx)
 			if err != nil {
 				// Log but don't stop on error - we don't want to halt all ingesters because of a typo
 				level.Error(om.logger).Log("msg", "failed to load config", "err", err)
@@ -144,8 +165,9 @@ func (om *Manager) loop(ctx context.Context) error {
 
 // loadConfig loads configuration using the loader function, and if successful,
 // stores it as current configuration and notifies listeners.
-func (om *Manager) loadConfig() error {
-	buf, err := os.ReadFile(om.cfg.LoadPath)
+func (om *Manager) loadConfig(ctx context.Context) error {
+	buf, err := om.loadConfigFromBucket(ctx)
+
 	if err != nil {
 		om.configLoadSuccess.Set(0)
 		return errors.Wrap(err, "read file")
@@ -165,10 +187,24 @@ func (om *Manager) loadConfig() error {
 	// expose hash of runtime config
 	om.configHash.Reset()
 	om.configHash.WithLabelValues(fmt.Sprintf("%x", hash[:])).Set(1)
-
 	return nil
 }
 
+func (om *Manager) loadConfigFromBucket(ctx context.Context) ([]byte, error) {
+	readCloser, err := om.bucketClient.Get(ctx, om.cfg.LoadPath)
+	if err != nil {
+		return nil, errors.Wrap(err, "open file")
+	}
+
+	buf, err := io.ReadAll(readCloser)
+	if err != nil {
+		return nil, errors.Wrap(err, "read entire file")
+	}
+
+	err = readCloser.Close()
+	return buf, err
+}
+
 func (om *Manager) setConfig(config interface{}) {
 	om.configMtx.Lock()
 	defer om.configMtx.Unlock()
diff --git a/tools/doc-generator/main.go b/tools/doc-generator/main.go
index a54626ed893..65ee0b8212a 100644
--- a/tools/doc-generator/main.go
+++ b/tools/doc-generator/main.go
@@ -36,6 +36,7 @@ import (
 	"github.com/cortexproject/cortex/pkg/storage/tsdb"
 	"github.com/cortexproject/cortex/pkg/storegateway"
 	"github.com/cortexproject/cortex/pkg/tracing"
+	"github.com/cortexproject/cortex/pkg/util/runtimeconfig"
 	"github.com/cortexproject/cortex/pkg/util/validation"
 )
 
@@ -168,6 +169,11 @@ var (
 			structType: reflect.TypeOf(rulestore.Config{}),
 			desc:       "The ruler_storage_config configures the Cortex ruler storage backend.",
 		},
+		{
+			name:       "runtime_configuration_storage_config",
+			structType: reflect.TypeOf(runtimeconfig.Config{}),
+			desc:       "The runtime_configuration_storage_config configures the storage backend for the runtime configuration file.",
+		},
 		{
 			name:       "s3_sse_config",
 			structType: reflect.TypeOf(s3.SSEConfig{}),