Repository: getsentry/sentry-infra-tools
Pull request #114: feat(sentry-kube): support combining files with keyword
Author: login: victoria-yining-huang
Created: 2025-04-29T20:12:40Z | Merged: 2025-05-09T16:37:40Z

**Context:**
streaming team needs github CODEOWNERS to consumer values so they can merge changes without depending on SRE aproval


**PR explained:**
for every level of overriding logic, find all files that start with the keyword (usually it will be the default value of `_values`) and combine them together, before proceeding with the original overriding logic

**Impacts:**
for services that only has one `_values.yaml` file, this changes nothing

**Rollout plan:**
after this PR is merged, cut a new release, update sentrykubelib version in ops, then make a new `_consumer_values.yaml` file in `getsentry` service, in all overriding levels


Diff:
diff --git a/libsentrykube/helm.py b/libsentrykube/helm.py
index 99d9ecb..758109a 100644
--- a/libsentrykube/helm.py
+++ b/libsentrykube/helm.py
@@ -99,7 +99,9 @@ def inner(*args, **kwargs):
     return inner
 
 
-def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.yaml"):
+def _consolidate_ctx(
+    region_name, service_name, cluster_name, src_files_prefix="_values"
+):
     from libsentrykube.service import (
         get_service_ctx,
         get_service_ctx_overrides,
@@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
         assert_customer_is_defined_at_most_once(
             service_name, region_name, namespace="helm"
         )
 
-    ctx = get_service_ctx(service_name, namespace="helm", src_file=src_file)
+    ctx = get_service_ctx(service_name, namespace="helm")
     ctx_overrides = get_service_ctx_overrides(
         service_name,
         region_name,
         cluster_name,
         namespace="helm",
-        src_file=src_file,
+        src_files_prefix=src_files_prefix,
         cluster_as_folder=True,
     )
     ctx_common = get_common_regional_override(
-        service_name, region_name, namespace="helm", src_file=src_file
+        service_name, region_name, namespace="helm", src_files_prefix=src_files_prefix
     )
     if ctx_overrides or ctx_common:
         deep_merge_dict(ctx, ctx_common)
         deep_merge_dict(ctx, ctx_overrides)
     else:
         ctx_hierarchical = get_hierarchical_value_overrides(
-            service_name, region_name, cluster_name, namespace="helm", src_file=src_file
+            service_name,
+            region_name,
+            cluster_name,
+            namespace="helm",
+            src_files_prefix=src_files_prefix,
         )
         deep_merge_dict(ctx, ctx_hierarchical)
     # get_tools_managed_service_value_overrides?
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
         ctx_region, _ = get_helm_service_data(region_name, service_name, cluster_name)
         deep_merge_dict(ctx, ctx_region)
     return ctx
@@ -144,7 +150,7 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
 def _helm_chart_ctx(region_name, service_name, cluster_name) -> list[HelmRelease]:
     rv = []
     ctx = _consolidate_ctx(
-        region_name, service_name, cluster_name, src_file="_helm.yaml"
+        region_name, service_name, cluster_name, src_files_prefix="_helm"
     )
     chart_spec = ctx.get("chart", {})
     if isinstance(chart_spec, str):
diff --git a/libsentrykube/kube.py b/libsentrykube/kube.py
index 36226e7..e52c5c8 100644
--- a/libsentrykube/kube.py
+++ b/libsentrykube/kube.py
@@ -120,6 +120,20 @@ def _consolidate_variables(
        preserve comments.
     6. overridden by the cluster file. Which is likely going to be replaced by 2 and 3.
 
+    ***
+    For all levels of overrides listed above, in the same directory of file system, we support
+    merging separete values files together, before proceding with the overriding logic
+    Most common examples (numbers refer to override level listed above):
+    1. `k8s/services/getsentry/_values.yaml` content will be combined with `k8s/services/getsentry/_values_consumers.yaml`
+    3. `k8s/services/getsentry/regional_overrides/s4s/_values.yaml` content will be combined with
+       `k8s/services/getsentry/regional_overrides/s4s/_values_consumers.yaml`
+    4. `k8s/services/seer/region_overrides/de/default.yaml` content will be combined with `k8s/services/seer/region_overrides/de/default_2.yaml`
+    Files that should be combined have constraints:
+    - files cannot have conflicting keys. It will fail.
+    - files must be in the same directory in the file system
+    - file names must start with `_values` to be combined with `_values.yaml` file, or must start with `<cluster>` to be combined with `<cluster>.yaml`
+
+
     TODO: write the minimum components of a yaml parser to remove step 3 and
           patch the regional override preserving comments.
     """
diff --git a/libsentrykube/service.py b/libsentrykube/service.py
index 838a8ce..a2a660c 100644
--- a/libsentrykube/service.py
+++ b/libsentrykube/service.py
@@ -95,30 +95,60 @@ def get_service_names(namespace: str | None = None) -> List[str]:
     return [s for s in _services.get(namespace, {}).keys()]
 
 
+def merge_values_files_no_conflict(base: dict, new: dict, file_name: str) -> dict:
+    """
+    All values files in each level of overriding should be flatly combined together.
+    There should be no key conflics
+    Example (these two files should have no key conflicts):
+    _values.yaml
+        workers:
+            rabbit-worker-1:
+                data
+
+    _values_consumer.yaml
+        consumer_groups:
+            consumer-1:
+                data
+    """
+    conflict_keys = base.keys() & new.keys()
+    if conflict_keys:
+        raise ValueError(
+            f"Conflict detected when merging file '{file_name}': duplicate keys {conflict_keys}"
+        )
+    base.update(new)
+    return base
+
+
 def get_service_ctx(
     service_name: str,
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
 ) -> dict:
     """
-    For the given service, return the values specified in the corresponding {src_file}.
+    For the given service, return the combined values from all _values*.yaml files in the service directory.
+
+    Raises an error if duplicate keys are found across files.
 
     If "external=True" is specified, treat the service name as the full service path.
     """
     if external:
-        service_path = workspace_root() / service_name
+        service_path_root = workspace_root() / service_name
     else:
-        service_path = get_service_path(service_name, namespace=namespace)
-    try:
-        with open(service_path / src_file, "rb") as f:
-            return yaml.safe_load(f) or {}
-    except FileNotFoundError:
-        return {}
+        service_path_root = get_service_path(service_name, namespace=namespace)
+
+    ctx: dict[str, dict[str, Any]] = {}
+    for file in service_path_root.glob("_values*.yaml"):
+        try:
+            with open(file, "rb") as f:
+                values = yaml.safe_load(f) or {}
+                ctx = merge_values_files_no_conflict(ctx, values, file.name)
+        except FileNotFoundError:
+            continue
+    return ctx
 
 
 def get_service_values(service_name: str, external: bool = False) -> dict:
-    return get_service_ctx(service_name, external=external, src_file="_values.yaml")
+    return get_service_ctx(service_name, external=external)
 
 
 def get_service_value_override_path(
@@ -151,29 +181,34 @@ def get_service_ctx_overrides(
     cluster_name: str = "default",
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
+    src_files_prefix: str = "_values",
     cluster_as_folder: bool = False,
 ) -> dict:
     """
     For the given service, return the values specified in the corresponding _values.yaml.
     If "external=True" is specified, treat the service name as the full service path.
     """
-    try:
-        override_path = get_service_value_override_path(
-            service_name, region_name, external, namespace=namespace
-        )
-        service_override_file = (
-            override_path / cluster_name / src_file
-            if cluster_as_folder
-            else override_path / f"{cluster_name}.yaml"
-        )
-
-        with open(service_override_file, "rb") as f:
-            values = yaml.safe_load(f) or {}
 
-        return values
-    except FileNotFoundError:
-        return {}
+    ctx: dict[str, dict[str, Any]] = {}
+    override_path = get_service_value_override_path(
+        service_name, region_name, external, namespace=namespace
+    )
+    service_override_file_root = (
+        override_path / cluster_name if cluster_as_folder else override_path
+    )
+    keyword_to_merge_files = (
+        f"{src_files_prefix}*.yaml" if cluster_as_folder else f"{cluster_name}*.yaml"
+    )
+    for file in service_override_file_root.glob(keyword_to_merge_files):
+        try:
+            if file.name.endswith("managed.yaml"):
+                continue
+            with open(file, "rb") as f:
+                values = yaml.safe_load(f) or {}
+                ctx = merge_values_files_no_conflict(ctx, values, file.name)
+        except FileNotFoundError:
+            raise
+    return ctx
 
 
 def get_service_value_overrides(
@@ -192,7 +227,7 @@ def get_common_regional_override(
     region_name: str,
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
+    src_files_prefix: str = "_values",
 ) -> dict:
     """
     Helper function to load common regional configuration values.
@@ -200,18 +235,18 @@ def get_common_regional_override(
     Looks for a '_values.yaml' file in the region's override directory that contains
     settings shared across all clusters in that region.
     """
-    try:
-        common_service_override_file = (
-            get_service_value_override_path(
-                service_name, region_name, external, namespace=namespace
-            )
-            / src_file
-        )
-
-        with open(common_service_override_file, "rb") as f:
-            return yaml.safe_load(f) or {}
-    except FileNotFoundError:
-        return {}
+    common_service_override_file_root = get_service_value_override_path(
+        service_name, region_name, external, namespace=namespace
+    )
+    ctx: dict[str, dict[str, Any]] = {}
+    for file in common_service_override_file_root.glob(f"{src_files_prefix}*.yaml"):
+        try:
+            with open(file, "rb") as f:
+                values = yaml.safe_load(f) or {}
+                ctx = merge_values_files_no_conflict(ctx, values, file.name)
+        except FileNotFoundError:
+            continue
+    return ctx
 
 
 def get_hierarchical_value_overrides(
@@ -220,7 +255,7 @@ def get_hierarchical_value_overrides(
     cluster_name: str = "default",
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
+    src_files_prefix: str = "_values",
 ) -> dict:
     """
     Enables hierarchical configuration overrides with shared base values.
@@ -255,13 +290,15 @@ def get_hierarchical_value_overrides(
         if not override_group.is_dir():
             continue
 
+        service_override_file_root = service_regions_path / override_group.name
+        base_values: dict[str, dict[str, Any]] = {}
         try:
-            service_override_file = (
-                service_regions_path / override_group.name / src_file
-            )
-
-            with open(service_override_file, "rb") as f:
-                base_values = yaml.safe_load(f) or {}
+            for file in service_override_file_root.glob(f"{src_files_prefix}*.yaml"):
+                with open(file, "rb") as f:
+                    values = yaml.safe_load(f) or {}
+                base_values = merge_values_files_no_conflict(
+                    base_values, values, file.name
+                )
         except FileNotFoundError:
             base_values = {}
 
@@ -275,12 +312,16 @@ def get_hierarchical_value_overrides(
             cluster_name,
             external,
             namespace=namespace,
-            src_file=src_file,
+            src_files_prefix=src_files_prefix,
             cluster_as_folder=namespace == "helm",
         )
 
         common_service_values = get_common_regional_override(
-            service_name, region_path, external, namespace=namespace, src_file=src_file
+            service_name,
+            region_path,
+            external,
+            namespace=namespace,
+            src_files_prefix=src_files_prefix,
         )
 
         # There must be either a cluster specific override file a _values.yaml in the region dir
diff --git a/libsentrykube/tests/k8s_root/services/service1/_values_consumers.yaml b/libsentrykube/tests/k8s_root/services/service1/_values_consumers.yaml
new file mode 100644
index 0000000..bedf478
--- /dev/null
+++ b/libsentrykube/tests/k8s_root/services/service1/_values_consumers.yaml
@@ -0,0 +1,6 @@
+consumer_key1: value1
+consumer_key2:
+  subkey2_1: value2_1
+  subkey2_2: 2
+  subkey2_3:
+    - value2_3_1
diff --git a/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer.managed.yaml b/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer.managed.yaml
index d90977f..677438f 100644
--- a/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer.managed.yaml
+++ b/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer.managed.yaml
@@ -3,3 +3,8 @@ key2:
     - value2_4_1_managed_replaced
   subkey2_5:
     - value2_5_1_managed_replaced
+consumer_key2:
+  subkey2_4:
+    - value2_4_1_managed_replaced
+  subkey2_5:
+    - value2_5_1_managed_replaced
diff --git a/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer_consumers.yaml b/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer_consumers.yaml
new file mode 100644
index 0000000..159f859
--- /dev/null
+++ b/libsentrykube/tests/k8s_root/services/service1/region_overrides/us/customer_consumers.yaml
@@ -0,0 +1,5 @@
+consumer_key2:
+  subkey2_3:
+    - value2_3_1_replaced
+  subkey2_4:
+    - value2_4_1_replaced
diff --git a/libsentrykube/tests/test_kube.py b/libsentrykube/tests/test_kube.py
index ef1196f..9e88ffb 100644
--- a/libsentrykube/tests/test_kube.py
+++ b/libsentrykube/tests/test_kube.py
@@ -22,6 +22,18 @@
                         "value2_5_1_managed_replaced"
                     ],  # From the managed file
                 },
+                "consumer_key1": "value1",
+                "consumer_key2": {
+                    "subkey2_1": "value2_1",  # From the value file
+                    "subkey2_2": 2,  # From the value file
+                    "subkey2_3": ["value2_3_1_replaced"],  # From the region override
+                    "subkey2_4": [
+                        "value2_4_1_managed_replaced"
+                    ],  # From the managed file
+                    "subkey2_5": [
+                        "value2_5_1_managed_replaced"
+                    ],  # From the managed file
+                },
             },
             "service2": {
                 "key3": "three",  # From the cluster file
diff --git a/libsentrykube/tests/test_service.py b/libsentrykube/tests/test_service.py
index b8ec497..ecef98b 100644
--- a/libsentrykube/tests/test_service.py
+++ b/libsentrykube/tests/test_service.py
@@ -1,3 +1,4 @@
+import pytest
 from libsentrykube.context import init_cluster_context
 import os
 from pathlib import Path
@@ -14,6 +15,7 @@
     get_service_path,
     get_service_template_files,
     write_managed_values_overrides,
+    merge_values_files_no_conflict,
 )
 from libsentrykube.utils import set_workspace_root_start
 from libsentrykube.utils import workspace_root
@@ -33,6 +35,12 @@
                     "subkey2_2": 2,
                     "subkey2_3": ["value2_3_1"],
                 },
+                "consumer_key1": "value1",
+                "consumer_key2": {
+                    "subkey2_1": "value2_1",
+                    "subkey2_2": 2,
+                    "subkey2_3": ["value2_3_1"],
+                },
             },
             "service2": {},
         }
@@ -48,7 +56,13 @@
                     "subkey2_4": [
                         "value2_4_1_replaced"  # From the managed file
                     ],
-                }
+                },
+                "consumer_key2": {
+                    "subkey2_3": ["value2_3_1_replaced"],
+                    "subkey2_4": [
+                        "value2_4_1_replaced"  # From the managed file
+                    ],
+                },
             },
             "service2": {},
         }
@@ -62,7 +76,11 @@
                 "key2": {
                     "subkey2_4": ["value2_4_1_managed_replaced"],
                     "subkey2_5": ["value2_5_1_managed_replaced"],
-                }
+                },
+                "consumer_key2": {
+                    "subkey2_4": ["value2_4_1_managed_replaced"],
+                    "subkey2_5": ["value2_5_1_managed_replaced"],
+                },
             },
             "service2": {},
         }
@@ -243,7 +261,7 @@ def test_get_helm_hierarchical_value_overrides(
         region_name="customer1",
         cluster_name="cluster1",
         namespace="helm",
-        src_file="_helm.yaml",
+        src_files_prefix="_helm",
     )
 
     assert returned_values == expected_hierarchical_and_regional_cluster_values
@@ -289,7 +307,7 @@ def test_helm_regional_cluster_value_overrides(
         region_name="customer1",
         cluster_name="cluster1",
         namespace="helm",
-        src_file="_helm.yaml",
+        src_files_prefix="_helm",
         cluster_as_folder=True,
     )
 
@@ -322,3 +340,24 @@ def test_get_service_template_files():
         assert Path("template2.yml") in templates
         assert Path("template3.yaml.j2") in templates
         assert Path("template4.yml.j2") in templates
+
+
+def test_verify_values_files_no_conflict_no_conflict():
+    base = {"workers": {"rabbit-worker-1": {"some": "data"}}}
+    new = {"consumer_groups": {"consumer-1": {"other": "data"}}}
+    expected = {
+        "workers": {"rabbit-worker-1": {"some": "data"}},
+        "consumer_groups": {"consumer-1": {"other": "data"}},
+    }
+    result = merge_values_files_no_conflict(base.copy(), new, "_values_consumers.yaml")
+    assert result == expected
+
+
+def test_verify_values_files_no_conflict_with_conflict():
+    base = {"workers": {"rabbit-worker-1": {"some": "data"}}}
+    new = {"workers": {"rabbit-worker-2": {"other": "data"}}}
+    with pytest.raises(ValueError) as excinfo:
+        merge_values_files_no_conflict(base.copy(), new, "_values_consumers.yaml")
+
+    assert "Conflict detected when merging file" in str(excinfo.value)
+    assert "duplicate keys {'workers'}" in str(excinfo.value)


Reviews:
user: login: fpacifici
body: Please see my comment inline. There are quite a few more cases to adjust.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T01:19:16Z
user: login: rgibert
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-30T13:51:21Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:28:47Z
user: login: mwarkentin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T15:16:08Z
user: login: rgibert
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-06T15:32:12Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:48:15Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:48:50Z
user: login: mwarkentin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T16:27:24Z
user: login: rgibert
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T19:40:48Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T19:58:20Z
user: login: fpacifici
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T22:06:50Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T18:19:20Z
user: login: victoria-yining-huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T18:33:20Z
user: login: fpacifici
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T22:20:47Z
user: login: fpacifici
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T21:52:51Z

Review comments:
diff_hunk: @@ -192,26 +222,25 @@ def get_common_regional_override(
     region_name: str,
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
 ) -> dict:
     """
     Helper function to load common regional configuration values.
 
-    Looks for a '_values.yaml' file in the region's override directory that contains
+    Merges all files that end with '_values.yaml' in the region's override directory that contains
path: libsentrykube/service.py
user: login: fpacifici
body: I think this is not the right place to support multiple file for regional overrides.
This loads a file that is common to all regions for a service. It is not the regional override file.
For example, this file is loading [this](https://github.com/getsentry/ops/tree/master/k8s/services/relay-pop/region_overrides/saas) value file.

The most commonly utilized override files are in each region directory:
Example: https://github.com/getsentry/ops/tree/master/k8s/services/relay-pop/region_overrides/saas/us

Taking a step back there are a number of corner cases to be taken into account to approach this problem, and several other places to update (follow [this](https://github.com/getsentry/sentry-infra-tools/blob/main/libsentrykube/kube.py#L107-L121) ) :
1. The service value file. The one in the service directory. Is this supposed to be broken down into more files ? If yes, `get_service_values` needs to be updated .
2. Is the common regional override (a per service override file common to all regions) to be broken? See here https://github.com/getsentry/sentry-infra-tools/blob/main/libsentrykube/kube.py#L139-L142
3. Is the regional override file supposed to be broken? Certainly yes.
4. Is the hierarchical override supposed to be broken ? Maybe not. https://github.com/getsentry/sentry-infra-tools/blob/main/libsentrykube/kube.py#L157
5. In several of the above the naming convention can be strict: see pops. The override file name means the name of the k8s cluster so you cannot just add other files. You'd have to allow multiple files per k8s cluster thus adding a new naming convention: See this example https://github.com/getsentry/ops/tree/master/k8s/services/relay-pop/region_overrides/saas/us. We already have multiple override files for pops. But they signify the k8s cluster.
created_at: 2025-04-30T01:18:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -192,26 +222,25 @@ def get_common_regional_override(
     region_name: str,
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
 ) -> dict:
     """
     Helper function to load common regional configuration values.
 
-    Looks for a '_values.yaml' file in the region's override directory that contains
+    Merges all files that end with '_values.yaml' in the region's override directory that contains
path: libsentrykube/service.py
user: login: rgibert
body: 4. Ideally we'd support this being broken, IMO every layer that takes a `_values` should support the new mechanism too for consistency and flexibility.
5. I think we should use something like a `_values_*` naming convention to support override files vs cluster files.  Today `_values` are defaults and all other files are considered cluster names.  If we have `_values_streaming.yaml` or similar it'd keep this division clear (and help with alpha-sorting files grouping the overrides vs cluster files).
created_at: 2025-04-30T13:51:21Z
author_association: MEMBER
diff_hunk: @@ -192,26 +222,25 @@ def get_common_regional_override(
     region_name: str,
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
 ) -> dict:
     """
     Helper function to load common regional configuration values.
 
-    Looks for a '_values.yaml' file in the region's override directory that contains
+    Merges all files that end with '_values.yaml' in the region's override directory that contains
path: libsentrykube/service.py
user: login: victoria-yining-huang
body: updated PR, every layer now supports combining files based on a prefix keyword such as `_values`
created_at: 2025-05-05T18:28:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,30 +95,60 @@ def get_service_names(namespace: str | None = None) -> List[str]:
     return [s for s in _services.get(namespace, {}).keys()]
 
 
+def merge_values_files_no_conflict(base: dict, new: dict, file_name: str) -> dict:
+    """
+    All values files in each level of overriding should be flatly combined together.
+    There should be no key conflics
+    Example (these two files should have no key conflicts):
+    _values.yaml
+        workers:
+            rabbit-worker-1:
+                data
+
+    _consumer_values.yaml
path: libsentrykube/service.py
user: login: rgibert
body: ```suggestion
    _values_consumer.yaml
```
created_at: 2025-05-06T15:15:11Z
author_association: MEMBER
diff_hunk: @@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
path: libsentrykube/helm.py
user: login: mwarkentin
body: Nit: isn't this more of a suffix? Or maybe `src_files_match`?
created_at: 2025-05-06T15:16:08Z
author_association: MEMBER
diff_hunk: @@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
path: libsentrykube/helm.py
user: login: rgibert
body: It's prefixed to all files like `_values_sre.yaml` and `_values_streaming.yaml`
created_at: 2025-05-06T15:31:47Z
author_association: MEMBER
diff_hunk: @@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
path: libsentrykube/helm.py
user: login: victoria-yining-huang
body: no, Richard had a prior comment about using the convention `_values_<logical_grouping>.yaml` so like `_values_consumers.yaml`. We both think this way the alphabetical ordering of files in the repo will be more apparent
created_at: 2025-05-06T15:48:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
path: libsentrykube/helm.py
user: login: victoria-yining-huang
body: the same thing goes for cluster file contents. `default.yaml` and `default_consumers.yaml`
created_at: 2025-05-06T15:48:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,33 +111,37 @@ def _consolidate_ctx(region_name, service_name, cluster_name, src_file="_values.
         assert_customer_is_defined_at_most_once,
     )
 
-    if src_file == "_values.yaml":
+    if src_files_prefix == "_values":
path: libsentrykube/helm.py
user: login: mwarkentin
body: Thanks, resolving!
created_at: 2025-05-06T16:27:24Z
author_association: MEMBER
diff_hunk: @@ -120,6 +120,20 @@ def _consolidate_variables(
        preserve comments.
     6. overridden by the cluster file. Which is likely going to be replaced by 2 and 3.
 
+    ***
+    For all levels of overrides listed above, in the same directory of file system, we support
+    merging separete values files together, before proceding with the overriding logic
path: libsentrykube/kube.py
user: login: rgibert
body: typo
```suggestion
    merging separate values files together, before proceding with the overriding logic
```
created_at: 2025-05-06T19:37:48Z
author_association: MEMBER
diff_hunk: @@ -120,6 +120,20 @@ def _consolidate_variables(
        preserve comments.
     6. overridden by the cluster file. Which is likely going to be replaced by 2 and 3.
 
+    ***
+    For all levels of overrides listed above, in the same directory of file system, we support
+    merging separete values files together, before proceding with the overriding logic
+    Most common examples (numbers refer to override level listed above):
+    1. `k8s/services/getsentry/_values.yaml` content will be combined with `k8s/services/getsentry/_values_consumers.yaml`
+    3. `k8s/services/getsentry/regional_overrides/s4s/_values.yaml` content will be combined with
+       `k8s/services/getsentry/regional_overrides/s4s/_values_consumers.yaml`
+    4. `sk8s/services/seer/region_overrides/de/default.yaml` content will be combined with ``sk8s/services/seer/region_overrides/de/default_2.yaml`
path: libsentrykube/kube.py
user: login: rgibert
body: typos
```suggestion
    4. `k8s/services/seer/region_overrides/de/default.yaml` content will be combined with `k8s/services/seer/region_overrides/de/default_2.yaml`
```

Is there a reason we use a `default_2` example instead of a `default_consumers`?
created_at: 2025-05-06T19:38:30Z
author_association: MEMBER
diff_hunk: @@ -120,6 +120,20 @@ def _consolidate_variables(
        preserve comments.
     6. overridden by the cluster file. Which is likely going to be replaced by 2 and 3.
 
+    ***
+    For all levels of overrides listed above, in the same directory of file system, we support
+    merging separete values files together, before proceding with the overriding logic
+    Most common examples (numbers refer to override level listed above):
+    1. `k8s/services/getsentry/_values.yaml` content will be combined with `k8s/services/getsentry/_values_consumers.yaml`
+    3. `k8s/services/getsentry/regional_overrides/s4s/_values.yaml` content will be combined with
+       `k8s/services/getsentry/regional_overrides/s4s/_values_consumers.yaml`
+    4. `sk8s/services/seer/region_overrides/de/default.yaml` content will be combined with ``sk8s/services/seer/region_overrides/de/default_2.yaml`
+    Files that should be combined have constraints:
+    - files cannot have conflicting keys. It will fail.
+    - files must be in the same direcotry in the file system
path: libsentrykube/kube.py
user: login: rgibert
body: ```suggestion
    - files must be in the same directory in the file system
```
created_at: 2025-05-06T19:39:24Z
author_association: MEMBER
diff_hunk: @@ -120,6 +120,20 @@ def _consolidate_variables(
        preserve comments.
     6. overridden by the cluster file. Which is likely going to be replaced by 2 and 3.
 
+    ***
+    For all levels of overrides listed above, in the same directory of file system, we support
+    merging separete values files together, before proceding with the overriding logic
+    Most common examples (numbers refer to override level listed above):
+    1. `k8s/services/getsentry/_values.yaml` content will be combined with `k8s/services/getsentry/_values_consumers.yaml`
+    3. `k8s/services/getsentry/regional_overrides/s4s/_values.yaml` content will be combined with
+       `k8s/services/getsentry/regional_overrides/s4s/_values_consumers.yaml`
+    4. `sk8s/services/seer/region_overrides/de/default.yaml` content will be combined with ``sk8s/services/seer/region_overrides/de/default_2.yaml`
path: libsentrykube/kube.py
user: login: victoria-yining-huang
body: just because i don't think the `seer` service would have a consumers file, i was trying to give real examples. Not sure what example to give here. 
created_at: 2025-05-06T19:58:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,30 +95,60 @@ def get_service_names(namespace: str | None = None) -> List[str]:
     return [s for s in _services.get(namespace, {}).keys()]
 
 
+def merge_values_files_no_conflict(base: dict, new: dict, file_name: str) -> dict:
+    """
+    All values files in each level of overriding should be flatly combined together.
+    There should be no key conflics
+    Example (these two files should have no key conflicts):
+    _values.yaml
+        workers:
+            rabbit-worker-1:
+                data
+
+    _values_consumer.yaml
+        consumer_groups:
+            consumer-1:
+                data
+    """
+    conflict_keys = base.keys() & new.keys()
path: libsentrykube/service.py
user: login: fpacifici
body: This is not enough. 
Let's say the issues tame takes over the ingest consumers and the subscription results consumers, it would not be possible to give them access to some specific consumers as we would have to break the value file according to a subkey of `consumer_groups`.

So you will need to allow some level of overlap. You could allow the same key to be defined in multiple places if it is a collection (like a map) as long as there is no conflict between the subkeys. The result of the merge would be a merge of the collections rather than a substitution.
created_at: 2025-05-06T21:33:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -151,29 +181,34 @@ def get_service_ctx_overrides(
     cluster_name: str = "default",
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
+    src_files_prefix: str = "_values",
     cluster_as_folder: bool = False,
 ) -> dict:
     """
     For the given service, return the values specified in the corresponding _values.yaml.
     If "external=True" is specified, treat the service name as the full service path.
     """
-    try:
-        override_path = get_service_value_override_path(
-            service_name, region_name, external, namespace=namespace
-        )
-        service_override_file = (
-            override_path / cluster_name / src_file
-            if cluster_as_folder
-            else override_path / f"{cluster_name}.yaml"
-        )
-
-        with open(service_override_file, "rb") as f:
-            values = yaml.safe_load(f) or {}
 
-        return values
-    except FileNotFoundError:
-        return {}
+    ctx: dict[str, dict[str, Any]] = {}
+    override_path = get_service_value_override_path(
+        service_name, region_name, external, namespace=namespace
+    )
+    service_override_file_root = (
+        override_path / cluster_name if cluster_as_folder else override_path
+    )
+    keyword_to_merge_files = (
+        f"{src_files_prefix}*.yaml" if cluster_as_folder else f"{cluster_name}*.yaml"
+    )
+    for file in service_override_file_root.glob(keyword_to_merge_files):
+        try:
+            if "managed" in file.name:
+                continue
path: libsentrykube/service.py
user: login: fpacifici
body: This is supposed to be a suffix, not a substring at any point in the filename.
created_at: 2025-05-06T22:05:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,30 +95,60 @@ def get_service_names(namespace: str | None = None) -> List[str]:
     return [s for s in _services.get(namespace, {}).keys()]
 
 
+def merge_values_files_no_conflict(base: dict, new: dict, file_name: str) -> dict:
+    """
+    All values files in each level of overriding should be flatly combined together.
+    There should be no key conflics
+    Example (these two files should have no key conflicts):
+    _values.yaml
+        workers:
+            rabbit-worker-1:
+                data
+
+    _values_consumer.yaml
+        consumer_groups:
+            consumer-1:
+                data
+    """
+    conflict_keys = base.keys() & new.keys()
path: libsentrykube/service.py
user: login: victoria-yining-huang
body: > Let's say the issues tame takes over the ingest consumers

is that really happening soon that we should build the support now? I think it's a good idea but I was only under the assumption that streaming team would own all consumers for the next little while
created_at: 2025-05-07T18:19:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -151,29 +181,34 @@ def get_service_ctx_overrides(
     cluster_name: str = "default",
     external: bool = False,
     namespace: str | None = None,
-    src_file: str = "_values.yaml",
+    src_files_prefix: str = "_values",
     cluster_as_folder: bool = False,
 ) -> dict:
     """
     For the given service, return the values specified in the corresponding _values.yaml.
     If "external=True" is specified, treat the service name as the full service path.
     """
-    try:
-        override_path = get_service_value_override_path(
-            service_name, region_name, external, namespace=namespace
-        )
-        service_override_file = (
-            override_path / cluster_name / src_file
-            if cluster_as_folder
-            else override_path / f"{cluster_name}.yaml"
-        )
-
-        with open(service_override_file, "rb") as f:
-            values = yaml.safe_load(f) or {}
 
-        return values
-    except FileNotFoundError:
-        return {}
+    ctx: dict[str, dict[str, Any]] = {}
+    override_path = get_service_value_override_path(
+        service_name, region_name, external, namespace=namespace
+    )
+    service_override_file_root = (
+        override_path / cluster_name if cluster_as_folder else override_path
+    )
+    keyword_to_merge_files = (
+        f"{src_files_prefix}*.yaml" if cluster_as_folder else f"{cluster_name}*.yaml"
+    )
+    for file in service_override_file_root.glob(keyword_to_merge_files):
+        try:
+            if "managed" in file.name:
+                continue
path: libsentrykube/service.py
user: login: victoria-yining-huang
body: fixed
created_at: 2025-05-07T18:33:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,30 +95,60 @@ def get_service_names(namespace: str | None = None) -> List[str]:
     return [s for s in _services.get(namespace, {}).keys()]
 
 
+def merge_values_files_no_conflict(base: dict, new: dict, file_name: str) -> dict:
+    """
+    All values files in each level of overriding should be flatly combined together.
+    There should be no key conflics
+    Example (these two files should have no key conflicts):
+    _values.yaml
+        workers:
+            rabbit-worker-1:
+                data
+
+    _values_consumer.yaml
+        consumer_groups:
+            consumer-1:
+                data
+    """
+    conflict_keys = base.keys() & new.keys()
path: libsentrykube/service.py
user: login: fpacifici
body: > is that really happening soon

Yes it is happening as part of the bearhug for celery workers and hopefully for consumers.
Actually this is already supposed to happen now. The streaming platform is not going to take over the metrics indexer (sns) and it is a getsentry consumer.
created_at: 2025-05-07T22:20:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: mwarkentin
created_at: 2025-05-06T15:19:08Z
body: Not sure where, but it would be nice to have some documentation about how our values merging works, maybe in https://github.com/getsentry/sentry-infra-tools/tree/main/libsentrykube/README.md? 

Do we provide any sort of ordering logic? Is it purely alphabetical? What happens in case of a conflict?
author_association: MEMBER
user: login: rgibert
created_at: 2025-05-06T15:52:05Z
body: > Not sure where, but it would be nice to have some documentation about how our values merging works, maybe in https://github.com/getsentry/sentry-infra-tools/tree/main/libsentrykube/README.md?
> 
> Do we provide any sort of ordering logic? Is it purely alphabetical? What happens in case of a conflict?

`merge_values_files_no_conflict()` should enforce no conflicts.
author_association: MEMBER
user: login: victoria-yining-huang
created_at: 2025-05-06T16:36:50Z
body: @mwarkentin in terms of documentation, I was pointed to this block of comments https://github.com/getsentry/sentry-infra-tools/blob/57a7eb15495e5101cb7663250f0c8fdc85a085ba/libsentrykube/kube.py#L107 
Not sure if there exists another doc like this
author_association: CONTRIBUTOR
user: login: mwarkentin
created_at: 2025-05-06T16:45:16Z
body: Ok, at a minimum we should be updating that block with the new functionality, however it does feel like we should have something a little more discoverable / usable for introducing people to how sentry-kube works (values generation being a big part).

I don't have a strong opinion but either in a sentry-infra-tools readme or docs/ dir, or a notion page.
author_association: MEMBER