Repository: elastic/rally
Pull request #1912: Preserve pre-existing `rally-metrics` index templates by default
Author: login: fressi-elastic
Labels: name: enhancement
description: Improves the status quo
name: highlight
description: A substantial improvement that is worth mentioning separately in release notes
Created: 2025-02-05T10:59:15Z | Merged: 2025-03-06T06:31:55Z

It handles issue #1900: Rally should not overwrite pre existing templates by default

When opening an EsMetricsStore, it creates the index template if any of following is true:

- the index template doesn't exist
- `reporting/datastore.overwrite_existing_templates` option is true and there are differences
  between existing and requested template.

It will preserve existing template on all the other cases.
It logs a warning when an existing index template is being replaced.
It logs index template differences between the existing one (if any) and configured one.


Diff:
diff --git a/docs/configuration.rst b/docs/configuration.rst
index 14b6a68ea..0abfd982b 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -80,6 +80,8 @@ The following settings are applicable only if ``datastore.type`` is set to "elas
 * ``datastore.probe.cluster_version`` (default: true): Enables automatic detection of the metric store's version.
 * ``datastore.number_of_shards`` (default: `Elasticsearch default value <https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings>`_): The number of primary shards that the ``rally-*`` indices should have. Any updates to this setting after initial index creation will only be applied to new ``rally-*`` indices.
 * ``datastore.number_of_replicas`` (default: `Elasticsearch default value <https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings>`_): The number of replicas each primary shard has. Defaults to . Any updates to this setting after initial index creation will only be applied to new ``rally-*`` indices.
+* ``datastore.overwrite_existing_templates`` (default: ``false``): Existing Rally index templates are replaced only when this option is ``true``.
+
 
 **Examples**
 
diff --git a/docs/migrate.rst b/docs/migrate.rst
index f85604e97..72e08bbfe 100644
--- a/docs/migrate.rst
+++ b/docs/migrate.rst
@@ -9,6 +9,12 @@ Minimum Python version is 3.9.0
 
 Rally 2.12.0 requires Python 3.9.0 or above. Check the :ref:`updated installation instructions <install_python>` for more details.
 
+The metrics store keeps existing index templates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Existing Rally index templates are replaced only when option ``datastore.overwrite_existing_templates`` in section ``reporting`` is ``true``.
+
+
 Migrating to Rally 2.10.1
 -------------------------
 
diff --git a/esrally/metrics.py b/esrally/metrics.py
index 969743e0f..63be29655 100644
--- a/esrally/metrics.py
+++ b/esrally/metrics.py
@@ -14,6 +14,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+from __future__ import annotations
 
 import collections
 import datetime
@@ -33,7 +34,7 @@
 import tabulate
 
 from esrally import client, config, exceptions, paths, time, types, version
-from esrally.utils import console, convert, io, versions
+from esrally.utils import console, convert, io, pretty, versions
 
 
 class EsClient:
@@ -47,6 +48,9 @@ def __init__(self, client, cluster_version=None):
         self._cluster_version = cluster_version
         self.retryable_status_codes = [502, 503, 504, 429]
 
+    def get_template(self, name):
+        return self.guarded(self._client.indices.get_index_template, name=name)
+
     def put_template(self, name, template):
         tmpl = json.loads(template)
         return self.guarded(self._client.indices.put_index_template, name=name, **tmpl)
@@ -898,8 +902,7 @@ def open(self, race_id=None, race_timestamp=None, track_name=None, challenge_nam
         self._index = self.index_name()
         # reduce a bit of noise in the metrics cluster log
         if create:
-            # always update the mapping to the latest version
-            self._client.put_template("rally-metrics", self._get_template())
+            self._ensure_index_template()
             if not self._client.exists(index=self._index):
                 self._client.create_index(index=self._index)
             else:
@@ -913,6 +916,34 @@ def open(self, race_id=None, race_timestamp=None, track_name=None, challenge_nam
         # ensure we can search immediately after opening
         self._client.refresh(index=self._index)
 
+    def _ensure_index_template(self):
+        new_template: str = self._get_template()
+
+        old_template: dict | None = None
+        if self._client.template_exists("rally-metrics"):
+            for t in self._client.get_template("rally-metrics").body.get("index_templates", []):
+                old_template = t.get("index_template", {}).get("template", {})
+                break
+
+        if old_template is None:
+            self.logger.info(
+                "Create index template:\n%s",
+                pretty.dump(json.loads(new_template).get("template", {}), pretty.Flag.FLAT_DICT),
+            )
+        else:
+            diff = pretty.diff(old_template, json.loads(new_template).get("template", {}), pretty.Flag.FLAT_DICT)
+            if diff == "":
+                self.logger.debug("Keep existing template (it is identical)")
+                return
+            if not convert.to_bool(
+                self._config.opts(section="reporting", key="datastore.overwrite_existing_templates", default_value=False, mandatory=False)
+            ):
+                self.logger.debug("Keep existing template (datastore.overwrite_existing_templates = false):\n%s", diff)
+                return
+            self.logger.warning("Overwrite existing index template (datastore.overwrite_existing_templates = true):\n%s", diff)
+
+        self._client.put_template("rally-metrics", new_template)
+
     def index_name(self):
         ts = time.from_iso8601(self._race_timestamp)
         return "rally-metrics-%04d-%02d" % (ts.year, ts.month)
diff --git a/esrally/types.py b/esrally/types.py
index 17f9ca393..eef958581 100644
--- a/esrally/types.py
+++ b/esrally/types.py
@@ -66,6 +66,7 @@
     "datastore.host",
     "datastore.number_of_replicas",
     "datastore.number_of_shards",
+    "datastore.overwrite_existing_templates",
     "datastore.password",
     "datastore.port",
     "datastore.probe.cluster_version",
diff --git a/pyproject.toml b/pyproject.toml
index daf818a9a..292b9a80f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -120,7 +120,7 @@ develop = [
     "black==24.10.0",
     # mypy
     "boto3-stubs==1.26.125",
-    "mypy==1.10.1",
+    "mypy==1.15.0",
     "types-psutil==5.9.4",
     "types-tabulate==0.8.9",
     "types-urllib3==1.26.19",
diff --git a/tests/metrics_test.py b/tests/metrics_test.py
index 524dd788a..df52f4c7f 100644
--- a/tests/metrics_test.py
+++ b/tests/metrics_test.py
@@ -15,6 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 # pylint: disable=protected-access
+from __future__ import annotations
 
 import datetime
 import json
@@ -27,6 +28,7 @@
 from dataclasses import dataclass
 from unittest import mock
 
+import elastic_transport
 import elasticsearch.exceptions
 import elasticsearch.helpers
 import pytest
@@ -34,10 +36,47 @@
 from esrally import config, exceptions, metrics, paths, track
 from esrally.metrics import GlobalStatsCalculator
 from esrally.track import Challenge, Operation, Task, Track
-from esrally.utils import opts
+from esrally.utils import cases, opts, pretty
+
+
+def rally_metric_template():
+    return {
+        "mappings": {
+            "_source": {"enabled": True},
+            "date_detection": False,
+            "dynamic_templates": [
+                {"strings": {"mapping": {"ignore_above": 8191, "type": "keyword"}, "match": "*", "match_mapping_type": "string"}}
+            ],
+            "properties": {
+                "@timestamp": {"format": "epoch_millis", "type": "date"},
+                "car": {"type": "keyword"},
+                "challenge": {"type": "keyword"},
+                "environment": {"type": "keyword"},
+                "job": {"type": "keyword"},
+                "max": {"type": "float"},
+                "mean": {"type": "float"},
+                "median": {"type": "float"},
+                "meta": {"properties": {"error-description": {"type": "wildcard"}}},
+                "min": {"type": "float"},
+                "name": {"type": "keyword"},
+                "operation": {"type": "keyword"},
+                "operation-type": {"type": "keyword"},
+                "race-id": {"type": "keyword"},
+                "race-timestamp": {"fields": {"raw": {"type": "keyword"}}, "format": "basic_date_time_no_millis", "type": "date"},
+                "relative-time": {"type": "float"},
+                "sample-type": {"type": "keyword"},
+                "task": {"type": "keyword"},
+                "track": {"type": "keyword"},
+                "unit": {"type": "keyword"},
+                "value": {"type": "float"},
+            },
+        },
+        "settings": {"index": {"mapping": {"total_fields": {"limit": "2000"}}, "number_of_replicas": "3", "number_of_shards": "3"}},
+    }
 
 
 class MockClientFactory:
+
     def __init__(self, cfg):
         self._es = mock.create_autospec(metrics.EsClient)
 
@@ -49,8 +88,8 @@ class DummyIndexTemplateProvider:
     def __init__(self, cfg):
         pass
 
-    def metrics_template(self):
-        return "metrics-test-template"
+    def metrics_template(self) -> str:
+        return json.dumps({"index_patterns": ["rally-metrics-*"], "template": provided_metrics_template()})
 
     def races_template(self):
         return "races-test-template"
@@ -59,6 +98,12 @@ def results_template(self):
         return "results-test-template"
 
 
+def provided_metrics_template() -> dict:
+    template = rally_metric_template()
+    template["settings"]["index"] = {"mapping.total_fields.limit": 2000, "number_of_shards": 1, "number_of_replicas": 1}
+    return template
+
+
 class StaticClock:
     NOW = 1453362707
 
@@ -438,6 +483,67 @@ def setup_method(self, method):
         # get hold of the mocked client...
         self.es_mock = self.metrics_store._client
         self.es_mock.exists.return_value = False
+        self.es_mock.template_exists.return_value = False
+        self.es_mock.get_template.return_value = mock.create_autospec(elastic_transport.ObjectApiResponse, body={"index_templates": []})
+        self.metrics_store.logger = mock.create_autospec(logging.Logger)
+
+    @dataclass
+    class OpenCase:
+        create: bool = True
+        template: dict | None = None
+        overwrite_templates: str | None = None
+        want_put_template: bool = False
+        want_logger_call: mock._Call | None = None
+
+    @cases.cases(
+        create_false=OpenCase(create=False),
+        default=OpenCase(
+            want_put_template=True,
+            want_logger_call=mock.call.info("Create index template:\n%s", pretty.dump(provided_metrics_template(), pretty.Flag.FLAT_DICT)),
+        ),
+        template_exists=OpenCase(
+            template=rally_metric_template(),
+            want_logger_call=mock.call.debug(
+                "Keep existing template (datastore.overwrite_existing_templates = false):\n%s",
+                pretty.diff(rally_metric_template(), provided_metrics_template(), pretty.Flag.FLAT_DICT),
+            ),
+        ),
+        keep_identical_template=OpenCase(
+            template=provided_metrics_template(), want_logger_call=mock.call.debug("Keep existing template (it is identical)")
+        ),
+        overwrite_templates_true=OpenCase(
+            template=rally_metric_template(),
+            overwrite_templates="true",
+            want_put_template=True,
+            want_logger_call=mock.call.warning(
+                "Overwrite existing index template (datastore.overwrite_existing_templates = true):\n%s",
+                pretty.diff(rally_metric_template(), provided_metrics_template(), pretty.Flag.FLAT_DICT),
+            ),
+        ),
+        overwrite_templates_false=OpenCase(
+            template=rally_metric_template(),
+            overwrite_templates="false",
+            want_logger_call=mock.call.debug(
+                "Keep existing template (datastore.overwrite_existing_templates = false):\n%s",
+                pretty.diff(rally_metric_template(), provided_metrics_template(), pretty.Flag.FLAT_DICT),
+            ),
+        ),
+    )
+    def test_open(self, case: OpenCase):
+        if case.template is not None:
+            self.metrics_store._client.template_exists.return_value = True
+            self.metrics_store._client.get_template.return_value.body["index_templates"] = [{"index_template": {"template": case.template}}]
+        if case.overwrite_templates is not None:
+            self.cfg.add(
+                scope=config.Scope.application,
+                section="reporting",
+                key="datastore.overwrite_existing_templates",
+                value=case.overwrite_templates,
+            )
+        self.metrics_store.open(self.RACE_ID, self.RACE_TIMESTAMP, "test", "append", "defaults", create=case.create)
+        assert case.want_put_template == self.metrics_store._client.put_template.called
+        if case.want_logger_call is not None:
+            assert self.metrics_store.logger.method_calls[-1:] == [case.want_logger_call]
 
     def test_put_value_without_meta_info(self):
         throughput = 5000
diff --git a/tests/types_test.py b/tests/types_test.py
index 9bae404e7..65e6d9378 100644
--- a/tests/types_test.py
+++ b/tests/types_test.py
@@ -126,5 +126,5 @@ def assert_annotations(obj, ident, *expects):
 class TestConfigTypeHint:
     def test_esrally_module_annotations(self):
         for module in project_root.glob_modules("esrally/**/*.py"):
-            assert_annotations(module, "cfg", types.Config)
+            assert_annotations(module, "cfg", types.Config, "types.Config")
             assert_annotations(module, "config", types.Config, Optional[types.Config], ConfigParser)


Reviews:
user: login: favilo
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T15:18:29Z
user: login: favilo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-07T17:03:10Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-10T17:31:13Z
user: login: gbanasiak
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-20T09:50:03Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-23T05:34:39Z
user: login: gbanasiak
body: This works great, thank you. I left some comments inline.

I think the same logic should apply to remaining templates - `rally-results`, `rally-races`, and `rally-annotations`, for consistency. WDYT?

Regarding `config` refactor my personal preference would be to simply avoid it in this PR due to its size and the fact the PR title does not match the content. I would simply use `convert.to_bool()` like in all other places (`datastore.overwrite_existing_templates` is not the first true/false setting in the INI file) and work on the refactor in another PR. Note we are note quite there yet, as now we should convert all the boolean config reads to not use `convert.to_bool()` which will extend the scope further.

I don't have strong feelings about `types_test.py`. I think its form is due to the context in which it came to be. There were no annotations at all and they were inserted surgically. Some of these tests were just to confirm that code edits were complete.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T15:36:23Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T05:46:34Z
user: login: gbanasiak
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T11:02:59Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:18:31Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:20:13Z
user: login: gareth-ellis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T14:28:03Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:50:06Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T15:25:00Z
user: login: gareth-ellis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T15:25:36Z
user: login: fressi-elastic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T15:31:59Z

Review comments:
diff_hunk: @@ -241,6 +245,37 @@ def test_can_migrate_outdated_config(self):
     def assert_equals_base_config(self, base_config, local_config, section, key):
         assert base_config.opts(section, key) == local_config.opts(section, key)
 
+    @dataclass
+    class BooleanCase:
+        case_id: str
+        value: str | None = None
path: tests/config_test.py
user: login: favilo
body: We are still supporting python 3.9, which doesn't have this type of annotation support, yet. Can you change this to `typing.Union`?

But yes, eventually I'd like to use this type of annotation. 

Or does `from _future_ import annotations` fix this? I suppose we could just run mypy with 3.10 and later, since that import turns annotations into strings... I'd have to do more research
created_at: 2025-02-07T17:03:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -241,6 +245,37 @@ def test_can_migrate_outdated_config(self):
     def assert_equals_base_config(self, base_config, local_config, section, key):
         assert base_config.opts(section, key) == local_config.opts(section, key)
 
+    @dataclass
+    class BooleanCase:
+        case_id: str
+        value: str | None = None
path: tests/config_test.py
user: login: fressi-elastic
body: Yes, `from _future_ import annotations` adds the support for the new union type annotation in Python 3.9. Test cases are passing nicely with `python3.9`. `mypy` and `pylint` are both running smoothly after adding such `import` statement. Without the annotation unit test execution fails, therefore running `mypy` or `pylint` is not required to catch this specific issue.

created_at: 2025-02-10T17:31:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -898,8 +902,31 @@ def open(self, race_id=None, race_timestamp=None, track_name=None, challenge_nam
         self._index = self.index_name()
         # reduce a bit of noise in the metrics cluster log
         if create:
-            # always update the mapping to the latest version
-            self._client.put_template("rally-metrics", self._get_template())
+            template = None
+            if self._client.template_exists("rally-metrics"):
+                for t in self._client.get_template("rally-metrics").body.get("index_templates", []):
+                    template = t.get("index_template", {}).get("template", {})
+                    break
+
+            new_template: str = self._get_template()
+            if template is None or self._config.boolean(section="reporting", key="datastore.overwrite_existing_templates", default=False):
+                if template is None:
+                    self.logger.info(
+                        "Create index template:\n%s",
+                        pretty.diff(old={}, new=json.loads(new_template).get("template", {}), flat_dict=True),
+                    )
+                else:
+                    self.logger.warning(
+                        "Overwrite existing index template (datastore.overwrite_existing_templates = true):\n%s",
+                        pretty.diff(old=template, new=json.loads(new_template).get("template", {}), flat_dict=True),
+                    )
+                self._client.put_template("rally-metrics", new_template)
+            else:
+                self.logger.warning(
+                    "Keep existing template (datastore.overwrite_existing_templates = false):\n%s",
+                    pretty.diff(old=template, new=json.loads(new_template).get("template", {}), flat_dict=True),
+                )
path: esrally/metrics.py
user: login: gbanasiak
body: I would lower this to debug level.
created_at: 2025-02-20T09:50:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +33,7 @@ repos:
           "--config",
           "pyproject.toml"
         ]
+        require_serial: true
path: .pre-commit-config.yaml
user: login: fressi-elastic
body: This is required to shut up a flaky internal `mypy` error appearing randomly when executing `make lint`.
created_at: 2025-02-23T05:34:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -176,51 +335,59 @@ def opts(self, section: types.Section, key: types.Key, default_value=None, manda
         mandatory properties. It must be ensured that a value exists. Default: True
         :return: The associated value.
         """
-        try:
-            scope = self._resolve_scope(section, key)
-            return self._opts[self._k(scope, section, key)]
-        except KeyError:
-            if not mandatory:
-                return default_value
-            else:
+        v = self._opts[section].get(key)
+        if v is None:
+            if mandatory:
                 raise exceptions.ConfigError(f"No value for mandatory configuration: section='{section}', key='{key}'")
+            return default_value
+        return v.value
+
+    def boolean(self, section: Section, key: Key, default: bool | None = None) -> bool:
+        """It reads a boolean value from the configuration.
path: esrally/config.py
user: login: gbanasiak
body: We're using `convert.to_bool()` for this purpose. Let's keep consistency, so either convert to new method everywhere, or revert to `convert.to_bool()` in this PR, and refactor in a follow-up.
created_at: 2025-02-28T11:06:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,7 +245,9 @@ def location(self):
         return os.path.join(self.config_dir, f"rally{config_name_suffix}.ini")
 
 
-def auto_load_local_config(base_config, additional_sections=None, config_file_class=ConfigFile, **kwargs) -> types.Config:
+def auto_load_local_config(
+    base_config: Config, additional_sections: list[Section] | None = None, config_file: ConfigFile | None = None
+) -> Config:
path: esrally/config.py
user: login: gbanasiak
body: Please update the doc string.
created_at: 2025-02-28T11:18:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +33,8 @@ repos:
           "--config",
           "pyproject.toml"
         ]
+        # This is required to shut up a flaky internal mypy error appearing randomly when executing make lint.
path: .pre-commit-config.yaml
user: login: gbanasiak
body: ```suggestion
        # This is required to suppress a flaky internal mypy error appearing randomly when executing make lint.
```
created_at: 2025-02-28T11:22:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -150,22 +307,24 @@ def add(self, scope, section: types.Section, key: types.Key, value):
         :param key: The configuration key within this section. Same keys in different sections will not collide.
         :param value: The associated value.
         """
-        self._opts[self._k(scope, section, key)] = value
+        if scope is None:
+            scope = Scope.application
+        v = self._opts[section].get(key)
+        if v is None or v.scope.value <= scope.value:
+            self._opts[section][key] = _V(value, scope)
path: esrally/config.py
user: login: gbanasiak
body: That's a nice and important simplification. We lose configuration values with broder scope (lower numerical value), but we don't seem to need them? That's the type of thing I would prefer to see in a PR with a different title. Do we need scopes at all?
created_at: 2025-02-28T11:55:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -68,34 +81,27 @@ def store_default_config(self):
             }
         )
 
-    def store(self, c):
-        self.present = True
-        self.config = configparser.ConfigParser()
-        self.config.read_dict(c)
path: tests/config_test.py
user: login: gbanasiak
body: I think I would keep this for completeness. Today it is not needed but `migrate()` from `config.py` has a potential of running `ConfigFile.store()` if only earliest supported config version becomes smaller than current config version.
created_at: 2025-02-28T13:24:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: gbanasiak
body: Nit: These "wants" are difficult to read, can we break these lines?
created_at: 2025-02-28T13:37:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -898,8 +902,41 @@ def open(self, race_id=None, race_timestamp=None, track_name=None, challenge_nam
         self._index = self.index_name()
         # reduce a bit of noise in the metrics cluster log
         if create:
-            # always update the mapping to the latest version
-            self._client.put_template("rally-metrics", self._get_template())
+            old_template: dict | None = None
+            if self._client.template_exists("rally-metrics"):
+                for t in self._client.get_template("rally-metrics").body.get("index_templates", []):
+                    old_template = t.get("index_template", {}).get("template", {})
+                    break
+
+            new_template: str = self._get_template()
+            if old_template is None or self._config.boolean(
+                section="reporting", key="datastore.overwrite_existing_templates", default=False
+            ):
+                if old_template is None:
+                    self.logger.info(
+                        "Create index template:\n%s",
+                        pretty.dump(json.loads(new_template).get("template", {}), pretty.Flag.FLAT_DICT),
+                    )
+                else:
+                    self.logger.warning(
+                        "Overwrite existing index template (datastore.overwrite_existing_templates = true):\n%s",
+                        pretty.diff(
+                            old=old_template,
+                            new=json.loads(new_template).get("template", {}),
+                            flags=pretty.Flag.FLAT_DICT | pretty.Flag.DUMP_EQUALS,
+                        ),
+                    )
+                self._client.put_template("rally-metrics", new_template)
path: esrally/metrics.py
user: login: gbanasiak
body: Can we avoid overwriting a template if it is equal to the one we want to write? And avoid the log in that case?
created_at: 2025-02-28T13:57:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -80,6 +80,8 @@ The following settings are applicable only if ``datastore.type`` is set to "elas
 * ``datastore.probe.cluster_version`` (default: true): Enables automatic detection of the metric store's version.
 * ``datastore.number_of_shards`` (default: `Elasticsearch default value <https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings>`_): The number of primary shards that the ``rally-*`` indices should have. Any updates to this setting after initial index creation will only be applied to new ``rally-*`` indices.
 * ``datastore.number_of_replicas`` (default: `Elasticsearch default value <https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings>`_): The number of replicas each primary shard has. Defaults to . Any updates to this setting after initial index creation will only be applied to new ``rally-*`` indices.
+* ``datastore.overwrite_existing_templates`` (default: ``false``): Existing index template is replaced only when this option is ``true``.
path: docs/configuration.rst
user: login: gbanasiak
body: ```suggestion
* ``datastore.overwrite_existing_templates`` (default: ``false``): Existing Rally index templates are replaced only when this option is ``true``.
```
created_at: 2025-02-28T14:03:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -9,6 +9,12 @@ Minimum Python version is 3.9.0
 
 Rally 2.12.0 requires Python 3.9.0 or above. Check the :ref:`updated installation instructions <install_python>` for more details.
 
+The metrics store keeps existing index templates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Existing index template is replaced only when option ``datastore.overwrite_existing_templates`` in section ``reporting`` is ``true``.
path: docs/migrate.rst
user: login: gbanasiak
body: ```suggestion
Existing Rally index templates are replaced only when option ``datastore.overwrite_existing_templates`` in section ``reporting`` is ``true``.
```
created_at: 2025-02-28T14:04:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -150,22 +307,24 @@ def add(self, scope, section: types.Section, key: types.Key, value):
         :param key: The configuration key within this section. Same keys in different sections will not collide.
         :param value: The associated value.
         """
-        self._opts[self._k(scope, section, key)] = value
+        if scope is None:
+            scope = Scope.application
+        v = self._opts[section].get(key)
+        if v is None or v.scope.value <= scope.value:
+            self._opts[section][key] = _V(value, scope)
path: esrally/config.py
user: login: fressi-elastic
body: I realized the lower scope value where stored but never read. Therefore I preferred to filter out lower scope value when putting new values instead when retrieving them. This should open to many more simplifications later on. For example at this point because we are using a dictionary of dictionaries we could probably use 3rd party code for the inner implementation as it is the way typically ini file are being loaded (I.E https://docs.python.org/3/library/configparser.html).
created_at: 2025-03-01T05:46:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -150,22 +307,24 @@ def add(self, scope, section: types.Section, key: types.Key, value):
         :param key: The configuration key within this section. Same keys in different sections will not collide.
         :param value: The associated value.
         """
-        self._opts[self._k(scope, section, key)] = value
+        if scope is None:
+            scope = Scope.application
+        v = self._opts[section].get(key)
+        if v is None or v.scope.value <= scope.value:
+            self._opts[section][key] = _V(value, scope)
path: esrally/config.py
user: login: gbanasiak
body: I can see how lower scope _could_ be useful if we say iterated through multiple benchmarks with slightly different settings, and wanted to override settings per-benchmark, then remove this override, then work with another override. But we're not doing this today. I think I'm OK with it unless @favilo has different opinion? We can return to this discussion in the expected new incoming PR.
created_at: 2025-03-03T11:02:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -68,34 +81,27 @@ def store_default_config(self):
             }
         )
 
-    def store(self, c):
-        self.present = True
-        self.config = configparser.ConfigParser()
-        self.config.read_dict(c)
path: tests/config_test.py
user: login: fressi-elastic
body: This method is not used for what is intended for the original method in the actual implementation class. This method was only used to set up the dataset for the test case, while the actual implementation used by the Config class is using it to save the dataset to a file. Because the purpose of the two is pretty different and we don't need this to populate the dataset in the test case, I decided to remove it, adding simplicity to the code. The original store method is still there and could be broken. This should not be an issue unless test cases don't call the method.
created_at: 2025-03-03T14:18:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: fressi-elastic
body: I tryed to rewrite this code to make it more readable but `black` pre-commit hook is overriding my changes making splitting these strings on multiple lines impossible. 
created_at: 2025-03-03T14:20:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: gareth-ellis
body: How about adding #fmt on and #fmt off around this section? https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#ignoring-sections

created_at: 2025-03-03T14:28:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: fressi-elastic
body: I just tried it. Didn't worked. Do you have some example?

created_at: 2025-03-03T14:50:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: fressi-elastic
body: I used it wrong and now I did mad it work. I finally did the change as asked as suggested by Gareth.
created_at: 2025-03-03T15:25:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: gareth-ellis
body: Not more than what is in the docs. I just tested and it works for me.
I tested first splitting over multiple lines and without they are split onto single lines, and with it stays as I wrote it 

```
# fmt: off 
... list of cases

# fmt: on
```
created_at: 2025-03-03T15:25:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,96 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# 	http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from esrally.utils import cases, pretty
+
+
+@dataclass
+class DumpCase:
+    o: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    null=DumpCase(None, want="null"),
+    string=DumpCase("string", want='"string"'),
+    integer=DumpCase(1, want="1"),
+    float=DumpCase(1.0, want="1.0"),
+    list=DumpCase([1, 2, 3], want="[\n  1,\n  2,\n  3\n]"),
+    tuple=DumpCase((2, 3), want="[\n  2,\n  3\n]"),
+    object=DumpCase({"a": "a", "b": 2}, want='{\n  "a": "a",\n  "b": 2\n}'),
+    flat_dict=DumpCase({"a": {"b": "c"}}, want='{\n  "a.b": "c"\n}', flags=pretty.Flag.FLAT_DICT),
+)
+def test_dump(case: DumpCase):
+    params: dict[str, Any] = {}
+    if case.flags is not None:
+        params["flags"] = case.flags
+    got = pretty.dump(case.o, **params)
+    assert got == case.want
+
+
+@dataclass
+class DiffCase:
+    old: Any
+    new: Any
+    want: str
+    flags: pretty.Flag | None = None
+
+
+@cases.cases(
+    none_and_none=DiffCase(None, None, want=""),
+    none_and_string=DiffCase(None, "something", want='- null\n+ "something"'),
+    strings=DiffCase("cat", "cut", want='- "cat"\n?   ^\n\n+ "cut"\n?   ^\n'),
+    equal_strings=DiffCase("same", "same", want=""),
+    integers=DiffCase(123, 132, want="- 123\n+ 132"),
+    equal_integers=DiffCase(42, 42, want=""),
+    floats=DiffCase(1.23, 13.2, want="- 1.23\n?    -\n\n+ 13.2\n?  +\n"),
+    equal_floats=DiffCase(3.140, 3.14e0, want=""),
+    float_and_integer=DiffCase(1.0, 1, want=""),
+    lists=DiffCase(old=[1, 2, 3], new=[1, 3, 4], want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_lists=DiffCase([2, 3], [2, 3], want=""),
+    tuples=DiffCase(old=(1, 2, 3), new=(1, 3, 4), want="  [\n    1,\n-   2,\n-   3\n+   3,\n?    +\n\n+   4\n  ]"),
+    equal_tuples=DiffCase((2, 3), (2, 3), want=""),
+    list_and_tuples=DiffCase((3, 4), [3, 4], want=""),
+    objects=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"b": 2, "c": 3},
+        want='  {\n-   "a": 1,\n-   "b": 2\n+   "b": 2,\n?         +\n\n+   "c": 3\n  }',
+    ),
+    flat_dict=DiffCase(
+        old={"a": {"b": "c"}},
+        new={"a": {"c": "d"}},
+        want='  {\n-   "a.b": "c"\n?      ^    ^\n\n+   "a.c": "d"\n?      ^    ^\n\n  }',
+        flags=pretty.Flag.FLAT_DICT,
+    ),
+    dump_equals=DiffCase(
+        old={"a": 1, "b": 2},
+        new={"a": 1, "b": 2},
+        want='  {\n    "a": 1,\n    "b": 2\n  }',
+        flags=pretty.Flag.DUMP_EQUALS,
+    ),
+)
path: tests/utils/pretty_test.py
user: login: fressi-elastic
body: By the way I moved the implementation of the pretty printing to PR #1924 and rebased this on top of it.
created_at: 2025-03-03T15:31:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: gareth-ellis
created_at: 2025-02-06T08:37:27Z
body: Could we get an update to the docs done too, to explain this feature? Obviously we want to have it tested too before we merge
author_association: MEMBER
user: login: fressi-elastic
created_at: 2025-02-06T10:34:22Z
body: > Could we get an update to the docs done too, to explain this feature? Obviously we want to have it tested too before we merge

I did tested and it works as expected in all cases except one: one the `overwrite_existing_templates` is false it looks like it act as it is true. I guess the code is parsing it as string instead of a boolean. I am investigating on it.
author_association: CONTRIBUTOR
user: login: gbanasiak
created_at: 2025-02-19T14:09:37Z
body: In addition to documentation I would also try adding new modules in here to enable full scope of mypy checks: https://github.com/elastic/rally/blob/2ef33ebf463745468defbc99f389cf31354f2445/pyproject.toml#L200-L206

Context: Until https://github.com/elastic/rally/pull/1798, so recently on Rally timescale, we had no annotations. They were introduced very surgically to reduce the scope. Then in https://github.com/elastic/rally/pull/1859 we introduced mypy overrides that restore full set of checks for selected modules - either new ones, or the ones we are revisiting. As we're planning to broaden annotations it would be best if all new modules were added there. Ultimately the overrides should be removed.
author_association: CONTRIBUTOR
user: login: gbanasiak
created_at: 2025-02-21T13:09:44Z
body: CI failure will be addressed by https://github.com/elastic/rally-tracks/pull/748.
author_association: CONTRIBUTOR
user: login: fressi-elastic
created_at: 2025-02-23T05:33:21Z
body: I enabled some more typing check in more modules (including `config.py`) and improved unit tests. While doing that I made some cleanup in the Config class to simplify it. Its unit test is still a mess on my opinion but I would avoid refactoring it in this change because it is out of scope and this is getting too big. I am considering splitting this PR in more parts to speed up the review process. Do you think it would help?
author_association: CONTRIBUTOR
user: login: gbanasiak
created_at: 2025-02-24T11:57:25Z
body: >  I am considering splitting this PR in more parts to speed up the review process. Do you think it would help?

I think `config.py` simplification and its merge with `types.py` should go into a separate PR with a more representative title.
author_association: CONTRIBUTOR
user: login: gbanasiak
created_at: 2025-02-24T12:24:46Z
body: For example, it seems we're dropping some useful tests from `types_test.py`? I think making sure the list of literals is sorted, or making sure that every literal is actually present somewhere in the code are useful. But this discussion should not block index template change I think.
author_association: CONTRIBUTOR
user: login: fressi-elastic
created_at: 2025-02-27T14:39:40Z
body: > For example, it seems we're dropping some useful tests from `types_test.py`? I think making sure the list of literals is sorted, or making sure that every literal is actually present somewhere in the code are useful. But this discussion should not block index template change I think.

@gbanasiak @gareth-ellis @favilo I think using Literals for defining sections and keys in the config file has been kind of naive and incomplete solution. A better one could be adopted (like for example using `dataclasses` and ad hoc section importers from a generic ini file parser).

We could have for example a loader for every section implemented near the code that is going to use it (for preserving code modularity). The ini file is first converted to a dict of dict and then passed to a `dataclass` before being used in the code. In this way we do have regular linters protecting us from refractory errors. There are no strong reasons why not to have independent conversion (dict to object) functions for every individual section, given a common dict of dict made from the original file.

For the purpose of scopes it is enough to made up the final objects from multiple set of dictionaries (one for every scope) and in the right order to keep scope priority.

I tried to reduce the complexity of the code, but this is still a naive solution, this testing I removed is very weak and it doesn't allow easy further refractory of the code.

Could we please forget about this test module and plan a true refactor of this part of the project after this change?


author_association: CONTRIBUTOR
user: login: fressi-elastic
created_at: 2025-03-01T09:17:16Z
body: I am still processing pending comments. I am going to split this PR.
author_association: CONTRIBUTOR
user: login: fressi-elastic
created_at: 2025-03-03T14:06:05Z
body: I took out parts of this PR to some new smaller ones:
- [x] PR #1919
- [x] PR #1922 
- [x] PR #1924
- [ ] PR #1925

author_association: CONTRIBUTOR
user: login: gbanasiak
created_at: 2025-03-05T13:50:59Z
body: @fressi-elastic Many thanks for the split. As mentioned earlier, I'm happy with how this works for `rally-metrics`. The only remaining concern is:

> I think the same logic should apply to remaining templates - `rally-results`, `rally-races`, and `rally-annotations`, for consistency. WDYT?

The rationale is the `datastore.overwrite_existing_templates` setting name is generic, and we have multiple templates, see `EsResultsStore` and `EsRaceStore` classes in `metrics.py`. We can address this in a follow-up PR if you prefer.
author_association: CONTRIBUTOR
user: login: fressi-elastic
created_at: 2025-03-06T04:42:20Z
body: I already started working in a follow-up PR address @gbanasiak comments. Thank you for addressing this point I didn't catch earlier.
author_association: CONTRIBUTOR