Repository: elastic/rally

Issue:
Remove 'index-settings' property
With Rally 0.9.0 we have introduced a new "create-index" operation which allows to define a "settings" property which makes the specific "index-settings" property obsolete. We'll hence remove it.

Discussion:
danielmitterdorfer: Note: There is no need to adapt our default tracks because this has already been done in https://github.com/elastic/rally-tracks/pull/41.

Patch:
diff --git a/docs/command_line_reference.rst b/docs/command_line_reference.rst
index dd1855d5..afecc539 100644
--- a/docs/command_line_reference.rst
+++ b/docs/command_line_reference.rst
@@ -82,16 +82,15 @@ Consider the following track snippet showing a single challenge::
 
     {
       "name": "index-only",
-      "index-settings": {
-        "index.number_of_replicas": {{ replica_count|default(0) }},
-        "index.number_of_shards": {{ shard_count|default(5) }},
-      },
       "schedule": [
-        {
-          "operation": "bulk-index",
-          "warmup-time-period": 120,
-          "clients": 8
-        }
+         {
+           "operation": {
+             "operation-type": "bulk",
+             "bulk-size": {{ bulk_size|default(5000) }}
+           },
+           "warmup-time-period": 120,
+           "clients": {{ clients|default(8) }}
+         }
       ]
     }
 
@@ -99,28 +98,28 @@ Rally tracks can use the Jinja templating language and the construct ``{{ some_v
 
 We can see that it defines two variables:
 
-* ``replica_count`` with a default value of 0
-* ``shard_count`` with a default value of 5
+* ``bulk_size`` with a default value of 5000
+* ``clients`` with a default value of 8
 
 When we run this track, we can override these defaults:
 
-* ``--track-params="replica_count:1,shard_count:3"`` will set the number of replicas to 1 and the number of shards to 3.
-* ``--track-params="replica_count:1"`` will just set the number of replicas to 1 and just keep the default value of 5 shards.
+* ``--track-params="bulk_size:2000,clients:16"`` will set the bulk size to 2000 and the number of clients for bulk indexing to 16.
+* ``--track-params="bulk_size:8000"`` will just set the bulk size to 8000 and keep the default value of 8 clients.
 * ``--track-params="params.json"`` will read the track parameters from a JSON file (defined below)
 
 Example JSON file::
 
    {
-      "replica_count": 1,
-      "shard_count": 3
+      "bulk_size": 2000,
+      "clients": 16
    }
 
 All track parameters are recorded for each metrics record in the metrics store. Also, when you run ``esrally list races``, it will show all track parameters::
 
-    Race Timestamp    Track    Track Parameters               Challenge            Car       User Tag
-    ----------------  -------  ------------------------------ -------------------  --------  ---------
-    20160518T122341Z  pmc      replica_count=1                append-no-conflicts  defaults
-    20160518T112341Z  pmc      replica_count=1,shard_count=3  append-no-conflicts  defaults
+    Race Timestamp    Track    Track Parameters          Challenge            Car       User Tag
+    ----------------  -------  ------------------------- -------------------  --------  ---------
+    20160518T122341Z  pmc      bulk_size=8000            append-no-conflicts  defaults
+    20160518T112341Z  pmc      bulk_size=2000,clients=16 append-no-conflicts  defaults
 
 Note that the default values are not recorded or shown (Rally does not know about them).
 
diff --git a/docs/metrics.rst b/docs/metrics.rst
index 9f952e04..d6ba4a61 100644
--- a/docs/metrics.rst
+++ b/docs/metrics.rst
@@ -149,4 +149,6 @@ Rally stores the following metrics:
 * ``indexing_throttle_time``: Total time that indexing has been throttled as reported by the indices stats API. Note that this is not Wall clock time.
 * ``refresh_total_time``: Total time used for index refresh as reported by the indices stats API. Note that this is not Wall clock time.
 * ``flush_total_time``: Total time used for index flush as reported by the indices stats API. Note that this is not Wall clock time.
-* ``final_index_size_bytes``: Final resulting index size after the benchmark.
+* ``final_index_size_bytes``: Final resulting index size on the file system after all nodes have been shutdown at the end of the benchmark. It includes all files in the nodes' data directories (actual index files and translog).
+* ``store_size_in_bytes``: The size in bytes of the index (excluding the translog) as reported by the indices stats API.
+* ``translog_size_in_bytes``: The size in bytes of the translog as reported by the indices stats API.
diff --git a/docs/summary_report.rst b/docs/summary_report.rst
index e1edb140..46e9c16e 100644
--- a/docs/summary_report.rst
+++ b/docs/summary_report.rst
@@ -81,9 +81,21 @@ Total Old Gen GC
 Index size
 ----------
 
-* **Definition**: Final resulting index size after the benchmark.
+* **Definition**: Final resulting index size on the file system after all nodes have been shutdown at the end of the benchmark. It includes all files in the nodes' data directories (actual index files and translog).
 * **Corresponding metrics key**: ``final_index_size_bytes``
 
+Store size
+----------
+
+* **Definition**: The size in bytes of the index (excluding the translog) as reported by the indices stats API.
+* **Corresponding metrics key**: ``store_size_in_bytes``
+
+Translog size
+-------------
+
+* **Definition**: The size in bytes of the translog as reported by the indices stats API.
+* **Corresponding metrics key**: ``translog_size_in_bytes``
+
 Totally written
 ---------------
 
diff --git a/docs/track.rst b/docs/track.rst
index 7af77b92..0bea3c76 100644
--- a/docs/track.rst
+++ b/docs/track.rst
@@ -665,7 +665,6 @@ Each challenge consists of the following properties:
 * ``name`` (mandatory): A descriptive name of the challenge. Should not contain spaces in order to simplify handling on the command line for users.
 * ``description`` (optional): A human readable description of the challenge.
 * ``default`` (optional): If true, Rally selects this challenge by default if the user did not specify a challenge on the command line. If your track only defines one challenge, it is implicitly selected as default, otherwise you need define ``"default": true`` on exactly one challenge.
-* ``index-settings`` (optional): Defines the index settings of the benchmark candidate when an index is created.
 * ``schedule`` (mandatory): Defines the concrete execution order of operations. It is described in more detail below.
 
 .. note::
diff --git a/esrally/client.py b/esrally/client.py
index f81d432b..23677f52 100644
--- a/esrally/client.py
+++ b/esrally/client.py
@@ -12,7 +12,12 @@ class EsClientFactory:
     Abstracts how the Elasticsearch client is created. Intended for testing.
     """
     def __init__(self, hosts, client_options):
-        logger.info("Creating ES client connected to %s with options [%s]" % (hosts, client_options))
+        masked_client_options = dict(client_options)
+        if "basic_auth_password" in masked_client_options:
+            masked_client_options["basic_auth_password"] = "*****"
+        if "http_auth" in masked_client_options:
+            masked_client_options["http_auth"] = (client_options["http_auth"][0], "*****")
+        logger.info("Creating ES client connected to %s with options [%s]", hosts, masked_client_options)
         self.hosts = hosts
         self.client_options = client_options
 
diff --git a/esrally/mechanic/telemetry.py b/esrally/mechanic/telemetry.py
index 6d8ae9c0..27bcdd82 100644
--- a/esrally/mechanic/telemetry.py
+++ b/esrally/mechanic/telemetry.py
@@ -708,7 +708,7 @@ class IndexStats(InternalTelemetryDevice):
         # the pipeline "benchmark-only" where we don't have control over the cluster and the user might not have restarted
         # the cluster so we can at least tell them.
         if self.first_time:
-            index_times = self.index_times(self.primaries_index_stats())
+            index_times = self.index_times(self.index_stats()["primaries"])
             for k, v in index_times.items():
                 if v > 0:
                     console.warn("%s is %d ms indicating that the cluster is not in a defined clean state. Recorded index time "
@@ -718,8 +718,9 @@ class IndexStats(InternalTelemetryDevice):
     def on_benchmark_stop(self):
         import json
         logger.info("Gathering indices stats for all primaries on benchmark stop.")
-        p = self.primaries_index_stats()
-        logger.info("Returned indices stats:\n%s" % json.dumps(p, indent=2))
+        index_stats = self.index_stats()
+        logger.info("Returned indices stats:\n%s" % json.dumps(index_stats, indent=2))
+        p = index_stats["primaries"]
         # actually this is add_count
         self.add_metrics(self.extract_value(p, ["segments", "count"]), "segments_count")
         self.add_metrics(self.extract_value(p, ["segments", "memory_in_bytes"]), "segments_memory_in_bytes", "byte")
@@ -733,12 +734,14 @@ class IndexStats(InternalTelemetryDevice):
         self.add_metrics(self.extract_value(p, ["segments", "terms_memory_in_bytes"]), "segments_terms_memory_in_bytes", "byte")
         self.add_metrics(self.extract_value(p, ["segments", "norms_memory_in_bytes"]), "segments_norms_memory_in_bytes", "byte")
         self.add_metrics(self.extract_value(p, ["segments", "points_memory_in_bytes"]), "segments_points_memory_in_bytes", "byte")
+        self.add_metrics(self.extract_value(index_stats, ["total", "store", "size_in_bytes"]), "store_size_in_bytes", "byte")
+        self.add_metrics(self.extract_value(index_stats, ["total", "translog", "size_in_bytes"]), "translog_size_in_bytes", "byte")
 
-    def primaries_index_stats(self):
+    def index_stats(self):
         # noinspection PyBroadException
         try:
             stats = self.client.indices.stats(metric="_all", level="shards")
-            return stats["_all"]["primaries"]
+            return stats["_all"]
         except BaseException:
             logger.exception("Could not retrieve index stats.")
             return {}
diff --git a/esrally/reporter.py b/esrally/reporter.py
index ee0b997a..c1bcfdca 100644
--- a/esrally/reporter.py
+++ b/esrally/reporter.py
@@ -168,9 +168,13 @@ class StatsCalculator:
         result.memory_points = self.median("segments_points_memory_in_bytes")
         result.memory_stored_fields = self.median("segments_stored_fields_memory_in_bytes")
 
-        # This metric will only be written for the last iteration (as it can only be determined after the cluster has been shut down)
         logger.debug("Gathering disk metrics.")
+        # This metric will only be written for the last iteration (as it can only be determined after the cluster has been shut down)
         result.index_size = self.sum("final_index_size_bytes")
+        # we need to use the median here because these two are captured with the indices stats API and thus once per lap. If we'd
+        # sum up the values we'd get wrong results for benchmarks that ran for multiple laps.
+        result.store_size = self.median("store_size_in_bytes")
+        result.translog_size = self.median("translog_size_in_bytes")
         result.bytes_written = self.sum("disk_io_write_bytes")
 
         # convert to int, fraction counts are senseless
@@ -264,6 +268,8 @@ class Stats:
         self.memory_stored_fields = self.v(d, "memory_stored_fields")
 
         self.index_size = self.v(d, "index_size")
+        self.store_size = self.v(d, "store_size")
+        self.translog_size = self.v(d, "translog_size")
         self.bytes_written = self.v(d, "bytes_written")
 
         self.segment_count = self.v(d, "segment_count")
@@ -497,6 +503,8 @@ class SummaryReporter:
 
     def report_disk_usage(self, stats):
         return self.join(
+            self.line("Store size", "", stats.store_size, "GB", convert.bytes_to_gb),
+            self.line("Translog size", "", stats.translog_size, "GB", convert.bytes_to_gb),
             self.line("Index size", "", stats.index_size, "GB", convert.bytes_to_gb),
             self.line("Totally written", "", stats.bytes_written, "GB", convert.bytes_to_gb)
         )
@@ -688,6 +696,10 @@ class ComparisonReporter:
 
     def report_disk_usage(self, baseline_stats, contender_stats):
         return self.join(
+            self.line("Store size", baseline_stats.store_size, contender_stats.store_size, "", "GB",
+                      treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
+            self.line("Translog size", baseline_stats.translog_size, contender_stats.translog_size, "", "GB",
+                      treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
             self.line("Index size", baseline_stats.index_size, contender_stats.index_size, "", "GB",
                       treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
             self.line("Totally written", baseline_stats.bytes_written, contender_stats.bytes_written, "", "GB",
diff --git a/esrally/resources/track-schema.json b/esrally/resources/track-schema.json
index f148778f..8e9c3829 100644
--- a/esrally/resources/track-schema.json
+++ b/esrally/resources/track-schema.json
@@ -29,10 +29,6 @@
           "type": "object",
           "description": "Defines the cluster settings of the benchmark candidate."
         },
-        "index-settings": {
-          "type": "object",
-          "description": "Defines the index settings of the benchmark candidate when an index is created."
-        },
         "schedule": {
           "type": "array",
           "minItems": 1,
diff --git a/esrally/track/loader.py b/esrally/track/loader.py
index 5adf6d71..d04c516d 100644
--- a/esrally/track/loader.py
+++ b/esrally/track/loader.py
@@ -829,14 +829,8 @@ class TrackSpecificationReader:
             meta_data = self._r(challenge_spec, "meta", error_ctx=name, mandatory=False)
             # if we only have one challenge it is treated as default challenge, no matter what the user has specified
             default = number_of_challenges == 1 or self._r(challenge_spec, "default", error_ctx=name, mandatory=False)
-            # TODO #381: Remove this setting
-            index_settings = self._r(challenge_spec, "index-settings", error_ctx=name, mandatory=False)
             cluster_settings = self._r(challenge_spec, "cluster-settings", error_ctx=name, mandatory=False)
 
-            if index_settings and self.name not in DEFAULT_TRACKS:
-                console.warn("Challenge [%s] in track [%s] defines the [index-settings] property which will be removed soon. For details "
-                             "please see the migration guide in the docs." % (name, self.name))
-
             if default and default_challenge is not None:
                 self._error("Both '%s' and '%s' are defined as default challenges. Please define only one of them as default."
                             % (default_challenge.name, name))
@@ -867,7 +861,6 @@ class TrackSpecificationReader:
                                         meta_data=meta_data,
                                         description=description,
                                         user_info=user_info,
-                                        index_settings=index_settings,
                                         cluster_settings=cluster_settings,
                                         default=default,
                                         schedule=schedule)
diff --git a/esrally/track/track.py b/esrally/track/track.py
index 2bd67231..1b9dde50 100644
--- a/esrally/track/track.py
+++ b/esrally/track/track.py
@@ -349,7 +349,6 @@ class Challenge:
                  name,
                  description=None,
                  user_info=None,
-                 index_settings=None,
                  cluster_settings=None,
                  default=False,
                  meta_data=None,
@@ -358,7 +357,6 @@ class Challenge:
         self.meta_data = meta_data if meta_data else {}
         self.description = description
         self.user_info = user_info
-        self.index_settings = index_settings if index_settings else {}
         self.cluster_settings = cluster_settings if cluster_settings else {}
         self.default = default
         self.schedule = schedule if schedule else []
@@ -379,13 +377,13 @@ class Challenge:
         return ", ".join(r)
 
     def __hash__(self):
-        return hash(self.name) ^ hash(self.description) ^ hash(self.index_settings) ^ hash(self.cluster_settings) ^ hash(self.default) ^ \
+        return hash(self.name) ^ hash(self.description) ^ hash(self.cluster_settings) ^ hash(self.default) ^ \
                hash(self.meta_data) ^ hash(self.schedule)
 
     def __eq__(self, othr):
         return (isinstance(othr, type(self)) and
-                (self.name, self.description, self.index_settings, self.cluster_settings, self.default, self.meta_data, self.schedule) ==
-                (othr.name, othr.description, othr.index_settings, othr.cluster_settings, othr.default, othr.meta_data, othr.schedule))
+                (self.name, self.description, self.cluster_settings, self.default, self.meta_data, self.schedule) ==
+                (othr.name, othr.description, othr.cluster_settings, othr.default, othr.meta_data, othr.schedule))
 
 
 @unique
diff --git a/run.sh b/run.sh
index 6fdd7311..5c952382 100755
--- a/run.sh
+++ b/run.sh
@@ -18,9 +18,9 @@ install_esrally_with_setuptools () {
     fi
 
     if [[ ${IN_VIRTUALENV} == 0 ]]; then
-        python3 setup.py -q develop --user
+        python3 setup.py -q develop --user --upgrade
     else
-        python3 setup.py -q develop
+        python3 setup.py -q develop --upgrade
     fi
 }
 


Test patch:
diff --git a/tests/mechanic/telemetry_test.py b/tests/mechanic/telemetry_test.py
index 28a2aae5..58f6e1eb 100644
--- a/tests/mechanic/telemetry_test.py
+++ b/tests/mechanic/telemetry_test.py
@@ -764,6 +764,17 @@ class IndexStatsTests(TestCase):
                     "flush": {
                         "total_time_in_millis": 100
                     }
+                },
+                "total": {
+                    "store": {
+                        "size_in_bytes": 2113867510
+                    },
+                    "translog": {
+                        "operations": 6840000,
+                        "size_in_bytes": 2647984713,
+                        "uncommitted_operations": 0,
+                        "uncommitted_size_in_bytes": 430
+                    }
                 }
             }
         })
@@ -784,7 +795,8 @@ class IndexStatsTests(TestCase):
             mock.call("segments_stored_fields_memory_in_bytes", 1024, "byte"),
             mock.call("segments_terms_memory_in_bytes", 256, "byte"),
             # we don't have norms, so nothing should have been called
-            mock.call("segments_points_memory_in_bytes", 512, "byte"),
+            mock.call("store_size_in_bytes", 2113867510, "byte"),
+            mock.call("translog_size_in_bytes", 2647984713, "byte"),
         ], any_order=True)
 
     @mock.patch("esrally.metrics.EsMetricsStore.put_value_cluster_level")
diff --git a/tests/metrics_test.py b/tests/metrics_test.py
index 1c676b4a..ea1224ca 100644
--- a/tests/metrics_test.py
+++ b/tests/metrics_test.py
@@ -668,7 +668,7 @@ class EsResultsStoreTests(TestCase):
 
         t = track.Track(name="unittest-track",
                         indices=[track.Index(name="tests", types=["test-type"])],
-                        challenges=[track.Challenge(name="index", default=True, index_settings=None, schedule=schedule)])
+                        challenges=[track.Challenge(name="index", default=True, schedule=schedule)])
 
         c = cluster.Cluster([], [], None)
         c.distribution_version = "5.0.0"
diff --git a/tests/track/loader_test.py b/tests/track/loader_test.py
index 85f8bb31..7afea9a7 100644
--- a/tests/track/loader_test.py
+++ b/tests/track/loader_test.py
@@ -691,7 +691,6 @@ class TrackPostProcessingTests(TestCase):
                 {
                     "name": "default-challenge",
                     "description": "Default challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -764,7 +763,6 @@ class TrackPostProcessingTests(TestCase):
                 {
                     "name": "default-challenge",
                     "description": "Default challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1024,7 +1022,6 @@ class TrackSpecificationReaderTests(TestCase):
             "challenges": [
                 {
                     "name": "default-challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1148,7 +1145,6 @@ class TrackSpecificationReaderTests(TestCase):
             "challenges": [
                 {
                     "name": "default-challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1327,9 +1323,6 @@ class TrackSpecificationReaderTests(TestCase):
                         "mixed": True,
                         "max-clients": 8
                     },
-                    "index-settings": {
-                        "index.number_of_replicas": 2
-                    },
                     "schedule": [
                         {
                             "clients": 8,
@@ -1414,8 +1407,6 @@ class TrackSpecificationReaderTests(TestCase):
         self.assertEqual(1, len(resulting_track.challenges))
         self.assertEqual("default-challenge", resulting_track.challenges[0].name)
         self.assertEqual("Default challenge", resulting_track.challenges[0].description)
-        self.assertEqual(1, len(resulting_track.challenges[0].index_settings))
-        self.assertEqual(2, resulting_track.challenges[0].index_settings["index.number_of_replicas"])
         self.assertEqual({"mixed": True, "max-clients": 8}, resulting_track.challenges[0].meta_data)
         self.assertEqual({"append": True}, resulting_track.challenges[0].schedule[0].operation.meta_data)
         self.assertEqual({"operation-index": 0}, resulting_track.challenges[0].schedule[0].meta_data)