Repository: elastic/rally

Issue:
Upgrade Elasticsearch client to 6.2.0
The Elasticsearch 6.1.1 client adds the ability to set custom HTTP headers which is necessary for some API calls (e.g. `msearch`, see also the [respective Discuss thread](https://discuss.elastic.co/t/msearch-capability/110702/11)).

When we upgrade, we should also reimplement SSL handling as described in https://github.com/elastic/elasticsearch-py/issues/712#issuecomment-357750570.

Discussion:

Patch:
diff --git a/docs/command_line_reference.rst b/docs/command_line_reference.rst
index dd1855d5..afecc539 100644
--- a/docs/command_line_reference.rst
+++ b/docs/command_line_reference.rst
@@ -82,16 +82,15 @@ Consider the following track snippet showing a single challenge::
 
     {
       "name": "index-only",
-      "index-settings": {
-        "index.number_of_replicas": {{ replica_count|default(0) }},
-        "index.number_of_shards": {{ shard_count|default(5) }},
-      },
       "schedule": [
-        {
-          "operation": "bulk-index",
-          "warmup-time-period": 120,
-          "clients": 8
-        }
+         {
+           "operation": {
+             "operation-type": "bulk",
+             "bulk-size": {{ bulk_size|default(5000) }}
+           },
+           "warmup-time-period": 120,
+           "clients": {{ clients|default(8) }}
+         }
       ]
     }
 
@@ -99,28 +98,28 @@ Rally tracks can use the Jinja templating language and the construct ``{{ some_v
 
 We can see that it defines two variables:
 
-* ``replica_count`` with a default value of 0
-* ``shard_count`` with a default value of 5
+* ``bulk_size`` with a default value of 5000
+* ``clients`` with a default value of 8
 
 When we run this track, we can override these defaults:
 
-* ``--track-params="replica_count:1,shard_count:3"`` will set the number of replicas to 1 and the number of shards to 3.
-* ``--track-params="replica_count:1"`` will just set the number of replicas to 1 and just keep the default value of 5 shards.
+* ``--track-params="bulk_size:2000,clients:16"`` will set the bulk size to 2000 and the number of clients for bulk indexing to 16.
+* ``--track-params="bulk_size:8000"`` will just set the bulk size to 8000 and keep the default value of 8 clients.
 * ``--track-params="params.json"`` will read the track parameters from a JSON file (defined below)
 
 Example JSON file::
 
    {
-      "replica_count": 1,
-      "shard_count": 3
+      "bulk_size": 2000,
+      "clients": 16
    }
 
 All track parameters are recorded for each metrics record in the metrics store. Also, when you run ``esrally list races``, it will show all track parameters::
 
-    Race Timestamp    Track    Track Parameters               Challenge            Car       User Tag
-    ----------------  -------  ------------------------------ -------------------  --------  ---------
-    20160518T122341Z  pmc      replica_count=1                append-no-conflicts  defaults
-    20160518T112341Z  pmc      replica_count=1,shard_count=3  append-no-conflicts  defaults
+    Race Timestamp    Track    Track Parameters          Challenge            Car       User Tag
+    ----------------  -------  ------------------------- -------------------  --------  ---------
+    20160518T122341Z  pmc      bulk_size=8000            append-no-conflicts  defaults
+    20160518T112341Z  pmc      bulk_size=2000,clients=16 append-no-conflicts  defaults
 
 Note that the default values are not recorded or shown (Rally does not know about them).
 
diff --git a/docs/metrics.rst b/docs/metrics.rst
index 9f952e04..d6ba4a61 100644
--- a/docs/metrics.rst
+++ b/docs/metrics.rst
@@ -149,4 +149,6 @@ Rally stores the following metrics:
 * ``indexing_throttle_time``: Total time that indexing has been throttled as reported by the indices stats API. Note that this is not Wall clock time.
 * ``refresh_total_time``: Total time used for index refresh as reported by the indices stats API. Note that this is not Wall clock time.
 * ``flush_total_time``: Total time used for index flush as reported by the indices stats API. Note that this is not Wall clock time.
-* ``final_index_size_bytes``: Final resulting index size after the benchmark.
+* ``final_index_size_bytes``: Final resulting index size on the file system after all nodes have been shutdown at the end of the benchmark. It includes all files in the nodes' data directories (actual index files and translog).
+* ``store_size_in_bytes``: The size in bytes of the index (excluding the translog) as reported by the indices stats API.
+* ``translog_size_in_bytes``: The size in bytes of the translog as reported by the indices stats API.
diff --git a/docs/summary_report.rst b/docs/summary_report.rst
index e1edb140..46e9c16e 100644
--- a/docs/summary_report.rst
+++ b/docs/summary_report.rst
@@ -81,9 +81,21 @@ Total Old Gen GC
 Index size
 ----------
 
-* **Definition**: Final resulting index size after the benchmark.
+* **Definition**: Final resulting index size on the file system after all nodes have been shutdown at the end of the benchmark. It includes all files in the nodes' data directories (actual index files and translog).
 * **Corresponding metrics key**: ``final_index_size_bytes``
 
+Store size
+----------
+
+* **Definition**: The size in bytes of the index (excluding the translog) as reported by the indices stats API.
+* **Corresponding metrics key**: ``store_size_in_bytes``
+
+Translog size
+-------------
+
+* **Definition**: The size in bytes of the translog as reported by the indices stats API.
+* **Corresponding metrics key**: ``translog_size_in_bytes``
+
 Totally written
 ---------------
 
diff --git a/docs/track.rst b/docs/track.rst
index 7af77b92..0bea3c76 100644
--- a/docs/track.rst
+++ b/docs/track.rst
@@ -665,7 +665,6 @@ Each challenge consists of the following properties:
 * ``name`` (mandatory): A descriptive name of the challenge. Should not contain spaces in order to simplify handling on the command line for users.
 * ``description`` (optional): A human readable description of the challenge.
 * ``default`` (optional): If true, Rally selects this challenge by default if the user did not specify a challenge on the command line. If your track only defines one challenge, it is implicitly selected as default, otherwise you need define ``"default": true`` on exactly one challenge.
-* ``index-settings`` (optional): Defines the index settings of the benchmark candidate when an index is created.
 * ``schedule`` (mandatory): Defines the concrete execution order of operations. It is described in more detail below.
 
 .. note::
diff --git a/esrally/client.py b/esrally/client.py
index f81d432b..8da59ff8 100644
--- a/esrally/client.py
+++ b/esrally/client.py
@@ -12,22 +12,47 @@ class EsClientFactory:
     Abstracts how the Elasticsearch client is created. Intended for testing.
     """
     def __init__(self, hosts, client_options):
-        logger.info("Creating ES client connected to %s with options [%s]" % (hosts, client_options))
+        masked_client_options = dict(client_options)
+        if "basic_auth_password" in masked_client_options:
+            masked_client_options["basic_auth_password"] = "*****"
+        if "http_auth" in masked_client_options:
+            masked_client_options["http_auth"] = (client_options["http_auth"][0], "*****")
+        logger.info("Creating ES client connected to %s with options [%s]", hosts, masked_client_options)
         self.hosts = hosts
         self.client_options = client_options
+        self.ssl_context = None
+
+        # we're using an SSL context now and it is not allowed to have use_ssl present in client options anymore
+        if client_options.pop("use_ssl", False):
+            import ssl
+            from elasticsearch.connection import create_ssl_context
+            logger.info("SSL support: on")
+            client_options["scheme"] = "https"
+
+            self.ssl_context = create_ssl_context(cafile=client_options.pop("ca_certs", certifi.where()))
+
+            if not client_options.pop("verify_certs", True):
+                logger.info("SSL certificate verification: off")
+                self.ssl_context.check_hostname = False
+                self.ssl_context.verify_mode = ssl.CERT_NONE
+
+                logger.warning("User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a "
+                               "benchmark. Disabling urllib warnings now to avoid a logging storm. "
+                               "See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings for details.")
+                # disable:  "InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly \
+                # advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings"
+                urllib3.disable_warnings()
+            else:
+                logger.info("SSL certificate verification: on")
+        else:
+            logger.info("SSL support: off")
+            client_options["scheme"] = "http"
 
-        if self._is_set(client_options, "use_ssl") and self._is_set(client_options, "verify_certs") and "ca_certs" not in client_options:
-            self.client_options["ca_certs"] = certifi.where()
-        elif self._is_set(client_options, "use_ssl") and not self._is_set(client_options, "verify_certs"):
-            logger.warning("User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark. "
-                           "Disabling urllib warnings now to avoid a logging storm. "
-                           "See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings for details.")
-            # disable:  "InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly \
-            # advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings"
-            urllib3.disable_warnings()
         if self._is_set(client_options, "basic_auth_user") and self._is_set(client_options, "basic_auth_password"):
-            # Maybe we should remove these keys from the dict?
-            self.client_options["http_auth"] = (client_options["basic_auth_user"], client_options["basic_auth_password"])
+            logger.info("HTTP basic authentication: on")
+            self.client_options["http_auth"] = (client_options.pop("basic_auth_user"), client_options.pop("basic_auth_password"))
+        else:
+            logger.info("HTTP basic authentication: off")
 
     def _is_set(self, client_opts, k):
         try:
@@ -55,8 +80,12 @@ class EsClientFactory:
             def __init__(self, compressed=False, **kwargs):
                 super(ConfigurableHttpConnection, self).__init__(**kwargs)
                 if compressed:
+                    logger.info("HTTP compression: on")
                     self.headers.update(urllib3.make_headers(accept_encoding=True))
                     self.headers.update({"Content-Encoding": "gzip"})
+                else:
+                    logger.info("HTTP compression: off")
                 self.pool = PoolWrap(self.pool, **kwargs)
 
-        return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection, **self.client_options)
+        return elasticsearch.Elasticsearch(hosts=self.hosts, connection_class=ConfigurableHttpConnection,
+                                           ssl_context=self.ssl_context, **self.client_options)
diff --git a/esrally/mechanic/telemetry.py b/esrally/mechanic/telemetry.py
index 6d8ae9c0..27bcdd82 100644
--- a/esrally/mechanic/telemetry.py
+++ b/esrally/mechanic/telemetry.py
@@ -708,7 +708,7 @@ class IndexStats(InternalTelemetryDevice):
         # the pipeline "benchmark-only" where we don't have control over the cluster and the user might not have restarted
         # the cluster so we can at least tell them.
         if self.first_time:
-            index_times = self.index_times(self.primaries_index_stats())
+            index_times = self.index_times(self.index_stats()["primaries"])
             for k, v in index_times.items():
                 if v > 0:
                     console.warn("%s is %d ms indicating that the cluster is not in a defined clean state. Recorded index time "
@@ -718,8 +718,9 @@ class IndexStats(InternalTelemetryDevice):
     def on_benchmark_stop(self):
         import json
         logger.info("Gathering indices stats for all primaries on benchmark stop.")
-        p = self.primaries_index_stats()
-        logger.info("Returned indices stats:\n%s" % json.dumps(p, indent=2))
+        index_stats = self.index_stats()
+        logger.info("Returned indices stats:\n%s" % json.dumps(index_stats, indent=2))
+        p = index_stats["primaries"]
         # actually this is add_count
         self.add_metrics(self.extract_value(p, ["segments", "count"]), "segments_count")
         self.add_metrics(self.extract_value(p, ["segments", "memory_in_bytes"]), "segments_memory_in_bytes", "byte")
@@ -733,12 +734,14 @@ class IndexStats(InternalTelemetryDevice):
         self.add_metrics(self.extract_value(p, ["segments", "terms_memory_in_bytes"]), "segments_terms_memory_in_bytes", "byte")
         self.add_metrics(self.extract_value(p, ["segments", "norms_memory_in_bytes"]), "segments_norms_memory_in_bytes", "byte")
         self.add_metrics(self.extract_value(p, ["segments", "points_memory_in_bytes"]), "segments_points_memory_in_bytes", "byte")
+        self.add_metrics(self.extract_value(index_stats, ["total", "store", "size_in_bytes"]), "store_size_in_bytes", "byte")
+        self.add_metrics(self.extract_value(index_stats, ["total", "translog", "size_in_bytes"]), "translog_size_in_bytes", "byte")
 
-    def primaries_index_stats(self):
+    def index_stats(self):
         # noinspection PyBroadException
         try:
             stats = self.client.indices.stats(metric="_all", level="shards")
-            return stats["_all"]["primaries"]
+            return stats["_all"]
         except BaseException:
             logger.exception("Could not retrieve index stats.")
             return {}
diff --git a/esrally/metrics.py b/esrally/metrics.py
index 1e284dd7..44ce00c6 100644
--- a/esrally/metrics.py
+++ b/esrally/metrics.py
@@ -132,18 +132,24 @@ class EsClientFactory:
         password = self._config.opts("reporting", "datastore.password")
         verify = self._config.opts("reporting", "datastore.ssl.verification_mode", default_value="full", mandatory=False) != "none"
         ca_path = self._config.opts("reporting", "datastore.ssl.certificate_authorities", default_value=None, mandatory=False)
-        if ca_path is None and verify:
-            ca_path = certifi.where()
 
+        from esrally import client
+
+        # Instead of duplicating code, we're just adapting the metrics store specific properties to match the regular client options.
+        client_options = {
+            "use_ssl": secure,
+            "verify_certs": verify,
+            "timeout": 120
+        }
+        if ca_path:
+            client_options["ca_certs"] = ca_path
         if user and password:
-            auth = (user, password)
-        else:
-            auth = None
-        logger.info("Creating connection to metrics store at %s:%s" % (host, port))
-        import elasticsearch
-        self._client = elasticsearch.Elasticsearch(hosts=[{"host": host, "port": port}],
-                                                   use_ssl=secure, http_auth=auth, verify_certs=verify, ca_certs=ca_path,
-                                                   timeout=120, request_timeout=120)
+            client_options["basic_auth_user"] = user
+            client_options["basic_auth_password"] = password
+
+        logger.info("Creating connection to metrics store at %s:%s", host, port)
+        factory = client.EsClientFactory(hosts=[{"host": host, "port": port}], client_options=client_options)
+        self._client = factory.create()
 
     def create(self):
         return EsClient(self._client)
diff --git a/esrally/reporter.py b/esrally/reporter.py
index ee0b997a..c1bcfdca 100644
--- a/esrally/reporter.py
+++ b/esrally/reporter.py
@@ -168,9 +168,13 @@ class StatsCalculator:
         result.memory_points = self.median("segments_points_memory_in_bytes")
         result.memory_stored_fields = self.median("segments_stored_fields_memory_in_bytes")
 
-        # This metric will only be written for the last iteration (as it can only be determined after the cluster has been shut down)
         logger.debug("Gathering disk metrics.")
+        # This metric will only be written for the last iteration (as it can only be determined after the cluster has been shut down)
         result.index_size = self.sum("final_index_size_bytes")
+        # we need to use the median here because these two are captured with the indices stats API and thus once per lap. If we'd
+        # sum up the values we'd get wrong results for benchmarks that ran for multiple laps.
+        result.store_size = self.median("store_size_in_bytes")
+        result.translog_size = self.median("translog_size_in_bytes")
         result.bytes_written = self.sum("disk_io_write_bytes")
 
         # convert to int, fraction counts are senseless
@@ -264,6 +268,8 @@ class Stats:
         self.memory_stored_fields = self.v(d, "memory_stored_fields")
 
         self.index_size = self.v(d, "index_size")
+        self.store_size = self.v(d, "store_size")
+        self.translog_size = self.v(d, "translog_size")
         self.bytes_written = self.v(d, "bytes_written")
 
         self.segment_count = self.v(d, "segment_count")
@@ -497,6 +503,8 @@ class SummaryReporter:
 
     def report_disk_usage(self, stats):
         return self.join(
+            self.line("Store size", "", stats.store_size, "GB", convert.bytes_to_gb),
+            self.line("Translog size", "", stats.translog_size, "GB", convert.bytes_to_gb),
             self.line("Index size", "", stats.index_size, "GB", convert.bytes_to_gb),
             self.line("Totally written", "", stats.bytes_written, "GB", convert.bytes_to_gb)
         )
@@ -688,6 +696,10 @@ class ComparisonReporter:
 
     def report_disk_usage(self, baseline_stats, contender_stats):
         return self.join(
+            self.line("Store size", baseline_stats.store_size, contender_stats.store_size, "", "GB",
+                      treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
+            self.line("Translog size", baseline_stats.translog_size, contender_stats.translog_size, "", "GB",
+                      treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
             self.line("Index size", baseline_stats.index_size, contender_stats.index_size, "", "GB",
                       treat_increase_as_improvement=False, formatter=convert.bytes_to_gb),
             self.line("Totally written", baseline_stats.bytes_written, contender_stats.bytes_written, "", "GB",
diff --git a/esrally/resources/track-schema.json b/esrally/resources/track-schema.json
index f148778f..8e9c3829 100644
--- a/esrally/resources/track-schema.json
+++ b/esrally/resources/track-schema.json
@@ -29,10 +29,6 @@
           "type": "object",
           "description": "Defines the cluster settings of the benchmark candidate."
         },
-        "index-settings": {
-          "type": "object",
-          "description": "Defines the index settings of the benchmark candidate when an index is created."
-        },
         "schedule": {
           "type": "array",
           "minItems": 1,
diff --git a/esrally/track/loader.py b/esrally/track/loader.py
index 5adf6d71..d04c516d 100644
--- a/esrally/track/loader.py
+++ b/esrally/track/loader.py
@@ -829,14 +829,8 @@ class TrackSpecificationReader:
             meta_data = self._r(challenge_spec, "meta", error_ctx=name, mandatory=False)
             # if we only have one challenge it is treated as default challenge, no matter what the user has specified
             default = number_of_challenges == 1 or self._r(challenge_spec, "default", error_ctx=name, mandatory=False)
-            # TODO #381: Remove this setting
-            index_settings = self._r(challenge_spec, "index-settings", error_ctx=name, mandatory=False)
             cluster_settings = self._r(challenge_spec, "cluster-settings", error_ctx=name, mandatory=False)
 
-            if index_settings and self.name not in DEFAULT_TRACKS:
-                console.warn("Challenge [%s] in track [%s] defines the [index-settings] property which will be removed soon. For details "
-                             "please see the migration guide in the docs." % (name, self.name))
-
             if default and default_challenge is not None:
                 self._error("Both '%s' and '%s' are defined as default challenges. Please define only one of them as default."
                             % (default_challenge.name, name))
@@ -867,7 +861,6 @@ class TrackSpecificationReader:
                                         meta_data=meta_data,
                                         description=description,
                                         user_info=user_info,
-                                        index_settings=index_settings,
                                         cluster_settings=cluster_settings,
                                         default=default,
                                         schedule=schedule)
diff --git a/esrally/track/track.py b/esrally/track/track.py
index 2bd67231..1b9dde50 100644
--- a/esrally/track/track.py
+++ b/esrally/track/track.py
@@ -349,7 +349,6 @@ class Challenge:
                  name,
                  description=None,
                  user_info=None,
-                 index_settings=None,
                  cluster_settings=None,
                  default=False,
                  meta_data=None,
@@ -358,7 +357,6 @@ class Challenge:
         self.meta_data = meta_data if meta_data else {}
         self.description = description
         self.user_info = user_info
-        self.index_settings = index_settings if index_settings else {}
         self.cluster_settings = cluster_settings if cluster_settings else {}
         self.default = default
         self.schedule = schedule if schedule else []
@@ -379,13 +377,13 @@ class Challenge:
         return ", ".join(r)
 
     def __hash__(self):
-        return hash(self.name) ^ hash(self.description) ^ hash(self.index_settings) ^ hash(self.cluster_settings) ^ hash(self.default) ^ \
+        return hash(self.name) ^ hash(self.description) ^ hash(self.cluster_settings) ^ hash(self.default) ^ \
                hash(self.meta_data) ^ hash(self.schedule)
 
     def __eq__(self, othr):
         return (isinstance(othr, type(self)) and
-                (self.name, self.description, self.index_settings, self.cluster_settings, self.default, self.meta_data, self.schedule) ==
-                (othr.name, othr.description, othr.index_settings, othr.cluster_settings, othr.default, othr.meta_data, othr.schedule))
+                (self.name, self.description, self.cluster_settings, self.default, self.meta_data, self.schedule) ==
+                (othr.name, othr.description, othr.cluster_settings, othr.default, othr.meta_data, othr.schedule))
 
 
 @unique
diff --git a/run.sh b/run.sh
index 6fdd7311..5c952382 100755
--- a/run.sh
+++ b/run.sh
@@ -18,9 +18,9 @@ install_esrally_with_setuptools () {
     fi
 
     if [[ ${IN_VIRTUALENV} == 0 ]]; then
-        python3 setup.py -q develop --user
+        python3 setup.py -q develop --user --upgrade
     else
-        python3 setup.py -q develop
+        python3 setup.py -q develop --upgrade
     fi
 }
 
diff --git a/setup.py b/setup.py
index 9ad27054..0accae0e 100644
--- a/setup.py
+++ b/setup.py
@@ -27,7 +27,7 @@ long_description = str_from_file("README.rst")
 #
 ################################################################################################
 install_requires = [
-    "elasticsearch==6.0.0",
+    "elasticsearch==6.2.0",
     "psutil==5.4.0",
     "py-cpuinfo==3.2.0",
     "tabulate==0.8.1",


Test patch:
diff --git a/tests/mechanic/telemetry_test.py b/tests/mechanic/telemetry_test.py
index 28a2aae5..58f6e1eb 100644
--- a/tests/mechanic/telemetry_test.py
+++ b/tests/mechanic/telemetry_test.py
@@ -764,6 +764,17 @@ class IndexStatsTests(TestCase):
                     "flush": {
                         "total_time_in_millis": 100
                     }
+                },
+                "total": {
+                    "store": {
+                        "size_in_bytes": 2113867510
+                    },
+                    "translog": {
+                        "operations": 6840000,
+                        "size_in_bytes": 2647984713,
+                        "uncommitted_operations": 0,
+                        "uncommitted_size_in_bytes": 430
+                    }
                 }
             }
         })
@@ -784,7 +795,8 @@ class IndexStatsTests(TestCase):
             mock.call("segments_stored_fields_memory_in_bytes", 1024, "byte"),
             mock.call("segments_terms_memory_in_bytes", 256, "byte"),
             # we don't have norms, so nothing should have been called
-            mock.call("segments_points_memory_in_bytes", 512, "byte"),
+            mock.call("store_size_in_bytes", 2113867510, "byte"),
+            mock.call("translog_size_in_bytes", 2647984713, "byte"),
         ], any_order=True)
 
     @mock.patch("esrally.metrics.EsMetricsStore.put_value_cluster_level")
diff --git a/tests/metrics_test.py b/tests/metrics_test.py
index 1c676b4a..ea1224ca 100644
--- a/tests/metrics_test.py
+++ b/tests/metrics_test.py
@@ -668,7 +668,7 @@ class EsResultsStoreTests(TestCase):
 
         t = track.Track(name="unittest-track",
                         indices=[track.Index(name="tests", types=["test-type"])],
-                        challenges=[track.Challenge(name="index", default=True, index_settings=None, schedule=schedule)])
+                        challenges=[track.Challenge(name="index", default=True, schedule=schedule)])
 
         c = cluster.Cluster([], [], None)
         c.distribution_version = "5.0.0"
diff --git a/tests/track/loader_test.py b/tests/track/loader_test.py
index 85f8bb31..7afea9a7 100644
--- a/tests/track/loader_test.py
+++ b/tests/track/loader_test.py
@@ -691,7 +691,6 @@ class TrackPostProcessingTests(TestCase):
                 {
                     "name": "default-challenge",
                     "description": "Default challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -764,7 +763,6 @@ class TrackPostProcessingTests(TestCase):
                 {
                     "name": "default-challenge",
                     "description": "Default challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1024,7 +1022,6 @@ class TrackSpecificationReaderTests(TestCase):
             "challenges": [
                 {
                     "name": "default-challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1148,7 +1145,6 @@ class TrackSpecificationReaderTests(TestCase):
             "challenges": [
                 {
                     "name": "default-challenge",
-                    "index-settings": {},
                     "schedule": [
                         {
                             "clients": 8,
@@ -1327,9 +1323,6 @@ class TrackSpecificationReaderTests(TestCase):
                         "mixed": True,
                         "max-clients": 8
                     },
-                    "index-settings": {
-                        "index.number_of_replicas": 2
-                    },
                     "schedule": [
                         {
                             "clients": 8,
@@ -1414,8 +1407,6 @@ class TrackSpecificationReaderTests(TestCase):
         self.assertEqual(1, len(resulting_track.challenges))
         self.assertEqual("default-challenge", resulting_track.challenges[0].name)
         self.assertEqual("Default challenge", resulting_track.challenges[0].description)
-        self.assertEqual(1, len(resulting_track.challenges[0].index_settings))
-        self.assertEqual(2, resulting_track.challenges[0].index_settings["index.number_of_replicas"])
         self.assertEqual({"mixed": True, "max-clients": 8}, resulting_track.challenges[0].meta_data)
         self.assertEqual({"append": True}, resulting_track.challenges[0].schedule[0].operation.meta_data)
         self.assertEqual({"operation-index": 0}, resulting_track.challenges[0].schedule[0].meta_data)