Repository: elastic/rally

Issue:
Allow to use Rally without knowing about challenges
_Note: This is a follow-up from #455._

### Purpose

In order to make it simpler for new users, "challenges" will become optional in Rally. We will however not get rid of them because it makes sense to keep them to provide structure for more complex tracks.

### Tasks

* [x] Simplify the track format so if there is only one challenge, users can specify the `schedule` element on top-level.
* [x] Hide the notion of a challenge from UI output in these simple cases.
* [x] State more clearly in the docs that challenges are a more advanced concept and that it is not necessary to use them when getting started.

Discussion:

Patch:
diff --git a/docs/adding_tracks.rst b/docs/adding_tracks.rst
index 43f9ab18..dde2e4ca 100644
--- a/docs/adding_tracks.rst
+++ b/docs/adding_tracks.rst
@@ -96,6 +96,124 @@ For details on the allowed syntax, see the Elasticsearch documentation on `mappi
 
 Finally, store the track as ``track.json`` in the tutorial directory::
 
+    {
+      "version": 2,
+      "description": "Tutorial benchmark for Rally",
+      "indices": [
+        {
+          "name": "geonames",
+          "body": "index.json",
+          "types": [ "docs" ]
+        }
+      ],
+      "corpora": [
+        {
+          "name": "rally-tutorial",
+          "documents": [
+            {
+              "source-file": "documents.json",
+              "document-count": 11658903,
+              "uncompressed-bytes": 1544799789
+            }
+          ]
+        }
+      ],
+      "schedule": [
+        {
+          "operation": {
+            "operation-type": "delete-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "create-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "cluster-health",
+            "request-params": {
+              "wait_for_status": "green"
+            }
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "bulk",
+            "bulk-size": 5000
+          },
+          "warmup-time-period": 120,
+          "clients": 8
+        },
+        {
+          "operation": {
+            "operation-type": "force-merge"
+          }
+        },
+        {
+          "operation": {
+            "name": "query-match-all",
+            "operation-type": "search",
+            "body": {
+              "query": {
+                "match_all": {}
+              }
+            }
+          },
+          "clients": 8,
+          "warmup-iterations": 1000,
+          "iterations": 1000,
+          "target-throughput": 100
+        }
+      ]
+    }
+
+
+The numbers under the ``documents`` property are needed to verify integrity and provide progress reports. Determine the correct document count with ``wc -l documents.json`` and the size in bytes with ``stat -f "%z" documents.json``.
+
+.. note::
+
+    You can store any supporting scripts along with your track. However, you need to place them in a directory starting with "_", e.g. "_support". Rally loads track plugins (see below) from any directory but will ignore directories starting with "_".
+
+.. note::
+
+    We have defined a `JSON schema for tracks <https://github.com/elastic/rally/blob/master/esrally/resources/track-schema.json>`_ which you can use to check how to define your track. You should also check the tracks provided by Rally for inspiration.
+
+The new track appears when you run ``esrally list tracks --track-path=~/rally-tracks/tutorial``::
+
+    dm@io:~ $ esrally list tracks --track-path=~/rally-tracks/tutorial
+
+        ____        ____
+       / __ \____ _/ / /_  __
+      / /_/ / __ `/ / / / / /
+     / _, _/ /_/ / / / /_/ /
+    /_/ |_|\__,_/_/_/\__, /
+                    /____/
+    Available tracks:
+
+    Name        Description                   Documents    Compressed Size  Uncompressed Size
+    ----------  ----------------------------- -----------  ---------------  -----------------
+    tutorial    Tutorial benchmark for Rally      11658903  N/A              1.4 GB
+
+Congratulations, you have created your first track! You can test it with ``esrally --distribution-version=6.0.0 --track-path=~/rally-tracks/tutorial``.
+
+.. _add_track_test_mode:
+
+Adding support for test mode
+----------------------------
+
+You can check your track very quickly for syntax errors when you invoke Rally with ``--test-mode``. Rally postprocesses its internal track representation as follows:
+
+* Iteration-based tasks run at most one warmup iteration and one measurement iteration.
+* Time-period-based tasks run at most for 10 seconds without warmup.
+
+Rally also postprocesses all data file names. Instead of ``documents.json``, Rally expects ``documents-1k.json`` and assumes the file contains 1.000 documents. You need to prepare these data files though. Pick 1.000 documents for every data file in your track and store them in a file with the suffix ``-1k``. We choose the first 1.000 with ``head -n 1000 documents.json > documents-1k.json``.
+
+Challenges
+----------
+
+To specify different workloads in the same track you can use so-called challenges. Instead of specifying the ``schedule`` property on top-level you specify a ``challenges`` array::
+
     {
       "version": 2,
       "description": "Tutorial benchmark for Rally",
@@ -174,46 +292,14 @@ Finally, store the track as ``track.json`` in the tutorial directory::
       ]
     }
 
-
-The numbers under the ``documents`` property are needed to verify integrity and provide progress reports. Determine the correct document count with ``wc -l documents.json`` and the size in bytes with ``stat -f "%z" documents.json``.
-
-.. note::
-
-    You can store any supporting scripts along with your track. However, you need to place them in a directory starting with "_", e.g. "_support". Rally loads track plugins (see below) from any directory but will ignore directories starting with "_".
-
 .. note::
 
-    We have defined a `JSON schema for tracks <https://github.com/elastic/rally/blob/master/esrally/resources/track-schema.json>`_ which you can use to check how to define your track. You should also check the tracks provided by Rally for inspiration.
-
-The new track appears when you run ``esrally list tracks --track-path=~/rally-tracks/tutorial``::
-
-    dm@io:~ $ esrally list tracks --track-path=~/rally-tracks/tutorial
-
-        ____        ____
-       / __ \____ _/ / /_  __
-      / /_/ / __ `/ / / / / /
-     / _, _/ /_/ / / / /_/ /
-    /_/ |_|\__,_/_/_/\__, /
-                    /____/
-    Available tracks:
-
-    Name        Description                   Documents    Compressed Size  Uncompressed Size  Default Challenge  All Challenges
-    ----------  ----------------------------- -----------  ---------------  -----------------  -----------------  ---------------
-    tutorial    Tutorial benchmark for Rally      11658903  N/A              1.4 GB            index-and-query    index-and-query
-
-Congratulations, you have created your first track! You can test it with ``esrally --distribution-version=6.0.0 --track-path=~/rally-tracks/tutorial``.
-
-.. _add_track_test_mode:
-
-Adding support for test mode
-----------------------------
-
-You can check your track very quickly for syntax errors when you invoke Rally with ``--test-mode``. Rally postprocesses its internal track representation as follows:
+    If you define multiple challenges, Rally runs the challenge where ``default`` is set to ``true``. If you want to run a different challenge, provide the command line option ``--challenge=YOUR_CHALLENGE_NAME``.
 
-* Iteration-based tasks run at most one warmup iteration and one measurement iteration.
-* Time-period-based tasks run at most for 10 seconds without warmup.
+When should you use challenges? Challenges are useful when you want to run completely different workloads based on the same track but for the majority of cases you should get away without using challenges:
 
-Rally also postprocesses all data file names. Instead of ``documents.json``, Rally expects ``documents-1k.json`` and assumes the file contains 1.000 documents. You need to prepare these data files though. Pick 1.000 documents for every data file in your track and store them in a file with the suffix ``-1k``. We choose the first 1.000 with ``head -n 1000 documents.json > documents-1k.json``.
+* To run only a subset of the tasks, you can use :ref:`task filtering <clr_include_tasks>`, e.g. ``--include-tasks="create-index,bulk"` will only run these two tasks in the track above.
+* To vary parameters, e.g. the number of clients, you can use :ref:`track parameters <clr_track_params>`
 
 Structuring your track
 ----------------------
@@ -477,10 +563,6 @@ The changes are:
 
     Rally's log file contains the fully rendered track after it has loaded it successfully.
 
-.. note::
-
-    If you define multiple challenges, Rally runs the challenge where ``default`` is set to ``true``. If you want to run a different challenge, provide the command line option ``--challenge=YOUR_CHALLENGE_NAME``.
-
 You can even use `Jinja2 variables <http://jinja.pocoo.org/docs/2.9/templates/#assignments>`_ but then you need to import the Rally helpers a bit differently. You also need to declare all variables before the ``import`` statement::
 
         {% set clients = 16 %}
diff --git a/docs/command_line_reference.rst b/docs/command_line_reference.rst
index 41dcd3ba..027d9d67 100644
--- a/docs/command_line_reference.rst
+++ b/docs/command_line_reference.rst
@@ -69,6 +69,8 @@ Selects the track repository that Rally should use to resolve tracks. By default
 
 Selects the track that Rally should run. By default the ``geonames`` track is run. For more details on how tracks work, see :doc:`adding tracks </adding_tracks>` or the :doc:`track reference </track>`. ``--track-path`` and ``--track-repository`` as well as ``--track`` are mutually exclusive.
 
+.. _clr_track_params:
+
 ``track-params``
 ~~~~~~~~~~~~~~~~
 
diff --git a/docs/track.rst b/docs/track.rst
index aaa93ee1..33eca156 100644
--- a/docs/track.rst
+++ b/docs/track.rst
@@ -112,7 +112,7 @@ A track JSON file consists of the following sections:
 * templates
 * corpora
 * operations
-* challenges
+* schedule
 
 In the ``indices`` and ``templates`` sections you define the relevant indices and index templates. These sections are optional but recommended if you want to create indices and index templates with the help of Rally.
 
@@ -120,7 +120,7 @@ In the ``corpora`` section you define all document corpora (i.e. data files) tha
 
 In the ``operations`` section you describe which operations are available for this track and how they are parametrized. This section is optional and you can also define any operations directly per challenge. You can use it, if you want to share operation definitions between challenges.
 
-In the ``challenge`` or ``challenges`` section you describe one or more execution schedules respectively. Each schedule either uses the operations defined in the ``operations`` block or defines the operations to execute inline. Think of a challenge as a scenario that you want to test for your data. An example challenge is to index with two clients at maximum throughput while searching with another two clients with ten operations per second.
+In the ``schedule`` section you describe the workload for the benchmark, for example index with two clients at maximum throughput while searching with another two clients with ten operations per second. The schedule either uses the operations defined in the ``operations`` block or defines the operations to execute inline.
 
 Track elements
 ==============
@@ -293,7 +293,7 @@ We can also define default values on document corpus level but override some of
 operations
 ..........
 
-The ``operations`` section contains a list of all operations that are available later when specifying challenges. Operations define the static properties of a request against Elasticsearch whereas the ``schedule`` element defines the dynamic properties (such as the target throughput).
+The ``operations`` section contains a list of all operations that are available later when specifying a schedule. Operations define the static properties of a request against Elasticsearch whereas the ``schedule`` element defines the dynamic properties (such as the target throughput).
 
 Each operation consists of the following properties:
 
@@ -658,27 +658,10 @@ With the operation ``raw-request`` you can execute arbitrary HTTP requests again
 * ``request-params`` (optional): A structure containing HTTP request parameters.
 * ``ignore`` (optional): An array of HTTP response status codes to ignore (i.e. consider as successful).
 
-challenge
-.........
-
-If you track has only one challenge, you can use the ``challenge`` element. If you have multiple challenges, you can define an array of ``challenges``.
-
-This section contains one or more challenges which describe the benchmark scenarios for this data set. A challenge can reference all operations that are defined in the ``operations`` section.
-
-Each challenge consists of the following properties:
-
-* ``name`` (mandatory): A descriptive name of the challenge. Should not contain spaces in order to simplify handling on the command line for users.
-* ``description`` (optional): A human readable description of the challenge.
-* ``default`` (optional): If true, Rally selects this challenge by default if the user did not specify a challenge on the command line. If your track only defines one challenge, it is implicitly selected as default, otherwise you need define ``"default": true`` on exactly one challenge.
-* ``schedule`` (mandatory): Defines the concrete execution order of operations. It is described in more detail below.
-
-.. note::
-   You should strive to minimize the number of challenges. If you just want to run a subset of the tasks in a challenge, use :ref:`task filtering <clr_include_tasks>`.
-
 schedule
 ~~~~~~~~
 
-The ``schedule`` element contains a list of tasks that are executed by Rally. Each task consists of the following properties:
+The ``schedule`` element contains a list of tasks that are executed by Rally, i.e. it describes the workload. Each task consists of the following properties:
 
 * ``name`` (optional): This property defines an explicit name for the given task. By default the operation's name is implicitly used as the task name but if the same operation is run multiple times, a unique task name must be specified using this property.
 * ``operation`` (mandatory): This property refers either to the name of an operation that has been defined in the ``operations`` section or directly defines an operation inline.
@@ -696,100 +679,91 @@ Defining operations
 
 In the following snippet we define two operations ``force-merge`` and a ``match-all`` query separately in an operations block::
 
-   {
-     "operations": [
-       {
-         "name": "force-merge",
-         "operation-type": "force-merge"
-       },
-       {
-         "name": "match-all-query",
-         "operation-type": "search",
-         "body": {
-           "query": {
-             "match_all": {}
-           }
-         }
-       }
-     ],
-     "challenge": {
-       "name": "just-query",
-       "schedule": [
-         {
-           "operation": "force-merge",
-           "clients": 1
-         },
-         {
-           "operation": "match-all-query",
-           "clients": 4,
-           "warmup-iterations": 1000,
-           "iterations": 1000,
-           "target-throughput": 100
-         }
-       ]
-     }
-   }
+    {
+      "operations": [
+        {
+          "name": "force-merge",
+          "operation-type": "force-merge"
+        },
+        {
+          "name": "match-all-query",
+          "operation-type": "search",
+          "body": {
+            "query": {
+              "match_all": {}
+            }
+          }
+        }
+      ],
+      "schedule": [
+        {
+          "operation": "force-merge",
+          "clients": 1
+        },
+        {
+          "operation": "match-all-query",
+          "clients": 4,
+          "warmup-iterations": 1000,
+          "iterations": 1000,
+          "target-throughput": 100
+        }
+      ]
+    }
 
 If we do not want to reuse these operations, we can also define them inline. Note that the ``operations`` section is gone::
 
-   {
-     "challenge": {
-       "name": "just-query",
-       "schedule": [
-         {
-           "operation": {
-             "name": "force-merge",
-             "operation-type": "force-merge"
-           },
-           "clients": 1
-         },
-         {
-           "operation": {
-             "name": "match-all-query",
-             "operation-type": "search",
-             "body": {
-               "query": {
-                 "match_all": {}
-               }
-             }
-           },
-           "clients": 4,
-           "warmup-iterations": 1000,
-           "iterations": 1000,
-           "target-throughput": 100
-         }
-       ]
-     }
-   }
+    {
+      "schedule": [
+        {
+          "operation": {
+            "name": "force-merge",
+            "operation-type": "force-merge"
+          },
+          "clients": 1
+        },
+        {
+          "operation": {
+            "name": "match-all-query",
+            "operation-type": "search",
+            "body": {
+              "query": {
+                "match_all": {}
+              }
+            }
+          },
+          "clients": 4,
+          "warmup-iterations": 1000,
+          "iterations": 1000,
+          "target-throughput": 100
+        }
+      ]
+    }
 
 Contrary to the ``query``, the ``force-merge`` operation does not take any parameters, so Rally allows us to just specify the ``operation-type`` for this operation. It's name will be the same as the operation's type::
 
-   {
-     "challenge": {
-       "name": "just-query",
-       "schedule": [
-         {
-           "operation": "force-merge",
-           "clients": 1
-         },
-         {
-           "operation": {
-             "name": "match-all-query",
-             "operation-type": "search",
-             "body": {
-               "query": {
-                 "match_all": {}
-               }
-             }
-           },
-           "clients": 4,
-           "warmup-iterations": 1000,
-           "iterations": 1000,
-           "target-throughput": 100
-         }
-       ]
-     }
-   }
+    {
+      "schedule": [
+        {
+          "operation": "force-merge",
+          "clients": 1
+        },
+        {
+          "operation": {
+            "name": "match-all-query",
+            "operation-type": "search",
+            "body": {
+              "query": {
+                "match_all": {}
+              }
+            }
+          },
+          "clients": 4,
+          "warmup-iterations": 1000,
+          "iterations": 1000,
+          "target-throughput": 100
+        }
+      ]
+    }
 
 Choosing a schedule
 ...................
@@ -831,6 +805,23 @@ All tasks in the ``schedule`` list are executed sequentially in the order in whi
 
     Specify the number of clients on each task separately. If you specify this number on the ``parallel`` element instead, Rally will only use that many clients in total and you will only want to use this behavior in very rare cases (see examples)!
 
+challenge
+.........
+
+If your track defines only one benchmarking scenario specify the ``schedule`` on top-level. Use the ``challenge`` element if you want to specify additional properties like a name or a description. You can think of a challenge as a benchmarking scenario. If you have multiple challenges, you can define an array of ``challenges``.
+
+This section contains one or more challenges which describe the benchmark scenarios for this data set. A challenge can reference all operations that are defined in the ``operations`` section.
+
+Each challenge consists of the following properties:
+
+* ``name`` (mandatory): A descriptive name of the challenge. Should not contain spaces in order to simplify handling on the command line for users.
+* ``description`` (optional): A human readable description of the challenge.
+* ``default`` (optional): If true, Rally selects this challenge by default if the user did not specify a challenge on the command line. If your track only defines one challenge, it is implicitly selected as default, otherwise you need to define ``"default": true`` on exactly one challenge.
+* ``schedule`` (mandatory): Defines the workload. It is described in more detail above.
+
+.. note::
+
+    You should strive to minimize the number of challenges. If you just want to run a subset of the tasks in a challenge, use :ref:`task filtering <clr_include_tasks>`.
 
 Examples
 ========
@@ -841,29 +832,26 @@ A track with a single task
 To get started with custom tracks, you can benchmark a single task, e.g. a match_all query::
 
     {
-      "challenge": {
-        "name": "just-search",
-        "schedule": [
-          {
-            "operation": {
-              "operation-type": "search",
-              "index": "_all",
-              "body": {
-                "query": {
-                  "match_all": {}
-                }
+      "schedule": [
+        {
+          "operation": {
+            "operation-type": "search",
+            "index": "_all",
+            "body": {
+              "query": {
+                "match_all": {}
               }
-            },
-            "warmup-iterations": 100,
-            "iterations": 100,
-            "target-throughput": 10
-          }
-        ]
-      }
+            }
+          },
+          "warmup-iterations": 100,
+          "iterations": 100,
+          "target-throughput": 10
+        }
+      ]
     }
 
 
-This track assumes that you have an existing cluster with pre-populated data. It will run the provided match_all query at 10 operations per second with one client and use 100 iterations as warmup and the next 100 iterations to measure.
+This track assumes that you have an existing cluster with pre-populated data. It will run the provided ``match_all`` query at 10 operations per second with one client and use 100 iterations as warmup and the next 100 iterations to measure.
 
 For the examples below, note that we do not show the operation definition but you should be able to infer from the operation name what it is doing.
 
diff --git a/esrally/metrics.py b/esrally/metrics.py
index 24dfa0e7..937f54e2 100644
--- a/esrally/metrics.py
+++ b/esrally/metrics.py
@@ -294,7 +294,7 @@ class MetricsStore:
         """
         Opens a metrics store for a specific trial, track, challenge and car.
 
-        :param trial_id: The trial id. This attribute is sufficient to uniquely identify a challenge.
+        :param trial_id: The trial id. This attribute is sufficient to uniquely identify a race.
         :param trial_timestamp: The trial timestamp as a datetime.
         :param track_name: Track name.
         :param challenge_name: Challenge name.
@@ -1053,7 +1053,7 @@ def list_races(cfg):
 
     races = []
     for race in race_store(cfg).list():
-        races.append([time.to_iso8601(race.trial_timestamp), race.track, format_dict(race.track_params), race.challenge, race.car_name,
+        races.append([time.to_iso8601(race.trial_timestamp), race.track, format_dict(race.track_params), race.challenge_name, race.car_name,
                       format_dict(race.user_tags)])
 
     if len(races) > 0:
@@ -1107,7 +1107,7 @@ class Race:
 
     @property
     def challenge_name(self):
-        return str(self.challenge)
+        return str(self.challenge) if self.challenge else None
 
     @property
     def car_name(self):
@@ -1140,12 +1140,13 @@ class Race:
             "pipeline": self.pipeline,
             "user-tags": self.user_tags,
             "track": self.track_name,
-            "challenge": self.challenge_name,
             "car": self.car,
             "total-laps": self.total_laps,
             "cluster": self.cluster.as_dict(),
             "results": self.results.as_dict()
         }
+        if not self.challenge.auto_generated:
+            d["challenge"] = self.challenge_name
         if self.track_params:
             d["track-params"] = self.track_params
         return d
@@ -1163,7 +1164,6 @@ class Race:
             "distribution-major-version": versions.major_version(self.cluster.distribution_version),
             "user-tags": self.user_tags,
             "track": self.track_name,
-            "challenge": self.challenge_name,
             "car": self.car_name,
             "node-count": len(self.cluster.nodes),
             # allow to logically delete records, e.g. for UI purposes when we only want to show the latest result on graphs
@@ -1176,6 +1176,8 @@ class Race:
         if plugins:
             result_template["plugins"] = list(plugins)
 
+        if not self.challenge.auto_generated:
+            result_template["challenge"] = self.challenge_name
         if self.track_params:
             result_template["track-params"] = self.track_params
 
@@ -1201,7 +1203,7 @@ class Race:
         # Don't restore a few properties like cluster because they (a) cannot be reconstructed easily without knowledge of other modules
         # and (b) it is not necessary for this use case.
         return Race(d["rally-version"], d["environment"], d["trial-id"], time.from_is8601(d["trial-timestamp"]), d["pipeline"], user_tags,
-                    d["track"], d.get("track-params"), d["challenge"], d["car"], d["total-laps"], results=d["results"])
+                    d["track"], d.get("track-params"), d.get("challenge"), d["car"], d["total-laps"], results=d["results"])
 
 
 class RaceStore:
diff --git a/esrally/racecontrol.py b/esrally/racecontrol.py
index 5971b0b6..bf76e16b 100644
--- a/esrally/racecontrol.py
+++ b/esrally/racecontrol.py
@@ -105,8 +105,12 @@ class BenchmarkActor(actor.RallyActor):
         self.metrics_store.meta_info = msg.system_meta_info
         cluster = msg.cluster_meta_info
         self.race.cluster = cluster
-        console.info("Racing on track [%s], challenge [%s] and car %s with version [%s].\n"
-                     % (self.race.track_name, self.race.challenge_name, self.race.car, self.race.cluster.distribution_version))
+        if self.race.challenge.auto_generated:
+            console.info("Racing on track [{}] and car {} with version [{}].\n"
+                         .format(self.race.track_name, self.race.car, self.race.cluster.distribution_version))
+        else:
+            console.info("Racing on track [{}], challenge [{}] and car {} with version [{}].\n"
+                         .format(self.race.track_name, self.race.challenge_name, self.race.car, self.race.cluster.distribution_version))
         # start running we assume that each race has at least one lap
         self.run()
 
@@ -198,7 +202,7 @@ class BenchmarkActor(actor.RallyActor):
         self.lap_counter = LapCounter(self.race, self.metrics_store, self.cfg)
         self.race_store = metrics.race_store(self.cfg)
         self.logger.info("Asking mechanic to start the engine.")
-        cluster_settings = self.race.challenge.cluster_settings
+        cluster_settings = challenge.cluster_settings
         self.send(self.mechanic, mechanic.StartEngine(self.cfg, self.metrics_store.open_context, cluster_settings, msg.sources, msg.build,
                                                       msg.distribution, msg.external, msg.docker))
 
diff --git a/esrally/reporter.py b/esrally/reporter.py
index 156f0bd3..67d371bd 100644
--- a/esrally/reporter.py
+++ b/esrally/reporter.py
@@ -554,12 +554,14 @@ class ComparisonReporter:
         print_internal("")
         print_internal("Comparing baseline")
         print_internal("  Race timestamp: %s" % r1.trial_timestamp)
-        print_internal("  Challenge: %s" % r1.challenge_name)
+        if r1.challenge_name:
+            print_internal("  Challenge: %s" % r1.challenge_name)
         print_internal("  Car: %s" % r1.car_name)
         print_internal("")
         print_internal("with contender")
         print_internal("  Race timestamp: %s" % r2.trial_timestamp)
-        print_internal("  Challenge: %s" % r2.challenge_name)
+        if r2.challenge_name:
+            print_internal("  Challenge: %s" % r2.challenge_name)
         print_internal("  Car: %s" % r2.car_name)
         print_internal("")
         print_header("------------------------------------------------------")
diff --git a/esrally/resources/track-schema.json b/esrally/resources/track-schema.json
index 8e9c3829..6078f355 100644
--- a/esrally/resources/track-schema.json
+++ b/esrally/resources/track-schema.json
@@ -4,6 +4,160 @@
   "description": "Specification of tracks for Rally",
   "type": "object",
   "definitions": {
+    "schedule": {
+      "title": "Schedule",
+      "type": "array",
+      "minItems": 1,
+      "description": "Defines the concrete execution order of operations.",
+      "items": {
+        "type": "object",
+        "properties": {
+          "parallel": {
+            "type": "object",
+            "description": "This element allows to define tasks that should be run in parallel. We do not support nested parallel tasks.",
+            "properties": {
+              "clients": {
+                "type": "integer",
+                "minimum": 1
+              },
+              "warmup-iterations": {
+                "type": "integer",
+                "minimum": 0
+              },
+              "iterations": {
+                "type": "integer",
+                "minimum": 1
+              },
+              "warmup-time-period": {
+                "type": "integer",
+                "minimum": 0,
+                "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
+              },
+              "time-period": {
+                "type": "integer",
+                "minimum": 1,
+                "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
+              },
+              "completed-by": {
+                "type": "string",
+                "description": "The name of an operation in the 'tasks' block. When this operation is completed, the whole parallel element is considered to be completed."
+              },
+              "tasks": {
+                "type": "array",
+                "minItems": 1,
+                "description": "Defines the operations that will be run in parallel",
+                "items": {
+                  "type": "object",
+                  "description": "Defines an individual operation that is executed",
+                  "properties": {
+                    "name": {
+                      "type": "string",
+                      "description": "Explicit name of the task. By default the operation's name is implicitly used as the task's name but if the same operation is run multiple times, a unique task name must be specified."
+                    },
+                    "operation": {
+                      "description": "The name of an operation that should be executed. This name must match the operation name in the 'operations' block."
+                    },
+                    "meta": {
+                      "type": "object",
+                      "description": "Meta-information which will be added to each metrics-record of this task."
+                    },
+                    "clients": {
+                      "type": "integer",
+                      "minimum": 1
+                    },
+                    "warmup-iterations": {
+                      "type": "integer",
+                      "minimum": 0,
+                      "description": "Defines the number of times to run the operation in order to warmup the benchmark candidate. Warmup iterations will not be considered in the benchmark result."
+                    },
+                    "iterations": {
+                      "type": "integer",
+                      "minimum": 1,
+                      "description": "Defines the number of times to run the operation."
+                    },
+                    "warmup-time-period": {
+                      "type": "integer",
+                      "minimum": 0,
+                      "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
+                    },
+                    "time-period": {
+                      "type": "integer",
+                      "minimum": 1,
+                      "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
+                    },
+                    "schedule": {
+                      "type": "string",
+                      "description": "Defines the scheduling strategy that is used for throughput throttled operations. Out of the box, Rally supports 'deterministic' (default) and 'poisson' but you can implement your own schedules."
+                    },
+                    "target-throughput": {
+                      "type": "number",
+                      "minimum": 0,
+                      "description": "Defines the number of operations per second that Rally should attempt to run."
+                    },
+                    "target-interval": {
+                      "type": "number",
+                      "minimum": 0,
+                      "description": "Defines the number of seconds to wait between operations (inverse of target-throughput). Only one of 'target-throughput' or 'target-interval' may be defined."
+                    }
+                  },
+                  "required": [
+                    "operation"
+                  ]
+                }
+              }
+            },
+            "required": [
+              "tasks"
+            ]
+          },
+          "name": {
+            "type": "string",
+            "description": "Explicit name of the task. By default the operation's name is implicitly used as the task's name but if the same operation is run multiple times, a unique task name must be specified."
+          },
+          "operation": {
+            "description": "The name of an operation that should be executed. This name must match the operation name in the 'operations' block."
+          },
+          "meta": {
+            "type": "object",
+            "description": "Meta-information which will be added to each metrics-record of this task."
+          },
+          "clients": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "warmup-iterations": {
+            "type": "integer",
+            "minimum": 0,
+            "description": "Defines the number of times to run the operation in order to warmup the benchmark candidate. Warmup iterations will not be considered in the benchmark result."
+          },
+          "iterations": {
+            "type": "integer",
+            "minimum": 1,
+            "description": "Defines the number of times to run the operation."
+          },
+          "warmup-time-period": {
+            "type": "integer",
+            "minimum": 0,
+            "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
+          },
+          "time-period": {
+            "type": "integer",
+            "minimum": 1,
+            "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
+          },
+          "target-throughput": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Defines the number of operations per second that Rally should attempt to run."
+          },
+          "target-interval": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Defines the number of seconds to wait between operations (inverse of target-throughput). Only one of 'target-throughput' or 'target-interval' may be defined."
+          }
+        }
+      }
+    },
     "challenge": {
       "title": "Challenge",
       "type": "object",
@@ -30,157 +184,7 @@
           "description": "Defines the cluster settings of the benchmark candidate."
         },
         "schedule": {
-          "type": "array",
-          "minItems": 1,
-          "description": "Defines the concrete execution order of operations.",
-          "items": {
-            "type": "object",
-            "properties": {
-              "parallel": {
-                "type": "object",
-                "description": "This element allows to define tasks that should be run in parallel. We do not support nested parallel tasks.",
-                "properties": {
-                  "clients": {
-                    "type": "integer",
-                    "minimum": 1
-                  },
-                  "warmup-iterations": {
-                    "type": "integer",
-                    "minimum": 0
-                  },
-                  "iterations": {
-                    "type": "integer",
-                    "minimum": 1
-                  },
-                  "warmup-time-period": {
-                    "type": "integer",
-                    "minimum": 0,
-                    "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
-                  },
-                  "time-period": {
-                    "type": "integer",
-                    "minimum": 1,
-                    "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
-                  },
-                  "completed-by": {
-                    "type": "string",
-                    "description": "The name of an operation in the 'tasks' block. When this operation is completed, the whole parallel element is considered to be completed."
-                  },
-                  "tasks": {
-                    "type": "array",
-                    "minItems": 1,
-                    "description": "Defines the operations that will be run in parallel",
-                    "items": {
-                      "type": "object",
-                      "description": "Defines an individual operation that is executed",
-                      "properties": {
-                        "name": {
-                          "type": "string",
-                          "description": "Explicit name of the task. By default the operation's name is implicitly used as the task's name but if the same operation is run multiple times, a unique task name must be specified."
-                        },
-                        "operation": {
-                          "description": "The name of an operation that should be executed. This name must match the operation name in the 'operations' block."
-                        },
-                        "meta": {
-                          "type": "object",
-                          "description": "Meta-information which will be added to each metrics-record of this task."
-                        },
-                        "clients": {
-                          "type": "integer",
-                          "minimum": 1
-                        },
-                        "warmup-iterations": {
-                          "type": "integer",
-                          "minimum": 0,
-                          "description": "Defines the number of times to run the operation in order to warmup the benchmark candidate. Warmup iterations will not be considered in the benchmark result."
-                        },
-                        "iterations": {
-                          "type": "integer",
-                          "minimum": 1,
-                          "description": "Defines the number of times to run the operation."
-                        },
-                        "warmup-time-period": {
-                          "type": "integer",
-                          "minimum": 0,
-                          "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
-                        },
-                        "time-period": {
-                          "type": "integer",
-                          "minimum": 1,
-                          "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
-                        },
-                        "schedule": {
-                          "type": "string",
-                          "description": "Defines the scheduling strategy that is used for throughput throttled operations. Out of the box, Rally supports 'deterministic' (default) and 'poisson' but you can implement your own schedules."
-                        },
-                        "target-throughput": {
-                          "type": "number",
-                          "minimum": 0,
-                          "description": "Defines the number of operations per second that Rally should attempt to run."
-                        },
-                        "target-interval": {
-                          "type": "number",
-                          "minimum": 0,
-                          "description": "Defines the number of seconds to wait between operations (inverse of target-throughput). Only one of 'target-throughput' or 'target-interval' may be defined."
-                        }
-                      },
-                      "required": [
-                        "operation"
-                      ]
-                    }
-                  }
-                },
-                "required": [
-                  "tasks"
-                ]
-              },
-              "name": {
-                "type": "string",
-                "description": "Explicit name of the task. By default the operation's name is implicitly used as the task's name but if the same operation is run multiple times, a unique task name must be specified."
-              },
-              "operation": {
-                "description": "The name of an operation that should be executed. This name must match the operation name in the 'operations' block."
-              },
-              "meta": {
-                "type": "object",
-                "description": "Meta-information which will be added to each metrics-record of this task."
-              },
-              "clients": {
-                "type": "integer",
-                "minimum": 1
-              },
-              "warmup-iterations": {
-                "type": "integer",
-                "minimum": 0,
-                "description": "Defines the number of times to run the operation in order to warmup the benchmark candidate. Warmup iterations will not be considered in the benchmark result."
-              },
-              "iterations": {
-                "type": "integer",
-                "minimum": 1,
-                "description": "Defines the number of times to run the operation."
-              },
-              "warmup-time-period": {
-                "type": "integer",
-                "minimum": 0,
-                "description": "Defines the time period in seconds to run the operation in order to warmup the benchmark candidate. The warmup time period will not be considered in the benchmark result."
-              },
-              "time-period": {
-                "type": "integer",
-                "minimum": 1,
-                "description": "Defines the time period in seconds to run the operation. Note that the parameter source may be exhausted before the specified time period has elapsed."
-              },
-              "target-throughput": {
-                "type": "number",
-                "minimum": 0,
-                "description": "Defines the number of operations per second that Rally should attempt to run."
-              },
-              "target-interval": {
-                "type": "number",
-                "minimum": 0,
-                "description": "Defines the number of seconds to wait between operations (inverse of target-throughput). Only one of 'target-throughput' or 'target-interval' may be defined."
-              }
-            }
-          }
+          "$ref": "#/definitions/schedule"
         }
       },
       "required": [
@@ -450,6 +454,9 @@
     },
     "challenge": {
       "$ref": "#/definitions/challenge"
+    },
+    "schedule": {
+      "$ref": "#/definitions/schedule"
     }
   }
 }
diff --git a/esrally/track/loader.py b/esrally/track/loader.py
index d7bed3c1..cfa25949 100644
--- a/esrally/track/loader.py
+++ b/esrally/track/loader.py
@@ -38,14 +38,25 @@ def tracks(cfg):
 
 
 def list_tracks(cfg):
+    available_tracks = tracks(cfg)
+    only_auto_generated_challenges = all(t.default_challenge.auto_generated for t in available_tracks)
+
+    data = []
+    for t in available_tracks:
+        line = [t.name, t.description, t.number_of_documents, convert.bytes_to_human_string(t.compressed_size_in_bytes),
+                convert.bytes_to_human_string(t.uncompressed_size_in_bytes)]
+        if not only_auto_generated_challenges:
+            line.append(t.default_challenge)
+            line.append(",".join(map(str, t.challenges)))
+        data.append(line)
+
+    headers = ["Name", "Description", "Documents", "Compressed Size", "Uncompressed Size"]
+    if not only_auto_generated_challenges:
+        headers.append("Default Challenge")
+        headers.append("All Challenges")
+
     console.println("Available tracks:\n")
-    console.println(tabulate.tabulate(
-        tabular_data=[
-            [t.name, t.description, t.number_of_documents, convert.bytes_to_human_string(t.compressed_size_in_bytes),
-             convert.bytes_to_human_string(t.uncompressed_size_in_bytes), t.default_challenge,
-             ",".join(map(str, t.challenges))] for t in tracks(cfg)
-        ],
-        headers=["Name", "Description", "Documents", "Compressed Size", "Uncompressed Size", "Default Challenge", "All Challenges"]))
+    console.println(tabulate.tabulate(tabular_data=data, headers=headers))
 
 
 def load_track(cfg):
@@ -819,7 +830,7 @@ class TrackSpecificationReader:
         challenges = []
         known_challenge_names = set()
         default_challenge = None
-        challenge_specs = self._get_challenge_specs(track_spec)
+        challenge_specs, auto_generated = self._get_challenge_specs(track_spec)
         number_of_challenges = len(challenge_specs)
         for challenge_spec in challenge_specs:
             name = self._r(challenge_spec, "name", error_ctx="challenges")
@@ -862,6 +873,7 @@ class TrackSpecificationReader:
                                         user_info=user_info,
                                         cluster_settings=cluster_settings,
                                         default=default,
+                                        auto_generated=auto_generated,
                                         schedule=schedule)
             if default:
                 default_challenge = challenge
@@ -874,17 +886,27 @@ class TrackSpecificationReader:
         return challenges
 
     def _get_challenge_specs(self, track_spec):
+        schedule = self._r(track_spec, "schedule", mandatory=False)
         challenge = self._r(track_spec, "challenge", mandatory=False)
         challenges = self._r(track_spec, "challenges", mandatory=False)
 
-        if challenge is not None and challenges is not None:
-            self._error("'challenge' and 'challenges' are defined but only one of them is allowed.")
+        count_defined = len(list(filter(lambda e: e is not None, [schedule, challenge, challenges])))
+
+        if count_defined == 0:
+            self._error("You must define 'challenge', 'challenges' or 'schedule' but none is specified.")
+        elif count_defined > 1:
+            self._error("Multiple out of 'challenge', 'challenges' or 'schedule' are defined but only one of them is allowed.")
         elif challenge is not None:
-            return [challenge]
+            return [challenge], False
         elif challenges is not None:
-            return challenges
+            return challenges, False
+        elif schedule is not None:
+            return [{
+                "name": "default",
+                "schedule": schedule
+            }], True
         else:
-            self._error("You must define either 'challenge' or 'challenges' but none is specified.")
+            raise AssertionError("Unexpected: schedule=[{}], challenge=[{}], challenges=[{}]".format(schedule, challenge, challenges))
 
     def parse_parallel(self, ops_spec, ops, challenge_name):
         # use same default values as #parseTask() in case the 'parallel' element did not specify anything
diff --git a/esrally/track/track.py b/esrally/track/track.py
index 39b1528f..dcdbe72b 100644
--- a/esrally/track/track.py
+++ b/esrally/track/track.py
@@ -348,6 +348,7 @@ class Challenge:
                  user_info=None,
                  cluster_settings=None,
                  default=False,
+                 auto_generated=False,
                  meta_data=None,
                  schedule=None):
         self.name = name
@@ -356,6 +357,7 @@ class Challenge:
         self.user_info = user_info
         self.cluster_settings = cluster_settings if cluster_settings else {}
         self.default = default
+        self.auto_generated = auto_generated
         self.schedule = schedule if schedule else []
 
     def prepend_tasks(self, tasks):
@@ -375,12 +377,12 @@ class Challenge:
 
     def __hash__(self):
         return hash(self.name) ^ hash(self.description) ^ hash(self.cluster_settings) ^ hash(self.default) ^ \
-               hash(self.meta_data) ^ hash(self.schedule)
+               hash(self.auto_generated) ^ hash(self.meta_data) ^ hash(self.schedule)
 
     def __eq__(self, othr):
         return (isinstance(othr, type(self)) and
-                (self.name, self.description, self.cluster_settings, self.default, self.meta_data, self.schedule) ==
-                (othr.name, othr.description, othr.cluster_settings, othr.default, othr.meta_data, othr.schedule))
+                (self.name, self.description, self.cluster_settings, self.default, self.auto_generated, self.meta_data, self.schedule) ==
+                (othr.name, othr.description, othr.cluster_settings, othr.default, othr.auto_generated, othr.meta_data, othr.schedule))
 
 
 @unique


Test patch:
diff --git a/tests/track/loader_test.py b/tests/track/loader_test.py
index 179bdbdd..e487640a 100644
--- a/tests/track/loader_test.py
+++ b/tests/track/loader_test.py
@@ -1074,7 +1074,7 @@ class TrackSpecificationReaderTests(TestCase):
         }))
         with self.assertRaises(loader.TrackSyntaxError) as ctx:
             reader("unittest", track_specification, "/mappings")
-        self.assertEqual("Track 'unittest' is invalid. You must define either 'challenge' or 'challenges' but none is specified.",
+        self.assertEqual("Track 'unittest' is invalid. You must define 'challenge', 'challenges' or 'schedule' but none is specified.",
                          ctx.exception.args[0])
 
     def test_parse_challenge_and_challenges_are_defined(self):
@@ -1109,8 +1109,8 @@ class TrackSpecificationReaderTests(TestCase):
         }))
         with self.assertRaises(loader.TrackSyntaxError) as ctx:
             reader("unittest", track_specification, "/mappings")
-        self.assertEqual("Track 'unittest' is invalid. 'challenge' and 'challenges' are defined but only one of them is allowed.",
-                         ctx.exception.args[0])
+        self.assertEqual("Track 'unittest' is invalid. Multiple out of 'challenge', 'challenges' or 'schedule' are defined but only "
+                         "one of them is allowed.", ctx.exception.args[0])
 
     def test_parse_with_mixed_warmup_time_period_and_iterations(self):
         track_specification = {
@@ -1629,6 +1629,28 @@ class TrackSpecificationReaderTests(TestCase):
         self.assertEqual("challenge", resulting_track.challenges[0].name)
         self.assertTrue(resulting_track.challenges[0].default)
 
+    def test_auto_generates_challenge_from_schedule(self):
+        track_specification = {
+            "description": "description for unit test",
+            "indices": [{"name": "test-index"}],
+            "operations": [
+                {
+                    "name": "index-append",
+                    "operation-type": "bulk"
+                }
+            ],
+            "schedule": [
+                {
+                    "operation": "index-append"
+                }
+            ]
+        }
+        reader = loader.TrackSpecificationReader()
+        resulting_track = reader("unittest", track_specification, "/mappings")
+        self.assertEqual(1, len(resulting_track.challenges))
+        self.assertTrue(resulting_track.challenges[0].auto_generated)
+        self.assertTrue(resulting_track.challenges[0].default)
+
     def test_inline_operations(self):
         track_specification = {
             "description": "description for unit test",