Repository: elastic/rally

Issue:
Alias `--user-tags` to `--user-tag`
Rally currently supports custom user tags via the [--user-tag](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=user-tag#user-tag) cli option.
This doesn't convey the concept that several tags can be user.

In order to not break BWC, I suggest we add an alias `--user-tags` acting as
a synonym to `--user-tag`.


Discussion:
dliappis: > Thank you for raising this!
> 
> I left some comments. One thing that is missing IMHO is that we should emit a "Deprecation warning" when `--user-tag` is use in the CLI. That way people who don't happen to check the migration guide or read docs will be reminded that they should be switching away.

I discussed with @pquentin and instead of doing what I proposed here, we could just keep the functionality for both i.e. no need for a depreciation warning and allow both for the foreseeable future. Given the large usage of Rally this would probably be the least disruptive change for users. 
dliappis: @elasticmachine test this please
pquentin: @elasticmachine test this please

Patch:
diff --git a/docs/adding_tracks.rst b/docs/adding_tracks.rst
index f2e8df9b..8bd6b455 100644
--- a/docs/adding_tracks.rst
+++ b/docs/adding_tracks.rst
@@ -15,6 +15,10 @@ If you already have a cluster with data in it you can use the ``create-track`` s
 
     esrally create-track --track=acme --target-hosts=127.0.0.1:9200 --indices="products,companies" --output-path=~/tracks
 
+Alternatively, to create a Rally track with data from the data streams ``logs-*`` and ``metrics-*``, issue the following command::
+
+    esrally create-track --track=acme --target-hosts=127.0.0.1:9200 --data-streams="metrics-*,logs-*" --output-path=~/tracks
+
 If you want to connect to a cluster with TLS and basic authentication enabled, for example via Elastic Cloud, also specify :ref:`--client-options <clr_client_options>` and change ``basic_auth_user`` and ``basic_auth_password`` accordingly::
 
     esrally create-track --track=acme --target-hosts=abcdef123.us-central-1.gcp.cloud.es.io:9243 --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'secret-password'" --indices="products,companies" --output-path=~/tracks
diff --git a/docs/command_line_reference.rst b/docs/command_line_reference.rst
index f36ef9ec..5572b540 100644
--- a/docs/command_line_reference.rst
+++ b/docs/command_line_reference.rst
@@ -258,7 +258,7 @@ Example JSON file::
 
 All track parameters are recorded for each metrics record in the metrics store. Also, when you run ``esrally list races``, it will show all track parameters::
 
-    Race Timestamp    Track    Track Parameters          Challenge            Car       User Tag
+    Race Timestamp    Track    Track Parameters          Challenge            Car       User Tags
     ----------------  -------  ------------------------- -------------------  --------  ---------
     20160518T122341Z  pmc      bulk_size=8000            append-no-conflicts  defaults
     20160518T112341Z  pmc      bulk_size=2000,clients=16 append-no-conflicts  defaults
@@ -902,38 +902,39 @@ Rally usually installs and launches an Elasticsearch cluster internally and wipe
 .. note::
    This option does only affect clusters that are provisioned by Rally. More specifically, if you use the pipeline ``benchmark-only``, this option is ineffective as Rally does not provision a cluster in this case.
 
-``user-tag``
-~~~~~~~~~~~~
+``user-tags``
+~~~~~~~~~~~~~
 
 This is only relevant when you want to run :doc:`tournaments </tournament>`. You can use this flag to attach an arbitrary text to the meta-data of each metric record and also the corresponding race. This will help you to recognize a race when you run ``esrally list races`` as you don't need to remember the concrete timestamp on which a race has been run but can instead use your own descriptive names.
 
-The required format is ``key`` ":" ``value``. You can choose ``key`` and  ``value`` freely.
+The required format is ``key`` ":" ``value``. You can choose ``key`` and  ``value`` freely. You can also specify multiple tags. They need to be separated by a comma.
 
 **Example**
 
  ::
 
-   esrally race --track=pmc --user-tag="intention:github-issue-1234-baseline,gc:cms"
-
-You can also specify multiple tags. They need to be separated by a comma.
+   esrally race --track=pmc --user-tags="intention:github-issue-1234-baseline,gc:cms"
 
 **Example**
 
  ::
 
-   esrally race --track=pmc --user-tag="disk:SSD,data_node_count:4"
+   esrally race --track=pmc --user-tags="disk:SSD,data_node_count:4"
 
 
 
 When you run ``esrally list races``, this will show up again::
 
-    Race Timestamp    Track    Track Parameters   Challenge            Car       User Tag
+    Race Timestamp    Track    Track Parameters   Challenge            Car       User Tags
     ----------------  -------  ------------------ -------------------  --------  ------------------------------------
     20160518T122341Z  pmc                         append-no-conflicts  defaults  intention:github-issue-1234-baseline
     20160518T112341Z  pmc                         append-no-conflicts  defaults  disk:SSD,data_node_count:4
 
 This will help you recognize a specific race when running ``esrally compare``.
 
+.. note::
+   This option used to be named `--user-tag` without an s, which was confusing as multiple tags are supported. While users are now encouraged to use `--user-tags` for clarity, Rally will continue to honor `--user-tag` in the future to avoid breaking backwards-compatibility.
+
 ``indices``
 ~~~~~~~~~~~
 
@@ -954,6 +955,29 @@ Use index patterns::
     esrally create-track --track=my-logs --indices="logs-*" --target-hosts=127.0.0.1:9200 --output-path=~/tracks
 
 
+.. note::
+   If the cluster requires authentication specify credentials via ``--client-options`` as described in the :ref:`command line reference <clr_client_options>`.
+
+``data-streams``
+~~~~~~~~~~~~~~~~
+
+A comma-separated list of data stream patterns to target when generating a track with the ``create-track`` subcommand.
+
+**Examples**
+
+Target a single data stream::
+
+    esrally create-track --track=acme --data-streams="metrics-system.cpu-default" --target-hosts=127.0.0.1:9200 --output-path=~/tracks
+
+Target multiple data streams::
+
+    esrally create-track --track=acme --data-streams="logs-aws.vpcflow-default,metrics-system.cpu-default" --target-hosts=127.0.0.1:9200 --output-path=~/tracks
+
+Use index patterns::
+
+    esrally create-track --track=my-logs --data-streams="logs-*,metrics-*" --target-hosts=127.0.0.1:9200 --output-path=~/tracks
+
+
 .. note::
    If the cluster requires authentication specify credentials via ``--client-options`` as described in the :ref:`command line reference <clr_client_options>`.
 
diff --git a/docs/metrics.rst b/docs/metrics.rst
index 9f58265a..af0ac6c5 100644
--- a/docs/metrics.rst
+++ b/docs/metrics.rst
@@ -108,7 +108,7 @@ Rally captures also some meta information for each metric record:
 * Node name: If Rally provisions the cluster, it will choose a unique name for each node.
 * Source revision: We always record the git hash of the version of Elasticsearch that is benchmarked. This is even done if you benchmark an official binary release.
 * Distribution version: We always record the distribution version of Elasticsearch that is benchmarked. This is even done if you benchmark a source release.
-* Custom tag: You can define one custom tag with the command line flag ``--user-tag``. The tag is prefixed by ``tag_`` in order to avoid accidental clashes with Rally internal tags.
+* Custom tags: You can define custom tags with the command line flag ``--user-tags``. The tags are prefixed by ``tag_`` in order to avoid accidental clashes with Rally internal tags.
 * Operation-specific: The optional substructure ``operation`` contains additional information depending on the type of operation. For bulk requests, this may be the number of documents or for searches the number of hits.
 
 Note that depending on the "level" of a metric record, certain meta information might be missing. It makes no sense to record host level meta info for a cluster wide metric record, like a query latency (as it cannot be attributed to a single node).
diff --git a/docs/tournament.rst b/docs/tournament.rst
index d0a5c406..461a465b 100644
--- a/docs/tournament.rst
+++ b/docs/tournament.rst
@@ -5,13 +5,13 @@ Suppose, we want to analyze the impact of a performance improvement.
 
 First, we need a baseline measurement. For example::
 
-    esrally race --track=pmc --revision=latest --user-tag="intention:baseline_github_1234"
+    esrally race --track=pmc --revision=latest --user-tags="intention:baseline_github_1234"
 
-Above we run the baseline measurement based on the latest source code revision of Elasticsearch. We can use the command line parameter ``--user-tag`` to provide a key-value pair to document the intent of a race.
+Above we run the baseline measurement based on the latest source code revision of Elasticsearch. We can use the command line parameter ``--user-tags`` to provide a key-value pair to document the intent of a race.
 
 Then we implement our changes and finally we want to run another benchmark to see the performance impact of the change. In that case, we do not want Rally to change our source tree and thus specify the pseudo-revision ``current``::
 
-    esrally race --track=pmc --revision=current --user-tag="intention:reduce_alloc_1234"
+    esrally race --track=pmc --revision=current --user-tags="intention:reduce_alloc_1234"
 
 After we've run both races, we want to know about the performance impact. With Rally we can analyze differences of two given races easily. First of all, we need to find two races to compare by issuing ``esrally list races``::
 
@@ -32,7 +32,7 @@ After we've run both races, we want to know about the performance impact. With R
     0cfb3576-3025-4c17-b672-d6c9e811b93e  20160518T101957Z  pmc                          append-no-conflicts  defaults
 
 
-We can see that the user tag helps us to recognize races. We want to compare the two most recent races and have to provide the two race IDs in the next step::
+We can see that the user tags help us to recognize races. We want to compare the two most recent races and have to provide the two race IDs in the next step::
 
     $ esrally compare --baseline=0bfd4542-3821-4c79-81a2-0858636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7
 
diff --git a/esrally/metrics.py b/esrally/metrics.py
index 3e46cf33..841b794a 100644
--- a/esrally/metrics.py
+++ b/esrally/metrics.py
@@ -310,7 +310,7 @@ def extract_user_tags_from_config(cfg):
     :param cfg: The current configuration object.
     :return: A dict containing user tags. If no user tags are given, an empty dict is returned.
     """
-    user_tags = cfg.opts("race", "user.tag", mandatory=False)
+    user_tags = cfg.opts("race", "user.tags", mandatory=False)
     return extract_user_tags_from_string(user_tags)
 
 
diff --git a/esrally/rally.py b/esrally/rally.py
index f07642b7..2ff9bf27 100644
--- a/esrally/rally.py
+++ b/esrally/rally.py
@@ -651,6 +651,7 @@ def create_arg_parser():
     )
     race_parser.add_argument(
         "--user-tag",
+        "--user-tags",
         help="Define a user-specific key-value pair (separated by ':'). It is added to each metric record as meta info. "
         "Example: intention:baseline-ticket-12345",
         default="",
@@ -1047,7 +1048,7 @@ def dispatch_sub_command(arg_parser, args, cfg):
             # use the race id implicitly also as the install id.
             cfg.add(config.Scope.applicationOverride, "system", "install.id", args.race_id)
             cfg.add(config.Scope.applicationOverride, "race", "pipeline", args.pipeline)
-            cfg.add(config.Scope.applicationOverride, "race", "user.tag", args.user_tag)
+            cfg.add(config.Scope.applicationOverride, "race", "user.tags", args.user_tag)
             cfg.add(config.Scope.applicationOverride, "driver", "profiling", args.enable_driver_profiling)
             cfg.add(config.Scope.applicationOverride, "driver", "assertions", args.enable_assertions)
             cfg.add(config.Scope.applicationOverride, "driver", "on.error", args.on_error)


Test patch:
diff --git a/tests/metrics_test.py b/tests/metrics_test.py
index 9a507377..7d4e0e72 100644
--- a/tests/metrics_test.py
+++ b/tests/metrics_test.py
@@ -116,21 +116,21 @@ class TestExtractUserTags:
 
     def test_missing_comma_raises_error(self):
         cfg = config.Config()
-        cfg.add(config.Scope.application, "race", "user.tag", "invalid")
+        cfg.add(config.Scope.application, "race", "user.tags", "invalid")
         with pytest.raises(exceptions.SystemSetupError) as ctx:
             metrics.extract_user_tags_from_config(cfg)
         assert ctx.value.args[0] == "User tag keys and values have to separated by a ':'. Invalid value [invalid]"
 
     def test_missing_value_raises_error(self):
         cfg = config.Config()
-        cfg.add(config.Scope.application, "race", "user.tag", "invalid1,invalid2")
+        cfg.add(config.Scope.application, "race", "user.tags", "invalid1,invalid2")
         with pytest.raises(exceptions.SystemSetupError) as ctx:
             metrics.extract_user_tags_from_config(cfg)
         assert ctx.value.args[0] == "User tag keys and values have to separated by a ':'. Invalid value [invalid1,invalid2]"
 
     def test_extracts_proper_user_tags(self):
         cfg = config.Config()
-        cfg.add(config.Scope.application, "race", "user.tag", "os:Linux,cpu:ARM")
+        cfg.add(config.Scope.application, "race", "user.tags", "os:Linux,cpu:ARM")
         assert metrics.extract_user_tags_from_config(cfg) == {"os": "Linux", "cpu": "ARM"}
 
 
@@ -364,7 +364,7 @@ class TestEsMetrics:
     def test_put_value_with_meta_info(self):
         throughput = 5000
         # add a user-defined tag
-        self.cfg.add(config.Scope.application, "race", "user.tag", "intention:testing,disk_type:hdd")
+        self.cfg.add(config.Scope.application, "race", "user.tags", "intention:testing,disk_type:hdd")
         self.metrics_store.open(self.RACE_ID, self.RACE_TIMESTAMP, "test", "append", "defaults", create=True)
 
         # Ensure we also merge in cluster level meta info
@@ -436,7 +436,7 @@ class TestEsMetrics:
 
     def test_put_doc_with_metadata(self):
         # add a user-defined tag
-        self.cfg.add(config.Scope.application, "race", "user.tag", "intention:testing,disk_type:hdd")
+        self.cfg.add(config.Scope.application, "race", "user.tags", "intention:testing,disk_type:hdd")
         self.metrics_store.open(self.RACE_ID, self.RACE_TIMESTAMP, "test", "append", "defaults", create=True)
 
         # Ensure we also merge in cluster level meta info
@@ -1667,7 +1667,7 @@ class TestStatsCalculator:
         cfg.add(config.Scope.application, "mechanic", "car.names", ["unittest_car"])
         cfg.add(config.Scope.application, "mechanic", "car.params", {})
         cfg.add(config.Scope.application, "mechanic", "plugin.params", {})
-        cfg.add(config.Scope.application, "race", "user.tag", "")
+        cfg.add(config.Scope.application, "race", "user.tags", "")
         cfg.add(config.Scope.application, "race", "pipeline", "from-sources")
         cfg.add(config.Scope.application, "track", "params", {})
 
@@ -1835,7 +1835,7 @@ class TestStatsCalculator:
         cfg.add(config.Scope.application, "mechanic", "car.names", ["unittest_car"])
         cfg.add(config.Scope.application, "mechanic", "car.params", {})
         cfg.add(config.Scope.application, "mechanic", "plugin.params", {})
-        cfg.add(config.Scope.application, "race", "user.tag", "")
+        cfg.add(config.Scope.application, "race", "user.tags", "")
         cfg.add(config.Scope.application, "race", "pipeline", "from-sources")
         cfg.add(config.Scope.application, "track", "params", {})