Repository: elastic/rally (Python)

Issue:
Do not forcefully include admin tasks when using --include-tasks and --exclude tasks
#1116 fixed a broken behavior where specifying `--exclude-tasks` also resulted excluding [administrative tasks](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/esrally/track/track.py#L528-L561).

While there are historical reasons behind this (primarily, implicitly creating indices for indices defined in the [index section](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/esrally/track/track.py#L528-L561)) this is no problem anymore, we need to specify explicit operations to create indices [already from version 0.9.0](https://esrally.readthedocs.io/en/stable/migrate.html#migrating-to-rally-0-9-0).

We should allow full user control over which operations should be executed when using `--include-tasks` or `--exclude-tasks`.

## Implementation hints

1. Forcefully including admin tasks is done [here](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/esrally/track/loader.py#L713-L714)
2. Administrative actions are specified [here](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/esrally/track/track.py#L528-L561)
3. Unit tests for task filtering can be found [here](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/tests/track/loader_test.py#L1293-L1488)
4. There is also an IT test [here](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/it/info_test.py#L32-L34). IT tests are "expensive" (i.e. take time to stand up/tear things down) so adding additional IT tests should be considered carefully and prefer, if possible, unit tests.
5. Docs for the two filtering cli arguments can be found [here](https://github.com/elastic/rally/blame/master/docs/command_line_reference.rst#L324-L352) and should be updated accordingly. Changes in behavior should also get documented [here](https://github.com/elastic/rally/blob/master/docs/migrate.rst).
6. Manual test is quite easy, without the need to actually run a benchmark (even without `--test-mode`) by just leveraging the [info subcommand](https://esrally.readthedocs.io/en/stable/command_line_reference.html#info) as shown in [this IT test](https://github.com/elastic/rally/blob/72fbb94a803ea476048290d31d0d5d742e6b7451/it/info_test.py#L32-L34).

Relates #1116 / https://github.com/elastic/rally/commit/f06991747ae48064d1433fd72d3b6ee3f4ceeb08


Discussion:
Example of problematic behavior:

```
$ esrally info --track=geonames --challenge=append-no-conflicts --include-tasks="delete-index,create-index,check-cluster-health"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

Showing details for track [geonames]:

* Description: POIs from Geonames
* Documents: 11,396,503
* Compressed Size: 252.9 MB
* Uncompressed Size: 3.3 GB

================================================
Challenge [append-no-conflicts] (run by default)
================================================

Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.

Schedule:
----------

1. delete-index
2. create-index
3. check-cluster-health
4. refresh-after-index
5. force-merge
6. refresh-after-force-merge

-------------------------------
[INFO] SUCCESS (took 1 seconds)
-------------------------------
```

Expected output:

```
1. delete-index
2. create-index
3. check-cluster-health
```
FYI @hub-cap 



Patch:
diff --git a/docs/command_line_reference.rst b/docs/command_line_reference.rst
index 9deb24026..517dda21b 100644
--- a/docs/command_line_reference.rst
+++ b/docs/command_line_reference.rst
@@ -326,7 +326,7 @@ A comma-separated list if IP:transport port pairs used to specify the seed hosts
 
 Each challenge consists of one or more tasks but sometimes you are only interested to run a subset of all tasks. For example, you might have prepared an index already and want only to repeatedly run search benchmarks. Or you want to run only the indexing task but nothing else.
 
-You can use ``--include-tasks`` to specify a comma-separated list of tasks that you want to run. Each item in the list defines either the name of a task or the operation type of a task. Only the tasks that match will be executed. Currently there is also no command that lists the tasks of a challenge so you need to look at the track source.
+You can use ``--include-tasks`` to specify a comma-separated list of tasks that you want to run. Each item in the list defines either the name of a task or the operation type of a task. Only the tasks that match will be executed. You can use the ``info`` subcommand to list the tasks of a challenge, or look at the track source.
 
 .. note::
 
diff --git a/docs/migrate.rst b/docs/migrate.rst
index 9235d8e1d..ad6168820 100644
--- a/docs/migrate.rst
+++ b/docs/migrate.rst
@@ -58,6 +58,11 @@ bulk-size metrics property is dropped
 
 Metrics records for bulk request don't contain the ``bulk-size`` property anymore. Please use the ``weight`` property instead and consider the ``unit`` property to interpret the value.
 
+--include-tasks and --exclude-tasks affect all operations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Prior to 2.0.3, administrative tasks (see :ref:`operations documentation<track_operations>`) were exempt from filtering and would run regardless of filtering. ``--include-tasks`` and ``--exclude-tasks`` flags now can affect all operations in a track. If you make use of include filters, it is advised to check that all desired operations are listed.
+
 Migrating to Rally 2.0.1
 ------------------------
 
diff --git a/esrally/track/loader.py b/esrally/track/loader.py
index 167d7d88e..8efbbc3f0 100644
--- a/esrally/track/loader.py
+++ b/esrally/track/loader.py
@@ -698,10 +698,7 @@ def filter_tasks(t, filters, exclude=False):
 
     logger = logging.getLogger(__name__)
 
-    def filter_out_match(task, user_defined_filters, force_include_filters, exclude):
-        for f in force_include_filters:
-            if task.matches(f):
-                return False
+    def filter_out_match(task, user_defined_filters, exclude):
 
         for f in user_defined_filters:
             if task.matches(f):
@@ -710,19 +707,16 @@ def filter_out_match(task, user_defined_filters, force_include_filters, exclude)
                 return exclude
         return not exclude
 
-    # always include administrative tasks
-    force_include_filters = [track.AdminTaskFilter()]
-
     for challenge in t.challenges:
         # don't modify the schedule while iterating over it
         tasks_to_remove = []
         for task in challenge.schedule:
-            if filter_out_match(task, filters, force_include_filters, exclude):
+            if filter_out_match(task, filters, exclude):
                 tasks_to_remove.append(task)
             else:
                 leafs_to_remove = []
                 for leaf_task in task:
-                    if filter_out_match(leaf_task, filters, force_include_filters, exclude):
+                    if filter_out_match(leaf_task, filters, exclude):
                         leafs_to_remove.append(leaf_task)
                 for leaf_task in leafs_to_remove:
                     logger.info("Removing sub-task [%s] from challenge [%s] due to task filter.", leaf_task, challenge)
diff --git a/esrally/track/track.py b/esrally/track/track.py
index a80869f6d..a683b074e 100644
--- a/esrally/track/track.py
+++ b/esrally/track/track.py
@@ -714,20 +714,6 @@ def __call__(cls, *args, **kwargs):
         return cls._instances[cls]
 
 
-class AdminTaskFilter(metaclass=Singleton):
-    """
-    Matches all tasks that execute administrative actions.
-    """
-    def matches(self, task):
-        try:
-            return OperationType.from_hyphenated_string(task.operation.type).admin_op
-        except KeyError:
-            return False
-
-    def __str__(self, *args, **kwargs):
-        return "filter for admin tasks"
-
-
 # Schedule elements
 class Parallel:
     def __init__(self, tasks, clients=None):


Test patch:
diff --git a/tests/track/loader_test.py b/tests/track/loader_test.py
index 018fc101f..24723c4c6 100644
--- a/tests/track/loader_test.py
+++ b/tests/track/loader_test.py
@@ -1394,11 +1394,10 @@ def test_filters_tasks(self):
                                                              ])
 
         schedule = filtered.challenges[0].schedule
-        self.assertEqual(4, len(schedule))
-        self.assertEqual("create-index", schedule[0].name)
-        self.assertEqual(["index-3", "match-all-parallel"], [t.name for t in schedule[1].tasks])
-        self.assertEqual("match-all-serial", schedule[2].name)
-        self.assertEqual("cluster-stats", schedule[3].name)
+        self.assertEqual(3, len(schedule))
+        self.assertEqual(["index-3", "match-all-parallel"], [t.name for t in schedule[0].tasks])
+        self.assertEqual("match-all-serial", schedule[1].name)
+        self.assertEqual("cluster-stats", schedule[2].name)
 
     def test_filters_exclude_tasks(self):
         track_specification = {
@@ -1478,15 +1477,146 @@ def test_filters_exclude_tasks(self):
         full_track = reader("unittest", track_specification, "/mappings")
         self.assertEqual(5, len(full_track.challenges[0].schedule))
 
-        filtered = loader.filter_tasks(full_track, [track.TaskNameFilter("index-3"), track.TaskOpTypeFilter("search")], exclude=True)
+        filters = [track.TaskNameFilter("index-3"), track.TaskOpTypeFilter("search"), track.TaskNameFilter("create-index")]
+        filtered = loader.filter_tasks(full_track, filters, exclude=True)
 
         schedule = filtered.challenges[0].schedule
-        self.assertEqual(4, len(schedule))
-        self.assertEqual("create-index", schedule[0].name)
-        self.assertEqual(["index-1", "index-2"], [t.name for t in schedule[1].tasks])
-        self.assertEqual("node-stats", schedule[2].name)
-        self.assertEqual("cluster-stats", schedule[3].name)
+        self.assertEqual(3, len(schedule))
+        self.assertEqual(["index-1", "index-2"], [t.name for t in schedule[0].tasks])
+        self.assertEqual("node-stats", schedule[1].name)
+        self.assertEqual("cluster-stats", schedule[2].name)
 
+    def test_unmatched_exclude_runs_everything(self):
+        track_specification = {
+            "description": "description for unit test",
+            "indices": [{"name": "test-index", "auto-managed": False}],
+            "operations": [
+                {
+                    "name": "create-index",
+                    "operation-type": "create-index"
+                },
+                {
+                    "name": "bulk-index",
+                    "operation-type": "bulk"
+                },
+                {
+                    "name": "node-stats",
+                    "operation-type": "node-stats"
+                },
+                {
+                    "name": "cluster-stats",
+                    "operation-type": "custom-operation-type"
+                },
+                {
+                    "name": "match-all",
+                    "operation-type": "search",
+                    "body": {
+                        "query": {
+                            "match_all": {}
+                        }
+                    }
+                },
+            ],
+            "challenges": [
+                {
+                    "name": "default-challenge",
+                    "schedule": [
+                        {
+                            "operation": "create-index"
+                        },
+                        {
+                            "operation": "bulk-index"
+                        },
+                        {
+                            "operation": "node-stats"
+                        },
+                        {
+                            "name": "match-all-serial",
+                            "operation": "match-all"
+                        },
+                        {
+                            "operation": "cluster-stats"
+                        }
+                    ]
+                }
+            ]
+        }
+
+        reader = loader.TrackSpecificationReader()
+        full_track = reader("unittest", track_specification, "/mappings")
+        self.assertEqual(5, len(full_track.challenges[0].schedule))
+
+        expected_schedule = full_track.challenges[0].schedule.copy()
+        filtered = loader.filter_tasks(full_track, [track.TaskNameFilter("nothing")], exclude=True)
+
+        schedule = filtered.challenges[0].schedule
+        self.assertEqual(expected_schedule, schedule)
+
+    def test_unmatched_include_runs_nothing(self):
+        track_specification = {
+            "description": "description for unit test",
+            "indices": [{"name": "test-index", "auto-managed": False}],
+            "operations": [
+                {
+                    "name": "create-index",
+                    "operation-type": "create-index"
+                },
+                {
+                    "name": "bulk-index",
+                    "operation-type": "bulk"
+                },
+                {
+                    "name": "node-stats",
+                    "operation-type": "node-stats"
+                },
+                {
+                    "name": "cluster-stats",
+                    "operation-type": "custom-operation-type"
+                },
+                {
+                    "name": "match-all",
+                    "operation-type": "search",
+                    "body": {
+                        "query": {
+                            "match_all": {}
+                        }
+                    }
+                },
+            ],
+            "challenges": [
+                {
+                    "name": "default-challenge",
+                    "schedule": [
+                        {
+                            "operation": "create-index"
+                        },
+                        {
+                            "operation": "bulk-index"
+                        },
+                        {
+                            "operation": "node-stats"
+                        },
+                        {
+                            "name": "match-all-serial",
+                            "operation": "match-all"
+                        },
+                        {
+                            "operation": "cluster-stats"
+                        }
+                    ]
+                }
+            ]
+        }
+
+        reader = loader.TrackSpecificationReader()
+        full_track = reader("unittest", track_specification, "/mappings")
+        self.assertEqual(5, len(full_track.challenges[0].schedule))
+
+        expected_schedule = []
+        filtered = loader.filter_tasks(full_track, [track.TaskNameFilter("nothing")], exclude=False)
+
+        schedule = filtered.challenges[0].schedule
+        self.assertEqual(expected_schedule, schedule)
 
 # pylint: disable=too-many-public-methods
 class TrackSpecificationReaderTests(TestCase):
diff --git a/tests/track/track_test.py b/tests/track/track_test.py
index 9d36c1238..73688fcfb 100644
--- a/tests/track/track_test.py
+++ b/tests/track/track_test.py
@@ -242,11 +242,6 @@ def search_task(self):
                           track.Operation("search-op",
                                           operation_type=track.OperationType.Search.to_hyphenated_string()))
 
-    def test_admin_task_filter(self):
-        f = track.AdminTaskFilter()
-        self.assertTrue(f.matches(self.create_index_task()))
-        self.assertFalse(f.matches(self.search_task()))
-
     def test_task_name_filter(self):
         f = track.TaskNameFilter("create-index-task")
         self.assertTrue(f.matches(self.create_index_task()))