Repository: EleutherAI/lm-evaluation-harness (language: python)

Issue:
Tasks of type `python_task` are not listed in `lm-eval --tasks list`
## Problem Description
When calling `TaskManager._initialize_tasks` I noticed that tasks which are managed by their own python class enter the `_task_index` with `type="python_task"`: 

https://github.com/EleutherAI/lm-evaluation-harness/blob/928e8bb6f50d1e93ef5d0bcaa81f8c5fd9a6f4d8/lm_eval/tasks/__init__.py#L452-L457

As a consequence, when printing the task list `lm-eval --tasks list` these are effectively omitted from any constructed table, which only consider types `group, task` or `tag`

https://github.com/EleutherAI/lm-evaluation-harness/blob/928e8bb6f50d1e93ef5d0bcaa81f8c5fd9a6f4d8/lm_eval/tasks/__init__.py#L39-L47

## How to reproduce
Simple way to notice this, take a task that is managed by a Python class, such as `fda` managed by `task.FDA`, I can run the following:
```bash
lm_eval --model hf  \
    --model_args pretrained=EleutherAI/pythia-160m,dtype="float" \
    --tasks fda \
    --device mps \
    --limit 10
```

Allthough, when I try to see `fda` in the list of tasks, e.g. `lm-eval --tasks list | grep fda` this doesn't appear in the table.

I would be happy to work on this but I'd like to hear a maintainer's opinion regarding whether these tasks should appear in tables and be treated the same way `task`-type tasks are treated, including their inclusion in the tag handling.


Discussion / hints:
Hi! This check

```
 self._all_subtasks = sorted( 
     [x for x in self._all_tasks if self._task_index[x]["type"] == "task"] 
 ) 
```
should indeed be
```
 self._all_subtasks = sorted( 
     [x for x in self._all_tasks if self._task_index[x]["type"] in ["task", "python_task"]] 
 ) 
```
.

If you have the bandwidth to PR and test a fix, we'd be grateful, but will fix this ourselves if not!

Fix patch:
diff --git a/lm_eval/tasks/__init__.py b/lm_eval/tasks/__init__.py
index 7eeec8bbc6..6669a35b9b 100644
--- a/lm_eval/tasks/__init__.py
+++ b/lm_eval/tasks/__init__.py
@@ -40,7 +40,11 @@ def __init__(
             [x for x in self._all_tasks if self._task_index[x]["type"] == "group"]
         )
         self._all_subtasks = sorted(
-            [x for x in self._all_tasks if self._task_index[x]["type"] == "task"]
+            [
+                x
+                for x in self._all_tasks
+                if self._task_index[x]["type"] in ["task", "python_task"]
+            ]
         )
         self._all_tags = sorted(
             [x for x in self._all_tasks if self._task_index[x]["type"] == "tag"]
@@ -436,6 +440,43 @@ def _get_task_and_group(self, task_dir: str):
         :return
             Dictionary of task names as key and task metadata
         """
+
+        def _populate_tags_and_groups(config, task, tasks_and_groups, print_info):
+            # TODO: remove group in next release
+            for attr in ["tag", "group"]:
+                if attr in config:
+                    if attr == "group" and print_info:
+                        self.logger.info(
+                            "`group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. "
+                            "The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. "
+                            "`group`s which aggregate across subtasks must be only defined in a separate group config file, "
+                            "which will be the official way to create groups that support cross-task aggregation as in `mmlu`. "
+                            "Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs "
+                            "for more information."
+                        )
+                        print_info = False
+                        # attr = "tag"
+
+                    attr_list = config[attr]
+                    if isinstance(attr_list, str):
+                        attr_list = [attr_list]
+
+                    for tag in attr_list:
+                        if tag not in tasks_and_groups:
+                            tasks_and_groups[tag] = {
+                                "type": "tag",
+                                "task": [task],
+                                "yaml_path": -1,
+                            }
+                        elif tasks_and_groups[tag]["type"] != "tag":
+                            self.logger.info(
+                                f"The tag {tag} is already registered as a group, this tag will not be registered. "
+                                "This may affect tasks you want to call."
+                            )
+                            break
+                        else:
+                            tasks_and_groups[tag]["task"].append(task)
+
         # TODO: remove group in next release
         print_info = True
         ignore_dirs = [
@@ -451,10 +492,14 @@ def _get_task_and_group(self, task_dir: str):
                     config = utils.load_yaml_config(yaml_path, mode="simple")
                     if self._config_is_python_task(config):
                         # This is a python class config
-                        tasks_and_groups[config["task"]] = {
+                        task = config["task"]
+                        tasks_and_groups[task] = {
                             "type": "python_task",
                             "yaml_path": yaml_path,
                         }
+                        _populate_tags_and_groups(
+                            config, task, tasks_and_groups, print_info
+                        )
                     elif self._config_is_group(config):
                         # This is a group config
                         tasks_and_groups[config["group"]] = {
@@ -483,41 +528,9 @@ def _get_task_and_group(self, task_dir: str):
                             "type": "task",
                             "yaml_path": yaml_path,
                         }
-
-                        # TODO: remove group in next release
-                        for attr in ["tag", "group"]:
-                            if attr in config:
-                                if attr == "group" and print_info:
-                                    self.logger.info(
-                                        "`group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. "
-                                        "The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. "
-                                        "`group`s which aggregate across subtasks must be only defined in a separate group config file, "
-                                        "which will be the official way to create groups that support cross-task aggregation as in `mmlu`. "
-                                        "Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs "
-                                        "for more information."
-                                    )
-                                    print_info = False
-                                    # attr = "tag"
-
-                                attr_list = config[attr]
-                                if isinstance(attr_list, str):
-                                    attr_list = [attr_list]
-
-                                for tag in attr_list:
-                                    if tag not in tasks_and_groups:
-                                        tasks_and_groups[tag] = {
-                                            "type": "tag",
-                                            "task": [task],
-                                            "yaml_path": -1,
-                                        }
-                                    elif tasks_and_groups[tag]["type"] != "tag":
-                                        self.logger.info(
-                                            f"The tag {tag} is already registered as a group, this tag will not be registered. "
-                                            "This may affect tasks you want to call."
-                                        )
-                                        break
-                                    else:
-                                        tasks_and_groups[tag]["task"].append(task)
+                        _populate_tags_and_groups(
+                            config, task, tasks_and_groups, print_info
+                        )
                     else:
                         self.logger.debug(f"File {f} in {root} could not be loaded")
 


Test patch:
diff --git a/tests/test_task_manager.py b/tests/test_task_manager.py
new file mode 100644
index 0000000000..1b10d3291a
--- /dev/null
+++ b/tests/test_task_manager.py
@@ -0,0 +1,73 @@
+import tempfile
+from pathlib import Path
+
+import pytest
+
+from lm_eval.tasks import TaskManager
+
+
+@pytest.fixture(scope="module")
+def custom_task_name():
+    return "zzz_my_python_task"
+
+
+@pytest.fixture(scope="module")
+def custom_task_tag():
+    return "zzz-tag"
+
+
+@pytest.fixture(scope="module")
+def task_yaml(pytestconfig, custom_task_name, custom_task_tag):
+    yield f"""include: {pytestconfig.rootpath}/lm_eval/tasks/hellaswag/hellaswag.yaml
+task: {custom_task_name}
+class: !function {custom_task_name}.MockPythonTask
+tag:
+  - {custom_task_tag}
+"""
+
+
+@pytest.fixture(scope="module")
+def task_code():
+    return """
+from lm_eval.tasks import ConfigurableTask
+
+class MockPythonTask(ConfigurableTask):
+
+    def __init__(
+        self,
+        data_dir=None,
+        cache_dir=None,
+        download_mode=None,
+        config=None,
+    ) -> None:
+        config.pop("class")
+        super().__init__(data_dir, cache_dir, download_mode, config)
+"""
+
+
+@pytest.fixture(scope="module")
+def custom_task_files_dir(task_yaml, task_code, custom_task_name):
+    with tempfile.TemporaryDirectory() as temp_dir:
+        yaml_path = Path(temp_dir) / f"{custom_task_name}.yaml"
+        with open(yaml_path, "w") as f:
+            f.write(task_yaml)
+        pysource_path = Path(temp_dir) / f"{custom_task_name}.py"
+        with open(pysource_path, "w") as f:
+            f.write(task_code)
+        yield temp_dir
+
+
+def test_python_task_inclusion(
+    custom_task_files_dir: Path, custom_task_name: str, custom_task_tag: str
+):
+    task_manager = TaskManager(
+        verbosity="INFO", include_path=str(custom_task_files_dir)
+    )
+    # check if python tasks enters the global task_index
+    assert custom_task_name in task_manager.task_index
+    # check if subtask is present
+    assert custom_task_name in task_manager.all_subtasks
+    # check if tag is present
+    assert custom_task_tag in task_manager.all_tags
+    # check if it can be loaded by tag (custom_task_tag)
+    assert custom_task_name in task_manager.load_task_or_group(custom_task_tag)