Repository: EleutherAI/lm-evaluation-harness
Pull request: Faster Task and Group Loading, Allow Recursive Groups
@haileyschoelkopf this is still a work in progress, but feel free to test this out. It makes task loading faster because the loading part only happens to list of task that the users want to evaluate. The `initialize_task()` is only for indexing. 

Updates:

1. Faster task/group loading. Only the tasks that are set by the user to evaluate are properly loaded.
2. Recursively build groups. Builds groups that can have other groups as subtasks.
3. Allow subtasks parameters to be modifier for that group config only.
4. Allow group-level parameter set (set `num_fewshot` on a group level, and apply that to all of it's subtasks)
5. Revamp on how Python-based tasks are registered.
6. Add documentation with references.

Should fix #1301 
Also fixes #1103

Problem statement:
Evaluation Error on Scrolls Task - 2
I am trying to evaluate LLM on long text understanding. 
I encountered the following error. 
I tried to resolve minor bugs but couldn't resolve this one. 

First I faced this issue
`acc_norm = 1.0 if np.argmax(results / completion_len) == gold else 0.0
                                ~~~~~~~~^~~~~~~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (3,2) (3,)`

I checked for the dimensions. Apparently, results parameter had a dimension of (3,2). It was present in this format 
`
[(-16.322248458862305, False), (-14.276965141296387, False), (-10.922859191894531, False)]
`
To resolve this I added another line of code 
`results = [result[0] for result in results]`


```
But after this, I faced another error, which I couldn't figure out. 

Traceback (most recent call last):
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 152, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 496, in evaluate
    results[task_name][metric + "_stderr" + "," + key] = stderr(items)
                                                         ^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/api/metrics.py", line 206, in mean_stderr
    return sample_stddev(arr) / math.sqrt(len(arr))
           ^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/api/metrics.py", line 202, in sample_stddev
    return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
```



Python task definitions are not imported by `include_task_folder`
As the title says -- I've tried copying the definition of `squadv2` (which is defined by a `task.py` file) in another folder, registering it under another name, and importing that folder via `include_task_folder`, but the new task does not get registered.

Can not add `mmlu` task inside benchmark?
Here is my config

```yaml
group: open_llm_leaderboard
task:
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: hellaswag_id
    task_alias: hellaswag_id 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: truthfulqa_mc2
    task_alias: truthfulqa 0 shot
    dataset_path: truthful_qa
    dataset_name: multiple_choice
    output_type: multiple_choice
    process_results: !function truthfulqa_utils.process_results_mc2
    metric_list:
      - metric: acc
  - task: mmlu
    task_alias: mmlu 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: winogrande
    task_alias: winogrande 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
```

Somehow because `mmlu` is not found, `lm_eval` skip all task after `mmlu` (including `mmlu` itself). But, if I do

```bash
lm_eval --tasks mmlu
```

It is working fine.

Hints from issue discussion:
Are you using `limit 1` for the second error? Might be because it divides by N - 1 to calculate the sample standard deviation.

cc @lintangsutawika 
Yeah, division by zero looks like an error from using only 1 sample. I should patch that.
Yes, thanks. Indeed using limit 1 was the issue. The issue is gone but I have encountered another issue. 

```
Traceback (most recent call last):
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 152, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 482, in evaluate
    results[task_name][metric_key] = agg_fn(items)
                                     ^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/tasks/scrolls/task.py", line 197, in compute_metrics
    computed = self.metric.compute(
               ^^^^^^^^^^^
AttributeError: 'GovReport' object has no attribute 'metric'
```
Will take a look
Sorry, reopening this as it turns out there is still an issue in the loading the metric.
Yeah it looks like `include_task_folder` only considers yaml task definitions: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py#L137 -- what's the best way to register python task definitions stored in different folders?
Python-based tasks still need to be added in the `__init__.py` manually.

https://github.com/EleutherAI/lm-evaluation-harness/blob/5133c9c4f5f3bf221d37dec2827aac0d5829d112/lm_eval/tasks/__init__.py#L18-L27
@lintangsutawika thanks, after importing them everything works like a charm!

BTW can we include all python tasks automatically within `initialize_tasks`?
The code would look something like this:

```python
import os
import importlib
import inspect

# Function to import all task classes decorated with @register_task in a given folder
def include_tasks_from_folder(folder_path):
    task_classes = []
    for filename in os.listdir(folder_path):
        if filename.endswith('.py') and not filename.startswith('__'):
            module_name = filename[:-3]
            module = importlib.import_module(f"{folder_path}.{module_name}")

            for name, cls in inspect.getmembers(module):
                if inspect.isclass(cls) and hasattr(obj, 'register_task'):
                    TASK_REGISTRY[name] = cls
                    ALL_TASKS.add(name)
                    func2task_index[cls.__name__] = name
```
@lintangsutawika do you know what might be going on here?
`mmlu` is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of `mmlu` with the adjusted parameters. (Use `group_alias` instead of `task_alias` and set to "mmlu 5 shot")



> `mmlu` is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of `mmlu` with the adjusted parameters. (Use `group_alias` instead of `task_alias` and set to "mmlu 5 shot")
> 
> 
> 

Could you give an example? is it like this?

```yaml
  - group: mmlu
    group_alias: mmlu 5 shot
```

Or like this

```yaml
  - group: mmlu
    group_alias: mmlu 5 shot
    task:
      - mmlu_stem
      - etc.
```
> mmlu is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of mmlu with the adjusted parameters. (Use group_alias instead of task_alias and set to "mmlu 5 shot")

Could we add a more descriptive failure message in this scenario then, rather than just failing to add it to the grouping?
@fahadh4ilyas the `num_fewshot: 5` would need to be set to each mmlu task unfortunately. Might need to copy the mmlu dir in `tasks/mmlu` and change the `num_fewshot`.

@haileyschoelkopf That would be appropriate. I'll make an issue for this. But also from the usecase here, I think it's also valid to make some changes so that the above yaml config would be possible.
@lintangsutawika thanks for the explanation on mmlu, I going through similar issue but one step before. I am trying to use the lm_eval in another repo, using similar config file and trying the following [used this doc for interface ref](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage)

```
 lm_eval.tasks.include_path("/data/home/external-repo/path-to-config/")
tasks_list = [
       "arc_challenge",
    ]
    # Evaluate
    results = evaluator.simple_evaluate(
        model=args.model,
        model_args= "pretrained=EleutherAI/pythia-160m",
        tasks=tasks_list, 
        limit=args.limit,  # Limit the number of examples per task
        
    )
```
running into couple of issues:
1- it complains about `KeyError: 'Missing task arc_challenge'` somehow doesn't pick it up. if I "tasks.initialize_tasks()" first then it does not pick up `num_fewshot` and reports n-shot 0

2- it also complains about not finding `'/data/home/external-repo/path-to-config/hellaswag_utils.py'`.

I would appreciate your help on what is the correct usage/ how should I use the open-llm config in an external repo.
Hey @HamidShojanazeri 
1. The correct way is in fact to run `tasks.initialize_tasks()` first so that it registers all tasks in the eval harness. As for the `num_fewshot`, If you're using the `evaluator.simple_evaluate` I think you should be able to use the `num_fewshot` arg in it. I haven't tried myself, but it's added there when the cli is used.
2. Is `hellaswag_utils.py` part of a custom task? The utils by itself can't be included like that of a python module, but rather used by a yaml task. The `include_path` is meant to register tasks outside of the eval harness.
thanks @lintangsutawika for the reply, 
re 1: , I was interested to have a single config as shared in this issue at the top to aggregate all the tasks I want to run where each task would have a different `num_fewshot` (as above). Would it be possible to use such a config file to pass different tasks with different `num_fewshots`?

I experimented to add a new config file for each task something like adding following to `tasks/arc/arc_challenge_fewshot.yaml`
it works fine there, so I was wondering  if the single config file like above can work. 

```
include: arc_challenge.yaml
task: arc_challenge_few_shot
task_alias: arc 25 shot
num_fewshot: 25
metric_list:
  - metric: acc_norm
 ```

So more specifically we should set `num_fewshot` to some number to activate it then it reads out from config for each specific task like [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L134-L145)?

```
results = evaluator.simple_evaluate(
        model=args.model,
        model_args= "pretrained=EleutherAI/pythia-160m",
        tasks=tasks_list, 
        num_fewshot=5,
    )
```
 
 re 2: yes, it is coming from ` process_docs: !function hellaswag_utils.process_docs` in the config file
 
 ```
group: open_llm_leaderboard
task:
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
```
> thanks @lintangsutawika for the reply, re 1: , I was interested to have a single config as shared in this issue at the top to aggregate all the tasks I want to run where each task would have a different `num_fewshot` (as above). Would it be possible to use such a config file to pass different tasks with different `num_fewshots`?
> 
> I experimented to add a new config file for each task something like adding following to `tasks/arc/arc_challenge_fewshot.yaml` it works fine there, so I was wondering if the single config file like above can work.
> 
> ```
> include: arc_challenge.yaml
> task: arc_challenge_few_shot
> task_alias: arc 25 shot
> num_fewshot: 25
> metric_list:
>   - metric: acc_norm
> ```
> 
> So more specifically we should set `num_fewshot` to some number to activate it then it reads out from config for each specific task like [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L134-L145)?
> 
> ```
> results = evaluator.simple_evaluate(
>         model=args.model,
>         model_args= "pretrained=EleutherAI/pythia-160m",
>         tasks=tasks_list, 
>         num_fewshot=5,
>     )
> ```
> 
> re 2: yes, it is coming from ` process_docs: !function hellaswag_utils.process_docs` in the config file
> 
> ```
> group: open_llm_leaderboard
> task:
>  - task: arc_challenge
>    task_alias: arc 25 shot
>    num_fewshot: 25
>    metric_list:
>      - metric: acc_norm
>  - task: hellaswag
>    task_alias: hellaswag 10 shot
>    process_docs: !function hellaswag_utils.process_docs
>    num_fewshot: 10
>    metric_list:
>      - metric: acc_norm
> ```

my solution is adding myself `hellaswag_utils.py` file. I got it from [lm_eval/tasks/hellaswag/utils.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/10488d0d2cfbfd2d73416e92eb30c4a74ac3920b/lm_eval/tasks/hellaswag/utils.py)
> @fahadh4ilyas the `num_fewshot: 5` would need to be set to each mmlu task unfortunately. Might need to copy the mmlu dir in `tasks/mmlu` and change the `num_fewshot`.
> 
> @haileyschoelkopf That would be appropriate. I'll make an issue for this. But also from the usecase here, I think it's also valid to make some changes so that the above yaml config would be possible.

I tried to do this

```yaml
group: open_llm_leaderboard
task:
  - group: mmlu
    group_alias: mmlu 5 shot
    task:
      - mmlu_stem
      - mmlu_other
      - mmlu_social_sciences
      - mmlu_humanities
    num_fewshot: 5
    metric_list:
      - metric: acc
```

and I got this error. Could you please give me an example?

```python
2024-01-18:12:50:41,971 INFO     [config.py:58] PyTorch version 2.1.2 available.
/home/fahadh/anaconda3/envs/merging/lib/python3.10/site-packages/torch/cuda/__init__.py:611: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
2024-01-18:12:50:46,266 INFO     [__main__.py:156] Verbosity set to DEBUG
2024-01-18:12:50:54,640 WARNING  [__init__.py:185] Unexpected error loading config in
                                 /home/fahadh/lm-evaluation-harness/lm_eval/tasks/benchmarks/open_llm_leaderboard.yaml
                                 Config will not be added to registry
                                 Error: unhashable type: 'list'
                                 Traceback: Traceback (most recent call last):
  File "/home/fahadh/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 173, in include_task_folder
    register_configurable_group(config, yaml_path)
  File "/home/fahadh/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 69, in register_configurable_group
    if task_name in ALL_TASKS:
TypeError: unhashable type: 'list'
```

Commit message: removed unused code

Patch to review:
diff --git a/lm_eval/__main__.py b/lm_eval/__main__.py
index 37fdabc6df..f001a4ab97 100644
--- a/lm_eval/__main__.py
+++ b/lm_eval/__main__.py
@@ -10,8 +10,7 @@
 import numpy as np
 
 from lm_eval import evaluator, utils
-from lm_eval.api.registry import ALL_TASKS
-from lm_eval.tasks import include_path, initialize_tasks
+from lm_eval.tasks import TaskManager
 from lm_eval.utils import make_table
 
 
@@ -156,44 +155,46 @@ def cli_evaluate(args: Union[argparse.Namespace, None] = None) -> None:
     eval_logger.info(f"Verbosity set to {args.verbosity}")
     os.environ["TOKENIZERS_PARALLELISM"] = "false"
 
-    initialize_tasks(args.verbosity)
+    # initialize_tasks(args.verbosity)
+    task_manager = TaskManager(args.verbosity, include_path=args.include_path)
 
     if args.limit:
         eval_logger.warning(
             " --limit SHOULD ONLY BE USED FOR TESTING."
             "REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT."
         )
-    if args.include_path is not None:
-        eval_logger.info(f"Including path: {args.include_path}")
-        include_path(args.include_path)
 
     if args.tasks is None:
-        task_names = ALL_TASKS
+        eval_logger.error("Need to specify task to evaluate.")
+        sys.exit()
     elif args.tasks == "list":
         eval_logger.info(
-            "Available Tasks:\n - {}".format("\n - ".join(sorted(ALL_TASKS)))
+            "Available Tasks:\n - {}".format("\n - ".join(task_manager.all_tasks()))
         )
-        sys.exit()
     else:
         if os.path.isdir(args.tasks):
             import glob
 
-            task_names = []
+            loaded_task_list = []
             yaml_path = os.path.join(args.tasks, "*.yaml")
             for yaml_file in glob.glob(yaml_path):
                 config = utils.load_yaml_config(yaml_file)
-                task_names.append(config)
+                loaded_task_list.append(config)
         else:
-            tasks_list = args.tasks.split(",")
-            task_names = utils.pattern_match(tasks_list, ALL_TASKS)
-            for task in [task for task in tasks_list if task not in task_names]:
+            input_task_list = args.tasks.split(",")
+            loaded_task_list = utils.pattern_match(
+                input_task_list, task_manager.all_tasks()
+            )
+            for task in [
+                task for task in input_task_list if task not in loaded_task_list
+            ]:
                 if os.path.isfile(task):
                     config = utils.load_yaml_config(task)
-                    task_names.append(config)
+                    loaded_task_list.append(config)
             task_missing = [
                 task
-                for task in tasks_list
-                if task not in task_names and "*" not in task
+                for task in input_task_list
+                if task not in loaded_task_list and "*" not in task
             ]  # we don't want errors if a wildcard ("*") task name was used
 
             if task_missing:
@@ -226,12 +227,15 @@ def cli_evaluate(args: Union[argparse.Namespace, None] = None) -> None:
     elif args.log_samples and not args.output_path:
         assert args.output_path, "Specify --output_path"
 
-    eval_logger.info(f"Selected Tasks: {task_names}")
+    eval_logger.info(f"Selected Tasks: {loaded_task_list}")
+    eval_logger.info("Loading selected tasks...")
+
+    all_tasks = task_manager.load_task_or_group(loaded_task_list)
 
     results = evaluator.simple_evaluate(
         model=args.model,
         model_args=args.model_args,
-        tasks=task_names,
+        tasks=all_tasks,
         num_fewshot=args.num_fewshot,
         batch_size=args.batch_size,
         max_batch_size=args.max_batch_size,
diff --git a/lm_eval/api/task.py b/lm_eval/api/task.py
index 5713d7ed7a..7a4a4d6c0b 100644
--- a/lm_eval/api/task.py
+++ b/lm_eval/api/task.py
@@ -111,6 +111,8 @@ def __post_init__(self) -> None:
                 }
 
         # TODO: how to make TaskConfigs be de- and re-serializable, even when using the !function constructor?
+        # if self.dataset_kwargs is None:
+        #     self.dataset_kwargs = {"trust_remote_code": True}
 
     def __getitem__(self, item):
         return getattr(self, item)
diff --git a/lm_eval/evaluator.py b/lm_eval/evaluator.py
index 0b715c5ab7..3e1387fcc9 100644
--- a/lm_eval/evaluator.py
+++ b/lm_eval/evaluator.py
@@ -38,6 +38,7 @@ def simple_evaluate(
     write_out: bool = False,
     log_samples: bool = True,
     gen_kwargs: str = None,
+    weight_by_size: bool = False,
 ):
     """Instantiate and evaluate a model on a list of tasks.
 
@@ -46,8 +47,8 @@ def simple_evaluate(
     :param model_args: Optional[str]
         String arguments for each model class, see LM.create_from_arg_string.
         Ignored if `model` argument is a LM object.
-    :param tasks: list[Union[str, Task]]
-        List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
+    :param tasks: list[Task]
+        List of Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
     :param num_fewshot: int
         Number of examples in few-shot context
     :param batch_size: int or str, optional
@@ -119,7 +120,7 @@ def simple_evaluate(
             + ".db",
         )
 
-    task_dict = lm_eval.tasks.get_task_dict(tasks)
+    task_dict = tasks
     for task_name in task_dict.keys():
         task_obj = task_dict[task_name]
         if type(task_obj) == tuple:
@@ -155,6 +156,7 @@ def simple_evaluate(
         decontamination_ngrams_path=decontamination_ngrams_path,
         write_out=write_out,
         log_samples=log_samples,
+        weight_by_size=weight_by_size,
     )
 
     if lm.rank == 0:
@@ -192,6 +194,7 @@ def evaluate(
     decontamination_ngrams_path=None,
     write_out: bool = False,
     log_samples: bool = True,
+    weight_by_size: bool = False,
 ):
     """Instantiate and evaluate a model on a list of tasks.
 
@@ -479,12 +482,14 @@ def evaluate(
                         if "alias" in metrics:
                             metrics.pop("alias")
 
-                        current_size = metrics.pop("samples")
                         # TODO: There should be a way for users
                         #       to toggle between weighted and
                         #       unweighted averaging
-                        # For unweighted averaging, use:
-                        #     current_size = 1
+                        if weight_by_size:
+                            current_size = metrics.pop("samples")
+                        else:
+                            metrics.pop("samples")
+                            current_size = 1
 
                         all_stderr = []
                         for metric in [
diff --git a/lm_eval/tasks/__init__.py b/lm_eval/tasks/__init__.py
index 3eb86f2d05..1c3f59095a 100644
--- a/lm_eval/tasks/__init__.py
+++ b/lm_eval/tasks/__init__.py
@@ -1,5 +1,9 @@
 import os
+import abc
 import yaml
+import collections
+
+from functools import partial, lru_cache
 from typing import List, Union, Dict
 
 from lm_eval import utils
@@ -10,7 +14,6 @@
     register_group,
     TASK_REGISTRY,
     GROUP_REGISTRY,
-    ALL_TASKS,
 )
 
 import logging
@@ -29,272 +32,203 @@
 eval_logger = utils.eval_logger
 
 
-def register_configurable_task(config: Dict[str, str]) -> int:
-    SubClass = type(
-        config["task"] + "ConfigurableTask",
-        (ConfigurableTask,),
-        {"CONFIG": TaskConfig(**config)},
-    )
+class TaskManager(abc.ABC):
 
-    if "task" in config:
-        task_name = "{}".format(config["task"])
-        register_task(task_name)(SubClass)
+    def __init__(
+        self,
+        verbosity="INFO",
+        include_path=None
+        ) -> None:
 
-    if "group" in config:
-        if config["group"] == config["task"]:
-            raise ValueError("task and group name cannot be the same")
-        elif type(config["group"]) == str:
-            group_name = [config["group"]]
-        else:
-            group_name = config["group"]
-
-        for group in group_name:
-            register_group(group)(SubClass)
-
-    return 0
-
-
-def register_configurable_group(config: Dict[str, str], yaml_path: str = None) -> int:
-    group = config["group"]
-    all_task_list = config["task"]
-    config_list = [task for task in all_task_list if type(task) != str]
-    task_list = [task for task in all_task_list if type(task) == str]
-
-    for task_config in config_list:
-
-        base_config = {}
-        task_name_config = {}
-        if "task" in task_config:
-            task_name = task_config["task"]
-            if task_name in ALL_TASKS:
-                task_obj = get_task_dict(task_name)[task_name]
-                if type(task_obj) == tuple:
-                    _, task_obj = task_obj
-
-                if task_obj is not None:
-                    base_config = task_obj._config.to_dict(keep_callable=True)
-                    task_name_config["task"] = f"{group}_{task_name}"
-
-        task_config = utils.load_yaml_config(yaml_path, task_config)
-        var_configs = check_prompt_config(
-            {
-                **base_config,
-                **task_config,
-                **{"group": group},
-                **task_name_config,
-            },
-            yaml_path=os.path.dirname(yaml_path),
-        )
-        for config in var_configs:
-            register_configurable_task(config)
-
-    task_names = utils.pattern_match(task_list, ALL_TASKS)
-    for task in task_names:
-        if (task in TASK_REGISTRY) or (task in GROUP_REGISTRY):
-            if group in GROUP_REGISTRY:
-                GROUP_REGISTRY[group].append(task)
-            else:
-                GROUP_REGISTRY[group] = [task]
-                ALL_TASKS.add(group)
-
-    return 0
-
-
-def check_prompt_config(
-    config: Dict[str, str], yaml_path: str = None
-) -> List[Dict[str, str]]:
-    all_configs = []
-    if "use_prompt" in config:
-        prompt_list = prompts.load_prompt_list(
-            use_prompt=config["use_prompt"],
-            dataset_name=config["dataset_path"],
-            subset_name=config["dataset_name"] if "dataset_name" in config else None,
-            yaml_path=yaml_path,
-        )
-        for idx, prompt_variation in enumerate(prompt_list):
-            all_configs.append(
-                {
-                    **config,
-                    **{"use_prompt": prompt_variation},
-                    **{
-                        "task": "_".join(
-                            [
-                                config["task"]
-                                if "task" in config
-                                else get_task_name_from_config(config),
-                                prompt_variation.split("/")[-1]
-                                if ".yaml" in prompt_variation
-                                else prompt_variation,
-                            ]
-                        )
-                    },
-                    **{"output_type": "generate_until"},
-                }
+        self.verbosity = verbosity
+        self.include_path = include_path
+        self.logger = eval_logger.setLevel(getattr(logging, f"{verbosity}"))
+
+        self.ALL_TASKS = self.initialize_tasks(
+            include_path=include_path
             )
-    else:
-        all_configs.append(config)
-    return all_configs
-
-
-def get_task_name_from_config(task_config: Dict[str, str]) -> str:
-    if "dataset_name" in task_config:
-        return "{dataset_path}_{dataset_name}".format(**task_config)
-    else:
-        return "{dataset_path}".format(**task_config)
-
-
-def include_task_folder(task_dir: str, register_task: bool = True) -> None:
-    """
-    Calling this function
-    """
-
-    # Track whether any tasks failed during loading
-    import_fail = False
-    for root, subdirs, file_list in os.walk(task_dir):
-        # if (subdirs == [] or subdirs == ["__pycache__"]) and (len(file_list) > 0):
-        for f in file_list:
-            if f.endswith(".yaml"):
-                yaml_path = os.path.join(root, f)
-                try:
-                    config = utils.load_yaml_config(yaml_path)
-
-                    if "task" not in config:
-                        continue
-
-                    all_configs = check_prompt_config(
-                        config, yaml_path=os.path.dirname(yaml_path)
-                    )
-                    for config in all_configs:
-                        if register_task:
-                            if type(config["task"]) == str:
-                                register_configurable_task(config)
-                        else:
-                            if type(config["task"]) == list:
-                                register_configurable_group(config, yaml_path)
-
-                # Log this silently and show it only when
-                # the user defines the appropriate verbosity.
-                except (ImportError, ModuleNotFoundError) as e:
-                    import_fail = True
-                    eval_logger.debug(
-                        f"{yaml_path}: {e}. Config will not be added to registry."
-                    )
-                except Exception as error:
-                    import traceback
-
-                    eval_logger.warning(
-                        "Unexpected error loading config in\n"
-                        f"                                 {yaml_path}\n"
-                        "                                 Config will not be added to registry\n"
-                        f"                                 Error: {error}\n"
-                        f"                                 Traceback: {traceback.format_exc()}"
-                    )
-
-    if import_fail:
-        eval_logger.warning(
-          "Some tasks could not be loaded due to missing dependencies."
-          " Run with `--verbosity DEBUG` for full details."
-          )
-    return 0
-
-
-def include_path(task_dir):
-    include_task_folder(task_dir)
-    # Register Benchmarks after all tasks have been added
-    include_task_folder(task_dir, register_task=False)
-    return 0
-
-
-def initialize_tasks(verbosity="INFO"):
-    eval_logger.setLevel(getattr(logging, f"{verbosity}"))
-
-    task_dir = os.path.dirname(os.path.abspath(__file__)) + "/"
-    include_path(task_dir)
-
-
-def get_task(task_name, config):
-    try:
-        return TASK_REGISTRY[task_name](config=config)
-    except KeyError:
-        eval_logger.info("Available tasks:")
-        eval_logger.info(list(TASK_REGISTRY) + list(GROUP_REGISTRY))
-        raise KeyError(f"Missing task {task_name}")
-
-
-def get_task_name_from_object(task_object):
-    for name, class_ in TASK_REGISTRY.items():
-        if class_ is task_object:
-            return name
-
-    # TODO: scrap this
-    # this gives a mechanism for non-registered tasks to have a custom name anyways when reporting
-    return (
-        task_object.EVAL_HARNESS_NAME
-        if hasattr(task_object, "EVAL_HARNESS_NAME")
-        else type(task_object).__name__
-    )
-
-
-# TODO: pass num_fewshot and other cmdline overrides in a better way
-def get_task_dict(task_name_list: List[Union[str, Dict, Task]], **kwargs):
-    config = {**kwargs}
-
-    task_name_from_registry_dict = {}
-    task_name_from_config_dict = {}
-    task_name_from_object_dict = {}
-
-    if type(task_name_list) != list:
-        task_name_list = [task_name_list]
-
-    for task_element in task_name_list:
-        if isinstance(task_element, str):
-            if task_element in GROUP_REGISTRY:
-                group_name = task_element
-                for task_name in GROUP_REGISTRY[task_element]:
-                    if task_name not in task_name_from_registry_dict:
-                        task_obj = get_task_dict(task_name)
-                        if task_name in task_obj.keys():
-                            task_dict = {
-                                task_name: (group_name, task_obj[task_name]),
-                            }
-                        else:
-                            task_dict = {
-                                task_name: (group_name, None),
-                                **task_obj,
-                            }
 
-                        task_name_from_registry_dict = {
-                            **task_name_from_registry_dict,
-                            **task_dict,
+    def initialize_tasks(self, include_path=None):
+
+        all_paths = [os.path.dirname(os.path.abspath(__file__)) + "/"]
+        if include_path is not None:
+            if isinstance(include_path, str):
+                include_path = [include_path]
+            all_paths.extend(include_path)
+
+        ALL_TASKS = {}
+        for task_dir in all_paths:
+            tasks = self._get_task_and_group(task_dir)
+            ALL_TASKS = {**tasks, **ALL_TASKS}
+
+        return ALL_TASKS
+
+    def all_tasks(self):
+        return sorted(list(self.ALL_TASKS.keys()))
+
+    def _name_is_registered(self, name):
+        if name in self.ALL_TASKS:
+            return True
+        return False
+
+    def _name_is_task(self, name):
+        if self.ALL_TASKS[name]["type"] == "task":
+            return True
+        return False
+
+    def _config_is_task(self, config):
+        if list(config.keys()) == ["group", "task"]:
+            return False
+        return True
+
+    def _get_yaml_path(self, name):
+        assert name in self.ALL_TASKS
+        return self.ALL_TASKS[name]["yaml_path"]
+
+    def _get_config(self, name):
+        assert name in self.ALL_TASKS
+        yaml_path = self._get_yaml_path(name)
+        return utils.load_yaml_config(yaml_path)
+
+    def _get_tasklist(self, name):
+        assert self._name_is_task(name) == False
+        return self.ALL_TASKS[name]["task"]
+
+    @lru_cache(None)
+    def _load_individual_task_or_group(self, name_or_config: Union[str, dict] = None, parent_name: str = None) -> ConfigurableTask:
+
+        def load_task(config, task, group=None):
+            task_object = ConfigurableTask(config=config)
+            if group is not None:
+                task_object = (group, task_object)
+            return {task: task_object}
+
+        if isinstance(name_or_config, str):
+            if self._name_is_task(name_or_config):
+                task_config = self._get_config(name_or_config)
+                return load_task(task_config, task=name_or_config, group=parent_name)
+            else:
+                group_name = name_or_config
+                subtask_list = self._get_tasklist(name_or_config)
+                if subtask_list == -1:
+                    subtask_list = self._get_config(name_or_config)["task"]
+
+        elif isinstance(name_or_config, dict):
+            if self._config_is_task(name_or_config):
+                task_name = name_or_config["task"]
+                if self._name_is_registered(task_name):
+                    base_task_config = self._get_config(task_name)
+                    task_config={
+                            **base_task_config,
+                            **name_or_config,
                         }
+                else:
+                    task_config = name_or_config
+                return load_task(task_config, task=name_or_config, group=parent_name)
             else:
-                task_name = task_element
-                if task_name not in task_name_from_registry_dict:
-                    task_name_from_registry_dict = {
-                        **task_name_from_registry_dict,
-                        task_name: get_task(task_name=task_element, config=config),
-                    }
-
-        elif isinstance(task_element, dict):
-            task_element.update(config)
-            task_name_from_config_dict = {
-                **task_name_from_config_dict,
-                get_task_name_from_config(task_element): ConfigurableTask(
-                    config=task_element
-                ),
-            }
-
-        elif isinstance(task_element, Task):
-            task_name_from_object_dict = {
-                **task_name_from_object_dict,
-                get_task_name_from_object(task_element): task_element,
-            }
-
-    assert set(task_name_from_registry_dict.keys()).isdisjoint(
-        set(task_name_from_object_dict.keys())
-    )
-    return {
-        **task_name_from_registry_dict,
-        **task_name_from_config_dict,
-        **task_name_from_object_dict,
-    }
+                group_name = name_or_config["group"]
+                subtask_list = name_or_config["task"]
+
+        if self._get_yaml_path(group_name) == -1:
+            all_subtasks = {group_name: (parent_name, None)}
+        else:
+            all_subtasks = {}
+
+        fn = partial(self._load_individual_task_or_group, parent_name=group_name)
+        all_subtasks = {**all_subtasks, **dict(collections.ChainMap(*map(fn, subtask_list)))}
+        return all_subtasks
+
+
+    def load_task_or_group(self, task_list: Union[str, list] = None) -> dict:
+
+        if isinstance(task_list, str):
+            task_list = [task_list]
+
+        all_loaded_tasks = dict(
+            collections.ChainMap(
+                *map(
+                    self._load_individual_task_or_group,
+                    task_list
+                )
+            )
+        )
+        return all_loaded_tasks
+
+    def _get_task_and_group(self, task_dir: str):
+        tasks_and_groups = collections.defaultdict()
+        for root, _, file_list in os.walk(task_dir):
+            for f in file_list:
+                if f.endswith(".yaml"):
+                    yaml_path = os.path.join(root, f)
+                    config = utils.simple_load_yaml_config(yaml_path)
+                    if list(config.keys()) == ["group", "task"]:
+                        # This is a group config
+                        tasks_and_groups[config["group"]] = {
+                            "type": "group",
+                            "task": -1, # This signals that
+                                        # we don't need to know
+                                        # the task list for indexing
+                                        # as it can be loaded
+                                        # when called.
+                            "yaml_path": yaml_path,
+                        }
+                    else:
+                        # This is a task config
+                        task = config["task"]
+                        tasks_and_groups[task] = {
+                            "type": "task",
+                            "yaml_path": yaml_path,
+                            }
+
+                        if "group" in config:
+                            groups = config["group"]
+                            if isinstance(config["group"], str):
+                                groups = [groups]
+
+                            for group in groups:
+                                if group not in tasks_and_groups:
+                                    tasks_and_groups[group] = {
+                                        "type": "group",
+                                        "task": [task],
+                                        "yaml_path": -1,
+                                    }
+                                else:
+                                    tasks_and_groups[group]["task"].append(task)
+
+        return tasks_and_groups
+
+
+# def check_prompt_config(
+#     config: Dict[str, str], yaml_path: str = None
+# ) -> List[Dict[str, str]]:
+#     all_configs = []
+#     if "use_prompt" in config:
+#         prompt_list = prompts.load_prompt_list(
+#             use_prompt=config["use_prompt"],
+#             dataset_name=config["dataset_path"],
+#             subset_name=config["dataset_name"] if "dataset_name" in config else None,
+#             yaml_path=yaml_path,
+#         )
+#         for idx, prompt_variation in enumerate(prompt_list):
+#             all_configs.append(
+#                 {
+#                     **config,
+#                     **{"use_prompt": prompt_variation},
+#                     **{
+#                         "task": "_".join(
+#                             [
+#                                 config["task"]
+#                                 if "task" in config
+#                                 else get_task_name_from_config(config),
+#                                 prompt_variation.split("/")[-1]
+#                                 if ".yaml" in prompt_variation
+#                                 else prompt_variation,
+#                             ]
+#                         )
+#                     },
+#                     **{"output_type": "generate_until"},
+#                 }
+#             )
+#     else:
+#         all_configs.append(config)
+#     return all_configs
diff --git a/lm_eval/tasks/benchmarks/flan/flan_anli.yaml b/lm_eval/tasks/benchmarks/flan/flan_anli.yaml
deleted file mode 100644
index 21278e1f69..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_anli.yaml
+++ /dev/null
@@ -1,17 +0,0 @@
-group: flan_anli
-task:
-  - include: yaml_templates/held_in_template_yaml
-    task: anli_r1
-    dataset_path: anli
-    use_prompt: prompt_templates/anli.yaml:*
-    validation_split: dev_r1
-  - include: yaml_templates/held_in_template_yaml
-    task: anli_r2
-    dataset_path: anli
-    use_prompt: prompt_templates/anli.yaml:*
-    validation_split: dev_r2
-  - include: yaml_templates/held_in_template_yaml
-    task: anli_r3
-    dataset_path: anli
-    use_prompt: prompt_templates/anli.yaml:*
-    validation_split: dev_r3
diff --git a/lm_eval/tasks/benchmarks/flan/flan_arc.yaml b/lm_eval/tasks/benchmarks/flan/flan_arc.yaml
deleted file mode 100644
index 4e73b7ce53..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_arc.yaml
+++ /dev/null
@@ -1,14 +0,0 @@
-group: flan_arc
-task:
-  - include: yaml_templates/held_in_template_yaml
-    task: arc_easy
-    dataset_path: ai2_arc
-    dataset_name: ARC-Easy
-    use_prompt: prompt_templates/arc.yaml:*
-    validation_split: validation
-  - include: yaml_templates/held_in_template_yaml
-    task: arc_challenge
-    dataset_path: ai2_arc
-    dataset_name: ARC-Challenge
-    use_prompt: prompt_templates/arc.yaml:*
-    validation_split: validation
diff --git a/lm_eval/tasks/benchmarks/flan/flan_boolq.yaml b/lm_eval/tasks/benchmarks/flan/flan_boolq.yaml
deleted file mode 100644
index 8fe36cd53c..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_boolq.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-group: flan_boolq
-task:
-  - include: yaml_templates/held_in_template_yaml
-    dataset_path: super_glue
-    dataset_name: boolq
-    use_prompt: prompt_templates/boolq.yaml:*
-    validation_split: validation
diff --git a/lm_eval/tasks/benchmarks/flan/flan_cot.yaml b/lm_eval/tasks/benchmarks/flan/flan_cot.yaml
deleted file mode 100644
index 10102d2461..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_cot.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-group: flan_cot
-task:
-  - include: yaml_templates/cot_template_yaml
-    dataset_path: gsmk
-    dataset_name: boolq
-    use_prompt: promptsource:*
-    validation_split: validation
-  - include: yaml_templates/cot_template_yaml
-    dataset_path: EleutherAI/asdiv
-    use_prompt: promptsource:*
-    validation_split: validation
diff --git a/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml b/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml
deleted file mode 100644
index 5465b58c24..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-group: flan_held_in
-task:
-  - flan_boolq
-  - flan_rte
-  - flan_anli
-  - flan_arc
diff --git a/lm_eval/tasks/benchmarks/flan/flan_held_in_yaml b/lm_eval/tasks/benchmarks/flan/flan_held_in_yaml
deleted file mode 100644
index a31a942e6a..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_held_in_yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-group: flan_held_in
-task:
-  - include: flan/yaml_templates/held_in_template_yaml
-    dataset_path: super_glue
-    dataset_name: boolq
-    use_prompt: flan/prompt_templates/boolq.yaml:*
-    validation_split: validation
-  - include: flan/yaml_templates/held_in_template_yaml
-    dataset_path: super_glue
-    dataset_name: rte
-    use_prompt: flan/prompt_templates/rte.yaml:*
-    validation_split: validation
-  - include: flan/yaml_templates/held_in_template_yaml
-    task: anli_r1
-    dataset_path: anli
-    use_prompt: flan/prompt_templates/anli.yaml:*
-    validation_split: dev_r1
-  - include: flan/yaml_templates/held_in_template_yaml
-    task: anli_r2
-    dataset_path: anli
-    use_prompt: flan/prompt_templates/anli.yaml:*
-    validation_split: dev_r2
-  - include: flan/yaml_templates/held_in_template_yaml
-    task: anli_r3
-    dataset_path: anli
-    use_prompt: flan/prompt_templates/anli.yaml:*
-    validation_split: dev_r3
-  - include: flan/yaml_templates/held_in_template_yaml
-    task: arc_easy
-    dataset_path: ai2_arc
-    dataset_name: ARC-Easy
-    use_prompt: flan/prompt_templates/arc.yaml:*
-    validation_split: validation
-  - include: flan/yaml_templates/held_in_template_yaml
-    task: arc_challenge
-    dataset_path: ai2_arc
-    dataset_name: ARC-Challenge
-    use_prompt: flan/prompt_templates/arc.yaml:*
-    validation_split: validation
diff --git a/lm_eval/tasks/benchmarks/flan/flan_held_out.yaml b/lm_eval/tasks/benchmarks/flan/flan_held_out.yaml
deleted file mode 100644
index e1429cdac7..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_held_out.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-group: flan_held_out
-task:
-  # BBH
-  - bbh_flan_zeroshot
-  - bbh_flan_fewshot
-  - bbh_flan_cot_fewshot
-  - bbh_flan_cot_zeroshot
-  # MMLU
-  - mmlu
-  - mmlu_flan_n_shot_generative
-  - mmlu_flan_n_shot_loglikelihood
-  - mmlu_flan_cot_zeroshot
-  - mmlu_flan_cot_fewshot
diff --git a/lm_eval/tasks/benchmarks/flan/flan_rte.yaml b/lm_eval/tasks/benchmarks/flan/flan_rte.yaml
deleted file mode 100644
index a4f407361d..0000000000
--- a/lm_eval/tasks/benchmarks/flan/flan_rte.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-group: flan_rte
-task:
-  - include: yaml_templates/held_in_template_yaml
-    dataset_path: super_glue
-    dataset_name: rte
-    use_prompt: prompt_templates/rte.yaml:*
-    validation_split: validation
diff --git a/lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml b/lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml
deleted file mode 100644
index 6ff7884013..0000000000
--- a/lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Flan Prompt Templates
-prompts:
-  "template-0":
-    doc_to_text: "{{premise}}\n\nChoose your answer: based on the paragraph above can we conclude that \"{{hypothesis}}\"?\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No\nI think the answer is"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-1":
-    doc_to_text: "{{premise}}\n\nBased on that paragraph can we conclude that this sentence is true?\n{{hypothesis}}\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-2":
-    doc_to_text: "{{premise}}\n\nCan we draw the following conclusion?\n{{hypothesis}}\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-3":
-    doc_to_text: "{{premise}}\nDoes this next sentence follow, given the preceding text?\n{{hypothesis}}\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-4":
-    doc_to_text: "{{premise}}\nCan we infer the following?\n{{hypothesis}}\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No\nThe answer is:"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-5":
-    doc_to_text: "Read the following paragraph and determine if the hypothesis is true:\n\n{{premise}}\n\nOPTIONS:\n- Yes\n- It's impossible to say\n- No\nHypothesis: {{hypothesis}}\n\n\n"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-6":
-    doc_to_text: "Read the text and determine if the sentence is true (see options at the end):\n\n{{premise}}\n\nSentence: {{hypothesis}}\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-7":
-    doc_to_text: "Can we draw the following hypothesis from the context (see options)? \n\nContext:\n\n{{premise}}\n\nHypothesis: {{hypothesis}}\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
-  "template-8":
-    doc_to_text: "Choose from options: Determine if the sentence is true based on the text below:\n{{hypothesis}}\n\n{{premise}}\nOPTIONS:\n- Yes\n- It's impossible to say\n- No"
-    doc_to_target: "{{[\"Yes\", \"It's impossible to say\", \"No\"][label]}}"
diff --git a/lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml b/lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml
deleted file mode 100644
index 4ee34e6592..0000000000
--- a/lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-# Flan Prompt Templates
-prompts:
-  "template-0":
-    doc_to_text: "{{question}}\n\nOPTIONS:\n- {{choices.text|join('\n- ')}}"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-1":
-    doc_to_text: "Question: {{question}}\nOPTIONS:\n- {{choices.text|join('\n- ')}}\nAnswer:"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-2":
-    doc_to_text: "Question: {{question}}\n\nWhat is the correct answer to the question from the following choices?\nOPTIONS:\n- {{choices.text|join('\n- ')}}"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-3":
-    doc_to_text: "Q: {{question}}\nWhat is the correct answer to this question?\nOPTIONS:\n- {{choices.text|join('\n- ')}}...A:"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-4":
-    doc_to_text: "Choose your answer?\n\n{{question}}\n\nOPTIONS:\n- {{choices.text|join('\n- ')}}"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-5":
-    doc_to_text: "Answer the question\n\n{{question}}\nOPTIONS:\n- {{choices.text|join('\n- ')}}"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
-  "template-6":
-    doc_to_text: "{{question}}\n\nPick the answer from these options\n\nOPTIONS:\n- {{choices.text|join('\n- ')}}"
-    doc_to_target: "{{choices.text[choices.label.index(answerKey)]}}"
diff --git a/lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml b/lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml
deleted file mode 100644
index f8c8ebfca8..0000000000
--- a/lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml
+++ /dev/null
@@ -1,33 +0,0 @@
-# Flan Prompt Templates
-prompts:
-  "template-0":
-    doc_to_text: "{{passage}}\n\nCan we conclude that {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-1":
-    doc_to_text: "{{passage}}\n\nIs it true that {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-2":
-    doc_to_text: "{{passage}}\n\n{{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-3":
-    doc_to_text: "Text: {{passage}}\n\nQuestion: {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-4":
-    doc_to_text: "{{passage}}\n\nWhat's the best answer to this question: {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-5":
-    doc_to_text: "{{passage}}\nBased on the above text what's the best answer to this question: {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-6":
-    doc_to_text: "{{passage}}\nAnswer this question making sure that the answer is supposed by the text: {{question}}?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-7":
-    doc_to_text: "{{passage}}\n\nIs the following statement correct based on the text\n\n{{question}}\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-8":
-    # doc_to_text: "{{title}}\n\n{{passage}}\n\nIs this statement correct \"{{question}}\"?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_text: "{{passage}}\n\nIs this statement correct \"{{question}}\"?\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
-  "template-9":
-    doc_to_text: "Is it true that {{question}} based on the following text?\n\n{{passage}}\n\nOPTIONS:\n- no\n- yes"
-    doc_to_target: "{{['no', 'yes'][label]}}"
diff --git a/lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml b/lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml
deleted file mode 100644
index 7893eae44c..0000000000
--- a/lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Flan Prompt Templates
-prompts:
-  "template-0":
-    doc_to_text: "{{premise}}\n\nQuestion with options: Based on the paragraph above can we conclude that \"{{hypothesis}}\"?\n\nOPTIONS:\n- yes\n- no"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-1":
-    doc_to_text: "{{premise}}\n\nBased on that paragraph can we conclude that the sentence below is true?\n{{hypothesis}}\n\nOPTIONS:\n- yes\n- no"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-2":
-    doc_to_text: "{{premise}}\n\nQ with options: Can we draw the following conclusion?\n{{hypothesis}}\n\nOPTIONS:\n- yes\n- no"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-3":
-    doc_to_text: "{{premise}}\nDoes this next sentence follow, given the preceding text?\n{{hypothesis}}\n\nOPTIONS:\n- yes\n- no"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-4":
-    doc_to_text: "{{premise}}\nOPTIONS:\n- yes\n- no\nQuestion: Can we infer the following?\n{{hypothesis}}"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-5":
-    doc_to_text: "Read the following paragraph and determine if the hypothesis is true. Select from options at the end:\n\n{{premise}}\n\nHypothesis: {{hypothesis}}\nOPTIONS:\n- yes\n- no\nThe answer is"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-6":
-    doc_to_text: "Read the text and determine if the sentence is true:\n\n{{premise}}\n\nSentence: {{hypothesis}}\nOPTIONS:\n- yes\n- no\nA:"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-7":
-    doc_to_text: "Question with options: can we draw the following hypothesis from the context? \n\nContext:\n\n{{premise}}\n\nHypothesis: {{hypothesis}}\nOPTIONS:\n- yes\n- no\nA:"
-    doc_to_target: "{{['yes', 'no'][label]}}"
-  "template-8":
-    doc_to_text: "Determine if the sentence is true based on the text below. Choose from options.\n{{hypothesis}}\n\n{{premise}}\nOPTIONS:\n- yes\n- no"
-    doc_to_target: "{{['yes', 'no'][label]}}"
diff --git a/lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml b/lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml
deleted file mode 100644
index a18efa5f7d..0000000000
--- a/lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-group: flan-cot
-output_type: generate_until
-validation_split: validation
-doc_to_target: "{{answer}}"
-metric_list:
-  - metric: exact_match
-    aggregation: mean
-    higher_is_better: true
-generation_kwargs:
-  until:
-    - "\n\n"
-  do_sample: false
-  temperature: 0.0
-filter_list:
-  - name: "get-answer"
-    filter:
-      - function: "regex"
-        regex_pattern: "The answer is (\\-?[0-9\\.\\,]+)"
-      - function: "take_first"
-metadata:
-  version: 1.0
diff --git a/lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml b/lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml
deleted file mode 100644
index d3d9f52633..0000000000
--- a/lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-output_type: generate_until
-validation_split: validation
-metric_list:
-  - metric: exact_match
-    aggregation: mean
-    higher_is_better: true
-generation_kwargs:
-  until:
-    - "
"
-  do_sample: false
-  temperature: 0.0
-metadata:
-  version: 1.0
diff --git a/lm_eval/tasks/benchmarks/test.yaml b/lm_eval/tasks/benchmarks/test.yaml
new file mode 100644
index 0000000000..1781d28004
--- /dev/null
+++ b/lm_eval/tasks/benchmarks/test.yaml
@@ -0,0 +1,13 @@
+group: grouptest
+task:
+  - boolq
+  - group: arc_stuff
+    task:
+      - arc_challenge
+      - glue
+      - task: arc_easy
+        metric_list:
+          - metric: acc
+        num_fewshot: 3
+  # - task: mmlu
+  #   num_fewshot: 2
diff --git a/lm_eval/utils.py b/lm_eval/utils.py
index f0ec330e02..0cb08efb32 100644
--- a/lm_eval/utils.py
+++ b/lm_eval/utils.py
@@ -489,11 +489,20 @@ def import_function(loader, node):
     return function
 
 
-# Add the import_function constructor to the YAML loader
-yaml.add_constructor("!function", import_function)
+def ignore_constructor(loader, node):
+    return node
+
+
+def simple_load_yaml_config(yaml_path=None, yaml_config=None, yaml_dir=None):
+    yaml.add_constructor("!function", ignore_constructor)
+    with open(yaml_path, "rb") as file:
+        yaml_config = yaml.full_load(file)
+    return yaml_config
 
 
 def load_yaml_config(yaml_path=None, yaml_config=None, yaml_dir=None):
+    # Add the import_function constructor to the YAML loader
+    yaml.add_constructor("!function", import_function)
     if yaml_config is None:
         with open(yaml_path, "rb") as file:
             yaml_config = yaml.full_load(file)


Code review comments:
diff_hunk: @@ -479,12 +482,14 @@ def evaluate(
                         if "alias" in metrics:
                             metrics.pop("alias")
 
-                        current_size = metrics.pop("samples")
                         # TODO: There should be a way for users
                         #       to toggle between weighted and
                         #       unweighted averaging
-                        # For unweighted averaging, use:
-                        #     current_size = 1
+                        if weight_by_size:
path: lm_eval/evaluator.py
text: @user1:
`weight_by_size` should be set in a benchmark's YAML file IMO. it shouldn't be controllable by CLI at least, that'll create weird discrepancies where one user might set it and another might not, resulting in a divergence in scores that comparing the config YAML files would not reveal

@author:
That makes sense

diff_hunk: @@ -46,8 +47,8 @@ def simple_evaluate(
     :param model_args: Optional[str]
         String arguments for each model class, see LM.create_from_arg_string.
         Ignored if `model` argument is a LM object.
-    :param tasks: list[Union[str, Task]]
-        List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
+    :param tasks: list[Task]
path: lm_eval/evaluator.py
text: @user1:
I think we should really not change the public interface in such a way--a lot of users rely on the `evaluate()` and `simple_evaluate()` APIs to use the library externally and would be affected by it changing on them.