Repository: EleutherAI/lm-evaluation-harness
Pull request: Create a means for caching task registration and request building. Ad…
Hey all, here's the caching proposal for task registration and request building.

Caching is optional, and has rewrite capabilities for task building and for task registration.

It works by using pickles, and by default all pickles are stored in lm_harness/caching/.cache. You can specify an alternate path via the env var `LM_HARNESSS_CACHE_PATH`. Which is really useful for docker containers.

I also added some tqdm's to track the progress of these steps.

In addition to this, I extended simple_evaluate to allow for you to take in a model_arg_dict, and added a class method to the Model class to allow for this. Passing a bunch of strings was a frustrating DX.

I also added some type annotations, and fixed a bug with the limit, where it doesn't return results if you set your float really low. At least one item in a dataset is run now if you set limit super low.

If something's off please LMK, but would love to merge this puppy! (Hence the test script, I'll clean up the commit.)

Best,
Aaron

Problem statement:
Is there a way to cache the building of datasets?
Hey all, once again, excellent framework.

I'm using the framework programatically.

I wrote code to do this myself, but I want to make sure I'm not being stupid and wanted to check that the package doesn't already support this.

I know it supports the caching of results from LLMs that have already been run, but don't see caching for anything else. Although there's a lot of code here to understand. So with that said, I've got code that caches the building/registration of tasks, groups, datasets before preprocessing, and datasets after post processing (request dicts).

Reduces the spin up time to start testing a model to about a second or so and that's a big win for my use case.

I know that PRing can be a bit of a hassle at times, but if this isn't already part of the package, if you all think it would be worthwhile for me to PR, would love to.

The basics of my proposal are to just add a .cache directory, a caching module, and then to store all of the cached files in the .cache directory. (They'd all be pickles)

All functions involving a cache then have a kwarg on them to allow for rewriting the cache if necessary.

A future enhancement would be to also rewrite the cache automatically if any of the target entities change (tasks, groups) and any subsequent caches that depend on them, using a hash reduced from the last time a file was modified, if any file changed, full logic runs and the cache is refreshed, or whatever the fastest/most robust approach would be.

But the ideal would be the user doesn't have to manage that themselves because I can see a few shooting oneself in the foot scenarios happening.

Hints from issue discussion:
Hi! Thank you for using the framework and for being interested in contributing!

I think it'd certainly be worthwhile to put that caching code in a fork and we can look to see if it makes sense to PR it. 

I think this would be a huge improvement for the mentioned reasons (minimizing non-inference runtimes and making rerunning evals much faster) but that we'd want to be very careful to minimize the amount that this causes bugs or prevents a user from observing an expected change when debugging things. One way to minimize shooting oneself in the foot would be to potentially make this, like model output caching, strictly opt-in

I also outlined a potential less invasive or less codechange-intensive version of this in #1286 , where we could piggyback off of HF datasets' existing caching tools to cache at least some intermediate results ( a large amount of slowness is due to frequent Jinja2 template rendering)-- I think this might achieve a good portion of the benefits while still getting a significant speedup.


In a similar vein, I'm curious if this still works (from the main README.md):

`python write_out.py \
    --tasks all_tasks \
    --num_fewshot 5 \
    --num_examples 10 \
    --output_base_path /path/to/output/folder`

I'm trying to a variant of this command to run+download all the benchmarks for local offline usage later, but this functionality seems to be faulty now, depsite being in the example usage. The error I get is:

`File "/home/user/.conda/envs/conda_env/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 221, in get_task
    raise KeyError(f"Missing task {task_name}")
KeyError: 'Missing task all_tasks'`

Would this be a viable way to get all of them downloaded?


> I'm curious if this still works (from the main README.md):

Ah sorry, I will clarify this in the README--`all_tasks` is a placeholder for `<the list of task names you care about>`.

If you truly want to download *all* the tasks, I think you can do so via `--tasks *` to pattern-match to everything. I'd recommend using this to download just the tasks you care about as it'll save storage + time, and then to use HF datasets and transformers offline settings to run: https://huggingface.co/docs/datasets/v2.16.1/loading#offline
> Ah sorry, I will clarify this in the README--`all_tasks` is a placeholder for `<the list of task names you care about>`.

Oh, thank you for that clarification. I guess I'm used to seeing placeholders marked \<like so\> haha. Your tip looks great though. I will go with that, and make and edit here if I hit any bumps. Thanks for the timely response!




Aight peeps, I submitted the PR, may our odds ever be in the favor of our wonderful maintainers!

Commit message: Remove extra S in cache path in caching module

Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

Patch to review:
diff --git a/.gitignore b/.gitignore
index 0e5028fb11..35a6381ea7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,3 +16,6 @@ temp
 # IPython
 profile_default/
 ipython_config.py
+
+Old/lm_eval
+lm_eval/caching/.cache
\ No newline at end of file
diff --git a/lm_eval/api/model.py b/lm_eval/api/model.py
index df829af592..340b4443fb 100644
--- a/lm_eval/api/model.py
+++ b/lm_eval/api/model.py
@@ -133,6 +133,28 @@ def create_from_arg_string(
         args2 = {k: v for k, v in additional_config.items() if v is not None}
         return cls(**args, **args2)
 
+    @classmethod
+    def create_from_arg_obj(
+        cls: Type[T], arg_dict: dict, additional_config: Optional[dict] = None
+    ) -> T:
+        """
+        Creates an instance of the LM class using the given arg_obj
+
+        Parameters:
+        - arg_obj: A dict containing arguments in the format key1=value1,key2=value2.
+        - additional_config: Optional dictionary containing additional configuration parameters.
+
+        Returns:
+        - Instance of the LM class.
+        """
+
+        additional_config = {} if additional_config is None else additional_config
+        additional_config = {
+            k: v for k, v in additional_config.items() if v is not None
+        }
+
+        return cls(**arg_dict, **additional_config)
+
     @property
     def rank(self):
         # used in the case of parallelism. Hardcoded to
diff --git a/lm_eval/api/task.py b/lm_eval/api/task.py
index 650f5e6956..1f870a64fa 100644
--- a/lm_eval/api/task.py
+++ b/lm_eval/api/task.py
@@ -10,6 +10,7 @@
 
 import datasets
 import numpy as np
+from tqdm import tqdm
 
 from lm_eval import utils
 from lm_eval.api import samplers
@@ -29,6 +30,7 @@
 )
 from lm_eval.filters import build_filter_ensemble
 from lm_eval.prompts import get_prompt
+from lm_eval.caching.cache import save_to_cache, load_from_cache
 
 
 ALL_OUTPUT_TYPES = [
@@ -351,20 +353,71 @@ def doc_to_text(self, doc):
     def doc_to_target(self, doc):
         pass
 
-    def build_all_requests(self, limit=None, rank=None, world_size=None) -> None:
+    def build_all_requests(
+        self,
+        task_name,
+        limit=None,
+        rank=None,
+        world_size=None,
+        use_builder_cache=False,
+        rewrite_builder_cache=False,
+    ) -> None:
         """Build a set of Instances for a task, and store them in task.instances"""
+
+        # used with caching
+        og_limit = limit
+
+        cache_key = f"requests-{task_name}"
+
+        cached_instances = load_from_cache(file_name=cache_key)
+
+        if use_builder_cache and cached_instances and not rewrite_builder_cache:
+            cached_instances = cached_instances[:limit]
+
+            flattened_instances = [
+                instance
+                for instance_group in cached_instances
+                for instance in instance_group
+            ]
+
+            self._instances = flattened_instances
+            return
+
         if self.has_test_docs():
             docs = self.test_docs()
         elif self.has_validation_docs():
             docs = self.validation_docs()
         else:
-            assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
+            assert (
+                False
+            ), f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
 
-        eval_logger.info(f"Building contexts for task on rank {rank}...")
+        eval_logger.info(f"Task: {task_name} - Building contexts on rank {rank}...")
 
         instances = []
-        for doc_id, doc in utils.create_iterator(
-            enumerate(docs), rank, world_size, limit
+
+        # process all documents when caching is specified for simplicity
+        if (
+            use_builder_cache
+            and (not cached_instances or rewrite_builder_cache)
+            and limit is not None
+        ):
+            limit = None
+
+        doc_id_docs = list(
+            utils.create_iterator(
+                enumerate(docs),
+                rank,
+                world_size,
+                limit,
+            )
+        )
+
+        num_docs = len(doc_id_docs)
+
+        for doc_id, doc in tqdm(
+            doc_id_docs,
+            total=num_docs,
         ):
             # sample fewshot context #TODO: need to offset doc_id by rank now!
             fewshot_ctx = self.fewshot_context(
@@ -382,11 +435,25 @@ def build_all_requests(self, limit=None, rank=None, world_size=None) -> None:
             if not isinstance(inst, list):
                 inst = [inst]
 
-            instances.extend(inst)
+            instances.append(inst)
+
+        # now flatten, this is to allow slicing to work with pickles
+
+        sliced_instances = instances[:og_limit]
+
+        flattened_instances = [
+            instance
+            for instance_group in sliced_instances
+            for instance in instance_group
+        ]
+
+        self._instances = flattened_instances
 
-        self._instances = instances
         assert len(self._instances) != 0, "task.build_requests() did not find any docs!"
 
+        if use_builder_cache and (not cached_instances or rewrite_builder_cache):
+            save_to_cache(file_name=cache_key, obj=instances)
+
     @abc.abstractmethod
     def construct_requests(self, doc, ctx, **kwargs):
         """Uses RequestFactory to construct Requests and returns an iterable of
@@ -682,7 +749,9 @@ def __init__(
         elif self.has_validation_docs():
             self.task_docs = self.validation_docs()
         else:
-            assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
+            assert (
+                False
+            ), f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
 
         # Test One Doc
         self.features = list(self.task_docs.features.keys())
@@ -1199,7 +1268,9 @@ def process_results(self, doc, results):
                             predictions=[result],
                             **self._metric_fn_kwargs[metric],
                         )
-                    except TypeError:  # needed for now in order to use a different interface between our own metrics and HF Evaluate metrics
+                    except (
+                        TypeError
+                    ):  # needed for now in order to use a different interface between our own metrics and HF Evaluate metrics
                         result_score = self._metric_fn_list[metric]([gold, result])
                     if isinstance(result_score, dict):
                         # TODO: this handles the case where HF evaluate returns a dict.
diff --git a/lm_eval/caching/cache.py b/lm_eval/caching/cache.py
new file mode 100644
index 0000000000..b09af4632b
--- /dev/null
+++ b/lm_eval/caching/cache.py
@@ -0,0 +1,34 @@
+import os
+import dill
+
+
+MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
+
+OVERRIDE_PATH = os.getenv("LM_HARNESS_CACHE_PATH")
+
+
+PATH = OVERRIDE_PATH if OVERRIDE_PATH else f"{MODULE_DIR}/.cache"
+
+
+def load_from_cache(file_name):
+    try:
+        path = f"{PATH}/{file_name}.pickle"
+
+        with open(path, "rb") as file:
+            cached_task_dict = dill.loads(file.read())
+            return cached_task_dict
+
+    except Exception as exception:
+        print(f"{file_name} is not cached, generating...")
+        pass
+
+
+def save_to_cache(file_name, obj):
+    if not os.path.exists(PATH):
+        os.mkdir(PATH)
+
+    file_path = f"{PATH}/{file_name}.pickle"
+
+    print(f"Saving {file_path} to cache...")
+    with open(file_path, "wb") as file:
+        file.write(dill.dumps(obj))
diff --git a/lm_eval/evaluator.py b/lm_eval/evaluator.py
index 657fdcfef0..454be7fcb7 100644
--- a/lm_eval/evaluator.py
+++ b/lm_eval/evaluator.py
@@ -1,3 +1,5 @@
+import math
+from typing import TYPE_CHECKING
 import random
 import itertools
 import collections
@@ -20,17 +22,24 @@
     eval_logger,
 )
 
+if TYPE_CHECKING:
+    from lm_eval.api.model import LM
+    from lm_eval.tasks import Task
+
 
 @positional_deprecated
 def simple_evaluate(
     model,
     model_args=None,
-    tasks=None,
+    model_arg_dict=None,
+    tasks=[],
     num_fewshot=None,
     batch_size=None,
     max_batch_size=None,
     device=None,
     use_cache=None,
+    use_builder_cache=False,
+    rewrite_builder_cache=False,
     limit=None,
     bootstrap_iters: int = 100000,
     check_integrity: bool = False,
@@ -80,8 +89,6 @@ def simple_evaluate(
         1234
     )  # TODO: this may affect training runs that are run with evaluation mid-run.
 
-    if tasks is None:
-        tasks = []
     assert (
         tasks != []
     ), "No tasks specified, or no tasks found. Please verify the task names."
@@ -97,14 +104,26 @@ def simple_evaluate(
     if isinstance(model, str):
         if model_args is None:
             model_args = ""
-        lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
-            model_args,
-            {
-                "batch_size": batch_size,
-                "max_batch_size": max_batch_size,
-                "device": device,
-            },
-        )
+
+        if model_arg_dict:
+            lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(
+                model_arg_dict,
+                {
+                    "batch_size": batch_size,
+                    "max_batch_size": max_batch_size,
+                    "device": device,
+                },
+            )
+
+        else:
+            lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
+                model_args,
+                {
+                    "batch_size": batch_size,
+                    "max_batch_size": max_batch_size,
+                    "device": device,
+                },
+            )
     else:
         assert isinstance(model, lm_eval.api.model.LM)
         lm = model
@@ -116,9 +135,7 @@ def simple_evaluate(
             use_cache
             # each rank receives a different cache db.
             # necessary to avoid multiple writes to cache at once
-            + "_rank"
-            + str(lm.rank)
-            + ".db",
+            + "_rank" + str(lm.rank) + ".db",
         )
 
     task_dict = lm_eval.tasks.get_task_dict(tasks)
@@ -153,6 +170,8 @@ def simple_evaluate(
         lm=lm,
         task_dict=task_dict,
         limit=limit,
+        use_builder_cache=use_builder_cache,
+        rewrite_builder_cache=rewrite_builder_cache,
         bootstrap_iters=bootstrap_iters,
         decontamination_ngrams_path=decontamination_ngrams_path,
         write_out=write_out,
@@ -192,9 +211,11 @@ def simple_evaluate(
 
 @positional_deprecated
 def evaluate(
-    lm,
+    lm: "LM",
     task_dict,
     limit=None,
+    use_builder_cache=False,
+    rewrite_builder_cache=False,
     bootstrap_iters: int = 100000,
     decontamination_ngrams_path=None,
     write_out: bool = False,
@@ -244,6 +265,8 @@ def evaluate(
 
     # get lists of each type of request
     for task_name, task in task_dict.items():
+        task: Task
+
         if isinstance(task, tuple):
             group_name, task = task
             task_hierarchy[group_name].append(task_name)
@@ -282,9 +305,19 @@ def evaluate(
                 task_docs = task.validation_docs()
             else:
                 raise RuntimeError("Task has neither test_docs nor validation_docs")
-            limit = int(len(task_docs) * limit) if limit < 1.0 else int(limit)
 
-        task.build_all_requests(limit=limit, rank=lm.rank, world_size=lm.world_size)
+            num_docs = len(task_docs) * limit
+            # ceil to prevent limit being equal to 0
+            limit = int(math.ceil(num_docs)) if limit < 1.0 else int(limit)
+
+        task.build_all_requests(
+            task_name=task_name,
+            limit=limit,
+            rank=lm.rank,
+            world_size=lm.world_size,
+            use_builder_cache=use_builder_cache,
+            rewrite_builder_cache=rewrite_builder_cache,
+        )
 
         eval_logger.debug(
             f"Task: {task_name}; number of requests on this rank: {len(task.instances)}"
@@ -442,7 +475,6 @@ def evaluate(
         vals = vals_torch
 
     if lm.rank == 0:
-
         ### Aggregate results over all datapoints ###
         # aggregate results ; run bootstrap CIs
         for (task_name, key, metric), items in vals.items():
@@ -513,7 +545,10 @@ def evaluate(
                                     + metric_score * current_size
                                 ) / (total_size + current_size)
                                 # $$s_z^2 = \frac{(n-1) s_x^2 + (m-1) s_y^2}{n+m-1} + \frac{nm(\bar x - \bar y)^2}{(n+m)(n+m-1)}.$$
-                                if var_score == "N/A" or results[group][stderr] == "N/A":
+                                if (
+                                    var_score == "N/A"
+                                    or results[group][stderr] == "N/A"
+                                ):
                                     results[group][stderr] = "N/A"
                                 else:
                                     results[group][stderr] = (
diff --git a/lm_eval/tasks/__init__.py b/lm_eval/tasks/__init__.py
index fed0c1d0ed..d98bb92761 100644
--- a/lm_eval/tasks/__init__.py
+++ b/lm_eval/tasks/__init__.py
@@ -26,6 +26,9 @@
     QMSum,
 )
 
+from lm_eval.caching.cache import save_to_cache, load_from_cache
+from tqdm import tqdm
+
 eval_logger = utils.eval_logger
 
 
@@ -61,7 +64,6 @@ def register_configurable_group(config: Dict[str, str], yaml_path: str = None) -
     task_list = [task for task in all_task_list if isinstance(task, str)]
 
     for task_config in config_list:
-
         base_config = {}
         task_name_config = {}
         if "task" in task_config:
@@ -150,7 +152,13 @@ def include_task_folder(task_dir: str, register_task: bool = True) -> None:
 
     # Track whether any tasks failed during loading
     import_fail = False
-    for root, subdirs, file_list in os.walk(task_dir):
+    # Convert os.walk to a list
+    walk_list = list(os.walk(task_dir))
+
+    # Now use tqdm to iterate over the list
+    for root, subdirs, file_list in tqdm(
+        walk_list, desc="Processing task configuration directories", unit="dir"
+    ):
         # if (subdirs == [] or subdirs == ["__pycache__"]) and (len(file_list) > 0):
         for f in file_list:
             if f.endswith(".yaml"):
@@ -192,32 +200,80 @@ def include_task_folder(task_dir: str, register_task: bool = True) -> None:
 
     if import_fail:
         eval_logger.warning(
-          "Some tasks could not be loaded due to missing dependencies."
-          " Run with `--verbosity DEBUG` for full details."
-          )
+            "Some tasks could not be loaded due to missing dependencies."
+            " Run with `--verbosity DEBUG` for full details."
+        )
     return 0
 
 
-def include_path(task_dir):
+def include_path(task_dir, use_cache=False, rewrite_cache=False) -> None:
+    prefix = "registry-"
+
+    task_registry_file_name = f"{prefix}TASK_REGISTRY"
+    cached_group_registry_file_name = f"{prefix}GROUP_REGISTRY"
+    cached_all_tasks_file_name = f"{prefix}ALL_TASKS"
+
+    cached_task_registry: dict = load_from_cache(file_name=task_registry_file_name)
+    cached_group_registry: dict = load_from_cache(
+        file_name=cached_group_registry_file_name
+    )
+    cached_all_tasks: set = load_from_cache(file_name=cached_all_tasks_file_name)
+
+    caches_exist = cached_task_registry and cached_group_registry and cached_all_tasks
+
+    if not rewrite_cache and caches_exist:
+        for key, value in cached_task_registry.items():
+            TASK_REGISTRY[key] = value
+
+        for key, value in cached_group_registry.items():
+            GROUP_REGISTRY[key] = value
+
+        for key in cached_all_tasks:
+            ALL_TASKS.add(key)
+
+        return
+
     include_task_folder(task_dir)
     # Register Benchmarks after all tasks have been added
     include_task_folder(task_dir, register_task=False)
-    return 0
 
+    if use_cache and (
+        # if not already cached, cache
+        not caches_exist
+        or rewrite_cache
+    ):
+        print("Rewriting tasks caches...")
+        save_to_cache(file_name=task_registry_file_name, obj=TASK_REGISTRY)
+        save_to_cache(file_name=cached_group_registry_file_name, obj=GROUP_REGISTRY)
+        save_to_cache(file_name=cached_all_tasks_file_name, obj=ALL_TASKS)
+
+    return
 
-def initialize_tasks(verbosity="INFO"):
+
+def initialize_tasks(
+    verbosity="INFO", use_cache: bool = False, rewrite_cache: bool = False
+):
     eval_logger.setLevel(getattr(logging, f"{verbosity}"))
 
     task_dir = os.path.dirname(os.path.abspath(__file__)) + "/"
-    include_path(task_dir)
+    include_path(task_dir, use_cache, rewrite_cache)
 
 
 def get_task(task_name, config):
     try:
-        return TASK_REGISTRY[task_name](config=config)
+        configurable_task_factory = TASK_REGISTRY[task_name]
+
+        task = configurable_task_factory(config=config)
+
+        return task
     except KeyError:
         eval_logger.info("Available tasks:")
-        eval_logger.info(list(TASK_REGISTRY) + list(GROUP_REGISTRY))
+
+        all_tasks = list(TASK_REGISTRY) + list(GROUP_REGISTRY)
+
+        all_tasks.sort()
+
+        eval_logger.info(all_tasks)
         raise KeyError(f"Missing task {task_name}")
 
 
@@ -270,11 +326,32 @@ def get_task_dict(task_name_list: List[Union[str, Dict, Task]], **kwargs):
             else:
                 task_name = task_element
                 if task_name not in task_name_from_registry_dict:
+                    cache_key = f"raw-{task_name}"
+
+                    if task_name == "lambada_openai":
+                        a = 2
+
+                    cached_task_name_from_registry_dict = load_from_cache(
+                        file_name=cache_key
+                    )
+
+                    if cached_task_name_from_registry_dict:
+                        task_name_from_registry_dict = {
+                            **task_name_from_registry_dict,
+                            **cached_task_name_from_registry_dict,
+                        }
+
+                        continue
+
                     task_name_from_registry_dict = {
                         **task_name_from_registry_dict,
                         task_name: get_task(task_name=task_element, config=config),
                     }
 
+                    save_to_cache(file_name=cache_key, obj=task_name_from_registry_dict)
+
+                    pass
+
         elif isinstance(task_element, dict):
             task_element.update(config)
             task_name_from_config_dict = {
@@ -293,6 +370,7 @@ def get_task_dict(task_name_list: List[Union[str, Dict, Task]], **kwargs):
     assert set(task_name_from_registry_dict.keys()).isdisjoint(
         set(task_name_from_object_dict.keys())
     )
+
     return {
         **task_name_from_registry_dict,
         **task_name_from_config_dict,
diff --git a/pyproject.toml b/pyproject.toml
index 3840445b11..09f2227dc6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -36,6 +36,7 @@ dependencies = [
     "tqdm-multiprocess",
     "transformers>=4.1",
     "zstandard",
+    "dill"
 ]
 
 [tool.setuptools.packages.find]
diff --git a/test_script.py b/test_script.py
new file mode 100644
index 0000000000..ffaf7ebd4c
--- /dev/null
+++ b/test_script.py
@@ -0,0 +1,94 @@
+import os
+
+MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
+
+# Used to specify alternate cache path, useful if run in a docker container
+# NOTE raw datasets will break if you try to transfer the cache from your host to an image
+LM_HARNESSS_CACHE_PATH = os.getenv("LM_HARNESSS_CACHE_PATH")
+
+import torch
+from transformers import (
+    pipeline as trans_pipeline,
+)
+import torch
+
+from lm_eval import simple_evaluate, tasks as tasks_class
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+model = "gpt2"
+
+local_models_path = os.getenv("MODELS_PATH")
+
+task = "text-generation"
+
+print("\n\n\n")
+
+print("Model is: ", model)
+
+print("This task is: ", task)
+
+
+def run_model(model: str, device: str):
+    trans_pipe = trans_pipeline(
+        task=task, model=model, device=device, trust_remote_code=True
+    )
+
+    model = trans_pipe.model
+    tokenizer = trans_pipe.tokenizer
+
+    tasks = [
+        # "squadv2",
+        # "swag",
+        "hellaswag",
+        "lambada_openai",
+        # "glue",
+        # "super-glue-lm-eval-v1",
+        # "wikitext",
+        # "winogrande"
+    ]
+
+    print("Initializing tasks...")
+
+    tasks_class.initialize_tasks(use_cache=True, rewrite_cache=False)
+
+    print("Tasks initialized")
+
+    eval_data = simple_evaluate(
+        model="hf-auto",
+        # model_args=",".join(
+        #     [
+        #         f"pretrained={model}",
+        #         f"revision={revision}" if revision else "",
+        #         # "batch_size=16",
+        #         # "num_beams=5",
+        #         # "parallelize=True",
+        #     ]
+        # ),
+        model_arg_dict={
+            "pretrained": model,
+            "tokenizer": tokenizer,
+        },
+        limit=0.01,
+        device=device,
+        use_builder_cache=True,
+        rewrite_builder_cache=False,
+        tasks=tasks,
+        write_out=True,
+    )
+
+    return eval_data
+
+
+if __name__ == "__main__":
+    eval_data = run_model(model=model, device=device)
+
+    results: dict = eval_data["results"]
+
+    for dataset, metrics_dict in results.items():
+        for key, value in metrics_dict.items():
+            if key == "acc,none":
+                print(f"Dataset: {dataset}, Accuracy: {value}")
+
+
+pass


Code review comments:
diff_hunk: @@ -192,9 +211,11 @@ def simple_evaluate(
 
 @positional_deprecated
 def evaluate(
-    lm,
+    lm: "LM",
     task_dict,
     limit=None,
+    use_builder_cache=False,
path: lm_eval/evaluator.py
text: @user1:
minor nit: these might make more sense as named cache_requests and rewrite_requests_cache ? Or something more along those lines