Repository: instructlab/instructlab

Issue:
Evaluate expects safetensors but is passed GGUF by default
**Describe the bug**
Evaluation only works on safetensors at the moment for MMLU and MMLUBranch, but we are passing a GGUF file by default and there is no error handling for it

**To Reproduce**
Steps to reproduce the behavior:
1. Run `ilab config init` with no changes to the defaults
2. Run `ilab model evaluate --benchmark mmlu`

**Expected behavior**
Command should run as expected

**Device Info (please complete the following information):**
```bash
sys.version: 3.11.7 (main, May 16 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)]
sys.platform: linux
os.name: posix
platform.release: 5.14.0-427.26.1.el9_4.x86_64
platform.machine: x86_64
os-release.ID: rhel
os-release.VERSION_ID: 9.4
os-release.PRETTY_NAME: Red Hat Enterprise Linux 9.4 (Plow)
instructlab.version: 0.18.0a3.dev10
instructlab-dolomite.version: 0.1.1
instructlab-eval.version: 0.1.0
instructlab-quantize.version: 0.1.0
instructlab-schema.version: 0.2.0
instructlab-sdg.version: 0.1.2
instructlab-training.version: 0.2.0
torch.version: 2.3.1+cu121
torch.backends.cpu.capability: AVX2
torch.version.cuda: 12.1
torch.version.hip: None
torch.cuda.available: True
torch.backends.cuda.is_built: True
torch.backends.mps.is_built: False
torch.backends.mps.is_available: False
torch.cuda.bf16: True
torch.cuda.current: 0
torch.cuda.0.name: NVIDIA A10G
torch.cuda.0.free: 21.7
torch.cuda.0.total: 22.0
torch.cuda.0.capability: 8.6
llama_cpp_python.version: 0.2.79
llama_cpp_python.supports_gpu_offload: False
```




Discussion:
According to @alimaredia this will be resolved by https://github.com/instructlab/eval/issues/50 which should add GGUF support for all benchmarks 
Additionally, I've noticed that the default is a path to a GGUF file, but it seems Evaluation is expecting either a local directory or a HuggingFace repo



Patch:
diff --git a/requirements.txt b/requirements.txt
index 0c12ebb778..5576b64f35 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -6,7 +6,7 @@ datasets>=2.18.0
 gguf>=0.6.0
 GitPython>=3.1.42
 httpx>=0.25.0
-instructlab-eval>=0.1.1
+instructlab-eval>=0.2.1
 instructlab-quantize>=0.1.0
 instructlab-schema>=0.4.0
 instructlab-sdg>=0.3.0
diff --git a/src/instructlab/configuration.py b/src/instructlab/configuration.py
index 9e98b200a2..5bbb5ef440 100644
--- a/src/instructlab/configuration.py
+++ b/src/instructlab/configuration.py
@@ -282,7 +282,7 @@ class _mtbench(BaseModel):
 
     judge_model: str = Field(
         default_factory=lambda: DEFAULTS.JUDGE_MODEL_MT,
-        description="Directory where model to be used as judge is stored.",
+        description="Judge model for mt_bench and mt_bench_branch.",
     )
     output_dir: str = Field(
         default_factory=lambda: DEFAULTS.EVAL_DATA_DIR,
@@ -319,11 +319,11 @@ class _evaluate(BaseModel):
     model_config = ConfigDict(extra="ignore", protected_namespaces=())
     model: Optional[str] = Field(
         default=None,
-        description="Model to be used for evaluation.",
+        description="Model to be evaluated",
     )
     base_model: str = Field(
         default=DEFAULTS.MODEL_REPO,
-        description="Directory where model to be used for evaluation is stored.",
+        description="Base model to compare with 'model' for mt_bench_branch and mmlu_branch.",
     )
     branch: Optional[str] = Field(
         default=None,
diff --git a/src/instructlab/model/evaluate.py b/src/instructlab/model/evaluate.py
index e1513eab33..466b35ee09 100644
--- a/src/instructlab/model/evaluate.py
+++ b/src/instructlab/model/evaluate.py
@@ -83,23 +83,9 @@ def get_evaluator(
                 fg="red",
             )
             raise click.exceptions.Exit(1)
-        if os.path.exists(model):
-            model = pathlib.Path(model)
-            valid_model = False
-            if model.is_dir():
-                valid_model = backends.is_model_safetensors(model)
-            elif model.is_file():
-                valid_model = backends.is_model_gguf(model)
-            if not valid_model:
-                click.secho(
-                    "MTBench and MTBenchBranch need to be passed either a safetensors directory or a GGUF file",
-                    fg="red",
-                )
-                raise click.exceptions.Exit(1)
-            click.secho(
-                f"Using local model found at '{model}' for '--model'",
-                fg="blue",
-            )
+
+        validate_model(model)
+        validate_model(judge_model, "--judge-model")
         if benchmark == Benchmark.MT_BENCH:
             # Third Party
             from instructlab.eval.mt_bench import MTBenchEvaluator
@@ -111,6 +97,9 @@ def get_evaluator(
                 max_workers,
                 merge_system_user_message=merge_system_user_message,
             )
+
+        validate_model(base_model, "--base-model")
+
         # Third Party
         from instructlab.eval.mt_bench import MTBenchBranchEvaluator
 
@@ -139,24 +128,8 @@ def get_evaluator(
                 fg="red",
             )
             raise click.exceptions.Exit(1)
-        # ensure user is passing full safetensors if they specify a local directory
-        # TODO: also allow GGUF once the following is resolved: https://github.com/instructlab/eval/issues/50
-        if os.path.isdir(model):
-            if not backends.is_model_safetensors(pathlib.Path(model)):
-                click.secho(
-                    "MMLU and MMLUBranch can currently only be used with a safetensors directory",
-                    fg="red",
-                )
-                raise click.exceptions.Exit(1)
-            click.secho(
-                f"Using local safetensors found at '{model}' for '--model'",
-                fg="blue",
-            )
-        else:
-            click.secho(
-                f"Using safetensors from Hugging Face repo '{model}' for '--model'",
-                fg="blue",
-            )
+
+        validate_model(model)
         if benchmark == Benchmark.MMLU:
             # Third Party
             from instructlab.eval.mmlu import MMLUEvaluator
@@ -175,6 +148,9 @@ def get_evaluator(
                     model, few_shots=few_shots, batch_size=batch_size
                 )
             return evaluator
+
+        validate_model(base_model, "--base-model")
+
         # Third Party
         from instructlab.eval.mmlu import MMLUBranchEvaluator
 
@@ -187,6 +163,32 @@ def get_evaluator(
         )
 
 
+def validate_model(model: str, model_arg: str = "--model"):
+    if os.path.exists(model):
+        model_path = pathlib.Path(model)
+        valid_model = False
+        if model_path.is_dir():
+            valid_model = backends.is_model_safetensors(model_path)
+        elif model_path.is_file():
+            valid_model = backends.is_model_gguf(model_path)
+        if not valid_model:
+            click.secho(
+                f"Evaluate '{model_arg}' needs to be passed either a safetensors directory or a GGUF file",
+                fg="red",
+            )
+            raise click.exceptions.Exit(1)
+        click.secho(
+            f"Using local model found at '{model_path}' for '{model_arg}'",
+            fg="blue",
+        )
+    else:
+        click.secho(
+            f"Model could not be found at '{model}' for '{model_arg}'",
+            fg="red",
+        )
+        raise click.exceptions.Exit(1)
+
+
 def sort_score(pairing: tuple[str, float]) -> float:
     """helper func for display_branch_eval_summary
     takes a tuple pairing and returns just the score
@@ -283,7 +285,7 @@ def launch_server(
     ctx: click.Context,
     model: str,
     model_name: str,
-    max_workers: int,
+    max_workers: int | None,
     gpus: int | None,
     backend: str | None,
     enable_serving_output: bool,
@@ -325,20 +327,21 @@ def launch_server(
                 )
         effective_gpus = effective_gpus or 1
 
-        # Recommend max-workers based on hardware configuration: min(#GPUs being used * 10, #CPU cores) +- 50%
-        # Edge cases:
-        # - Many GPUs, not many CPUs: Unlikely, workers might not be able to keep the GPUs busy but recommendation can be ignored.
-        # - Many CPUs, not many GPUs: More likely, 10 workers per GPU should still be reasonable.
-        target_max_workers = min(effective_gpus * 10, get_cpu_count())
-        recommended_min_workers = max(target_max_workers // 2, 1)
-        recommended_max_workers = max(int(target_max_workers // 0.5), 1)
-        if (
-            max_workers > recommended_max_workers
-            or max_workers < recommended_min_workers
-        ):
-            logger.warning(
-                f"Based on your hardware configuration, when using vLLM, we recommend setting max-workers between {recommended_min_workers} and {recommended_max_workers} for optimal performance"
-            )
+        if max_workers is not None:
+            # Recommend max-workers based on hardware configuration: min(#GPUs being used * 10, #CPU cores) +- 50%
+            # Edge cases:
+            # - Many GPUs, not many CPUs: Unlikely, workers might not be able to keep the GPUs busy but recommendation can be ignored.
+            # - Many CPUs, not many GPUs: More likely, 10 workers per GPU should still be reasonable.
+            target_max_workers = min(effective_gpus * 10, get_cpu_count())
+            recommended_min_workers = max(target_max_workers // 2, 1)
+            recommended_max_workers = max(int(target_max_workers // 0.5), 1)
+            if (
+                max_workers > recommended_max_workers
+                or max_workers < recommended_min_workers
+            ):
+                logger.warning(
+                    f"Based on your hardware configuration, when using vLLM, we recommend setting max-workers between {recommended_min_workers} and {recommended_max_workers} for optimal performance"
+                )
     elif backend == backends.LLAMA_CPP:
         if ctx.obj.config.serve.llama_cpp.max_ctx_size < 5120:
             eval_serve.llama_cpp.max_ctx_size = 5120
@@ -346,11 +349,12 @@ def launch_server(
                 "Evaluate requires a context size of >= 5120, ignoring serve configuration for max_ctx_size"
             )
         # llama-cpp fails fast on too many incoming requests and returns errors to client
-        recommended_workers = max(get_cpu_count() // 2, 1)
-        if max_workers > recommended_workers:
-            logger.warning(
-                f"Based on your hardware configuration, when using llama-cpp, we recommend setting max-workers to a maximum of {recommended_workers}"
-            )
+        if max_workers is not None:
+            recommended_workers = max(get_cpu_count() // 2, 1)
+            if max_workers > recommended_workers:
+                logger.warning(
+                    f"Based on your hardware configuration, when using llama-cpp, we recommend setting max-workers to a maximum of {recommended_workers}"
+                )
         if gpus:
             logger.debug("Ignoring --gpus option for llama-cpp serving")
 
@@ -691,7 +695,21 @@ def evaluate(
             display_error_rate((error_rate + base_error_rate) / 2)
 
         elif benchmark == Benchmark.MMLU:
-            overall_score, individual_scores = evaluator.run()
+            server = None
+            try:
+                server, api_base = launch_server(
+                    ctx,
+                    model,
+                    model,
+                    None,
+                    gpus,
+                    backend,
+                    enable_serving_output,
+                )
+                overall_score, individual_scores = evaluator.run(api_base)
+            finally:
+                if server is not None:
+                    server.shutdown()
 
             print("# KNOWLEDGE EVALUATION REPORT")
             display_model(model)
@@ -721,9 +739,23 @@ def evaluate(
             overall_scores = []
             individual_scores_list = []
             for evaluator in evaluators:
-                overall_score, individual_scores = evaluator.run()
-                overall_scores.append(overall_score)
-                individual_scores_list.append(individual_scores)
+                server = None
+                try:
+                    server, api_base = launch_server(
+                        ctx,
+                        model,
+                        model,
+                        None,
+                        gpus,
+                        backend,
+                        enable_serving_output,
+                    )
+                    overall_score, individual_scores = evaluator.run(api_base)
+                    overall_scores.append(overall_score)
+                    individual_scores_list.append(individual_scores)
+                finally:
+                    if server is not None:
+                        server.shutdown()
 
             overall_score = overall_scores[0]
             base_overall_score = overall_scores[1]


Test patch:
diff --git a/tests/test_lab_evaluate.py b/tests/test_lab_evaluate.py
index f14f7de2be..4438c766e7 100644
--- a/tests/test_lab_evaluate.py
+++ b/tests/test_lab_evaluate.py
@@ -102,6 +102,8 @@ def run_mt_bench(cli_runner, error_rate):
             "instructlab/merlinite-7b-lab",
         ],
     )
+    if result.exit_code != 0:
+        print(result.output)
     assert result.exit_code == 0
     expected = textwrap.dedent(
         """\
@@ -156,6 +158,8 @@ def run_mt_bench_branch(cli_runner, error_rate):
             "taxonomy",
         ],
     )
+    if result.exit_code != 0:
+        print(result.output)
     assert result.exit_code == 0
     expected = textwrap.dedent(
         """\
@@ -197,6 +201,7 @@ def run_mt_bench_branch(cli_runner, error_rate):
     assert result.output == expected
 
 
+@patch("instructlab.model.evaluate.validate_model")
 @patch(
     "instructlab.model.evaluate.launch_server",
     return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
@@ -213,18 +218,22 @@ def test_evaluate_mt_bench(
     judge_answers_mock,
     gen_answers_mock,
     launch_server_mock,
+    validate_model_mock,
     cli_runner: CliRunner,
 ):
     run_mt_bench(cli_runner, 0)
+    assert validate_model_mock.call_count == 2
     judge_answers_mock.assert_called_once()
     gen_answers_mock.assert_called_once()
     assert launch_server_mock.call_count == 2
     run_mt_bench(cli_runner, 0.768)
+    assert validate_model_mock.call_count == 4
     assert judge_answers_mock.call_count == 2
     assert gen_answers_mock.call_count == 2
     assert launch_server_mock.call_count == 4
 
 
+@patch("instructlab.model.evaluate.validate_model")
 @patch(
     "instructlab.model.evaluate.launch_server",
     return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
@@ -243,23 +252,33 @@ def test_evaluate_mt_bench_branch(
     judge_answers_mock,
     gen_answers_mock,
     launch_server_mock,
+    validate_model_mock,
     cli_runner: CliRunner,
 ):
     run_mt_bench_branch(cli_runner, 0)
+    assert validate_model_mock.call_count == 3
     assert judge_answers_mock.call_count == 2
     assert gen_answers_mock.call_count == 2
     assert launch_server_mock.call_count == 3
     run_mt_bench_branch(cli_runner, 0.4567)
+    assert validate_model_mock.call_count == 6
     assert judge_answers_mock.call_count == 4
     assert gen_answers_mock.call_count == 4
     assert launch_server_mock.call_count == 6
 
 
+@patch("instructlab.model.evaluate.validate_model")
+@patch(
+    "instructlab.model.evaluate.launch_server",
+    return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
+)
 @patch(
     "instructlab.eval.mmlu.MMLUEvaluator.run",
     return_value=(0.5, {"task1": {"score": 0.1}, "task2": {"score": 0.9}}),
 )
-def test_evaluate_mmlu(run_mock, cli_runner: CliRunner):
+def test_evaluate_mmlu(
+    run_mock, launch_server_mock, validate_model_mock, cli_runner: CliRunner
+):
     result = cli_runner.invoke(
         lab.ilab,
         [
@@ -272,11 +291,12 @@ def test_evaluate_mmlu(run_mock, cli_runner: CliRunner):
             "instructlab/granite-7b-lab",
         ],
     )
+    validate_model_mock.assert_called_once()
+    launch_server_mock.assert_called_once()
     run_mock.assert_called_once()
     assert result.exit_code == 0
     expected = textwrap.dedent(
         """\
-        Using safetensors from Hugging Face repo 'instructlab/granite-7b-lab' for '--model'
         # KNOWLEDGE EVALUATION REPORT
 
         ## MODEL
@@ -293,6 +313,11 @@ def test_evaluate_mmlu(run_mock, cli_runner: CliRunner):
     assert result.output == expected
 
 
+@patch("instructlab.model.evaluate.validate_model")
+@patch(
+    "instructlab.model.evaluate.launch_server",
+    return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
+)
 @patch(
     "instructlab.eval.mmlu.MMLUBranchEvaluator.run",
     side_effect=[
@@ -300,7 +325,9 @@ def test_evaluate_mmlu(run_mock, cli_runner: CliRunner):
         (0.6, gen_individual_scores(False)),
     ],
 )
-def test_evaluate_mmlu_branch(run_mock, cli_runner: CliRunner):
+def test_evaluate_mmlu_branch(
+    run_mock, launch_server_mock, validate_model_mock, cli_runner: CliRunner
+):
     result = cli_runner.invoke(
         lab.ilab,
         [
@@ -317,11 +344,12 @@ def test_evaluate_mmlu_branch(run_mock, cli_runner: CliRunner):
             "generated",
         ],
     )
+    assert validate_model_mock.call_count == 2
+    assert launch_server_mock.call_count == 2
     assert run_mock.call_count == 2
     assert result.exit_code == 0
     expected = textwrap.dedent(
         """\
-        Using safetensors from Hugging Face repo 'instructlab/granite-7b-lab' for '--model'
         # KNOWLEDGE EVALUATION REPORT
 
         ## BASE MODEL
@@ -348,7 +376,8 @@ def test_evaluate_mmlu_branch(run_mock, cli_runner: CliRunner):
     assert result.output == expected
 
 
-def test_no_model_mt_bench(cli_runner: CliRunner):
+@patch("instructlab.model.evaluate.validate_model")
+def test_no_model_mt_bench(_, cli_runner: CliRunner):
     result = cli_runner.invoke(
         lab.ilab,
         [
@@ -394,7 +423,7 @@ def test_invalid_model_mt_bench(cli_runner: CliRunner):
             "instructlab/merlinite-7b-lab",
         ],
     )
-    assert "Failed to determine backend:" in result.output
+    assert "Model could not be found at 'invalid'" in result.output
     assert result.exit_code != 0
 
 
@@ -411,15 +440,16 @@ def test_invalid_model_mmlu(cli_runner: CliRunner):
             "invalid",
         ],
     )
-    assert "Model could not be found" in result.output
+    assert "Model could not be found at 'invalid'" in result.output
     assert result.exit_code != 0
 
 
+@patch("instructlab.model.evaluate.validate_model")
 @patch(
     "instructlab.eval.mmlu.MMLUEvaluator",
     side_effect=Exception("Exiting to check call_args"),
 )
-def test_int_batchsize_mmlu(mmlu_mock, cli_runner: CliRunner):
+def test_int_batchsize_mmlu(mmlu_mock, _, cli_runner: CliRunner):
     cli_runner.invoke(
         lab.ilab,
         [
@@ -437,11 +467,12 @@ def test_int_batchsize_mmlu(mmlu_mock, cli_runner: CliRunner):
     assert mmlu_mock.call_args_list[0][1]["batch_size"] == 1
 
 
+@patch("instructlab.model.evaluate.validate_model")
 @patch(
     "instructlab.model.evaluate.launch_server",
     return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
 )
-def test_invalid_taxonomy_mt_bench_branch(launch_server_mock, cli_runner: CliRunner):
+def test_invalid_taxonomy_mt_bench_branch(launch_server_mock, _, cli_runner: CliRunner):
     result = cli_runner.invoke(
         lab.ilab,
         [
@@ -469,11 +500,12 @@ def test_invalid_taxonomy_mt_bench_branch(launch_server_mock, cli_runner: CliRun
     assert result.exit_code != 0
 
 
+@patch("instructlab.model.evaluate.validate_model")
 @patch(
     "instructlab.model.evaluate.launch_server",
     return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
 )
-def test_invalid_branch_mt_bench_branch(launch_server_mock, cli_runner: CliRunner):
+def test_invalid_branch_mt_bench_branch(launch_server_mock, _, cli_runner: CliRunner):
     setup_taxonomy("taxonomy")
     result = cli_runner.invoke(
         lab.ilab,
@@ -502,7 +534,12 @@ def test_invalid_branch_mt_bench_branch(launch_server_mock, cli_runner: CliRunne
     assert result.exit_code != 0
 
 
-def test_invalid_tasks_dir(cli_runner: CliRunner):
+@patch("instructlab.model.evaluate.validate_model")
+@patch(
+    "instructlab.model.evaluate.launch_server",
+    return_value=(mock.MagicMock(), "http://127.0.0.1:8000/v1"),
+)
+def test_invalid_tasks_dir(_, __, cli_runner: CliRunner):
     result = cli_runner.invoke(
         lab.ilab,
         [
@@ -539,7 +576,7 @@ def test_invalid_model_path_mmlu(cli_runner: CliRunner, tmp_path):
         ],
     )
     assert (
-        "MMLU and MMLUBranch can currently only be used with a safetensors directory"
+        "Evaluate '--model' needs to be passed either a safetensors directory or a GGUF file"
         in result.output
     )
     assert result.exit_code != 0
@@ -561,13 +598,14 @@ def test_invalid_model_path_mt_bench(cli_runner: CliRunner, tmp_path):
         ],
     )
     assert (
-        "MTBench and MTBenchBranch need to be passed either a safetensors directory or a GGUF file"
+        "Evaluate '--model' needs to be passed either a safetensors directory or a GGUF file"
         in result.output
     )
     assert result.exit_code != 0
 
 
-def test_vllm_args_null(cli_runner: CliRunner):
+@patch("instructlab.model.evaluate.validate_model")
+def test_vllm_args_null(_, cli_runner: CliRunner):
     fname = common.setup_gpus_config(section_path="serve", vllm_args=lambda: None)
     args = common.vllm_setup_test(
         cli_runner,
diff --git a/tests/testdata/default_config.yaml b/tests/testdata/default_config.yaml
index 80e007271e..db2eab6358 100644
--- a/tests/testdata/default_config.yaml
+++ b/tests/testdata/default_config.yaml
@@ -32,7 +32,7 @@ evaluate:
   # Base taxonomy branch
   # Default: None
   base_branch:
-  # Directory where model to be used for evaluation is stored.
+  # Base model to compare with 'model' for mt_bench_branch and mmlu_branch.
   # Default: instructlab/granite-7b-lab
   base_model: instructlab/granite-7b-lab
   # Taxonomy branch containing custom skills/knowledge that should be used for
@@ -58,12 +58,12 @@ evaluate:
     # Directory where custom MMLU tasks are stored.
     # Default: /data/instructlab/datasets
     tasks_dir: /data/instructlab/datasets
-  # Model to be used for evaluation.
+  # Model to be evaluated
   # Default: None
   model:
   # Multi-turn benchmarking settings for skills.
   mt_bench:
-    # Directory where model to be used as judge is stored.
+    # Judge model for mt_bench and mt_bench_branch.
     # Default: prometheus-eval/prometheus-8x7b-v2.0
     judge_model: prometheus-eval/prometheus-8x7b-v2.0
     # Number of workers to use for evaluation.