tutorials/llm/evaluation/simple-evals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "31a4a82d-d3a3-4d6c-b7dc-9255b89fdacf",
   "metadata": {},
   "source": [
    "# Evaluating a NeMo checkpoint with simple-evals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a24f252f-b967-4ce0-a2c6-151d6e92a714",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to extend the evaluation capabilities within the NeMo Framework container.\n",
    "It guides you through installing an additional evaluation harness and various methods for specifying benchmarks.\n",
    "\n",
    "For a better understanding of in-framework deployment and the distinction between completion and chat endpoints, please refer to the tutorial on [\"Evaluating a NeMo checkpoint with lm-eval\"](mmlu.ipynb) first.\n",
    "\n",
    "In this tutorial, we will evaluate an LLM on the [HumanEval benchmark](https://arxiv.org/abs/2107.03374) available in [NVIDIA Evals Factory simple-evals](https://pypi.org/project/nvidia-simple-evals/) package.\n",
    "HumanEval comprises 164 manually created Python programming problems, each defined by a function signature and a docstring explaining its purpose.\n",
    "This benchmark assesses the functional correctness of generated code by comparing it against unit tests, rather than relying on textual similarity to a reference solution.\n",
    "\n",
    "We will utilize the chat variant of the benchmark, specifically designed for evaluating the coding abilities of instruction-tuned (chat) models.\n",
    "\n",
    "> Note: It is recommended to run this notebook within a [NeMo Framework container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo), as it includes all necessary dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e33f164-bff3-4f8f-87e1-7375a2ce48c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "\n",
    "import json\n",
    "import requests\n",
    "import signal\n",
    "import subprocess\n",
    "\n",
    "import core_evals\n",
    "from nemo.collections.llm import api\n",
    "from nemo.collections.llm.evaluation.api import EvaluationConfig, EvaluationTarget\n",
    "from nemo.collections.llm.evaluation.base import list_available_evaluations, find_framework\n",
    "from nemo.utils import logging\n",
    "\n",
    "logging.setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf8a9827-dca0-48de-8d6a-e811330e3b4a",
   "metadata": {},
   "source": [
    "## 1. Adding evaluation harness"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cc126c1-4c4a-48ef-96a5-2cd3f66c8284",
   "metadata": {},
   "source": [
    "Let's begin by exploring the available evaluations.\n",
    "\n",
    "First, we'll examine the benchmarks that come pre-installed with the [NeMo Framework container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo).\n",
    "\n",
    "The function `list_available_evaluations` identifies all tasks across all installed evaluation frameworks.\n",
    "Initially, it will only display `lm_evaluation_harness`.\n",
    "\n",
    "We can also use the `find_framework` function to locate the framework defining a specified task.\n",
    "Note that by default, it can find `mmlu` but cannot locate a framework for executing `humaneval`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78f70e4c-5a6c-4b56-8e62-484f584e5f32",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"frameworks:\", list(list_available_evaluations()))\n",
    "for task in (\"mmlu\", \"humaneval\"):\n",
    "    try:\n",
    "        print(f\"{task} found in {find_framework(task)}\")\n",
    "    except Exception as e:\n",
    "        print(\"Error:\", e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a708d8b-bbae-40ec-a388-77cd57d4863a",
   "metadata": {},
   "source": [
    "Now, we will install an additional evaluation framework: [NVIDIA Evals Factory simple-evals](https://pypi.org/project/nvidia-simple-evals/).\n",
    "\n",
    "It can be added by simply installing the package using `pip`.\n",
    "\n",
    "We will then reload the `core_evals` module to ensure the changes are reflected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fe7b577-45bf-4a98-bb65-716e89cba4ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -q nvidia-simple-evals\n",
    "importlib.reload(core_evals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84071b3-809d-4eed-9081-ae3dc6bbafb6",
   "metadata": {},
   "source": [
    "If we repeat the same checks as before, we can now see the newly installed framework and find an implementation for the `humaneval` task.\n",
    "\n",
    "At the same time, since both `lm-evaluation-harness` and `simple-evals` implement `mmlu`, we need to specify the version of this task if we want to execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2142d574-8bed-4298-af43-cec92ed82658",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"frameworks:\", list(list_available_evaluations()))\n",
    "for task in (\"mmlu\", \"humaneval\"):\n",
    "    try:\n",
    "        print(f\"{task} found in {find_framework(task)}\")\n",
    "    except Exception as e:\n",
    "        print(\"Error:\", e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dcb5525-30ff-4a4e-89a3-988f6b962f73",
   "metadata": {},
   "source": [
    "## 2. Deploying the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29cb5b17-b71f-478a-bafb-aeecbc816772",
   "metadata": {},
   "source": [
    "We are now ready to deploy and evaluate the model.\n",
    "\n",
    "First, you need to prepare a NeMo 2 checkpoint of the model you would like to evaluate.\n",
    "For the purpose of this tutorial, we will use the Llama 3.2 1B Instruct checkpoint, which you can download from the [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/llama-3_2-1b-instruct).\n",
    "Ensure that you mount the directory containing the checkpoint when starting the container.\n",
    "In this tutorial, we assume that the checkpoint is accessible under the path `\"/checkpoints/llama-3_2-1b-instruct_v2.0\"`.\n",
    "\n",
    "> Note: You can learn more about deployment and available server endpoints from the [\"Evaluating a NeMo checkpoint with lm-eval\"](mmlu.ipynb) tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf964980-69ba-447d-a6d8-1412726c768a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# modify this variable to point to your checkpoint\n",
    "CHECKPOINT_PATH = \"/checkpoints/llama-3_2-1b-instruct_v2.0\"\n",
    "\n",
    "# if you are not using NeMo FW container, modify this path to point to scripts directory\n",
    "SCRIPTS_PATH = \"/opt/NeMo/scripts\"\n",
    "\n",
    "# modify this path if you would like to save results in a different directory\n",
    "WORKSPACE = \"/workspace\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dca87531-5e91-4857-a06f-b2cac4b6f61c",
   "metadata": {},
   "outputs": [],
   "source": [
    "deploy_script = f\"{SCRIPTS_PATH}/deploy/nlp/deploy_in_fw_oai_server_eval.py\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1ae4669-6218-47d9-9a02-70c24fbb25d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "deploy_process = subprocess.Popen(\n",
    "    [\"python\", deploy_script, \"--nemo_checkpoint\", CHECKPOINT_PATH, \"--max_input_len\", \"8192\"], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4176cf73-706e-4c8f-8400-59d73c2495de",
   "metadata": {},
   "source": [
    "## 3. Evaluating the chat endpoint on HumanEval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b605b434-3428-406f-9284-340c1cc570a0",
   "metadata": {},
   "source": [
    "Once the model is deployed, we can evaluate it.\n",
    "\n",
    "Simple-evals provides a \"chat\" variant of the HumanEval benchmark, so we will send queries to the URL `\"http://0.0.0.0:8886/v1/chat/completions/\"`.\n",
    "\n",
    "To learn more about the difference between \"completions\" and \"chat\" benchmarks, see the tutorial on [\"Evaluating a NeMo checkpoint with lm-eval\"](mmlu.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "250733b3-fbe2-4da3-bb18-6f357331c241",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"triton_model\"\n",
    "chat_url = \"http://0.0.0.0:8886/v1/chat/completions/\"\n",
    "\n",
    "target_config = EvaluationTarget(api_endpoint={\"url\": chat_url, \"type\": \"chat\"})\n",
    "eval_config = EvaluationConfig(\n",
    "    type=\"humaneval\",\n",
    "    output_dir=f\"{WORKSPACE}/humaneval\",\n",
    ")\n",
    "\n",
    "results = api.evaluate(target_cfg=target_config, eval_cfg=eval_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0fd7fa4-9419-491d-ae1f-b4f197418279",
   "metadata": {},
   "source": [
    "When the job finishes, we can close the server and inspect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e55c94d0-f302-48d1-bb40-8cbdea323e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "deploy_process.send_signal(signal.SIGINT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e46cf9eb-db58-4e83-937f-5dca3259ff82",
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39bd207e-8b0b-411c-bc7e-41a9e365073e",
   "metadata": {},
   "source": [
    "We can also examine the artifacts produced by the evaluation job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83ac47db-f0e1-4eb5-a0c8-e37923544577",
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls {WORKSPACE}/humaneval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e706a9c0-f331-4db1-835f-618d79e29c5b",
   "metadata": {},
   "source": [
    "Inside the output directory, you can find a detailed report in HTML format: [humaneval.html](humaneval/humaneval.html).\n",
    "\n",
    "This report contains a summary of the metrics as well as the input-output pairs for all samples used in the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78efa6f2-4dc2-49bd-afa1-9d89e5f09fdf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}