Repository: NVIDIA/NeMo
Pull request #13191: Refine evaluation docs
Author: login: athitten
Created: 2025-04-22T23:10:23Z | Merged: 2025-04-24T18:42:21Z

> [!IMPORTANT]  
> The `Update branch` button must only be pressed in very rare occassions.
> An outdated branch is never blocking the merge of a PR.
> Please reach out to the automation team before pressing that button.

# What does this PR do ?

1) Incorporates suggestions on `docs/source/evaluation/evaluation_doc.rst` from #12902 
2) Adds `evaluation_doc.rst` to `docs/source/index.rst` so that it appears in the [NeMo FW user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html)

**Collection**: [Note which collection this PR will affect]

# Changelog

- Add specific line by line info of high level changes in this PR.

# Usage

- You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"

**Pre checks**:

- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:

- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed.
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information

- Related to # (issue)



Diff:
diff --git a/docs/source/evaluation/evaluation_doc.rst b/docs/source/evaluation/evaluation_doc.rst
index da1e900c8665..cc849698add1 100644
--- a/docs/source/evaluation/evaluation_doc.rst
+++ b/docs/source/evaluation/evaluation_doc.rst
@@ -3,14 +3,14 @@ Evaluate NeMo 2.0 Checkpoints
 
 This guide provides detailed instructions on evaluating NeMo 2.0 checkpoints using the `NVIDIA Evals Factory
 <https://pypi.org/project/nvidia-lm-eval/>`__ within the NeMo Framework. Supported benchmarks include:
-    * GPQA,
-    * GSM8K,
-    * IFEval,
-    * MGSM,
-    * MMLU,
-    * MMLU-Pro,
-    * MMLU-Redux, and
-    * Wikilingua.
+    * GPQA
+    * GSM8K
+    * IFEval
+    * MGSM
+    * MMLU
+    * MMLU-Pro
+    * MMLU-Redux
+    * Wikilingua
 
 
 Introduction
@@ -18,30 +18,30 @@ Introduction
 
 The evaluation process employs a server-client approach, comprising two main phases.
 In Phase 1, the NeMo 2.0 checkpoint is deployed in-framework on a PyTriton server by exposing
-OpenAI API (OAI) compatible endppoints. Both completions (`v1/completions`) and chat-completions
-(`v1/chat/completions`) endpoints are exposed enabling to evaluate on both completion and chat benchmarks.
+OpenAI API (OAI) compatible endpoints. Both completions (`v1/completions`) and chat-completions
+(`v1/chat/completions`) endpoints are exposed, enabling evaluation on both completion and chat benchmarks.
 Phase 2 involves running the evaluation on the model using the OAI endpoint and port.
 
-Some of the benchmarks (e.g. GPQA) use a gated dataset. In order to use them, you must authenticate to 
-`HuggingFace Hub <https://huggingface.co/docs/huggingface_hub/quick-start#authentication>`__
+Some of the benchmarks (e.g. GPQA) use a gated dataset. To use them, you must authenticate to the
+`Hugging Face Hub <https://huggingface.co/docs/huggingface_hub/quick-start#authentication>`__
 before launching the evaluation.
 
-NVIDIA Evals Factory provides following predefined configurations for evaluating completions endpoint:
-    * `gsm8k`,
-    * `mgsm`,
-    * `mmlu`,
-    * `mmlu_pro`, and
+The NVIDIA Evals Factory provides the following predefined configurations for evaluating the completions endpoint:
+    * `gsm8k`
+    * `mgsm`
+    * `mmlu`
+    * `mmlu_pro`
     * `mmlu_redux`
 
-...and the following configurations for evaluating the chat endpoint:
-    * `gpqa_diamond_cot`,
-    * `gsm8k_cot_instruct`,
-    * `ifeval`,
-    * `mgsm_cot`,
-    * `mmlu_instruct`,
-    * `mmlu_pro_instruct`,
-    * `mmlu_redux_instruct`, and
-    * `wikilingua`.
+It also provides the following configurations for evaluating the chat endpoint:
+    * `gpqa_diamond_cot`
+    * `gsm8k_cot_instruct`
+    * `ifeval`
+    * `mgsm_cot`
+    * `mmlu_instruct`
+    * `mmlu_pro_instruct`
+    * `mmlu_redux_instruct`
+    * `wikilingua`
 
 Run Evaluations without NeMo-Run
 ---------------------------------
@@ -83,11 +83,7 @@ the processes from being killed and aborting the runs.
         evaluate(target_cfg=eval_target, eval_cfg=eval_config)
 
 .. note::
-
-  Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument
-  options as these are just sample commands and don't share all arguments and their default settings. For more details
-  on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py
-  <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
+    Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument options as these are just sample commands and don't share all arguments and their default settings. For more details on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
 
 Run Evaluations with NeMo-Run
 ------------------------------
@@ -112,11 +108,10 @@ To run evaluations on your local workstation, use the following command:
 
 .. code-block:: bash
 
-    python scripts/llm/evaluation.py --nemo_checkpoint='/workspace/hf_llama3_8b_nemo2.nemo'
+    python scripts/llm/evaluation.py --nemo_checkpoint '/workspace/hf_llama3_8b_nemo2/' --eval_task 'gsm8k' --devices 2
 
 .. note::
-
-  When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
+    When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
 
 Run on Slurm-based Clusters
 ##########################
@@ -135,20 +130,20 @@ environments.
 
 
 
-Run Legacy Evaluations with `lm-evaluation-harness <https://github.com/EleutherAI/lm-evaluation-harness>`__
+Run Legacy Evaluations with lm-evaluation-harness
 -----------------------------------------------------------------------------------------------------------
 
 You can also run evaluations of NeMo 2.0 checkpoints using the integrated `lm-evaluation-harness
 <https://github.com/EleutherAI/lm-evaluation-harness>`__ within the NeMo Framework. Supported benchmarks include
 ``MMLU``, ``GSM8k``, ``lambada_openai``, ``winogrande``, ``arc_challenge``, ``arc_easy``, and ``copa``.
-Please note that this path is deprecated and will be removed in NeMo 25.06 release.
+Please note that this path is deprecated and will be removed in the NeMo Framework 25.06 release.
 
 The evaluation process employs a server-client approach, comprising two main phases. In Phase 1, the NeMo 2.0
 checkpoint is deployed on a PyTriton server by exporting it to TRT-LLM. Phase 2 involves running the evaluation
 on the model using the deployed URL and port.
 
 
-In order to deploy a model, use the following command. Make sure to pass ``backend="trtllm"``:
+To deploy a model, use the following command. Make sure to pass ``backend="trtllm"``:
 
 .. code-block:: python
 
@@ -164,8 +159,8 @@ In order to deploy a model, use the following command. Make sure to pass ``backe
 
 
 The ``evaluate`` method defined in ``nemo/collections/llm/api.py`` supports the legacy way of evaluating the models.
-To run evaluations on the deployed model, use the following command. Make sure to pass `nemo_checkpoint_path` and
-`url` parameters as they are needed to use the legacy evaluation code. Make sure to open a new terminal within the
+To run evaluations on the deployed model, use the following command. Make sure to pass the `nemo_checkpoint_path` and
+the `url` parameters as they are needed to use the legacy evaluation code. Make sure to open a new terminal within the
 same container to execute it. For longer evaluations, it is advisable to run both the deploy and evaluate commands
 in tmux sessions to prevent the processes from being killed and aborting the runs.
 
diff --git a/docs/source/index.rst b/docs/source/index.rst
index d015796096fc..8010caf9ae7e 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -65,6 +65,14 @@ For more information, browse the developer docs for your area of interest in the
 
    checkpoints/intro
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Evaluation
+   :name: evaluation
+   :titlesonly:
+
+   evaluation/evaluation_doc.rst
+
 .. toctree::
    :maxdepth: 1
    :caption: APIs


Reviews:
user: login: athitten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T02:18:06Z
user: login: athitten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T02:18:26Z
user: login: jgerh
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-24T17:32:17Z
user: login: jgerh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T17:38:47Z
user: login: jgerh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T17:44:01Z
user: login: athitten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:01:01Z
user: login: athitten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:01:32Z
user: login: chtruong814
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:41:54Z

Review comments:
diff_hunk: @@ -82,12 +82,7 @@ the processes from being killed and aborting the runs.
     if __name__ == "__main__":
         evaluate(target_cfg=eval_target, eval_cfg=eval_config)
 
-.. note::
-
-  Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument
-  options as these are just sample commands and don't share all arguments and their default settings. For more details
-  on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py
-  <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
+.. note:: Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument options as these are just sample commands and don't share all arguments and their default settings. For more details on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
path: docs/source/evaluation/evaluation_doc.rst
user: login: athitten
body: @jgerh it seems if we add note like below, the Note keyword does not appear.

.. note::
Please refer to deploy and evaluate method in nemo/collections/llm/api.py to check all the argument

So I changed to this and now the note keyword appears
.. note:: Please refer to deploy and evaluate method in nemo/collections/llm/api.py
created_at: 2025-04-24T02:18:06Z
author_association: COLLABORATOR
diff_hunk: @@ -112,11 +107,9 @@ To run evaluations on your local workstation, use the following command:
 
 .. code-block:: bash
 
-    python scripts/llm/evaluation.py --nemo_checkpoint='/workspace/hf_llama3_8b_nemo2.nemo'
-
-.. note::
+    python scripts/llm/evaluation.py --nemo_checkpoint '/workspace/hf_llama3_8b_nemo2/' --eval_task 'gsm8k' --devices 2
 
-  When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
+.. note:: When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
path: docs/source/evaluation/evaluation_doc.rst
user: login: athitten
body: Same comment as above for the note
created_at: 2025-04-24T02:18:26Z
author_association: COLLABORATOR
diff_hunk: @@ -112,11 +107,9 @@ To run evaluations on your local workstation, use the following command:
 
 .. code-block:: bash
 
-    python scripts/llm/evaluation.py --nemo_checkpoint='/workspace/hf_llama3_8b_nemo2.nemo'
-
-.. note::
+    python scripts/llm/evaluation.py --nemo_checkpoint '/workspace/hf_llama3_8b_nemo2/' --eval_task 'gsm8k' --devices 2
 
-  When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
+.. note:: When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
path: docs/source/evaluation/evaluation_doc.rst
user: login: jgerh
body: Please check the note syntax here: https://aschilling.gitlab-master-pages.nvidia.com/documentation/repo_docs/latest/rst-guide.html and correct the formatting.
created_at: 2025-04-24T17:38:47Z
author_association: COLLABORATOR
diff_hunk: @@ -82,12 +82,7 @@ the processes from being killed and aborting the runs.
     if __name__ == "__main__":
         evaluate(target_cfg=eval_target, eval_cfg=eval_config)
 
-.. note::
-
-  Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument
-  options as these are just sample commands and don't share all arguments and their default settings. For more details
-  on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py
-  <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
+.. note:: Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument options as these are just sample commands and don't share all arguments and their default settings. For more details on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
path: docs/source/evaluation/evaluation_doc.rst
user: login: jgerh
body: Please check the note syntax here: https://aschilling.gitlab-master-pages.nvidia.com/documentation/repo_docs/latest/rst-guide.html in the admonitions section.
created_at: 2025-04-24T17:44:01Z
author_association: COLLABORATOR
diff_hunk: @@ -82,12 +82,7 @@ the processes from being killed and aborting the runs.
     if __name__ == "__main__":
         evaluate(target_cfg=eval_target, eval_cfg=eval_config)
 
-.. note::
-
-  Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument
-  options as these are just sample commands and don't share all arguments and their default settings. For more details
-  on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py
-  <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
+.. note:: Please refer to ``deploy`` and ``evaluate`` method in ``nemo/collections/llm/api.py`` to check all the argument options as these are just sample commands and don't share all arguments and their default settings. For more details on arguments in the ApiEndpoint and ConfigParams classes for evaluation, refer to `nemo/collections/llm/evaluation/api.py <https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/evaluation/api.py>`__.
path: docs/source/evaluation/evaluation_doc.rst
user: login: athitten
body: Thanks, fixed and its now rendering correctly:
<img width="797" alt="Screenshot 2025-04-24 at 11 00 51 AM" src="https://github.com/user-attachments/assets/2c760cf2-c3e4-4daa-8292-39a4138e3c7b" />

created_at: 2025-04-24T18:01:01Z
author_association: COLLABORATOR
diff_hunk: @@ -112,11 +107,9 @@ To run evaluations on your local workstation, use the following command:
 
 .. code-block:: bash
 
-    python scripts/llm/evaluation.py --nemo_checkpoint='/workspace/hf_llama3_8b_nemo2.nemo'
-
-.. note::
+    python scripts/llm/evaluation.py --nemo_checkpoint '/workspace/hf_llama3_8b_nemo2/' --eval_task 'gsm8k' --devices 2
 
-  When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
+.. note:: When running locally with NeMo-Run, you will need to manually terminate the deploy process once evaluations are complete.
path: docs/source/evaluation/evaluation_doc.rst
user: login: athitten
body: Thanks, fixed and its now rendering correctly:
<img width="850" alt="Screenshot 2025-04-24 at 11 01 29 AM" src="https://github.com/user-attachments/assets/cfcc727d-61ca-492b-a32d-794fe4cf4843" />

created_at: 2025-04-24T18:01:32Z
author_association: COLLABORATOR

PR comments:
user: login: jgerh
created_at: 2025-04-24T18:04:10Z
body: Approved with changes.
author_association: COLLABORATOR
user: login: chtruong814
created_at: 2025-04-24T18:42:17Z
body: Fast merging since docs changes. It passes build and link checks.
author_association: COLLABORATOR