Repository: EleutherAI/lm-evaluation-harness
Pull request: bugfix and docs for API
closes #2138 and also added the option to add a custom tokenizer path. Also added an explainer to alleviate confusion. 

Problem statement:
Bug when hashing prompt when using openai-chatcompletions and log_samples
When I use openai-chatcompletions with --apply_chat_template and log_samples, it encounters an error 
```
Traceback (most recent call last):
  File "/root/anaconda3/envs/dschat/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/evaluator.py", line 525, in evaluate
    "prompt_hash": hash_string(requests[0].arguments[0]),
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 36, in hash_string
    return hashlib.sha256(string.encode("utf-8")).hexdigest()
AttributeError: 'JsonChatStr' object has no attribute 'encode'
```
It happens in 
```
"prompt_hash": hash_string(requests[0].arguments[0])
```
It seems that hash_string only receives string input rather than a list of messages. But open-chatcompletions needs an list input.

Hints from issue discussion:
Sorry about that. The PR should fix it!

Commit message: Update API_guide.md

Patch to review:
diff --git a/README.md b/README.md
index 1fbec92d9b..2eb3b1775d 100644
--- a/README.md
+++ b/README.md
@@ -6,7 +6,7 @@
 
 *Latest News 📣*
 
-- [2024/07] API model support has been updated and refactored, introducing support for batched and async requests, and making it significantly easier to customize and use for your own purposes. **To run Llama 405B, we recommend using VLLM's OpenAI-compliant API to host the model, and use the `local-completions` model type to evaluate the model.**
+- [2024/07] [API model](docs/API_guide.md) support has been updated and refactored, introducing support for batched and async requests, and making it significantly easier to customize and use for your own purposes. **To run Llama 405B, we recommend using VLLM's OpenAI-compliant API to host the model, and use the `local-completions` model type to evaluate the model.**
 - [2024/07] New Open LLM Leaderboard tasks have been added ! You can find them under the [leaderboard](lm_eval/tasks/leaderboard/README.md) task group.
 
 ---
diff --git a/docs/API_guide.md b/docs/API_guide.md
new file mode 100644
index 0000000000..4699caa1d4
--- /dev/null
+++ b/docs/API_guide.md
@@ -0,0 +1,198 @@
+# TemplateAPI Usage Guide
+
+The `TemplateAPI` class is a versatile superclass designed to facilitate the integration of various API-based language models into the lm-evaluation-harness framework. This guide will explain how to use and extend the `TemplateAPI` class to implement your own API models. If your API implements the OpenAI API you can use the `local-completions` or the `local-chat-completions` (defined [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/openai_completions.py)) model types, which can also serve as examples of how to effectively subclass this template.
+
+## Overview
+
+The `TemplateAPI` class provides a template for creating API-based model implementations. It handles common functionalities such as:
+
+- Tokenization (optional)
+- Batch processing
+- Caching
+- Retrying failed requests
+- Parsing API responses
+
+To use this class, you typically need to subclass it and implement specific methods for your API.
+
+## Key Methods to Implement
+
+When subclassing `TemplateAPI`, you need to implement the following methods:
+
+1. `_create_payload`: Creates the JSON payload for API requests.
+2. `parse_logprobs`: Parses log probabilities from API responses.
+3. `parse_generations`: Parses generated text from API responses.
+4. `headers`: Returns the headers for the API request.
+
+You may also need to override other methods or properties depending on your API's specific requirements.
+
+> [!NOTE]
+> Currently logliklehood and MCQ based tasks (such as MMLU) are only supported for completion endpoints. Not for chat-completion — those that expect a list of dicts — endpoints! Completion APIs which support instruct tuned models can be evaluated with the `--apply_chat_template` option.
+
+# TemplateAPI Usage Guide
+
+## TemplateAPI Arguments
+
+When initializing a `TemplateAPI` instance or a subclass, you can provide several arguments to customize its behavior. Here's a detailed explanation of some important arguments:
+
+- `model` or `pretrained` (str):
+   - The name or identifier of the model to use.
+   - `model` takes precedence over `pretrained` when both are provided.
+
+- `base_url` (str):
+   - The base URL for the API endpoint.
+
+- `tokenizer` (str, optional):
+  - The name or path of the tokenizer to use.
+  - If not provided, it defaults to using the same tokenizer name as the model.
+
+- `num_concurrent` (int):
+   - Number of concurrent requests to make to the API.
+   - Useful for APIs that support parallel processing.
+   - Default is 1 (sequential processing).
+
+- `tokenized_requests` (bool):
+  - Determines whether the input is pre-tokenized. Defaults to `True`.
+  - Requests can be sent in either tokenized form (`list[list[int]]`) or as text (`list[str]`, or `str` for batch_size=1).
+  - For loglikelihood-based tasks, prompts require tokenization to calculate the context length. If `False` prompts are decoded back to text before being sent to the API.
+  - Not as important for `generate_until` tasks.
+  - Ignored for chat formatted inputs (list[dict...]) or if tokenizer_backend is None.
+
+- `tokenizer_backend` (str, optional):
+  - Required for loglikelihood-based or MCQ tasks.
+  - Specifies the tokenizer library to use. Options are "tiktoken", "huggingface", or None.
+  - Default is "huggingface".
+
+- `max_length` (int, optional):
+  - Maximum length of input + output.
+  - Default is 2048.
+
+- `max_retries` (int, optional):
+   - Maximum number of retries for failed API requests.
+   - Default is 3.
+
+- `max_gen_toks` (int, optional):
+  - Maximum number of tokens to generate in completion tasks.
+  - Default is 256 or set in task yaml.
+
+- `batch_size` (int or str, optional):
+  - Number of requests to batch together (if the API supports batching).
+  - Can be an integer or "auto" (which defaults to 1 for API models).
+  - Default is 1.
+
+- `seed` (int, optional):
+  - Random seed for reproducibility.
+  - Default is 1234.
+
+- `add_bos_token` (bool, optional):
+  - Whether to add the beginning-of-sequence token to inputs (when tokenizing).
+  - Default is False.
+
+- `custom_prefix_token_id` (int, optional):
+  - Custom token ID to use as a prefix for inputs.
+  - If not provided, uses the model's default BOS or EOS token (if `add_bos_token` is True).
+
+
+Example usage:
+
+```python
+class MyAPIModel(TemplateAPI):
+    def __init__(self, **kwargs):
+        super().__init__(
+            model="my-model",
+            base_url="https://api.mymodel.com/v1/completions",
+            tokenizer_backend="huggingface",
+            num_concurrent=5,
+            max_retries=5,
+            batch_size=10,
+            **kwargs
+        )
+
+    # Implement other required methods...
+```
+
+When subclassing `TemplateAPI`, you can override these arguments in your `__init__` method to set default values specific to your API. You can also add additional arguments as needed for your specific implementation.
+
+## Example Implementation: OpenAI API
+
+The `OpenAICompletionsAPI` and `OpenAIChatCompletion` ([here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/openai_completions.py) classes demonstrate how to implement API models using the `TemplateAPI` class. Here's a breakdown of the key components:
+
+### 1. Subclassing and Initialization
+
+```python
+@register_model("openai-completions")
+class OpenAICompletionsAPI(LocalCompletionsAPI):
+    def __init__(
+        self,
+        base_url="https://api.openai.com/v1/completions",
+        tokenizer_backend="tiktoken",
+        **kwargs,
+    ):
+        super().__init__(
+            base_url=base_url, tokenizer_backend=tokenizer_backend, **kwargs
+        )
+```
+
+### 2. Implementing API Key Retrieval
+
+```python
+@cached_property
+def api_key(self):
+    key = os.environ.get("OPENAI_API_KEY", None)
+    if key is None:
+        raise ValueError(
+            "API key not found. Please set the OPENAI_API_KEY environment variable."
+        )
+    return key
+```
+
+### 3. Creating the Payload
+
+```python
+def _create_payload(
+    self,
+    messages: Union[List[List[int]], List[dict], List[str], str],
+    generate=False,
+    gen_kwargs: Optional[dict] = None,
+    **kwargs,
+) -> dict:
+    if generate:
+        # ... (implementation for generation)
+    else:
+        # ... (implementation for log likelihood)
+```
+
+### 4. Parsing API Responses
+
+```python
+@staticmethod
+def parse_logprobs(
+    outputs: Union[Dict, List[Dict]],
+    tokens: List[List[int]] = None,
+    ctxlens: List[int] = None,
+    **kwargs,
+) -> List[Tuple[float, bool]]:
+    # ... (implementation)
+
+@staticmethod
+def parse_generations(outputs: Union[Dict, List[Dict]], **kwargs) -> List[str]:
+    # ... (implementation)
+```
+
+The requests are initiated in the `model_call` or the `amodel_call` methods.
+
+## Implementing Your Own API Model
+
+To implement your own API model:
+
+1. Subclass `TemplateAPI` or one of its subclasses (e.g., `LocalCompletionsAPI`).
+2. Override the `__init__` method if you need to set specific parameters.
+3. Implement the `_create_payload` and `header` methods to create the appropriate payload for your API.
+4. Implement the `parse_logprobs` and `parse_generations` methods to parse your API's responses.
+5. Override the `api_key` property if your API requires authentication.
+6. Override any other methods as necessary to match your API's behavior.
+
+## Best Practices
+
+1. Use the `@register_model` decorator to register your model with the framework (and import it in `lm_eval/models/__init__.py`!).
+3. Use environment variables for sensitive information like API keys.
+4. Properly handle batching and concurrent requests if supported by your API.
diff --git a/lm_eval/models/anthropic_llms.py b/lm_eval/models/anthropic_llms.py
index 81efbe902a..7b22b6a979 100644
--- a/lm_eval/models/anthropic_llms.py
+++ b/lm_eval/models/anthropic_llms.py
@@ -356,7 +356,7 @@ def tok_encode(
     ) -> List[str]:
         return [string]
 
-    def _loglikelihood_tokens(self, requests, **kwargs):
+    def loglikelihood(self, requests, **kwargs):
         raise NotImplementedError(
-            "Anthropic Chat Completions API does not support the return of log"
+            "Anthropic Chat Completions API does not support the return of loglikelihood"
         )
diff --git a/lm_eval/models/api_models.py b/lm_eval/models/api_models.py
index af7dae59d6..042b70eada 100644
--- a/lm_eval/models/api_models.py
+++ b/lm_eval/models/api_models.py
@@ -3,7 +3,6 @@
 import copy
 import itertools
 import json
-from collections import namedtuple
 from functools import cached_property
 from typing import (
     Any,
@@ -13,6 +12,7 @@
     Iterable,
     List,
     Literal,
+    NamedTuple,
     Optional,
     Tuple,
     Union,
@@ -38,7 +38,15 @@
 
 
 LogLikelihoodInputs = Tuple[Tuple[str, str], List[int], List[int]]
-JsonChatStr = namedtuple("JsonChatStr", ["prompt"])
+
+
+# utility class to keep track of json encoded chats
+class JsonChatStr(NamedTuple):
+    prompt: str
+
+    def encode(self, encoding):
+        return self.prompt.encode(encoding)
+
 
 eval_logger = utils.eval_logger
 
@@ -113,29 +121,37 @@ def __init__(
             self.tokenizer = None
             self.tokenized_requests = False
         else:
-            if self.tokenizer_backend == "huggingface":
+            if self.tokenizer is None:
+                if self.tokenizer_backend == "huggingface":
+                    import transformers
+
+                    self.tokenizer = transformers.AutoTokenizer.from_pretrained(
+                        self.tokenizer if self.tokenizer else self.model
+                    )
+                    # Not used as the API will handle padding but to mirror the behavior of the HFLM
+                    self.tokenizer = configure_pad_token(self.tokenizer)
+                elif self.tokenizer_backend == "tiktoken":
+                    try:
+                        import tiktoken
+
+                        self.tokenizer = tiktoken.encoding_for_model(self.model)
+                    except ModuleNotFoundError as e:
+                        raise Exception(
+                            "Attempted to use 'openai' LM type, but the package `tiktoken` is not installed. "
+                            "Please install it via `pip install lm-eval[api]` or `pip install -e .[api]`."
+                        ) from e
+                    if "openai" not in self.base_url:
+                        eval_logger.warning(
+                            f"Passed `base_url={self.base_url}` but using (OpenAI) Tiktoken tokenizer backend. "
+                            "Pass `tokenizer_backend=huggingface` and provide the HF tokenizer name if your model does not use Tiktoken."
+                        )
+            else:
                 import transformers
 
+                assert isinstance(tokenizer, str), "tokenizer must be a string"
                 self.tokenizer = transformers.AutoTokenizer.from_pretrained(
-                    self.tokenizer if self.tokenizer else self.model
+                    tokenizer,
                 )
-                # Not used as the API will handle padding but to mirror the behavior of the HFLM
-                self.tokenizer = configure_pad_token(self.tokenizer)
-            elif self.tokenizer_backend == "tiktoken":
-                try:
-                    import tiktoken
-
-                    self.tokenizer = tiktoken.encoding_for_model(self.model)
-                except ModuleNotFoundError as e:
-                    raise Exception(
-                        "Attempted to use 'openai' LM type, but the package `tiktoken` is not installed. "
-                        "Please install it via `pip install lm-eval[api]` or `pip install -e .[api]`."
-                    ) from e
-                if "openai" not in self.base_url:
-                    eval_logger.warning(
-                        f"Passed `base_url={self.base_url}` but using (OpenAI) Tiktoken tokenizer backend. "
-                        "Pass `tokenizer_backend=huggingface` and provide the HF tokenizer name if your model does not use Tiktoken."
-                    )
 
     @abc.abstractmethod
     def _create_payload(
diff --git a/lm_eval/models/openai_completions.py b/lm_eval/models/openai_completions.py
index c2e81c8737..492056f60e 100644
--- a/lm_eval/models/openai_completions.py
+++ b/lm_eval/models/openai_completions.py
@@ -145,7 +145,7 @@ def tok_encode(
     ) -> Union[List[str], List[int], Any]:
         return string
 
-    def _loglikelihood_tokens(self, requests, **kwargs):
+    def loglikelihood(self, requests, **kwargs):
         raise NotImplementedError(
             "Loglikelihood is not supported for chat completions. Consider using the completions API instead."
         )
@@ -175,14 +175,14 @@ def api_key(self):
             )
         return key
 
-    def _loglikelihood_tokens(self, requests, **kwargs):
+    def loglikelihood(self, requests, **kwargs):
         assert (
             self.model != "gpt-3.5-turbo"
         ), "Loglikelihood is not supported for gpt-3.5-turbo"
-        return super()._loglikelihood_tokens(requests, **kwargs)
+        return super().loglikelihood(requests, **kwargs)
 
 
-@register_model("openai-chatcompletions")
+@register_model("openai-chat-completions")
 class OpenAIChatCompletion(LocalChatCompletion):
     def __init__(
         self,


Code review comments:
diff_hunk: @@ -0,0 +1,198 @@
+# TemplateAPI Usage Guide
+
+The `TemplateAPI` class is a versatile superclass designed to facilitate the integration of various API-based language models into the lm-evaluation-harness framework. This guide will explain how to use and extend the `TemplateAPI` class to implement your own API models. If your API implements the OpenAI API you can use the `local-completions` or the `local-chat-completions` (defined [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/openai_completions.py)) model types, which can also serve as examples of how to effectively subclass this template.
+
+## Overview
+
+The `TemplateAPI` class provides a template for creating API-based model implementations. It handles common functionalities such as:
+
+- Tokenization (optional)
+- Batch processing
+- Caching
+- Retrying failed requests
+- Parsing API responses
+
+To use this class, you typically need to subclass it and implement specific methods for your API.
+
+## Key Methods to Implement
+
+When subclassing `TemplateAPI`, you need to implement the following methods:
+
+1. `_create_payload`: Creates the JSON payload for API requests.
+2. `parse_logprobs`: Parses log probabilities from API responses.
+3. `parse_generations`: Parses generated text from API responses.
+4. `headers`: Returns the headers for the API request.
+
+You may also need to override other methods or properties depending on your API's specific requirements.
+
+> [!NOTE]
+> Currently logliklehood and MCQ based tasks (such as MMLU) are only supported for completion endpoints. Not for chat-completion — those that expect a list of dicts — endpoints! Completion APIs which support instruct tuned models can be evaluated with the `--apply_chat_template` option.
path: docs/API_guide.md
text: @user1:
```suggestion
> Currently loglikelihood and MCQ based tasks (such as MMLU) are only supported for completion endpoints. Not for chat-completion — those that expect a list of dicts — endpoints! Completion APIs which support instruct tuned models can be evaluated with the `--apply_chat_template` option.
```