Repository: stanford-crfm/helm

Issue:
Switch AnthropicTokenizer to use client.beta.messages.count_tokens()
When `anthropic` is upgraded to 0.39.0 or newer, the following error happens:

```
src/helm/tokenizers/anthropic_tokenizer.py:26: error: "Anthropic" has no attribute "get_tokenizer"  [attr-defined]
```

This is because of [this breaking change](https://github.com/anthropics/anthropic-sdk-python/blob/93cbbbde964e244f02bf1bd2b579c5fabce4e267/CHANGELOG.md?plain=1#L97-L100):

```
### ⚠ BREAKING CHANGES

* **client:** remove legacy `client.count_tokens()` & `client.get_tokenizer()` methods ([#726](https://github.com/anthropics/anthropic-sdk-python/issues/726))
  * This functionality has been replaced by the `client.beta.messages.count_tokens()` API which supports newer models and all content functionality, such as images and PDFs.
```

The temporary workaround is to limit `anthropic` to versions before 0.39.0. The long term change is to limit `anthropic` to after 0.39.0 and switch `AnthropicTokenizer` to use `client.beta.messages.count_tokens()`.


Discussion:
Looks like this is infeasible because the new beta API does not support encoding and decoding. The much longer solution is to change the tokenizer framework allow tokenizers to only support length counting.
`# error log :
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/helm/tokenizers/anthropic_tokenizer.py", line 26, in _init_
tokenizer_object=anthropic.Anthropic().get_tokenizer()
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/client.py", line 122, in __init_
super()._init_(
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/base_client.py", line 856, in __init_
self._client = http_client or SyncHttpxClientWrapper(
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/base_client.py", line 754, in __init_
super()._init_(**kwargs)
TypeError: _init_() got an unexpected keyword argument 'proxies'`

I got the above error and upgraded to anthropic 0.40.0.
But then I got the issue on this page. 
`# error log:
  File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/helm/tokenizers/anthropic_tokenizer.py", line 26, in __init__
    tokenizer_object=anthropic.Anthropic().get_tokenizer()
AttributeError: 'Anthropic' object has no attribute 'get_tokenizer'`

So I downgraded to 0.39.0 again, but then I got the first issue again.
Is there a solution for this problem?
You need a version of `httpx` before the `proxies` argument was deleted. According to the [httpx CHANGELOG](https://github.com/encode/httpx/blob/9e8ab40369bd3ec2cc8bff37ab79bf5769c8b00f/CHANGELOG.md#0280-28th-november-2024) version 0.27.2 and before should work. Could you try running:

```python
pip install httpx==0.27.2
```
@yifanmai wow! Thank you so much. It solved the problem I was struggling with all day.



Patch:
diff --git a/setup.cfg b/setup.cfg
index 8dd1450eb7d..9983f77346d 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -137,7 +137,7 @@ amazon =
     botocore~=1.34.131
 
 anthropic =
-    anthropic~=0.17,<0.39  # TODO(#3212): Limit anthropic to >=0.39 after resolving #3212.
+    anthropic~=0.39
     websocket-client~=1.3.2  # For legacy stanford-online-all-v4-s3
     httpx<0.28.0  # TODO(#3324): Remove this tepmorary workaround
 
diff --git a/src/helm/config/tokenizer_configs.yaml b/src/helm/config/tokenizer_configs.yaml
index 6e648b18f54..a4a0f34fe19 100644
--- a/src/helm/config/tokenizer_configs.yaml
+++ b/src/helm/config/tokenizer_configs.yaml
@@ -86,7 +86,9 @@ tokenizer_configs:
   # Anthropic
   - name: anthropic/claude
     tokenizer_spec:
-      class_name: "helm.tokenizers.anthropic_tokenizer.AnthropicTokenizer"
+      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
+      args:
+        pretrained_model_name_or_path: Xenova/claude-tokenizer
     end_of_text_token: "
"
     prefix_token: "
"
 
diff --git a/src/helm/tokenizers/anthropic_tokenizer.py b/src/helm/tokenizers/anthropic_tokenizer.py
deleted file mode 100644
index d34cd841dd6..00000000000
--- a/src/helm/tokenizers/anthropic_tokenizer.py
+++ /dev/null
@@ -1,52 +0,0 @@
-from typing import Any, Dict
-
-import threading
-
-from helm.common.cache import CacheConfig
-from helm.common.optional_dependencies import handle_module_not_found_error
-from helm.tokenizers.caching_tokenizer import CachingTokenizer
-from transformers import PreTrainedTokenizerBase, PreTrainedTokenizerFast
-
-try:
-    import anthropic
-except ModuleNotFoundError as e:
-    handle_module_not_found_error(e, ["anthropic"])
-
-
-class AnthropicTokenizer(CachingTokenizer):
-    LOCK: threading.Lock = threading.Lock()
-    """Global lock for the Anthropic tokenizer.
-
-    The Anthropic tokenizer is a wrapper around a single global Hugging Face tokenizer, which is thread-hostile."""
-
-    def __init__(self, cache_config: CacheConfig) -> None:
-        super().__init__(cache_config)
-        with AnthropicTokenizer.LOCK:
-            self._tokenizer: PreTrainedTokenizerBase = PreTrainedTokenizerFast(
-                tokenizer_object=anthropic.Anthropic().get_tokenizer()
-            )
-
-    def _tokenize_do_it(self, request: Dict[str, Any]) -> Dict[str, Any]:
-        if request["encode"]:
-            if request["truncation"]:
-                with AnthropicTokenizer.LOCK:
-                    tokens = self._tokenizer.encode(
-                        request["text"],
-                        truncation=request["truncation"],
-                        max_length=request["max_length"],
-                        add_special_tokens=False,
-                    )
-            else:
-                with AnthropicTokenizer.LOCK:
-                    tokens = self._tokenizer.encode(request["text"], add_special_tokens=False)
-        else:
-            # No encoding, just return the token strings
-            tokens = [self._tokenizer.convert_tokens_to_string([i]) for i in self._tokenizer.tokenize(request["text"])]
-        return {"tokens": tokens}
-
-    def _decode_do_it(self, request: Dict[str, Any]) -> Dict[str, Any]:
-        with AnthropicTokenizer.LOCK:
-            text = self._tokenizer.decode(
-                request["tokens"], clean_up_tokenization_spaces=request["clean_up_tokenization_spaces"]
-            )
-        return {"text": text}
diff --git a/src/helm/tokenizers/caching_tokenizer.py b/src/helm/tokenizers/caching_tokenizer.py
index 1d325e1dfdf..9826aed03c2 100644
--- a/src/helm/tokenizers/caching_tokenizer.py
+++ b/src/helm/tokenizers/caching_tokenizer.py
@@ -1,6 +1,6 @@
 from abc import abstractmethod
 from dataclasses import asdict
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List
 
 from helm.common.cache import Cache, CacheConfig
 from helm.common.request import wrap_request_time
@@ -153,31 +153,3 @@ def decode(self, request: DecodeRequest) -> DecodeRequestResult:
             )
         except Exception as error:
             raise ValueError(f"Failed to decode tokens with {self.__class__.__name__} tokenizer: {error}") from error
-
-
-def cleanup_str(token: str, tokenizer_name: Optional[str] = None) -> str:
-    """
-    Certain tokenizers introduce special characters to represent spaces, such as
-    "Ġ" or "▁". This function removes those characters.
-    """
-    if tokenizer_name in [
-        "TsinghuaKEG/ice",
-        "bigscience/T0pp",
-        "google/t5-11b",
-        "google/flan-t5-xxl",
-        "google/ul2",
-        "Yandex/yalm",
-        "ai21/j1",
-        "together",
-    ]:
-        return token.replace("▁", " ")
-    elif tokenizer_name is not None and tokenizer_name.startswith("huggingface"):
-        return token.replace("Ġ", " ")
-    return token
-
-
-def cleanup_tokens(tokens: List[str], tokenizer_name: Optional[str] = None) -> List[str]:
-    """
-    Applies `cleanup_str` to each token in `tokens`.
-    """
-    return [cleanup_str(token, tokenizer_name) for token in tokens]
diff --git a/src/helm/tokenizers/tokenizer.py b/src/helm/tokenizers/tokenizer.py
index 72f193dc527..91fae41ef22 100644
--- a/src/helm/tokenizers/tokenizer.py
+++ b/src/helm/tokenizers/tokenizer.py
@@ -41,7 +41,9 @@ def cleanup_str(token: str, tokenizer_name: Optional[str] = None) -> str:
         "together",
     ]:
         return token.replace("▁", " ")
-    elif tokenizer_name is not None and tokenizer_name.startswith("huggingface"):
+    elif tokenizer_name is not None and (
+        tokenizer_name.startswith("huggingface") or tokenizer_name == "anthropic/claude"
+    ):
         return token.replace("Ġ", " ")
     return token
 


Test patch:
diff --git a/src/helm/tokenizers/test_anthropic_tokenizer.py b/src/helm/tokenizers/test_anthropic_tokenizer.py
index 79ca089b844..24d7386985d 100644
--- a/src/helm/tokenizers/test_anthropic_tokenizer.py
+++ b/src/helm/tokenizers/test_anthropic_tokenizer.py
@@ -10,7 +10,7 @@
     TokenizationRequest,
     TokenizationRequestResult,
 )
-from helm.tokenizers.anthropic_tokenizer import AnthropicTokenizer
+from helm.tokenizers.huggingface_tokenizer import HuggingFaceTokenizer
 
 
 class TestAnthropicTokenizer:
@@ -21,7 +21,11 @@ class TestAnthropicTokenizer:
     def setup_method(self, method):
         cache_file = tempfile.NamedTemporaryFile(delete=False)
         self.cache_path: str = cache_file.name
-        self.tokenizer = AnthropicTokenizer(SqliteCacheConfig(self.cache_path))
+        self.tokenizer = HuggingFaceTokenizer(
+            SqliteCacheConfig(self.cache_path),
+            tokenizer_name="anthropic/claude",
+            pretrained_model_name_or_path="Xenova/claude-tokenizer",
+        )
 
     def teardown_method(self, method):
         os.remove(self.cache_path)