Repository: opendatahub-io/vllm-tgis-adapter (language: python)

Issue:
Make ADD_SPECIAL_TOKENS true by default
Before this change the ADD_SPECIAL_TOKENS acted as a tristate variable where the default (not set) would be to add special tokens only if the model didn't have a chat template.

However, this this default broke existing integration tests so, after some deliberation,  the decision was made to make the default to be "true". Due to the way that environment variables work, this means that this setting now only has 2 states: true and false.

bug: tokenization in Tokenize and Generate does not match if ADD_SPECIAL_TOKENS=False
The usage of `add_special_tokens` in Generate is not replicated in Tokenize. This means that the tokenization result/count will be different if `ADD_SPECIAL_TOKENS=False`.

Let's make sure to add a unit test and integration test for it if applicable!


Discussion / hints:
## [Codecov](https://app.codecov.io/gh/opendatahub-io/vllm-tgis-adapter/pull/66?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatahub-io) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 61.36%. Comparing base [(`09290e6`)](https://app.codecov.io/gh/opendatahub-io/vllm-tgis-adapter/commit/09290e6791db9ea0365506903b7c42f0d0205f64?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatahub-io) to head [(`f960051`)](https://app.codecov.io/gh/opendatahub-io/vllm-tgis-adapter/commit/f9600512ef916565233f34f7d15c201433c06518?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatahub-io).
> Report is 4 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main      #66      +/-   ##
==========================================
+ Coverage   61.29%   61.36%   +0.07%     
==========================================
  Files          20       20              
  Lines        1186     1183       -3     
  Branches      210      209       -1     
==========================================
- Hits          727      726       -1     
+ Misses        383      382       -1     
+ Partials       76       75       -1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/opendatahub-io/vllm-tgis-adapter/pull/66?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatahub-io).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatahub-io).



Fix patch:
diff --git a/src/vllm_tgis_adapter/grpc/grpc_server.py b/src/vllm_tgis_adapter/grpc/grpc_server.py
index 1ee9bccc..99c981a5 100644
--- a/src/vllm_tgis_adapter/grpc/grpc_server.py
+++ b/src/vllm_tgis_adapter/grpc/grpc_server.py
@@ -855,7 +855,9 @@ async def Tokenize(
         # other threads
         for req in request.requests:
             batch_encoding = tokenizer.encode_plus(
-                text=req.text, return_offsets_mapping=request.return_offsets
+                text=req.text,
+                return_offsets_mapping=request.return_offsets,
+                add_special_tokens=ADD_SPECIAL_TOKENS,
             )
 
             # Tokenize the input text


Test patch:
diff --git a/tests/test_grpc_server.py b/tests/test_grpc_server.py
index e64431eb..a2b2d5ed 100644
--- a/tests/test_grpc_server.py
+++ b/tests/test_grpc_server.py
@@ -25,6 +25,14 @@ def test_generation_request(grpc_client):
     assert response.stop_reason is not None
 
 
+def test_tokenize_request(grpc_client):
+    response_tokenize = grpc_client.make_request_tokenize(
+        text="Please answer the following question.\nhow far is Paris from New York?",
+    )
+
+    assert response_tokenize.token_count
+
+
 def test_generation_request_stream(grpc_client):
     streaming_response = grpc_client.make_request_stream(
         "The answer to life the universe and everything is ",
diff --git a/tests/utils.py b/tests/utils.py
index e848cb38..6217b5e3 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -11,11 +11,13 @@
 
 from vllm_tgis_adapter.grpc.pb.generation_pb2 import (
     BatchedGenerationRequest,
+    BatchedTokenizeRequest,
     GenerationRequest,
     ModelInfoRequest,
     Parameters,
     SingleGenerationRequest,
     StoppingCriteria,
+    TokenizeRequest,
 )
 from vllm_tgis_adapter.grpc.pb.generation_pb2_grpc import GenerationServiceStub
 
@@ -25,6 +27,7 @@
     from vllm_tgis_adapter.grpc.pb.generation_pb2 import (
         GenerationResponse,
         ModelInfoResponse,
+        TokenizeResponse,
     )
 
 _T = TypeVar("_T")
@@ -173,6 +176,30 @@ def make_request_stream(
         except grpc._channel._MultiThreadedRendezvous as exc:  # noqa: SLF001
             raise RuntimeError(exc.details()) from exc
 
+    def make_request_tokenize(
+        self,
+        text: str | list[str],
+        model_id: str | None = None,
+        adapter_id: str | None = None,
+    ) -> TokenizeResponse | Sequence[TokenizeResponse]:
+        if single_request := isinstance(text, str):
+            text = [text]
+
+        request = BatchedTokenizeRequest(
+            model_id=model_id,
+            requests=[TokenizeRequest(text=piece) for piece in text],
+            adapter_id=adapter_id,
+        )
+
+        response = self.generation_service_stub.Tokenize(
+            request=request,
+        )
+
+        if single_request:
+            return response.responses[0]
+
+        return response.responses
+
     def __enter__(self):  # noqa: D105
         return self