Repository: kserve/kserve

Issue:
Explicitly specify pad token id when generating tokens
/kind bug

**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]
We have added a [fallback pad token if it is not already present in the tokenizer](https://github.com/kserve/kserve/blob/11e0ab2ca4d852ec154156fad4f9d220c8768d78/python/huggingfaceserver/huggingfaceserver/model.py#L130-L131) as part of the PR #3459. But it does not explicitly specifies pad_token_id when invoking generate method which leads to huggingface using eos_token_id as the pad_token_id. The log from huggingface server is show below. To avoid this we should explicitly specify the pad_token_id
```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
```

**What did you expect to happen:**
pad_token_id should be set accordingly.

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]
```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-gpt2
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - "--model_name=gpt2"
        - "--model_id=openai-community/gpt2"
        - "--tensor_input_names=input_ids"
        - "--disable_vllm"
        - "--task=6"
      env:
        - name: CUDA_LAUNCH_BLOCKING
          value: "1"
      resources:
        limits:
          nvidia.com/gpu: "1"
        requests:
          nvidia.com/gpu: "1"
```

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version: master
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/encoder_model.py b/python/huggingfaceserver/huggingfaceserver/encoder_model.py
--- a/python/huggingfaceserver/huggingfaceserver/encoder_model.py
+++ b/python/huggingfaceserver/huggingfaceserver/encoder_model.py
@@ -146,11 +146,8 @@ def load(self) -> bool:
             do_lower_case=self.do_lower_case,
             **tokenizer_kwargs,
         )
-
-        if not self._tokenizer.pad_token:
-            self._tokenizer.add_special_tokens({"pad_token": "[PAD]"})
-
         logger.info("Successfully loaded tokenizer")
+
         # load huggingface model using from_pretrained for inference mode
         if not self.predictor_host:
             model_cls = get_model_class_for_task(self.task)
@@ -162,6 +159,16 @@ def load(self) -> bool:
             )
             self._model.eval()
             self._model.to(self._device)
+            if not self._tokenizer.pad_token:
+                pad_token_str = "[PAD]"
+                logger.warning(
+                    f"Tokenizer does not have a padding token defined. Adding fall back pad token `{pad_token_str}`"
+                )
+                # Add fallback pad token [PAD]
+                self._tokenizer.add_special_tokens({"pad_token": pad_token_str})
+                # When adding new tokens to the vocabulary, we should make sure to also resize the token embedding
+                # matrix of the model so that its embedding matrix matches the tokenizer.
+                self._model.resize_token_embeddings(len(self._tokenizer))
             logger.info(
                 f"Successfully loaded huggingface model from path {model_id_or_path}"
             )
diff --git a/python/huggingfaceserver/huggingfaceserver/generative_model.py b/python/huggingfaceserver/huggingfaceserver/generative_model.py
--- a/python/huggingfaceserver/huggingfaceserver/generative_model.py
+++ b/python/huggingfaceserver/huggingfaceserver/generative_model.py
@@ -198,6 +198,7 @@ def load(self) -> bool:
             # https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688
             # https://github.com/Vision-CAIR/MiniGPT-4/issues/129
             # https://github.com/huggingface/transformers/blob/1248f0925234f97da9eee98da2aa22f7b8dbeda1/src/transformers/generation/utils.py#L1376-L1388
+            logger.info("Decoder-only model detected. Setting padding side to left.")
             tokenizer_kwargs["padding_side"] = "left"
 
         if self.trust_remote_code:
@@ -212,9 +213,6 @@ def load(self) -> bool:
             **tokenizer_kwargs,
         )
 
-        if not self._tokenizer.pad_token:
-            self._tokenizer.add_special_tokens({"pad_token": "[PAD]"})
-
         logger.info("Successfully loaded tokenizer")
         # load huggingface model using from_pretrained for inference mode
         model_cls = get_model_class_for_task(self.task)
@@ -226,6 +224,16 @@ def load(self) -> bool:
         )
         self._model.eval()
         self._model.to(self._device)
+        if not self._tokenizer.pad_token:
+            pad_token_str = "[PAD]"
+            logger.warning(
+                f"Tokenizer does not have a padding token defined. Adding fall back pad token `{pad_token_str}`"
+            )
+            # Add fallback pad token [PAD]
+            self._tokenizer.add_special_tokens({"pad_token": pad_token_str})
+            # When adding new tokens to the vocabulary, we should make sure to also resize the token embedding
+            # matrix of the model so that its embedding matrix matches the tokenizer.
+            self._model.resize_token_embeddings(len(self._tokenizer))
         logger.info(
             f"Successfully loaded huggingface model from path {model_id_or_path}"
         )
@@ -340,8 +348,12 @@ def validate_supported_completion_params(self, params: CreateCompletionRequest):
     def build_generation_config(
         self, params: CreateCompletionRequest
     ) -> GenerationConfig:
-        kwargs = {}
-        kwargs["max_new_tokens"] = params.max_tokens
+        kwargs = {
+            "max_new_tokens": params.max_tokens,
+            "top_p": params.top_p,
+            "temperature": params.temperature,
+            "pad_token_id": self._tokenizer.pad_token_id,
+        }
         if params.presence_penalty and params.presence_penalty > 0:
             kwargs["repetition_penalty"] = params.presence_penalty
         if params.logit_bias is not None:
@@ -349,9 +361,6 @@ def build_generation_config(
             kwargs["sequence_bias"] = {
                 tuple(token): bias for token, bias in params.logit_bias.items()
             }
-        kwargs["top_p"] = params.top_p
-        kwargs["temperature"] = params.temperature
-
         return GenerationConfig(**kwargs)
 
     def apply_chat_template(


Test patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/test_model.py b/python/huggingfaceserver/huggingfaceserver/test_model.py
--- a/python/huggingfaceserver/huggingfaceserver/test_model.py
+++ b/python/huggingfaceserver/huggingfaceserver/test_model.py
@@ -89,6 +89,19 @@ def bert_token_classification():
     model.stop()
 
 
+@pytest.fixture(scope="module")
+def openai_gpt_model():
+    model = HuggingfaceGenerativeModel(
+        "openai-gpt",
+        model_id_or_path="openai-community/openai-gpt",
+        task=MLTask.text_generation,
+        max_length=512,
+    )
+    model.load()
+    yield model
+    model.stop()
+
+
 def test_unsupported_model():
     config = AutoConfig.from_pretrained("google/tapas-base-finetuned-wtq")
     with pytest.raises(ValueError) as err_info:
@@ -335,3 +348,27 @@ async def test_input_truncation(bert_base_yelp_polarity: HuggingfaceEncoderModel
     request = "good " * 600
     response = await bert_base_yelp_polarity({"instances": [request]}, headers={})
     assert response == {"predictions": [1]}
+
+
+@pytest.mark.asyncio
+async def test_input_padding_with_pad_token_not_specified(
+    openai_gpt_model: HuggingfaceGenerativeModel,
+):
+    # inputs with different lengths will throw an error
+    # unless padding token is configured.
+    # openai-gpt model does not specify the pad token, so the fallback pad token should be added.
+    assert openai_gpt_model._tokenizer.pad_token == "[PAD]"
+    assert openai_gpt_model._tokenizer.pad_token_id is not None
+    params = CreateCompletionRequest(
+        model="openai-gpt",
+        prompt=["Sun rises in the east, sets in the", "My name is Teven and I am"],
+        stream=False,
+        temperature=0,
+    )
+    request = CompletionRequest(params=params)
+    response = await openai_gpt_model.create_completion(request)
+    assert (
+        response.choices[0].text
+        == "west, and the sun sets in the west. \n the sun rises in the"
+    )
+    assert "a member of the royal family." in response.choices[1].text