Repository: kserve/kserve

Issue:
Pad left for decode-only architecture llm models
/kind bug

**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]
With decoder-only models, such as GPT-2 padding should be done on the left. This is because the output is a continuation of the input prompt -- there would be gaps in the output without left padding. Huggingface will warn if we don't use left padding for decoder models. 
https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688
The huggingface will throw a warning about this
```
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer
```

**What did you expect to happen:**
It should pad left for decoder only models.


**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]
```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-gpt2
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - "--model_name=gpt2"
        - "--model_id=openai-community/gpt2"
        - "--tensor_input_names=input_ids"
        - "--disable_vllm"
        - "--task=6"
      resources:
        limits:
          nvidia.com/gpu: "1"
        requests:
          nvidia.com/gpu: "1"
```

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version: 0.12.0
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/model.py b/python/huggingfaceserver/huggingfaceserver/model.py
--- a/python/huggingfaceserver/huggingfaceserver/model.py
+++ b/python/huggingfaceserver/huggingfaceserver/model.py
@@ -125,8 +125,16 @@ def load(self) -> bool:
         if self.model._no_split_modules:
             self.device_map = "auto"
         # load huggingface tokenizer
-        self.tokenizer = AutoTokenizer.from_pretrained(
-            model_id_or_path, do_lower_case=self.do_lower_case, device_map=self.device_map)
+        if not model_config.is_encoder_decoder:
+            # Pad left for decode-only architecture models.
+            # https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688
+            # https://github.com/Vision-CAIR/MiniGPT-4/issues/129
+            # https://github.com/huggingface/transformers/blob/1248f0925234f97da9eee98da2aa22f7b8dbeda1/src/transformers/generation/utils.py#L1376-L1388
+            self.tokenizer = AutoTokenizer.from_pretrained(
+                model_id_or_path, do_lower_case=self.do_lower_case, device_map=self.device_map, padding_side="left")
+        else:
+            self.tokenizer = AutoTokenizer.from_pretrained(
+                model_id_or_path, do_lower_case=self.do_lower_case, device_map=self.device_map)
         if not self.tokenizer.pad_token:
             self.tokenizer.add_special_tokens({'pad_token': '[PAD]'})
         logger.info(f"successfully loaded tokenizer for task: {self.task}")


Test patch: