Repository: kserve/kserve

Issue:
ValueError when running HuggingfaceModel with multiple inputs of different length
/kind bug

**What steps did you take and what happened:**
- Modify `test_bert_sequence_classification()` in `huggingfaceserver/test_model.py` such that the two input instances have different length:
```
% git diff huggingfaceserver/test_model.py
--- a/python/huggingfaceserver/huggingfaceserver/test_model.py
+++ b/python/huggingfaceserver/huggingfaceserver/test_model.py
@@ -62,7 +62,8 @@ def test_bert_sequence_classification():
     model.load()
 
     request = "Hello, my dog is cute."
-    response = asyncio.run(model({"instances": [request, request]}, headers={}))
+    request_2 = "Hello, my dog is cute. I love him."
+    response = asyncio.run(model({"instances": [request, request_2]}, headers={}))
     assert response == {"predictions": [1, 1]}
```
- Run the modified test: `pytest -v -W ignore -k "test_bert_sequence_classification"`
- This will fail with:
```
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True'
to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) 
have excessive nesting (inputs type `list` where type `int` is expected).
```
Full logs: [test.txt](https://github.com/kserve/kserve/files/14293621/test.txt)

- In short, the problem is that because the inputs have different lengths, they can’t be converted to fixed-size tensors

**What did you expect to happen:**
I expect that we are able to supply inputs of different length in a single batch.

**Potential fix:**
Based on the error, it seems like we need to add paddings, which is documented [here](https://huggingface.co/docs/transformers/en/pad_truncation) by hugging face.

> Batched inputs are often different lengths, so they can’t be converted to fixed-size tensors. Padding and truncation are strategies for dealing with this problem, to create rectangular tensors from batches of varying lengths. Padding adds a special padding token to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model. Truncation works in the other direction by truncating long sequences.
>
> In most cases, padding your batch to the length of the longest sequence and truncating to the maximum length a model can accept works pretty well. However, the API supports more strategies if you need them. The three arguments you need to are: padding, truncation and max_length.

So we should add `padding=True` (or `"longest"`, same behaviour, but more explicit) when calling `self.tokenizer()`, which will pad the inputs to the longest sequence in the batch, so that all inputs will have the same length.

I'm willing to contribute a fix for this. I've tested it locally and it works, but I'll add more rigorous tests as well, perhaps also see if we need truncation as well or just padding.

**Environment:**

- KServe Version: commit [e21b913](https://github.com/kserve/kserve/commit/e21b913fe10b377d72e67fbf92955dbcedd6f465)


Discussion:
Feel free to submit a fix!
/assign

Resolving patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/model.py b/python/huggingfaceserver/huggingfaceserver/model.py
--- a/python/huggingfaceserver/huggingfaceserver/model.py
+++ b/python/huggingfaceserver/huggingfaceserver/model.py
@@ -153,6 +153,8 @@ def preprocess(self, payload: Union[Dict, InferRequest], context: Dict[str, Any]
                 add_special_tokens=self.add_special_tokens,
                 return_tensors=TensorType.NUMPY,
                 return_token_type_ids=self.return_token_type_ids,
+                padding=True,
+                truncation=True,
             )
             context["payload"] = payload
             context["input_ids"] = inputs["input_ids"]
@@ -171,6 +173,8 @@ def preprocess(self, payload: Union[Dict, InferRequest], context: Dict[str, Any]
                 add_special_tokens=self.add_special_tokens,
                 return_tensors=TensorType.PYTORCH,
                 return_token_type_ids=self.return_token_type_ids,
+                padding=True,
+                truncation=True,
             )
             context["payload"] = payload
             context["input_ids"] = inputs["input_ids"]
@@ -192,7 +196,7 @@ async def generate(self, generate_request: GenerateRequest, headers: Dict[str, s
                 prompt,
                 max_length=self.max_length,
                 add_special_tokens=self.add_special_tokens,
-                return_tensors="pt",
+                return_tensors=TensorType.PYTORCH,
             )
             input_batch = input_batch.to(self.device)
             if headers.get("streaming", "false") == "true":


Test patch:
diff --git a/python/huggingfaceserver/huggingfaceserver/test_model.py b/python/huggingfaceserver/huggingfaceserver/test_model.py
--- a/python/huggingfaceserver/huggingfaceserver/test_model.py
+++ b/python/huggingfaceserver/huggingfaceserver/test_model.py
@@ -89,5 +89,33 @@ def test_bloom():
     assert response.text_output == "Hello, my dog is cute.\n- Hey, my dog is cute.\n- Hey, my dog"
 
 
+def test_input_padding():
+    model = HuggingfaceModel("bert-base-uncased-yelp-polarity",
+                             {"model_id": "textattack/bert-base-uncased-yelp-polarity",
+                              "task": MLTask.sequence_classification.value})
+    model.load()
+
+    # inputs with different lengths will throw an error
+    # unless we set padding=True in the tokenizer
+    request_one = "Hello, my dog is cute."
+    request_two = "Hello there, my dog is cute."
+    response = asyncio.run(model({"instances": [request_one, request_two]}, headers={}))
+    assert response == {"predictions": [1, 1]}
+
+
+def test_input_truncation():
+    model = HuggingfaceModel("bert-base-uncased-yelp-polarity",
+                             {"model_id": "textattack/bert-base-uncased-yelp-polarity",
+                              "task": MLTask.sequence_classification.value})
+    model.load()
+
+    # bert-base-uncased has a max length of 512 (tokenizer.model_max_length).
+    # this request exceeds that, so it will throw an error
+    # unless we set truncation=True in the tokenizer
+    request = "good " * 600
+    response = asyncio.run(model({"instances": [request]}, headers={}))
+    assert response == {"predictions": [1]}
+
+
 if __name__ == '__main__':
     unittest.main()