Repository: huggingface/optimum

Issue:
Better Transformer with QA pipeline returns padding issue
### System Info

```shell
Optimum version: 1.6.4
Platform: Linux
Python version: 3.10
Transformers version: 4.26.1
Accelerate version: 0.16.0
Torch version: 1.13.1+cu117
```


### Who can help?

@philschmid 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Notebook link to reproduce the same: https://colab.research.google.com/drive/1g-EzDtvEMIO1VjYBFJDlYWuKoBjaHxDd?usp=sharing

Code snippet:
```python
from optimum.pipelines import pipeline
qa_model = "bert-large-uncased-whole-word-masking-finetuned-squad"
reader = pipeline("question-answering", qa_model, accelerator="bettertransformer")
reader(question=["What is your name?", "What do you like to do in your free time?"] * 10, context=["My name is Bookworm and I like to read books."] * 20, batch_size=16)
```

Error persists on both cpu and cuda device. Works as expected if batches passed in require no padding.

Error received:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/question_answering.py", line 393, in __call__
    return super().__call__(examples, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1065, in __call__
    outputs = [output for output in final_iterator]
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1065, in <listcomp>
    outputs = [output for output in final_iterator]
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/app/.venv/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/app/.venv/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/app/.venv/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    return self.collate_fn(data)
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 169, in inner
    padded[key] = _pad(items, key, _padding_value, padding_side)
  File "/app/.venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 92, in _pad
    tensor = torch.zeros((batch_size, max_length), dtype=dtype) + padding_value
TypeError: unsupported operand type(s) for +: 'Tensor' and 'NoneType'
```

The system versions mentioned above are from my server setup although it seems reproducible from the notebook with different torch/cuda installations!

### Expected behavior

Expected behavior is to produce correct results without error.


Discussion:
Better Transformer Pipeline seems to work as expected when passing in model and tokenizer explicitly: https://colab.research.google.com/drive/1mKRunYh2RdPJ6Qi2bE055UYhaopV9OkT?usp=sharing
The issue seems to be with this line:
https://github.com/huggingface/optimum/blob/dbb43fb622727f2206fa2a2b3b479f6efe82945b/optimum/pipelines.py#L48

SUPPORTED_TASKS is not initialized to any particular value when an onnx_runtime is not available. 
This means when [this code snippet](https://github.com/huggingface/optimum/blob/dbb43fb622727f2206fa2a2b3b479f6efe82945b/optimum/pipelines.py#L153) executes:

```python
NO_FEATURE_EXTRACTOR_TASKS = set()
NO_TOKENIZER_TASKS = set()
for task, values in SUPPORTED_TASKS.items():
    if values["type"] == "text":
        NO_FEATURE_EXTRACTOR_TASKS.add(task)
    elif values["type"] == "image":
        NO_TOKENIZER_TASKS.add(task)
    elif values["type"] != "multimodal":
        raise ValueError(f"SUPPORTED_TASK {task} contains invalid type {values['type']}")
```

The NO_FEATURE_EXTRACTOR_TASKS are always empty. This causes [this code here](https://github.com/huggingface/optimum/blob/dbb43fb622727f2206fa2a2b3b479f6efe82945b/optimum/pipelines.py#L324) to evaluate to True.
```
    if targeted_task in NO_FEATURE_EXTRACTOR_TASKS:
        load_feature_extractor = False
    else:
        load_feature_extractor = True
        
```

This loads and sends the tokenizer as a feature_extractor to the hugging face pipeline class:
```python
    if feature_extractor is None and load_feature_extractor:
        feature_extractor = get_preprocessor(model_id)

    return transformers_pipeline(
        task,
        model=model,
        tokenizer=tokenizer,
        feature_extractor=feature_extractor,
        use_fast=use_fast,
        use_auth_token=use_auth_token,
        **kwargs,
    )
    
   ```
   
   This causes an issue in the [transformers pipeline class](https://github.com/huggingface/transformers/blob/619d51e01f326d298687912d1f65f8d460f2a6e2/src/transformers/pipelines/base.py#L129) which is looking for a feature extractor padding_value key which cannot be found cause we passed in the tokenizer (which doesn't have such a key) and hence that values gets assigned to None
   ```python
       if feature_extractor is not None:
        # Feature extractor can be images, where no padding is expected
        f_padding_value = getattr(feature_extractor, "padding_value", None)
        f_padding_side = getattr(feature_extractor, "padding_side", None)
```

which gets assigned as the default padding value a few lines below cause again the feature extractor got passed in incorrectly
```python
       for key in keys:
            if key in {"input_ids"}:
                # ImageGPT uses a feature extractor
                if feature_extractor is not None:
                    _padding_value = f_padding_value
                else:
                    _padding_value = t_padding_value
```
       

Resolving patch:
diff --git a/optimum/onnxruntime/runs/__init__.py b/optimum/onnxruntime/runs/__init__.py
--- a/optimum/onnxruntime/runs/__init__.py
+++ b/optimum/onnxruntime/runs/__init__.py
@@ -8,7 +8,7 @@
 
 from onnxruntime.quantization import QuantFormat, QuantizationMode, QuantType
 
-from ...pipelines import SUPPORTED_TASKS
+from ...pipelines import ORT_SUPPORTED_TASKS
 from ...pipelines import pipeline as _optimum_pipeline
 from ...runs_base import Run, TimeBenchmark, get_autoclass_name, task_processing_map
 from .. import ORTQuantizer
@@ -35,7 +35,7 @@ def __init__(self, run_config):
             operators_to_quantize=run_config["operators_to_quantize"],
         )
 
-        onnx_model = SUPPORTED_TASKS[self.task]["class"][0].from_pretrained(
+        onnx_model = ORT_SUPPORTED_TASKS[self.task]["class"][0].from_pretrained(
             run_config["model_name_or_path"], from_transformers=True
         )
 
diff --git a/optimum/pipelines.py b/optimum/pipelines.py
--- a/optimum/pipelines.py
+++ b/optimum/pipelines.py
@@ -45,8 +45,6 @@
 from .utils.file_utils import find_files_matching_pattern
 
 
-SUPPORTED_TASKS = {}
-
 if is_onnxruntime_available():
     from .onnxruntime import (
         ORTModelForCausalLM,
@@ -63,7 +61,7 @@
     )
     from .onnxruntime.modeling_ort import ORTModel
 
-    SUPPORTED_TASKS = {
+    ORT_SUPPORTED_TASKS = {
         "feature-extraction": {
             "impl": FeatureExtractionPipeline,
             "class": (ORTModelForFeatureExtraction,),
@@ -152,12 +150,12 @@
 
 NO_FEATURE_EXTRACTOR_TASKS = set()
 NO_TOKENIZER_TASKS = set()
-for task, values in SUPPORTED_TASKS.items():
+for task, values in TRANSFORMERS_SUPPORTED_TASKS.items():
     if values["type"] == "text":
         NO_FEATURE_EXTRACTOR_TASKS.add(task)
     elif values["type"] == "image":
         NO_TOKENIZER_TASKS.add(task)
-    elif values["type"] != "multimodal":
+    elif values["type"] not in ["multimodal", "audio", "video"]:
         raise ValueError(f"SUPPORTED_TASK {task} contains invalid type {values['type']}")
 
 
@@ -179,11 +177,11 @@ def load_bettertransformer(
         model_kwargs = {}
 
     if model is None:
-        model_id = TRANSFORMERS_SUPPORTED_TASKS[targeted_task]["default"]
-        model = TRANSFORMERS_SUPPORTED_TASKS[targeted_task]["pt"][0].from_pretrained(model_id, **model_kwargs)
+        model_id = SUPPORTED_TASKS[targeted_task]["default"]
+        model = SUPPORTED_TASKS[targeted_task]["pt"][0].from_pretrained(model_id, **model_kwargs)
     elif isinstance(model, str):
         model_id = model
-        model = TRANSFORMERS_SUPPORTED_TASKS[targeted_task]["pt"][0].from_pretrained(model, **model_kwargs)
+        model = SUPPORTED_TASKS[targeted_task]["pt"][0].from_pretrained(model, **model_kwargs)
     else:
         raise ValueError(
             f"""Model {model} is not supported. Please provide a valid model either as string or ORTModel.
@@ -285,9 +283,9 @@ def pipeline(
     targeted_task = "translation" if task.startswith("translation") else task
 
     if accelerator == "ort":
-        if targeted_task not in list(SUPPORTED_TASKS.keys()):
+        if targeted_task not in list(ORT_SUPPORTED_TASKS.keys()):
             raise ValueError(
-                f"Task {targeted_task} is not supported. Supported tasks are { list(SUPPORTED_TASKS.keys())}"
+                f"Task {targeted_task} is not supported for the ONNX Runtime pipeline. Supported tasks are { list(ORT_SUPPORTED_TASKS.keys())}"
             )
 
     if accelerator not in MAPPING_LOADING_FUNC:
@@ -309,6 +307,7 @@ def pipeline(
     else:
         load_feature_extractor = True
 
+    supported_tasks = ORT_SUPPORTED_TASKS if accelerator == "ort" else TRANSFORMERS_SUPPORTED_TASKS
     model, model_id, tokenizer, feature_extractor = MAPPING_LOADING_FUNC[accelerator](
         model,
         targeted_task,
@@ -316,7 +315,7 @@ def pipeline(
         tokenizer,
         feature_extractor,
         load_feature_extractor,
-        SUPPORTED_TASKS,
+        SUPPORTED_TASKS=supported_tasks,
         *model_kwargs,
         **kwargs,
     )


Test patch:
diff --git a/tests/bettertransformer/test_integration.py b/tests/bettertransformer/test_integration.py
--- a/tests/bettertransformer/test_integration.py
+++ b/tests/bettertransformer/test_integration.py
@@ -14,10 +14,12 @@
 # limitations under the License.
 
 import unittest
+from unittest.mock import patch
 
 from transformers import AutoModel
 
 from optimum.bettertransformer import BetterTransformer
+from optimum.pipelines import pipeline
 
 
 class BetterTransformerIntegrationTests(unittest.TestCase):
@@ -28,3 +30,17 @@ def test_raise_error_on_double_transform_call(self):
             bt_model = BetterTransformer.transform(model)
             bt_model = BetterTransformer.transform(bt_model)
         self.assertTrue("was called on a model already using Better Transformer" in str(cm.exception))
+
+    @patch("optimum.utils.import_utils.is_onnxruntime_available")
+    def test_direct_pipleine_initialization_without_onnx_installed(self, mock_onnxruntime_availability):
+        mock_onnxruntime_availability.return_value = False
+        pipe = pipeline(
+            "question-answering",
+            "hf-internal-testing/tiny-random-BertModel",
+            accelerator="bettertransformer",
+        )
+        pipe(
+            question=["Is huggingface getting better?", "Will it ever stop getting better?"],
+            context=["Huggingface will never stop getting better."] * 2,
+            batch_size=10,
+        )