Unexpected keyword argument 'trust_remote_code' when using `table-question-answering` pipeline
Author: philschmid
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/tapex-base-finetuned-wtq").save_pretrained("test")
model = AutoModelForSeq2SeqLM.from_pretrained("microsoft/tapex-base-finetuned-wtq").save_pretrained("test")
```
2. create pipeline object
```python
from transformers import pipeline
tq = pipeline("table-question-answering",model="test")
```
3. receive error
```python
pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    655         task=task,
    656         **hub_kwargs,
--> 657         **model_kwargs,
    658     )
    659 

[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/base.py](https://localhost:8080/#) in infer_framework_load_model(model, config, model_classes, task, framework, **model_kwargs)
    255 
    256             try:
--> 257                 model = model_class.from_pretrained(model, **kwargs)
    258                 if hasattr(model, "eval"):
    259                     model = model.eval()

[/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2104 
   2105         with ContextManagers(init_contexts):
-> 2106             model = cls(config, *model_args, **model_kwargs)
   2107 
   2108         if device_map == "auto":

TypeError: __init__() got an unexpected keyword argument 'trust_remote_code'
```

### Expected behavior

The model should normally load

Comments:
It is not only for local model broken

```python
from transformers import pipeline
tq = pipeline("table-question-answering",model="microsoft/tapex-base-finetuned-wtq")
```
creates the same error 


cc @NielsRogge @Narsil 

Hi @philschmid, I tried your code and raised the same error, but after I did some debugging, I found some info that may be useful to you.Here we take the code `pipeline("table-question-answering",model="microsoft/tapex-base-finetuned-wtq")`for example. 

Under the hood, the pipeline for table question answering will infer the config type base on your model name which is `microsoft/tapex-base-finetuned-wtq` here. 
https://github.com/huggingface/transformers/blob/f0d496828d3da3bf1e3c8fbed394d7847e839fa6/src/transformers/pipelines/__init__.py#L574
But unfortunately, the config type of this model is `BartConfig`. To initialize the pipeline, we need to provide the model which can infer the type of config is `TapasConfig`, for example, model `google/tapas-base`. I think you can try to initialize the pipeline with the following code:

`tq = pipeline("table-question-answering", model="google/tapas-base")`

@aRyBernAlTEglOTRO  it works in `4.20.1` with `tapex` and the issue comes from `trust_remote_code`, which might be missing somewhere in the files.

Hi @philschmid, After I downgrade the transformers to 4.20.1, although I can run the code with `microsoft/tapex-base-finetuned-wtq`. but I will raise the info below:

`The model 'BartForConditionalGeneration' is not supported for table-question-answering. Supported models are ['TapasForQuestionAnswering'].`

Therefore, I think even if you can initialize the pipeline, the pipeline may work in the wrong way. I still think we should initialize the pipeline with the model which can infer the `tapasConfig`.

Hi @philschmid, if you insist to initialize the pipeline with `microsoft/tapex-base-finetuned-wtq`. I found some info that may be useful. When `AutoModelForTableQuestionAnswering` try to init the model, it will remove the `trust_remote_code` from `kwargs`. you can check the following code:
https://github.com/huggingface/transformers/blob/30992ef0d911bdeca425969d210771118a5cd1ac/src/transformers/models/auto/auto_factory.py#L420
Therefore, If you want to initialize the pipeline with `microsoft/tapex-base-finetuned-wtq`, which will build a model `BartForConditionalGeneration` under the hood, thus you need to add some modifications to the `from_pretrained` method of `BartForConditionalGeneration`, which is code mentioned below:
https://github.com/huggingface/transformers/blob/30992ef0d911bdeca425969d210771118a5cd1ac/src/transformers/modeling_utils.py#L1606
and you may already found that they add this update in version `4.22.0.dev0`
https://github.com/huggingface/transformers/blob/30992ef0d911bdeca425969d210771118a5cd1ac/src/transformers/modeling_utils.py#L1829

After you upgrade the transformers to version `4.22.0.dev0`, everything should work fine, but you still get the info that:

`The model 'BartForConditionalGeneration' is not supported for table-question-answering. Supported models are ['TapasForQuestionAnswering'].`


@philschmid I can't be able to reproduce on `@main` branch, so the issues seems to have been fixed. 

I confirm the error exists in `4.21.1` though, I am not sure how to backport things or workaround this.

`trust_remote_code` was added recently @sgugger so he might know more.

As for the warning, the warnings is a bit outdated as the pipeline does support BartGeneration.
Added a PR to fix the warningS: https://github.com/huggingface/transformers/pull/18711

Hi @Narsil, thank you for your reply. I found the warning raised from the code below:
https://github.com/huggingface/transformers/blob/0f257a87749e0a72bda260c6f319a45dae1e7c4d/src/transformers/pipelines/table_question_answering.py#L102
which means we need to update the `OrderDict` in the following code to remove warning:
https://github.com/huggingface/transformers/blob/0f257a87749e0a72bda260c6f319a45dae1e7c4d/src/transformers/models/auto/modeling_auto.py#L590
I don't know except for the `BartForConditionalGeneration` model, what else model should be added to this `OrderDict`?

TAPEX wasn't added to the MODEL_FOR_TABLE_QUESTION_ANSWERING_MAPPING_NAMES because that mapping defines which models are supported by the `AutoModelForTableQuestionAnswering` class.

However, the table QA pipeline does support both TAPAS and TAPEX, so we may want to suppress this warning.

@aRyBernAlTEglOTRO @NielsRogge The PR to fix the warnings is here. https://github.com/huggingface/transformers/pull/18711
 
Basically any Seq2seq model should work (Well only the trained models will actually provide good results, but the pipeline WILL work.)

In general pipelines tries not to look at individual models, but only type for model (`ForXXX`)

The error was fixed in https://github.com/huggingface/transformers/pull/18428 @philschmid.

I'll likely do a patch PR later today containing this fix (v4.21.2).

Closing as solved by https://github.com/huggingface/transformers/pull/18428