Loading tapas model into pipeline from directory gives different result
Author: mchari
Pull request: no

Hi,
   I am using the following versions of these packages : 
transformers = 4.3.2
pytorch = 1.6.0

I am using the following code to download and save a pretrained model:

```py
from transformers import TapasConfig,TapasTokenizer,TapasForQuestionAnswering
import torch
config = TapasConfig.from_pretrained('google/tapas-base-finetuned-wtq',from_pt=True)
model = TapasForQuestionAnswering.from_pretrained('google/tapas-base', config=config)
tokenizer=TapasTokenizer.from_pretrained("google/tapas-base-finetuned-wtq",from_pt=True)
import sys

outdir = sys.argv[1]

model.save_pretrained(outdir)
tokenizer.save_pretrained(outdir)
config.save_pretrained(outdir)
```
When I then feed the model directory into pipeline, I don't get any result for the table illustrated in the documentation....
If I let pipeline download the model on-the-fly, I get results. Here is the code to feed model directory into pipeline...


```py
import sys
from transformers import pipeline

nlp = pipeline(task="table-question-answering",framework="pt",model="tapas_model_dir")
#nlp = pipeline(task="table-question-answering")

import pandas as pd

data= {    "actors": ["brad pitt", "leonardo di caprio", "george clooney"],
    "age": ["56", "45", "59"],
    "number of movies": ["87", "53", "69"],
    "date of birth": ["7 february 1967", "10 june 1996", "28 november 1967"]
}

import numpy as np
table = pd.DataFrame.from_dict(data)
print(np.shape(table))
result = nlp(query=["How many movies has Brad Pitt acted in","What is Leonardo di caprio's age"],table=table)

print(result)
```

Comments:
I noticed that the size of pytorch_model.bin that was downloaded into .cache is of size 442791751. When I save_pretrained() the model, the size is 442792154. 
If I copy the first model into the model directory, I get valid results....

Hello! You're using `'google/tapas-base'` in order to initialize weights, why don't you use the variant fine-tuned on WTQ like in the configuration and tokenizer? 

Unless you're using a fine-tuned version, you won't benefit from the best possible predictions, like you have seen here.

@LysandreJik , I fixed that typo and changed it to 
model = TapasForQuestionAnswering.from_pretrained('google/tapas-base-finetuned-wtq', config=config), but I get the same discrepancy in results.
I am trying to figure out why I get no results when I load the model from a directory(that was generated as above).

I'm running your code and I get some sensible results:

```py
from transformers import TapasConfig,TapasTokenizer,TapasForQuestionAnswering
import torch
config = TapasConfig.from_pretrained('google/tapas-base-finetuned-wtq',from_pt=True)
model = TapasForQuestionAnswering.from_pretrained('google/tapas-base-finetuned-wtq', config=config)
tokenizer=TapasTokenizer.from_pretrained("google/tapas-base-finetuned-wtq",from_pt=True)
import sys

outdir = "tmp"

model.save_pretrained(outdir)
tokenizer.save_pretrained(outdir)
config.save_pretrained(outdir)

from transformers import pipeline

nlp = pipeline(task="table-question-answering",framework="pt",model=outdir, tokenizer=outdir)
#nlp = pipeline(task="table-question-answering")

import pandas as pd

data= {    "actors": ["brad pitt", "leonardo di caprio", "george clooney"],
    "age": ["56", "45", "59"],
    "number of movies": ["87", "53", "69"],
    "date of birth": ["7 february 1967", "10 june 1996", "28 november 1967"]
}

import numpy as np
table = pd.DataFrame.from_dict(data)
print(np.shape(table))
result = nlp(query=["How many movies has Brad Pitt acted in","What is Leonardo di caprio's age"],table=table)

print(result)
```

Results in:
```
[{'answer': 'SUM > 87', 'coordinates': [(0, 2)], 'cells': ['87'], 'aggregator': 'SUM'}, {'answer': 'AVERAGE > 45', 'coordinates': [(1, 1)], 'cells': ['45'], 'aggregator': 'AVERAGE'}]
```

The aggregator are a bit off but the results are correct. Brad Pitt has played in 87 movies and Leonardo di Caprio is 45.

I did additionally specify `tokenizer=outdir`, could that be the source of your issue?

@LysandreJik , that was it ! 
I will look into the code, but any insight into why tokenizer=outdir needs to be specified (for this pipeline task only) ?

@LysandreJik , what is the size of your pytorch_model.bin in outdir ?


The tokenizer needs to be specified for every pipeline tasks. You should always specify the checkpoint for the tokenizer as well as for the model. The size is `442.79 MB`!