Tapas not working with tables exceeding token limit
Author: bogdankostic
Pull request: no

## Environment info

- `transformers` version: 4.3.0
- Platform: MacOS
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik @sgugger @NielsRogge 

## Information

Model I am using (Bert, XLNet ...): TaPas

## To reproduce
When executing the following code, using this [table](https://gist.github.com/bogdankostic/387d1c7a0e8ce25ea302395756df11b3), I get an `IndexError: index out of range in self`.
```python
from transformers import AutoTokenizer, AutoModelForTableQuestionAnswering
import pandas as pd

tokenizer = AutoTokenizer.from_pretrained("google/tapas-base-finetuned-wtq", drop_rows_to_fit=True)
model = AutoModelForTableQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")

df = pd.read_csv("table.tsv", sep="\t").astype(str)

queries = ["How big is Ardeen?"]
inputs = tokenizer(table=df, queries=queries, padding="max_length", truncation=True, return_tensors="pt")

outputs = model(**inputs)
```


I am not completely sure about the cause of the error but I suspect that the column rank vectors are not correctly generated. (`torch.max(token_type_ids[:, :, 4])` returns 298 and `torch.max(token_type_ids[:, :, 5])` returns 302, the Embedding Models for column rank and inverse column rank, however, allow a max value of 255)


Comments:
Hi,

Yes the column ranks may cause issues when a table is too big, as the vocab size is only 256. See also [my reply](https://github.com/huggingface/transformers/issues/9221#issuecomment-749093391) on #9221.

Actually, the authors of TAPAS did release a new method in a [follow-up paper](https://arxiv.org/abs/2010.00571) to prune columns that are not relevant to a question to be able to serve large tables to the BERT-like model, so this is something that maybe could be added in the future.

My suggestion would be to compute the column ranks on the truncated table. (Not sure if and how this is feasible.)
Otherwise I would suggest returning a more informative error message.

Yes, good suggestion. I've investigated this a bit and it seems that the original implementation also computes the column ranks on the original table, rather than the truncated one. I've asked the original authors [here](https://github.com/google-research/tapas/issues/106#issue-804538477). Will keep you updated.

So the author replied:

> IIRC, then we compute them before pruning the table.
That was by design so that those ranks would match the original numeric rank (pre-pruning).
It's true that the rank could thus exceed the vocab size.
We could add some trimming to prevent that.

So this is something that could be added in the future (together with the `prune_columns` option). I put it on my to-do list for now.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge Thanks for the explanations above. Has there been any update on this issue? I have also run into this issue when running Tapas on the WTQ dataset, and it took me a lot of efforts to get to the bottom of this and realize that this is an issue with the `column_rank` IDs from oversized tables.

The painful part is that there is currently no guard or no warning against feeding oversized tables into the tokenizer, and the issue will only come out as a "CUDA error: device-side assert triggered" message when the Tapas forward pass is run.

I think there are several potential ways to solve this or make this less painful:

1. Computing the column rank after the table truncation (as already suggested by another comment above). This makes a ton of sense because the table will only be presented to the model after truncation in the tokenizer anyway, so there is no point to maintain a non-continuous column rank for large tables (with some ranks removed due to truncation). I understand that the original TF implementation might not handle this, but can this be added as a behavior in the Huggingface implementation?

2. Add an option to re-map all the large column ranks to the max rank value. This can be implemented in this tokenizer function: https://github.com/huggingface/transformers/blob/7fcee113c163a95d1b125ef35dc49a0a1aa13a50/src/transformers/models/tapas/tokenization_tapas.py#L1487
This is less ideal than 1, but can make sure that the model won't crash due to an index-out-of-range error.

3. The easiest fix would be to add some warning/exception in the tokenizer that reminds users about this. Or let the tokenizer return a `None` value in the output, or return a special boolean variable such as `table_oversized`. This does not solve anything, but can make the capture of this issue much easier.

Look forward to some updates on this issue.

Is there any way to bypass the token limit ?

@KML1337,

I do not have sure if you could consider the following approach:

First, you split the table into n-subtables that generate tokens under the limit tokens;
Then, process each subtable with the model;
Finally, aggregate all responses and select the one with the highest logit score.