Tapas Tokenizer  makes DataFrame iterrows() iterator crazy ...
Author: jeromemassot
Pull request: no

## Environment info

- `transformers` version: 4.3.2 
- Platform: Colab Pro
- Python version: Python 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101    torch-scatter  2.0.5          
- Tensorflow version (GPU?): 2.4.1  
- Using GPU in script?: Tesla P100
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
@n1t0, @LysandreJik
 -->

## Information

Model I am using : Tapas

Very very very strange (at least for me, a Computer Science newbie) with the function when the table ingested has been resampled with the pd.DataFrame.sample() method.

In the following block of code, the rows iterator returns corrupted rows with my table.
I have check the iterrows() outside the Tapas Tokenizer and the rows returned are correct.
But inside the Tokenizer, the rows are sometimes ok but sometimes Cell objects and corresponding to wrong rows !!

```
# Second, replace cell values by Cell objects
for row_index, row in table.iterrows():
    for col_index, cell in enumerate(row):
            table.iloc[row_index, col_index] = Cell(text=cell)
```
The direct result in my case is a crash in the normalize_for_match() method :
AttributeError: 'Cell' object has no attribute 'lower'
which is normal since several rows in the table now are of Cell type and not str.

I cannot see why the rows iterator suddenly returns corrupted data, for both Type and Values.

Thanks

Best regards

Jerome

The problem arises when using:
* [ X] my own modified scripts: I am using the Tapas Tokenizer with shuffled Pandas DataFrame for table.

The tasks I am working on is:
* [ X] my own task or dataset: Total R&D

## To reproduce

Steps to reproduce the behavior:

1. Use a standard Pandas DataFrame read from csv
2. Shuffle this DataFrame by using sample with frac=1
3. Tokenize the DataFrame as table using the Tapas Tokenizer

# Second, replace cell values by Cell objects
```
for row_index, row in table.iterrows():
    for col_index, cell in enumerate(row):
            table.iloc[row_index, col_index] = Cell(text=cell)
```

## Expected behavior
The iterrows() is returning unconsistent row information for both type and content.

<!-- A clear and concise description of what you would expect to happen. -->
The iterrows() should return consistent row values.


Comments:
Hi,

Can you provide the table on which you tried this?

To add numeric value information to the table, each cell in the table is replaced by a `Cell` object. A `Cell` object has 2 attributes: `text` (the original string corresponding to the cell value) and an optional `numeric_value` (which can be a `float_value` or a `date`). 

Did you apply `.astype(str)` on your Pandas dataframe before providing it to `TapasTokenizer`? Since this is required before encoding the table.

Hi Niels,
Indeed, the code changes the DataFrame content to Cell format... but the `iterrows() `returns sometimes a correct row format which is transformed into Cell format... but sometimes a Cell object which is transformed into a Cell (of a Cell) object with text attribute initialized to the original Cell object !! :( 

I have applied the `.astype(str)` yeap, before and after the sample() call... just to be sure :)

Here is the table with ; as separator : 

Water injected volume (% P.V.);Oil recovery (% I.O.I.P.);Watercut (%)
214.11;61;98
215.23;61;98
216.36;61.1;99
217.49;61.1;99
218.62;61.2;98
219.75;61.2;99
220.88;61.2;98
222.02;61.3;98
223.15;61.3;99
224.28;61.4;98
225.41;61.4;99
226.55;61.4;98
227.67;61.4;99
228.8;61.5;99
229.94;61.5;99
231.07;61.6;98
232.2;61.6;99
233.33;61.6;98
234.47;61.7;99
235.6;61.7;98
236.73;61.8;99
237.86;61.8;99
239;61.8;99
240.11;61.9;99
241.24;61.9;99
242.39;61.9;99
243.51;62;99
244.64;62;99
245.77;62;99
246.9;62.1;99
248.03;62.1;99

How to repeat : 

table = pd.read_csv(os.path.join(DATA_PATH, "Table_01.csv"), sep=";").astype(str)
table = table.sample(frac=1.0, random_state=42, replace=False).astype(str)

inputs = tokenizer(table=table, queries=questions, padding='max_length', return_tensors="pt")

Good luck :)

I was able to reproduce it, however when I reset the indices after sampling, it works:

`table = table.sample(frac=1.0, random_state=42, replace=False).reset_index(drop=True).astype(str)`

Will look into why it can't handle without resetting the row indices.


Hi Niels,
I have also tried the reset index... and in my side it was crashing the same. But it was withtout the drop=True.
This behavior is very strange. I have check the Pandas documentation and normally sample() should returned a DataFrame object... nothing fancy here. 
And it does because the iterrows() outside the Tapas Tokenizer works fine :)
So Tapas Tokenizer is doing something on the DataFrame modified by the sample() function :)

Here's a notebook illustrating the issue, and fixing it:

https://colab.research.google.com/drive/10MbZiMKyEWUGk2Y1fvIj0Y0_lB42NO38?usp=sharing

The reason why you're getting the error is because in the part where each cell of the table is replaced by a Cell object:

https://github.com/huggingface/transformers/blob/97e688bc220514cd5ea072f06b186401c9cfbbd0/src/transformers/models/tapas/tokenization_tapas.py#L2742-L2745, the row indices are used. 

This can be fixed by replacing `table` with `table.reset_index(drop=True)` in the first line (or resetting the index of the table before providing it to the tokenizer). Another solution is to replace the final line by `table.iloc[row_index, col_index] = Cell(text=table.iloc[row_index, col_index])`. Will make a small PR to add this.

Thank you for spotting the error!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.