Repository: huggingface/transformers

Issue:
Tapas tokenizer
## Environment info

- `transformers` version: 4.8.2 and 4.1.1
- Platform: linux-ubuntu 20.04
- Python version: 3.7.5
- PyTorch version (GPU?): 1.9.0+cu111
- Tensorflow version (GPU?): none
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- tapas: @NielsRogge  @LysandreJik  @sgugger 

Library:

- tokenizers:  @LysandreJik

 -->

## Information

Model I am using : TAPAS

The problem arises when using:

* [ X] my own modified scripts: (give details below)


## To reproduce

Steps to reproduce the behavior:

1. The truncation fails because of maximum length because the truncation is not applied on the query and only on the table.

```python
from transformers import TapasTokenizer
import pandas as pd
tokenizer = TapasTokenizer.from_pretrained(pretrained_model_name_or_path="google/tapas-base-finetuned-tabfact")

large_text: str = '''

The standard Lorem Ipsum passage, used since the 1500s

"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
Section 1.10.32 of "de Finibus Bonorum et Malorum", written by Cicero in 45 BC

"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"
1914 translation by H. Rackham

"But I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?"
Section 1.10.33 of "de Finibus Bonorum et Malorum", written by Cicero in 45 BC

"At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat."
1914 translation by H. Rackham

"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains."

'''

tokenized = tokenizer(table=pd.DataFrame(), queries=large_text, max_length=512, truncation=True)

```

This raises the error:

```
~/.pyenv/versions/3.7.5/envs/plop/lib/python3.7/site-packages/transformers/models/tapas/tokenization_tapas.py in __call__(self, table, queries, answer_coordinates, answer_text, add_special_tokens, padding, truncation, max_length, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    633                 return_length=return_length,
    634                 verbose=verbose,
--> 635                 **kwargs,
    636             )
    637 

~/.pyenv/versions/3.7.5/envs/plop/lib/python3.7/site-packages/transformers/models/tapas/tokenization_tapas.py in encode_plus(self, table, query, answer_coordinates, answer_text, add_special_tokens, padding, truncation, max_length, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    981             return_length=return_length,
    982             verbose=verbose,
--> 983             **kwargs,
    984         )
    985 

~/.pyenv/versions/3.7.5/envs/plop/lib/python3.7/site-packages/transformers/models/tapas/tokenization_tapas.py in _encode_plus(self, table, query, answer_coordinates, answer_text, add_special_tokens, padding, truncation, max_length, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   1036             return_special_tokens_mask=return_special_tokens_mask,
   1037             return_length=return_length,
-> 1038             verbose=verbose,
   1039         )
   1040 

~/.pyenv/versions/3.7.5/envs/plop/lib/python3.7/site-packages/transformers/models/tapas/tokenization_tapas.py in prepare_for_model(self, raw_table, raw_query, tokenized_table, query_tokens, answer_coordinates, answer_text, add_special_tokens, padding, truncation, max_length, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis, **kwargs)
   1137         if max_length is not None and len(input_ids) > max_length:
   1138             raise ValueError(
-> 1139                 "Could not encode the query and table header given the maximum length. Encoding the query and table"
   1140                 f"header results in a length of {len(input_ids)} which is higher than the max_length of {max_length}"
   1141             )

ValueError: Could not encode the query and table header given the maximum length. Encoding the query and tableheader results in a length of 1235 which is higher than the max_length of 512

```


## Expected behavior

There should be a parameter to truncate the query as well so as to avoid errors when the query is too large. There are workarounds by changing the model max length but it requires doing the truncation afterwards by hand, so this is defeating the purpose of having truncation parameter.



Discussion:

Resolving patch:
diff --git a/src/transformers/models/tapas/tokenization_tapas.py b/src/transformers/models/tapas/tokenization_tapas.py
--- a/src/transformers/models/tapas/tokenization_tapas.py
+++ b/src/transformers/models/tapas/tokenization_tapas.py
@@ -262,7 +262,10 @@ class TapasTokenizer(PreTrainedTokenizer):
             Whether to add empty strings instead of column names.
         update_answer_coordinates (:obj:`bool`, `optional`, defaults to :obj:`False`):
             Whether to recompute the answer coordinates from the answer text.
-
+        min_question_length (:obj:`int`, `optional`):
+            Minimum length of each question in terms of tokens (will be skipped otherwise).
+        max_question_length (:obj:`int`, `optional`):
+            Maximum length of each question in terms of tokens (will be skipped otherwise).
     """
 
     vocab_files_names = VOCAB_FILES_NAMES
@@ -288,6 +291,8 @@ def __init__(
         max_row_id: int = None,
         strip_column_names: bool = False,
         update_answer_coordinates: bool = False,
+        min_question_length=None,
+        max_question_length=None,
         model_max_length: int = 512,
         additional_special_tokens: Optional[List[str]] = None,
         **kwargs
@@ -318,6 +323,8 @@ def __init__(
             max_row_id=max_row_id,
             strip_column_names=strip_column_names,
             update_answer_coordinates=update_answer_coordinates,
+            min_question_length=min_question_length,
+            max_question_length=max_question_length,
             model_max_length=model_max_length,
             additional_special_tokens=additional_special_tokens,
             **kwargs,
@@ -346,6 +353,8 @@ def __init__(
         self.max_row_id = max_row_id if max_row_id is not None else self.model_max_length
         self.strip_column_names = strip_column_names
         self.update_answer_coordinates = update_answer_coordinates
+        self.min_question_length = min_question_length
+        self.max_question_length = max_question_length
 
     @property
     def do_lower_case(self):
@@ -729,6 +738,19 @@ def batch_encode_plus(
             **kwargs,
         )
 
+    def _get_question_tokens(self, query):
+        """Tokenizes the query, taking into account the max and min question length."""
+
+        query_tokens = self.tokenize(query)
+        if self.max_question_length is not None and len(query_tokens) > self.max_question_length:
+            logger.warning("Skipping query as its tokens are longer than the max question length")
+            return "", []
+        if self.min_question_length is not None and len(query_tokens) < self.min_question_length:
+            logger.warning("Skipping query as its tokens are shorter than the min question length")
+            return "", []
+
+        return query, query_tokens
+
     def _batch_encode_plus(
         self,
         table,
@@ -757,8 +779,9 @@ def _batch_encode_plus(
         table_tokens = self._tokenize_table(table)
 
         queries_tokens = []
-        for query in queries:
-            query_tokens = self.tokenize(query)
+        for idx, query in enumerate(queries):
+            query, query_tokens = self._get_question_tokens(query)
+            queries[idx] = query
             queries_tokens.append(query_tokens)
 
         batch_outputs = self._batch_prepare_for_model(
@@ -1015,7 +1038,7 @@ def _encode_plus(
             )
 
         table_tokens = self._tokenize_table(table)
-        query_tokens = self.tokenize(query)
+        query, query_tokens = self._get_question_tokens(query)
 
         return self.prepare_for_model(
             table,


Test patch:
diff --git a/tests/test_tokenization_tapas.py b/tests/test_tokenization_tapas.py
--- a/tests/test_tokenization_tapas.py
+++ b/tests/test_tokenization_tapas.py
@@ -1076,6 +1076,37 @@ def test_tapas_truncation_integration_test(self):
         self.assertListEqual(new_encoded_inputs, dropped_encoded_inputs)
         self.assertLessEqual(len(new_encoded_inputs), 20)
 
+    @slow
+    def test_min_max_question_length(self):
+        data = {
+            "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
+            "Age": ["56", "45", "59"],
+            "Number of movies": ["87", "53", "69"],
+            "Date of birth": ["18 december 1963", "11 november 1974", "6 may 1961"],
+        }
+        queries = "When was Brad Pitt born?"
+        table = pd.DataFrame.from_dict(data)
+
+        # test max_question_length
+        tokenizer = TapasTokenizer.from_pretrained("lysandre/tapas-temporary-repo", max_question_length=2)
+
+        encoding = tokenizer(table=table, queries=queries)
+
+        # query should not be tokenized as it's longer than the specified max_question_length
+        expected_results = [101, 102]
+
+        self.assertListEqual(encoding.input_ids[:2], expected_results)
+
+        # test min_question_length
+        tokenizer = TapasTokenizer.from_pretrained("lysandre/tapas-temporary-repo", min_question_length=30)
+
+        encoding = tokenizer(table=table, queries=queries)
+
+        # query should not be tokenized as it's shorter than the specified min_question_length
+        expected_results = [101, 102]
+
+        self.assertListEqual(encoding.input_ids[:2], expected_results)
+
     @is_pt_tf_cross_test
     def test_batch_encode_plus_tensors(self):
         tokenizers = self.get_tokenizers(do_lower_case=False)