Repository: huggingface/transformers

Issue:
Zero-shot classification pipeline truncation support
Transformers 4.10.0 brought [a change](https://github.com/huggingface/transformers/pull/13299/files#diff-c5af53af9b08fb383b49d7a07c1a56c890198b5cd48adc97aeef753fe2e7d60dR91) that modified the default truncation strategy to TruncationStrategy.DO_NOT_TRUNCATE for the ZeroShotClassificationPipeline.

That uncovered an issue in that the [ZeroShotClassificationPipeline](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/zero_shot_classification.py#L217 ) doesn't appear to pass kwargs to the parent's call method. So even when calling the pipeline with truncation=True, it doesn't allow for truncation.

Thank you for the assistance in advance, appreciate all the work you guys do. 


Discussion:
One additional related question: this warning gets printed out each zsl pipeline call now:

logger.warning("The tokenizer {self.tokenizer} does not have a pad token, we're not running it as a batch")

Would that make more sense to raise on init of the object vs each call? Otherwise, it can get pretty noisy even if you reuse the same zsl pipeline multiple times. 
cc @Narsil 
What model/tokenizer were you using ? Using ONLY_FIRST on tokenizer that do not have pad_token should have resulted in error before, but I could be mistaken.

For the warning, as it was supposed to be a new path (and not an old one), it is indeed a bit noisy, and should be cleaned up with the new refactor of pipelines. https://github.com/huggingface/transformers/pull/13308

What you are claiming is that this was an unexpected regression here, so I would like to test out what was exactly wrong, so we can have a real test for this case before fixing it.
Appreciate the quick response. Here is code that works in 4.9.2 but fails in 4.10.0

```python
from transformers import pipeline

nlp = pipeline("zero-shot-classification", model="roberta-large-mnli", tokenizer="roberta-large-mnli")
nlp(["Very long text" * 1000, "Happy to hear"], ["negative", "positive"], truncation=True)

nlp = pipeline("zero-shot-classification")
nlp(["Very long text" * 1000, "Happy to hear"], ["negative", "positive"], truncation=True)
```

I don't think the truncation param ever did anything but changing the default tokenization param from ONLY_FIRST to DO_NOT_TRUNCATE seems to have exposed the issue. 
Ok.

I can confirm that truncation went from being on by default to not by default.

@LysandreJik that was to enable all tokenizers that don't have a `pad_token` (and can't pad anyway). However, changing the default for tokenizer that can was an oversight on my part.

We can go different routes:

1. Revert the change of default, and override to DO_NOT_TRUNCATE only in the `pad_token` missing path (I think this is what should have been done in the first place, my bad here).
2. Simply roll with it as it was released, but fix the passing around of the truncation argument (it does require a change, to pass on the kwargs to __call__ simply, but might break existing code where some kwargs where just ignored before and would wind up in the `tokenizer(...) ` call triggering new errors).
I think we can go with 1. and then release a patch (v4.10.1) to keep the previous behavior

Resolving patch:
diff --git a/src/transformers/pipelines/zero_shot_classification.py b/src/transformers/pipelines/zero_shot_classification.py
--- a/src/transformers/pipelines/zero_shot_classification.py
+++ b/src/transformers/pipelines/zero_shot_classification.py
@@ -88,7 +88,7 @@ def _parse_and_tokenize(
         hypothesis_template,
         padding=True,
         add_special_tokens=True,
-        truncation=TruncationStrategy.DO_NOT_TRUNCATE,
+        truncation=TruncationStrategy.ONLY_FIRST,
         **kwargs
     ):
         """
@@ -113,13 +113,31 @@ def _parse_and_tokenize(
                 )
                 inputs.append(model_input)
         else:
-            inputs = self.tokenizer(
-                sequence_pairs,
-                add_special_tokens=add_special_tokens,
-                return_tensors=return_tensors,
-                padding=padding,
-                truncation=truncation,
-            )
+            try:
+                inputs = self.tokenizer(
+                    sequence_pairs,
+                    add_special_tokens=add_special_tokens,
+                    return_tensors=return_tensors,
+                    padding=padding,
+                    truncation=truncation,
+                )
+            except Exception as e:
+                if "too short" in str(e):
+                    # tokenizers might yell that we want to truncate
+                    # to a value that is not even reached by the input.
+                    # In that case we don't want to truncate.
+                    # It seems there's not a really better way to catch that
+                    # exception.
+
+                    inputs = self.tokenizer(
+                        sequence_pairs,
+                        add_special_tokens=add_special_tokens,
+                        return_tensors=return_tensors,
+                        padding=padding,
+                        truncation=TruncationStrategy.DO_NOT_TRUNCATE,
+                    )
+                else:
+                    raise e
 
         return inputs
 


Test patch:
diff --git a/tests/test_pipelines_zero_shot.py b/tests/test_pipelines_zero_shot.py
--- a/tests/test_pipelines_zero_shot.py
+++ b/tests/test_pipelines_zero_shot.py
@@ -105,6 +105,20 @@ def run_entailment_id(self, zero_shot_classifier: Pipeline):
         zero_shot_classifier.model.config.label2id = original_label2id
         self.assertEqual(original_entailment, zero_shot_classifier.entailment_id)
 
+    @require_torch
+    def test_truncation(self):
+        zero_shot_classifier = pipeline(
+            "zero-shot-classification",
+            model="sshleifer/tiny-distilbert-base-cased-distilled-squad",
+            framework="pt",
+        )
+        # There was a regression in 4.10 for this
+        # Adding a test so we don't make the mistake again.
+        # https://github.com/huggingface/transformers/issues/13381#issuecomment-912343499
+        zero_shot_classifier(
+            "Who are you voting for in 2020?" * 100, candidate_labels=["politics", "public health", "science"]
+        )
+
     @require_torch
     def test_small_model_pt(self):
         zero_shot_classifier = pipeline(