Pipeline feature extraction: tensor size mismatch
Author: ioana-blue
Pull request: no

## Environment info

I'm using pipelines for the first time with feature extraction, it seems to work fine for my toy samples that I used to debug the code. However, when I started running with some real data, I got the following stack trace:

```
File "../../models//text_embeddings_feature_extraction.py", line 215, in main
    outputs = feature_extractor(predict_text, padding=True, truncation=TruncationStrategy.ONLY_FIRST)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 73, in __call__
    return super().__call__(*args, **kwargs)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/base.py", line 908, in __call__
    outputs = [output for output in final_iterator]
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/base.py", line 908, in <listcomp>
    outputs = [output for output in final_iterator]
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/base.py", line 631, in __next__
    item = next(self.iterator)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/base.py", line 632, in __next__
    processed = self.infer(item, **self.params)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/base.py", line 871, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 53, in _forward
    model_outputs = self.model(**model_inputs)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 988, in forward
    embedding_output = self.embeddings(
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/mimic/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 221, in forward
    embeddings += position_embeddings
RuntimeError: The size of tensor a (578) must match the size of tensor b (512) at non-singleton dimension 1
```

To me, it looks like a truncation issue. The embedding is longer than the max 512 (I'm using a BERT model). I have long samples in my data. I tried both `truncation = True` and `truncation = TruncationStrategy.ONLY_FIRST` as in the code below, both ended up with the same error. Not sure why this happens

I'm tagging tokenizers and pipelines... 
- Tokenizers: @LysandreJik
- Pipelines: @Narsil

Thanks in advance!

===========

Steps to reproduce the behavior:

Here it is pretty much all my code:

```
   # Load pretrained model and tokenizer                                                                                                                                        
    #                                                                                                                                                                            
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently                                                               
    # download model & vocab.                                                                                                                                                    
    # num_labels hard-coded; shouldn't be used                                                                                                                                   
    num_labels = 2
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=None,   # no task                                                                                                                                        
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
        output_hidden_states=True
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
    )

    predict_dataset = raw_datasets["test"]
    if data_args.max_predict_samples is not None:
        predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))

    # extract only the text                                                                                                                                                      
    predict_text = []
    for item in predict_dataset:
        predict_text.append(item['text'])

    # Log a few random samples from the testing set:                                                                                                                             
    for index in random.sample(range(len(predict_dataset)), 3):
        logger.info(f"Sample {index} of the test set: {predict_dataset[index]}.")

    logger.info("*** Generate embeddings ***")

    # use pipelines and feature extraction                                                                                                                                       
    feature_extractor = pipeline(
            task="feature-extraction", model=model_args.model_name_or_path, config = config, tokenizer = tokenizer, framework="pt"
        )
    outputs = feature_extractor(predict_text, padding=True, truncation=TruncationStrategy.ONLY_FIRST)
```

Comments:
I'm pretty sure that's the problem. I managed to reproduce with something like this:
```
# debug
    predict_text = "this " * 1000
    print(predict_text)
    outputs = feature_extractor(predict_text, padding="longest", truncation=TruncationStrategy.ONLY_FIRST)
```

I get `RuntimeError: The size of tensor a (1002) must match the size of tensor b (512) at non-singleton dimension 1`

On a related note, padding also seems to be ignored. I tried this:
```
    # debug
    predict_text = "this " * 20
    print(predict_text)
    outputs = feature_extractor(predict_text, padding=True, truncation=True)

    print(outputs)
    print(len(outputs[0]))
```
In my understanding, this should output a len of 512 (max len in BERT's case), but it gets 22... 

Hello @ioana-blue, thank you for the issue. Could you share your environment details including the transformers version? You can do so by running `transformers-cli env` in your local environment.

Thanks!

Sure! 
- `transformers` version: 4.11.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.7

Meanwhile, I tried adding the params in the tokenizer, also in the pipeline, they seem to be ignored everywhere which is rather strange (both padding and truncation).

Is it true that any special params are dropped for feature extraction because of this line?

https://github.com/huggingface/transformers/blob/c63fcabfe92229eac983ce743f4e9fce863c84dd/src/transformers/pipelines/feature_extraction.py#L44

Even if this is the case, I don't understand why the `truncation=True` is not sticky from the initialization of the tokenizer. 

I'm pretty sure the method above is buggy and there are no tests to catch this bug. There is a test for summarization here:

https://github.com/huggingface/transformers/blob/879fe8fa75e662ffd85a567a98522bc9cffe0c6c/tests/test_pipelines_summarization.py#L39

And the corresponding method can be found here:
https://github.com/huggingface/transformers/blob/026866df92afe40cdf928839864111015a62d3b5/src/transformers/pipelines/text2text_generation.py#L54

Note the truncation assignment:
https://github.com/huggingface/transformers/blob/026866df92afe40cdf928839864111015a62d3b5/src/transformers/pipelines/text2text_generation.py#L65

I also think the padding parameter should be copied as part of preprocessing. 

If you agree with my assessment, I can try to submit a PR with a few changes: change the _sanitize_parameters in feature extraction and also add a test similar to the summarization text. I haven't submitted a PR so I may need help to follow whatever protocol you have in place. Let me know. 

Am I the only one using the feature extraction pipeline? :) Or the only one with long samples? :) I got really happy when I found out about pipelines, it's a neat concept. 

I also think the preprocessing method needs to change:

https://github.com/huggingface/transformers/blob/c63fcabfe92229eac983ce743f4e9fce863c84dd/src/transformers/pipelines/feature_extraction.py#L47

It needs to take in truncation and padding. 

Hm, we can add the truncation, but it feels a bit odd to just ignore the rest of the sentence, no ?

We already have some pipelines (`question-answering` and `zero-shot-classification`) that are able to chunk their inputs into sub elements for processing in chunks, enabling to process large inputs in a seemless fashion.

Wouldn't that be a better solution ?

The only drawback I can come up with, it the interaction with `aggregation_strategy` where we are attempting to handle word boundaries, so the chunking should aim for "word" boundaries to enable `aggregation_strategy` to work correctly. That might be a bit delicate if the long input does not contain words (imagine protein inputs/models).




@Narsil I don't think I understand your comment. LMs are trained with a max length size and they can't support anything bigger. The code will crash as it crashed above. I don't see any other solution than truncating the input to the max of the supported size by the LM. How else is it going to work?

@Narsil Sure, in the long run, we could split longer sentences in chunks of max seq length, process each chunk, concatenate later. However, this is more involved. Right now, I'd like a version of the code that truncates and doesn't crash. The way it is right now, I can't run my code. I could have a workaround in my code to truncate the text before it's sent to the pipeline, but that's suboptimal because it's hard to estimate how much to truncate since the tokenizer doesn't split at word level,, it works on BPEs (for BERT, IIRC). 

I think it's manageable to do the "better" version earlier.

If you need a quick hack don't hesitate to override the pipeline

```
class MyPipeline(TokenClassificationPipeline):
    def preprocess(....)
          .....
          self.tokenizer(...., truncation=True)
          ......

```

I do understand this is a hacky patch but should work for you in the mean time if you're ok with it.

The problem with adding that to master, is that it will become a supported option, and we will have to support that for the foreseeable future. It's easier to add the better option first (as it shouldn't be much longer)

Thanks for suggesting this, I think it's a "clean" hack :) 

I personally still think that it would be nice if the truncation would be supported. For whoever doesn't want truncation, sure, go ahead and split in chunks, etc. That's why there is a `truncation` parameter to begin with. There are plenty of cases where one wants just an embedding from the last hidden layer, usually the embedding corresponding to the CLS token. Or even if one is interested in longer embeddings, you usually use these in some downstream task, I see benefit for them to be limited in size and of the same size. 

What use case do you have in mind where you think you'd need variable-size, potentially long embeddings?

Bottom line the code crashes right now as it is for longer sequences, and it needs to be fixed + test cases in the test suite. Locally, I'll do the hack you suggested, that should work for now, thank you!

Crashing is actually ok, and it's expected, it is the standard behavior of most pipelines when the input cannot be handled in a sane way by models (for instance text-generation).

Sequence length IS a limit of most models, and some pipelines just can't sanely recover from it, it's a real limitation that has to be taken care of. 

For `token-classification`, we probably can get away with chunking properly and stop failing (probably will come with it's own set of warnings though).

But, maybe I misunderstood your use case, Do you just want the `embeddings` of the last layer, only for the CLS token, is that right ? What's the downstream use ?
There is `feature-extraction` which should enable you to not do any post-processing and recover the raw embeddings.

If there is a "legitimate" use for the truncation, adding it and supporting it is more than desired, I simply thought it was a workaround for the crashing, not the real goal.

I am using `feature-extraction` and that's the one crashing. Did you look at my code above in the original posting? @Narsil

Also all the problems I pointed out with code lines and fixes I was suggesting are in the `feature-extraction` class.

Brain got confused ! Thought it was `token-classification` for some reason, on `feature-extraction` for sure truncation makes sense !

@Narsil Now you're making sense to me :) I was surprised that you said it doesn't make sense... Now that we agree, do you have time to fix it, or shall I go ahead and work on it and you can help me with the PR?

Your hack is still useful and I may try that first to unblock myself. I really want to get these embeddings and while I got them "manually", it's so much nicer to use the pipelines!

@Narsil I'm not sure if I can use your hack. IIUC, to create a pipeline, you have to provide a task name and the task names are predefined, mapped to predefined pipelines. I don't think providing your own pipeline is supported, right? So in theory, it seems like a good idea, in practice, I don't think it's supported. 

You can call pipelines on their own, it's a bit less convenient,

```python

pipe = MyPipeline(model=AutoModel.from_pretrained(...), tokenizer=AutoTokenizer.from_pretrained(....)
```

There are other ways to hack this, it's python after all.

Not trying to reopen this and I know this fix should roll out soon, but if you end up on this issue and you can want an environment workaround, 4.10.3 was the last version where truncation passed into a feature extraction pipeline just works.

I think the test that was added should have a longer string. I think right now it's under 512 which is usually the max length and I don't think it's testing the truncation. 

On my local test, I had to pass in a parameter to the tokenizer for the `model_max_length` - for reasons I didn't understand, it didn't get it from the config file from the model.