Output of pipeline feature extraction
Author: Stuffooh
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

Hi,

I am using the new pipeline feature of transformers for feature extraction and I have to say it's amazing. However I would like to alter the output of the pipeline slightly but I am not sure how to and I was hoping some people of the community could point me into the right direction.

I am using the following code snippet in my script:

```
nlp = pipeline('feature-extraction', model=args.model, config=args.config, tokenizer=args.tokenizer, device=args.device)

features = nlp(value)
features = np.squeeze(features)
features = features.astype('float32')

h5f = h5py.File(args.output_path, 'a')
h5f.create_dataset(key, data=features)
h5f.close()
```
The output for every value I put into the pipeline has the shape of (512, 768) no matter the length of the sentence I put into the pipeline.

I understand the shape of 512 is caused by padding and 768 because of the number of hidden states.

However I would like to have the output to be (15, 768) or (312, 768) for example depending on the input instead of always (512, 768). I know this is not standard but for my purpose I need this format.

Could someone please point me to the right direction how to achieve this?

Thanks a lot! I'm really at a loss here.


Comments:
I cannot reproduce this. Can you post a verifiable example with full code that I can just copy and paste?

Here you see that I just get 7 tokens back:

```python
import numpy as np
from transformers import AutoTokenizer, AutoModel, pipeline


model = AutoModel.from_pretrained('distilbert-base-uncased')
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
nlp = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

features = nlp('Do you like cookies ?')
print(features)
features = np.squeeze(features)
print(features.shape)
# (7, 768)
```

@BramVanroy Hi thanks for the quick reply.

I have used the code example you provided and get the same output again:

```
import numpy as np
from transformers import AutoTokenizer, AutoModel, pipeline


model = AutoModel.from_pretrained('distilbert-base-uncased')
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
nlp = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

features = nlp('Do you like cookies ?')
features = np.squeeze(features)
print(feature.shape)
# (512, 768)
```

Yesterday I deleted my transformers directory and cloned the github repo of transformers again and used pip --install . to set everything up so I should be on the most recent version with no differences.

Hm, that is odd. I have tested with 2.5.0 and 2.5.1 and both give me 7 tokens back.

Can you run `python transformers-cli env` and paste the result here?

I actually have a similar issue, but with the Fast Tokenizer for the `bert-base-uncased` model

```
model = BertModel.from_pretrained('bert-base-uncased')

tokenizer_fast = BertTokenizerFast.from_pretrained('bert-base-uncased', 
add_special_tokens=False)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', 
add_special_tokens=False)

sentence = 'We are very happy to include pipeline into the transformers repository.'

nlp = pipeline('feature-extraction', model=model, tokenizer=tokenizer, device=0)

nlp2 = pipeline('feature-extraction', model=model, tokenizer=tokenizer_fast, device=0)

tnsor = np.squeeze(nlp(sentence))
# (14, 768)

tnsor = np.squeeze(nlp2(sentence))
# (512, 768)

```

The "slow" tokenizer gives me the expected 14 tokens (which is strange too because I set `add_special_tokens=False` but not relevant for this question) but the fast tokenizer gives me the padded 512 tokens.

Any ideas? Thanks!

cc @mfuntowicz Users in this thread report that the behaviour of the fast tokenizer differs from the slow ones with respect to the pipeline. When the pipeline is used for feature extraction, the fast tokenizers return the fully padded output (512) instead of the expected output (number of subtokens). Not sure if related to https://github.com/huggingface/transformers/pull/2961

@BramVanroy I decided to clone and rebuild transformers again to make 100% sure I'm on the most recent version and have a clean working environment. After doing so I got the expected result of shape (<512, 768).

In the end I'm not sure what the problem was. Should I close this issue or keep it open for @mabergerx ?

@mabergerx Try cloning the code into a new directory and rebuild from source. This ended up fixing the problem for me.

You can keep it open for now, because it seems to indicate some inconsistencies between versions or even commits. I will have a closer look when I find the time.

Regarding `add_special_tokens` behaviour, this is one of the major difference between fast and slow tokenizers. Fast requires the parameter to be defined at creation time (as you did), for the slow ones, it should be provided while calling methods like `encode`, `encode_plus`, `batch_encode_plus`. 

For the initial padding issue, we fixed some stuff related to padding on transformers 2.5.1 which might also have resolved your issue. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.