Performance question for pipelines (feature extraction)
Author: ioana-blue
Pull request: no

Me again and my pipelines :) Feature extraction. I ran some small run on my laptop 1000 samples, it took about 10 mins. I ran the same on a gpu and it took 35 mins (batch_size was 16). I'm not sure what the problem is and wondering if anyone has a hunch. I ended up "batching" the inputs myself, something along these lines:

```
 # batch the inputs
    crt_batch = 0
    for i in range(0, len(predict_inputs), batch_size):
        last = i + batch_size if i + batch_size < len(predict_inputs) else len(predict_inputs)
        outputs = feature_extractor(inputs = predict_inputs[i:last], truncation=True)
```

Comments:
I believe using the pipeline as an iterator should result in greatly improved results - as shown here (using a `dataset`): https://huggingface.co/transformers/main_classes/pipelines.html#the-pipeline-abstraction

Pinging @Narsil for advice

@LysandreJik Thank you for your suggestion, I'll take a look. That may also take care of batching for me. 

Yes, as @LysandreJik said, using a real Dataset will help.

Using a list will work too, but less convenient since you need to wait for the whole list to be processed to be able to work on your items, the Dataset should work out of the box. Lists need to work that way for backward compatiblity.

The manual batching yields lower results for exactly the reason mentioned in the docs, you are stopping too long and don't keep the GPU fed when using recurring calls to the pipeline.

You should also get a warning being displayed when running the loop displayed as you showed.

@Narsil Unfortunately, when I don't use the manual batching and I provide the whole list as inputs (using a python list), it crashed with the OOM exception that I mentioned in #14327, which made me think that there might be a problem with batching. 

Is your model a tensorflow model ?

Can you really provide a full example displaying the error please ?

`batch_size` is working as it should, and there is definitely no way it would batch things if not for `batch_size` actually.
That being said, I trust you're seeing an issue, but without the *full code* to reproduce it's very hard to help you.

I'll switch back to no (manual) batching and I'll share with you what I get - it's really cryptic, sometimes I see no output, just the job being killed. I'll get to this later today, sorry, meetings today. Thanks a lot for your help, highly appreciate it! 

My model is a pytorch model. I'm using a model from the Hub. 

@Narsil I think in a way I got rid of the out of memory issue. I added the batch size as a parameter to my model and that seems to do something, as in, I don't have the jobs crashing. However, I still experience the very long time to get embeddings from 1K samples (more than 30 mins). The `batch_size` parameter seems to be ignored when used in the feature extractor, but once I put it in the model config, that fixed it. 

I also ran with batch sizes of 100, which I think are too big to fit in the gpu memory, and it still worked. I wonder if somewhere this batch size is overwritten. I'm attaching my script, maybe you can spot a problem. Thank you for your help! 


My script: 
[text_embeddings.zip](https://github.com/huggingface/transformers/files/7528576/text_embeddings.zip)



@ioana-blue Sorry for the late reply, but the error seems simple enough.

The `batch_size` (and `num_workers` are `__call__` arguments.


`pipe = pipeline(...., batch_size=10); pipe(data)` -> `pipe = pipeline(...); pipe(data, batch_size=10)`

This is an understandable error for sure as most arguments can be defined in both places.
I'll think of the best approach to solve the issue and propose a PR:

- Either purely doc oriented
- Make it possible to do
- Add warnings for unused arguments maybe ?

Add warnings for unused arguments is great as it will catch more errors. I'll try what you suggested and let you know the results. Thank you!

I edited as you suggested and now I run it like this: @Narsil 

```
outputs = feature_extractor(inputs = predict_inputs, truncation=True, batch_size = data_args.batch_size)
```

It's been running for long enough that I know it's still experiencing the same issue. Perhaps it is not running on the gpu at all? I'm running out of ideas. 

With the new way of calling it, it took more than 1h to compute embeddings for 1K samples. 

Are you adding `pipe = pipeline(...., device=0)` for instance to run on your GPU ? 

https://huggingface.co/transformers/main_classes/pipelines.html?highlight=pipeline#transformers.Pipeline
(This docs needs to be updated a bit to be more readable, but it does specify the `device=0` to use GPU.)

Nope :/ That's probably it since the default is -1 which means cpu. brb with updates... Thanks for your help!

That was it, thanks a lot, now it's super fast. 

Sorry the doc wasn't more explicit :(

Cheers !

I had a similar issue where the batch_size wasn't changing regardless of what argument I entered. I fixed this by changing the batch and batch_eval values in the config object that you pass into  input_pipeline.get_data_from_tfds()