Add warning if fast image processing is not done on the best available device
Author: yonigozlan
Pull request: yes

# What does this PR do?

As the title says. This is a softer version of this PR https://github.com/huggingface/transformers/pull/36268
The main goal here is to advertise that fast image processing can be done on cuda, where it is much much faster.
Specifying on which device the image processing is done was not supported on slow image processors, so it is not obvious that this is a possibility, hence the importance of advertising it :) 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36314). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qubvel or @molbap would have better insights than me 🙏 (although I am also happy to continue the review)

@yonigozlan thanks for working on this! 

I'm not sure this is a good idea because, for fine-tuning (and in some cases for inference) it's **common** to perform preprocessing on the CPU using a torch DataLoader with multiprocessing. As a result, we will be spammed with warnings from multiple workers. However, using the CPU is the expected behavior since we are splitting the load between the CPU and GPU.  

Additionally, as far as I remember, CUDA do not work with pure python multiprocessing, so switching the device will not work.

> I'm not sure this is a good idea because, for fine-tuning (and in some cases for inference) it's common to perform preprocessing on the CPU using a torch DataLoader with multiprocessing. As a result, we will be spammed with warnings from multiple workers. However, using the CPU is the expected behavior since we are splitting the load between the CPU and GPU.
> 
> Additionally, as far as I remember, CUDA do not work with pure python multiprocessing, so switching the device will not work.

Thanks a lot for your answers! Indeed using a dataloader with multiple workers and processing on cuda will raise errors, so thanks for flagging this @qubvel that's good to know! It's another strong argument against this: https://github.com/huggingface/transformers/pull/36268

I still think having a way to advertise processing on cuda would be nice, because it is often 10 to 100x faster, so it might be better that using multiple workers even when finetuning, plus it's a way not to have idle GPUs...
I'd say a warning could be warranted and not too annoying in inference mode at least? No sure how/if we can detect if we are training or not in the processors though...

 @qubvel made a very good point! This reminds me some old work I have done on Kaggle where I made data transformation on GPU with batch (but with TensorFlow : 😆)

https://www.kaggle.com/code/yihdarshieh/make-chris-deotte-s-data-augmentation-faster
https://www.kaggle.com/code/yihdarshieh/batch-implementation-of-more-data-augmentations

I was able to get 2x-5x speedup.

However, when I tired to chain data transformation and modeling training (only single GPU), it doesn't really get any meaningful speedup as total (which is reasonable).

If we would still advertise fast processor, we could make the message more precise in which scenario it would be good to use it.

BTW, it might be a good idea to prepare a simple example to demonstrate this `(it is often 10 to 100x faster)`, and provide a link to that example in the warning message.

> I still think having a way to advertise processing on cuda would be nice, because it is often 10 to 100x faster, so it might be better that using multiple workers even when finetuning, plus it's a way not to have idle GPUs...
I'd say a warning could be warranted and not too annoying in inference mode at least? No sure how/if we can detect if we are training or not in the processors though...

The idea behind using a dataloader and preprocessing on the CPU is that it happens in parallel with GPU computations. If preprocessing is not too heavy, you benefit from having it done on the CPU.

Simplified example, suppose we have:

 - preprocessing taking 40 ms on the CPU or 5 ms on the GPU
 - model batch processing on the GPU taking 100 ms
 
For GPU only, we will have 5 + 100 ms (sequential) for each step. While for CPU + GPU, we will still have 100 ms per step, because batches are prepared asynchronously on the CPU, resulting in no idle GPU time.

The only constraint is that CPU preprocessing should be faster than batch processing by the model. And that's not only a training case, evaluation is also common to make with multiprocessing dataloader set up. As an example, we recently integrated SigLIP2 model and evaluated it with [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark), where it's not possible to set GPU device for image processor because of the multiprocessing dataloader.

I agree, we should advertise that processing can be done on GPU as well, but that should be optional, and we should have both options enabled without warning. IMO adding more examples to model cards, documentation, and comms should help.

I would say that GPU preprocessing reduces the latency, so any scenarios where we need faster response will benefit, e.g., real-time api/applications. However, in terms of throughput, it would be more beneficial to utilize the CPU.

Thanks for the detailed explanations @qubvel! I didn't know dataloaders could run in parallel when running on CPU.
Makes perfect sense then not to add a warning. It also makes speed improvement on CPU important, such as the one in this PR https://github.com/huggingface/transformers/pull/36356!
Closing this for now then :)