🚨🚨🚨 [`Refactor`] Move third-party related utility files into `integrations/` folder 🚨🚨🚨
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title and to address https://github.com/huggingface/transformers/pull/25077#discussion_r1283042392 
Let's move all third party libs (outside HF ecosystem) related utility files inside `lib_integrations/`, currently to the best of my knowledge bitsandbytes and deepspeed are the only 2 third party libs that we use as a plugin integration and is outside HF ecosystem

cc @sgugger and @stas00 as it touches DS related code

Let me see first if the CI passes

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi Younes,

Unless I'm mistaken these would be BC breaking changes.

```
$ PYTHONPATH=src python -c "from transformers.deepspeed import HfDeepSpeedConfig"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers.deepspeed'
```

Changing the doc to use the new API won't unbreak users' code.

(In general for deepspeed integration issues please tag @pacman100 who is the current maintainer of the integration.)

unrelated - perhaps there could be a neater name than `lib_integrations` - this is awkward IMHO.

Perhaps just `integrations` or `integration` or `external` - in other words one word would be smoother. it's obvious that those are `libs`. all python packages are.

```
- from transformers.lib_integrations.deepspeed import HfDeepSpeedConfig
+ from transformers.integration.deepspeed import HfDeepSpeedConfig
```

Yes I had mentioned integrations as well as a name for the folder. Note that we usually do not guarantee backward compatibility with imports not at the init level (anything not imported in the main init or a subfolder init is considered private), but we can keep a deepspeed module that reimports `HfDeepSpeedConfig` if this line is in a lot of places.

I did not used `integrations` as there is already a module named `integrations.py` in the same place. Maybe renaming `lib_integrations` to `external` would be better, WDYT? 

Use a single `integration`? Since once you add the specifics it's a clear `transformers.integration.what`

as in `transformers.integration.deepspeed` aka transformers' integration of deepspeed

As long as the objects are re-imported, it's completely fine (and non-breaking) to change `integrations.py` to an `integrations` folder.

Ah yes makes sense, modified it accordingly

Actually sorry, forgot to do it for peft, will do it now

_The documentation is not available anymore as the PR was closed or merged._

@sgugger the PR is now ready again ! 🙏 
The failing CI is related with the Hub rate limit issue we have been discussing internally (cc @ydshieh), would like to have a review if possible and I can put the PR on hold until the issue gets solved ! 

Thank you all! Feedback should be addressed, @pacman100 I am happy to propose a patch in accelerate after this gets merged, regarding the docs, please see: https://github.com/younesbelkada/transformers/blob/move-integrations/docs/source/en/main_classes/deepspeed.md which has been adapted with the refactor, let me know if there are some other place where the doc should be updated

@younesbelkada as said before, we will need to keep the current DeepSpeed module to import back objects for backward compatibility (liek `file_utils.py` does), it's too breaking otherwise.

Ah makes sense, reverted that and added a comment in the file

Thanks everyone for your reviews, regarding the large diff on the doctest file my intuition is that it got large because I added a new line that changed the alphabetical order of the file but I am not sure. (maybe @ydshieh can confirm)
I also realised we might need to keep `utils/bitsandbytes.py` otherwise it would be too breaking, so I reverted it back and did a similar approach than `deepspeed.py` 

The large diff is fixed with https://github.com/huggingface/transformers/pull/25680 thanks to @ydshieh 

@sgugger , ran the daily CI (with a reduced number of models tests) thanks to @ydshieh and you can see the results here: https://github.com/huggingface/transformers/actions/runs/5965296109 
compared the report with the report of the same day and I don't see any surprising difference / test failure caused by this PR. Therefore I think we can merge it ! Can you please have a final look ? 🙏 Thanks!