importing `xgboost` before `transformers.pipeline` breaks pipeline loading for `image-feature-extraction` pipelines
Author: dan-jacobson
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.2
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.2
- Huggingface_hub version: 0.25.0
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

 @Rocketknight1, @amyeroberts, @qubvel

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
mkdir pipeline_test && cd pipeline_test
uv init
uv add transformers torch pillow xgboost
uv run python

import xgboost
from transformers import pipeline
pipe = pipeline(task="image-feature-extraction")
```
This fails silently in my terminal. For some reason I can't get a good error. In VSCode I can sometimes get: 
`[1]    88800 segmentation fault  python`.

Interestingly, it's the order of `xgboost`/`transformers` import that matters. This works on my machine:
```
from transformers import pipeline
import xgboost

pipe = pipeline(task="image-feature-extraction")
```

A couple other things I noticed:
- Does not appear to be `model` specific. I was able to get failures using multiple `dinov2` models, as well as the base model, `google/vit-base-patch16-224`.
-  Does not appear to be a problem for all `pipelines`. I tried `task="feature-extraction"` and that worked. I haven't exhaustively tried other `pipelines`.

### Expected behavior

I expected my `pipeline` to load.

Comments:
Hey @dan-jacobson, I'm not sure it's `transformers`, I have reason to believe it's `torch` :grin: 

Can you let me know if doing `import xgboost` followed by `import torch` fails?

Or, another potential thing to look into: uninstall `torch` and install `tensorflow` as a backend for pipelines; now do the same: import xgboost and pipeline. Does that still fail?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

i stumbled on an adjacent problem, which probably hits the same underlying cause.

```python
import xgboost
import torch
copied_tensor = torch.randn(8000, 2000).clone()
```
this fails, but switching the torch and xgboost imports "fixes" the issue.

my usecase relies on loading models (as expected when you use `transformers`) and copying large-ish tensors is likely involved in the process.

the example expectedly fails with these versions (latest available for the python version): 
- `torch==2.8.0`
- `xgboost==3.0.5`
- `python=3.11.11`