Transformers 4.49.0 breaks nvdiffrast plugin loading
Author: drake7707
Pull request: no
Labels: name: bug

### System Info

Using pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel docker image

Compiled and installed the latest nvdiffrast wheel, along with transformers==4.49.0




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


See https://github.com/NVlabs/nvdiffrast/issues/216#issue-2914327965

Posting it here as well because something changed in transformers between version 4.48.3 and 4.49.0 that affects loading a compiled plugin with torch.cpp_extension.load

Steps to reproduce:

1. Built & installed the latest nvdiffrast wheel
2. Installed transformers==4.49.0 (this is the only difference in pip packages, if I install  4.48.3 it works fine)
3. Run:

```python
from transformers import pipeline
import nvdiffrast.torch as dr
glctx = dr.RasterizeCudaContext()
```
Will give segfault:
```
Fatal Python error: Segmentation fault

Current thread 0x00007fa849bd3740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1233 in create_module
  File "<frozen importlib._bootstrap>", line 573 in module_from_spec
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2140 in _import_module_from_library
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1746 in _jit_compile
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1314 in load
  File "/opt/conda/lib/python3.11/site-packages/nvdiffrast/torch/ops.py", line 125 in _get_plugin
  File "/opt/conda/lib/python3.11/site-packages/nvdiffrast/torch/ops.py", line 184 in __init__
  File "<stdin>", line 1 in <module>

Extension modules: _brotli, zstandard.backend_c, yaml._yaml, markupsafe._speedups, PIL._imaging, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.utils, av.option, av.descriptor, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, av.filter.loudnorm, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, google._upb._message, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5o, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5l, h5py._selector, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, psutil._psutil_linux, psutil._psutil_posix, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, kiwisolver._cext, sklearn.__check_build._check_build, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code (total: 260)
Segmentation fault (core dumped)
```

If you omit the transformers import or do the exact same with 4.48.3 it loads fine



### Expected behavior

Not crash the entire process with segfault

Comments:
hi @drake7707, to help me assign this one would you be willing to use [git bisect](https://git-scm.com/docs/git-bisect) to find the commit/PR that caused this issue?

Interestingly enough, I was going to so I cloned the repo and ran `pip install .` to use the latest commit and it works fine.

However if I then uninstall transformers and install from the pip repository with `pip install transformers==4.49.0` the test script segfaults, so this might either be already fixed or packaging for pip is doing something weird.

~Is there a way to find the exact commit the transformers==4.49.0 pip package corresponds to?~

Okay, commit a22a4378d97d06b7a1d9abad6e0086d30fdea199 is faulty so i'll try to bisect in reverse to newer commits and see which one fixed it


*some bisects later*

```
6a876462c308bd7cd7d3ca8e93abaa7d5b02e90e is the first bad commit
commit 6a876462c308bd7cd7d3ca8e93abaa7d5b02e90e
Author: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Date:   Thu Feb 27 12:53:42 2025 +0000

    Lazy import libraries in `src/transformers/image_utils.py` (#36435)

    * Lazy import libraries in `src/transformers/image_utils.py`

    * `make fixup`

    Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

    * Protect imports

    Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

    ---------

    Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

 src/transformers/image_utils.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
```

This is the commit (6a876462c308bd7cd7d3ca8e93abaa7d5b02e90e) that fixed it (I set bisect bad if it worked and bisect good if it segfaulted because bisect doesn't work in reverse). I'll try to find the commit between 4.48.3 and 4.49.0 that broke it now


*some more bisecting later*


This is the commit (e0646f3dce229a2552c56746eac618baf8210efa) that started causing segfault:

```
e0646f3dce229a2552c56746eac618baf8210efa is the first bad commit
commit e0646f3dce229a2552c56746eac618baf8210efa
Author: Raushan Turganbay <raushan@huggingface.co>
Date:   Fri Jan 10 11:05:29 2025 +0100

    Chat template: return vectorized output in processors (#34275)

    * update chat template

    * style

    * fix tests

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * typehints + docs

    * fix tests

    * remove unnecessary warnings

    * forgot code style :(

    * allow users to pass backend and num frames

    * Update docs/source/en/chat_templating.md

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/image_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * Update src/transformers/processing_utils.py

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

    * typo fix

    * style

    * address comments

    * align with "pipeline" template

    * update docs

    * update docs

    * unpack for all kwargs?

    * wrong conflict resolution while rebasing

    * tmp

    * update docs

    * Update docs/source/en/chat_templating.md

    Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

    * Update docs/source/en/chat_templating.md

    Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

    * Update docs/source/en/chat_templating.md

    Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

    * Update docs/source/en/chat_templating.md

    Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

    ---------

    Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>
    Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

 benchmark.py                                       | 132 +++++++++++++
 docs/source/en/chat_templating.md                  |  44 ++++-
 read_video.py                                      |  77 ++++++++
 run.py                                             | 107 ++++++++++
 src/transformers/image_utils.py                    | 215 +++++++++++++++++++++
 .../llava_onevision/processing_llava_onevision.py  |   2 +-
 src/transformers/processing_utils.py               | 137 ++++++++++++-
 src/transformers/utils/__init__.py                 |   2 +
 src/transformers/utils/import_utils.py             |  28 +++
 tests/models/llava/test_processor_llava.py         |  56 +++++-
 .../test_processor_llava_onevision.py              |  64 +++++-
 tests/models/mllama/test_processor_mllama.py       |  62 +++---
 12 files changed, 880 insertions(+), 46 deletions(-)
 create mode 100644 benchmark.py
 create mode 100644 read_video.py
 create mode 100644 run.py
```


Whew, this is the first time using bisect, neat. Hope it helps pinpointing what caused it


Ok, new repro, it's related to decord

``` python
from decord import VideoReader, cpu
import nvdiffrast.torch as dr
glctx = dr.RasterizeCudaContext()
```
Compiling decord from source and installing the wheel fixed it (as mentioned in https://github.com/dmlc/decord/issues/293).

Got it, and thanks for the diagnosis! cc @zucchini-nlp for visibility since we added the `decord` import in #34275. The issue seems to be a recent `decord` bug rather than anything we can fix ourselves, so I don't think any action is needed right now, but we might need to rework the import order or refactor things if they don't release a fixed version.

Oh yeah, historically decord always had some other issues when importing with torch, for which reason we default to `pyav` now. Thanks for tagging, we'll need to keep an eye on how decord is imported whenever video processing is updated/refactored

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.