AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node' during import in transformers
Author: tijmenvanetten
Pull request: no

### System Info

System Info: Apple M1 Max, MacOS Sonoma 14.4
Python version: 3.9
Environment installationss:
2024-05-16T08:35:09: Package                   Version
2024-05-16T08:35:09: ------------------------- ------------------
2024-05-16T08:35:09: absl-py                   2.1.0
2024-05-16T08:35:09: accelerate                0.30.1
2024-05-16T08:35:09: aiohttp                   3.9.5
2024-05-16T08:35:09: aiosignal                 1.3.1
2024-05-16T08:35:09: antlr4-python3-runtime    4.9.3
2024-05-16T08:35:09: apache-beam               2.56.0
2024-05-16T08:35:09: appdirs                   1.4.4
2024-05-16T08:35:09: async-timeout             4.0.3
2024-05-16T08:35:09: attrs                     23.2.0
2024-05-16T08:35:09: beautifulsoup4            4.12.3
2024-05-16T08:35:09: bitsandbytes              0.43.1
2024-05-16T08:35:09: black                     21.4b2
2024-05-16T08:35:09: blinker                   1.8.2
2024-05-16T08:35:09: blis                      0.7.11
2024-05-16T08:35:09: botshop                   0.0.43
2024-05-16T08:35:09: catalogue                 2.0.10
2024-05-16T08:35:09: certifi                   2024.2.2
2024-05-16T08:35:09: cffi                      1.16.0
2024-05-16T08:35:09: charset-normalizer        3.3.2
2024-05-16T08:35:09: click                     8.1.7
2024-05-16T08:35:09: cloudpickle               2.2.1
2024-05-16T08:35:09: contourpy                 1.2.1
2024-05-16T08:35:09: crcmod                    1.7
2024-05-16T08:35:09: cryptography              42.0.7
2024-05-16T08:35:09: cycler                    0.12.1
2024-05-16T08:35:09: cymem                     2.0.8
2024-05-16T08:35:09: datasets                  2.19.1
2024-05-16T08:35:09: detectron2                0.5
2024-05-16T08:35:09: dill                      0.3.1.1
2024-05-16T08:35:09: dnspython                 2.6.1
2024-05-16T08:35:09: docopt                    0.6.2
2024-05-16T08:35:09: docutils                  0.21.2
2024-05-16T08:35:09: en-core-web-sm            3.3.0
2024-05-16T08:35:09: et-xmlfile                1.1.0
2024-05-16T08:35:09: evaluate                  0.4.2
2024-05-16T08:35:09: exceptiongroup            1.2.1
2024-05-16T08:35:09: fairscale                 0.4.13
2024-05-16T08:35:09: fastavro                  1.9.4
2024-05-16T08:35:09: fasteners                 0.19
2024-05-16T08:35:09: file-stores               0.0.4
2024-05-16T08:35:09: filelock                  3.14.0
2024-05-16T08:35:09: Flask                     3.0.3
2024-05-16T08:35:09: Flask-Cors                4.0.1
2024-05-16T08:35:09: fonttools                 4.51.0
2024-05-16T08:35:09: form-recognizer           0.0.2
2024-05-16T08:35:09: frozenlist                1.4.1
2024-05-16T08:35:09: fsspec                    2024.3.1
2024-05-16T08:35:09: future                    1.0.0
2024-05-16T08:35:09: fvcore                    0.1.5.post20221221
2024-05-16T08:35:09: grpcio                    1.63.0
2024-05-16T08:35:09: hdfs                      2.7.3
2024-05-16T08:35:09: httplib2                  0.22.0
2024-05-16T08:35:09: huggingface-hub           0.23.0
2024-05-16T08:35:09: hydra-core                1.3.2
2024-05-16T08:35:09: idna                      3.7
2024-05-16T08:35:09: importlib_metadata        7.1.0
2024-05-16T08:35:09: importlib_resources       6.4.0
2024-05-16T08:35:09: iniconfig                 2.0.0
2024-05-16T08:35:09: iopath                    0.1.10
2024-05-16T08:35:09: itsdangerous              2.2.0
2024-05-16T08:35:09: Jinja2                    3.1.4
2024-05-16T08:35:09: joblib                    1.4.2
2024-05-16T08:35:09: Js2Py                     0.74
2024-05-16T08:35:09: jsonpickle                3.0.4
2024-05-16T08:35:09: jsonschema                4.22.0
2024-05-16T08:35:09: jsonschema-specifications 2023.12.1
2024-05-16T08:35:09: kiwisolver                1.4.5
2024-05-16T08:35:09: langcodes                 3.4.0
2024-05-16T08:35:09: language_data             1.2.0
2024-05-16T08:35:09: layoutparser              0.3.4
2024-05-16T08:35:09: marisa-trie               1.1.1
2024-05-16T08:35:09: Markdown                  3.6
2024-05-16T08:35:09: markdownify               0.12.1
2024-05-16T08:35:09: MarkupSafe                2.1.5
2024-05-16T08:35:09: matplotlib                3.9.0
2024-05-16T08:35:09: ml-training-base          0.0.6
2024-05-16T08:35:09: mlpug                     0.0.56
2024-05-16T08:35:09: mmh3                      3.0.0
2024-05-16T08:35:09: mpmath                    1.3.0
2024-05-16T08:35:09: msgpack                   1.0.8
2024-05-16T08:35:09: multi-device              0.0.1
2024-05-16T08:35:09: multidict                 6.0.5
2024-05-16T08:35:09: multiprocess              0.70.9
2024-05-16T08:35:09: murmurhash                1.0.10
2024-05-16T08:35:09: mypy-extensions           1.0.0
2024-05-16T08:35:09: networkx                  3.2.1
2024-05-16T08:35:09: neural-conversations      0.0.16
2024-05-16T08:35:09: nltk                      3.8.1
2024-05-16T08:35:09: numpy                     1.26.4
2024-05-16T08:35:09: nvidia-cublas-cu12        12.1.3.1
2024-05-16T08:35:09: nvidia-cuda-cupti-cu12    12.1.105
2024-05-16T08:35:09: nvidia-cuda-nvrtc-cu12    12.1.105
2024-05-16T08:35:09: nvidia-cuda-runtime-cu12  12.1.105
2024-05-16T08:35:09: nvidia-cudnn-cu12         8.9.2.26
2024-05-16T08:35:09: nvidia-cufft-cu12         11.0.2.54
2024-05-16T08:35:09: nvidia-curand-cu12        10.3.2.106
2024-05-16T08:35:09: nvidia-cusolver-cu12      11.4.5.107
2024-05-16T08:35:09: nvidia-cusparse-cu12      12.1.0.106
2024-05-16T08:35:09: nvidia-nccl-cu12          2.18.1
2024-05-16T08:35:09: nvidia-nvjitlink-cu12     12.4.127
2024-05-16T08:35:09: nvidia-nvtx-cu12          12.1.105
2024-05-16T08:35:09: objsize                   0.7.0
2024-05-16T08:35:09: omegaconf                 2.3.0
2024-05-16T08:35:09: opencv-python             4.9.0.80
2024-05-16T08:35:09: openpyxl                  3.1.2
2024-05-16T08:35:09: orjson                    3.10.3
2024-05-16T08:35:09: packaging                 24.0
2024-05-16T08:35:09: pandas                    2.2.2
2024-05-16T08:35:09: pathlib_abc               0.1.1
2024-05-16T08:35:09: pathspec                  0.12.1
2024-05-16T08:35:09: pathy                     0.11.0
2024-05-16T08:35:09: pdf2image                 1.17.0
2024-05-16T08:35:09: pdfminer.six              20231228
2024-05-16T08:35:09: pdfplumber                0.11.0
2024-05-16T08:35:09: Pillow                    9.5.0
2024-05-16T08:35:09: pip                       24.0
2024-05-16T08:35:09: platformdirs              4.2.2
2024-05-16T08:35:09: pluggy                    1.5.0
2024-05-16T08:35:09: portalocker               2.8.2
2024-05-16T08:35:09: preshed                   3.0.9
2024-05-16T08:35:09: proto-plus                1.23.0
2024-05-16T08:35:09: protobuf                  3.20.3
2024-05-16T08:35:09: psutil                    5.9.8
2024-05-16T08:35:09: py-mongodb                0.0.1
2024-05-16T08:35:09: pyarrow                   14.0.2
2024-05-16T08:35:09: pyarrow-hotfix            0.6
2024-05-16T08:35:09: pycocotools               2.0.7
2024-05-16T08:35:09: pycparser                 2.22
2024-05-16T08:35:09: pydantic                  1.8.2
2024-05-16T08:35:09: pydot                     1.4.2
2024-05-16T08:35:09: pyjsparser                2.7.1
2024-05-16T08:35:09: pymilvus                  2.1.0
2024-05-16T08:35:09: pymongo                   4.7.2
2024-05-16T08:35:09: pyparsing                 3.1.2
2024-05-16T08:35:09: pypdfium2                 4.30.0
2024-05-16T08:35:09: pytest                    8.2.0
2024-05-16T08:35:09: python-dateutil           2.9.0.post0
2024-05-16T08:35:09: python-dotenv             1.0.1
2024-05-16T08:35:09: pytz                      2024.1
2024-05-16T08:35:09: PyYAML                    6.0.1
2024-05-16T08:35:09: ray                       2.22.0
2024-05-16T08:35:09: redis                     5.0.4
2024-05-16T08:35:09: referencing               0.35.1
2024-05-16T08:35:09: regex                     2024.5.15
2024-05-16T08:35:09: requests                  2.31.0
2024-05-16T08:35:09: rpds-py                   0.18.1
2024-05-16T08:35:09: safetensors               0.4.3
2024-05-16T08:35:09: scikit-learn              1.4.2
2024-05-16T08:35:09: scipy                     1.13.0
2024-05-16T08:35:09: sentence-transformers     2.7.0
2024-05-16T08:35:09: setfit                    1.0.3
2024-05-16T08:35:09: setuptools                58.1.0
2024-05-16T08:35:09: shell-nlp                 0.0.8
2024-05-16T08:35:09: six                       1.16.0
2024-05-16T08:35:09: smart-open                6.4.0
2024-05-16T08:35:09: soupsieve                 2.5
2024-05-16T08:35:09: spacy                     3.3.3
2024-05-16T08:35:09: spacy-legacy              3.0.12
2024-05-16T08:35:09: spacy-loggers             1.0.5
2024-05-16T08:35:09: srsly                     2.4.8
2024-05-16T08:35:09: statistics                1.0.3.5
2024-05-16T08:35:09: sympy                     1.12
2024-05-16T08:35:09: tabulate                  0.9.0
2024-05-16T08:35:09: tensorboard               2.16.2
2024-05-16T08:35:09: tensorboard-data-server   0.7.2
2024-05-16T08:35:09: tensorboardX              2.6.2.2
2024-05-16T08:35:09: termcolor                 2.4.0
2024-05-16T08:35:09: thinc                     8.0.17
2024-05-16T08:35:09: threadpoolctl             3.5.0
2024-05-16T08:35:09: timm                      1.0.3
2024-05-16T08:35:09: tokenizers                0.19.1
2024-05-16T08:35:09: toml                      0.10.2
2024-05-16T08:35:09: tomli                     2.0.1
2024-05-16T08:35:09: torch                     2.1.2
2024-05-16T08:35:09: torchvision               0.16.2
2024-05-16T08:35:09: tqdm                      4.66.4
2024-05-16T08:35:09: transformers              4.40.2
2024-05-16T08:35:09: triton                    2.1.0
2024-05-16T08:35:09: typer                     0.4.2
2024-05-16T08:35:09: typing_extensions         4.11.0
2024-05-16T08:35:09: tzdata                    2024.1
2024-05-16T08:35:09: tzlocal                   5.2
2024-05-16T08:35:09: ujson                     5.4.0
2024-05-16T08:35:09: urllib3                   2.2.1
2024-05-16T08:35:09: virtual-sme               0.0.15
2024-05-16T08:35:09: visionscaper-pybase       0.3.5
2024-05-16T08:35:09: wasabi                    0.10.1
2024-05-16T08:35:09: Werkzeug                  3.0.3
2024-05-16T08:35:09: wheel                     0.43.0
2024-05-16T08:35:09: xxhash                    3.4.1
2024-05-16T08:35:09: yacs                      0.1.8
2024-05-16T08:35:09: yarl                      1.9.4
2024-05-16T08:35:09: zipp                      3.18.1
2024-05-16T08:35:09: zstandard                 0.22.0
 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Traceback (most recent call last):
  File "/mnt/azureml/cr/j/abf6e4fd321b4fa295293387592a8351/exe/wd/create_embeddings.py", line 30, in <module>
    from shell_nlp.text_structure_extraction.table_detection import (
  File "/usr/local/lib/python3.9/site-packages/shell_nlp/text_structure_extraction/table_detection.py", line 15, in <module>
    **from transformers import pipeline**
  File "/usr/local/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/usr/local/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/usr/local/lib/python3.9/site-packages/transformers/utils/__init__.py", line 33, in <module>
    from .generic import (
  File "/usr/local/lib/python3.9/site-packages/transformers/utils/generic.py", line 478, in <module>
    _torch_pytree.register_pytree_node(
AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node'

### Expected behavior

I would expect the script to run without encountering any errors. Specifically, the import statements and function calls should execute successfully, allowing the create_embeddings.py script to complete its intended operation of creating embeddings. The transformers library should properly import the pipeline module without any issues, and the torch.utils._pytree module should have the necessary attributes, including register_pytree_node, to avoid any AttributeError during the execution of the script.

Comments: