Lib conflicts with ray and spacy

aman1: <p>I’m deploying my ML model on Anyscale which uses ray, transformers and spacy.</p>
<p>I’m getting this error while trying to run my process:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/tmp/ray/session_2024-01-04_07-06-56_749724_4614/runtime_resources/working_dir_files/_ray_pkg_9b970ef0493576c1/processor_kwextraction.py", line 229, in &lt;module&gt;
    model = en_core_web_trf.load()
  File "/home/ray/anaconda3/lib/python3.9/site-packages/en_core_web_trf/__init__.py", line 10, in load
    return load_model_from_init_py(__file__, **overrides)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy/util.py", line 649, in load_model_from_init_py
    return load_model_from_path(
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy/util.py", line 506, in load_model_from_path
    nlp = load_model_from_config(
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy/util.py", line 554, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy/language.py", line 1788, in from_config
    nlp = lang_cls(vocab=vocab, create_tokenizer=create_tokenizer, meta=meta)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy/language.py", line 163, in __init__
    util.registry._entry_point_factories.get_all()
  File "/home/ray/anaconda3/lib/python3.9/site-packages/catalogue/__init__.py", line 110, in get_all
    result.update(self.get_entry_points())
  File "/home/ray/anaconda3/lib/python3.9/site-packages/catalogue/__init__.py", line 125, in get_entry_points
    result[entry_point.name] = entry_point.load()
  File "/home/ray/anaconda3/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/home/ray/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "&lt;frozen importlib._bootstrap&gt;", line 1030, in _gcd_import
  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 972, in _find_and_load_unlocked
  File "&lt;frozen importlib._bootstrap&gt;", line 228, in _call_with_frames_removed
  File "&lt;frozen importlib._bootstrap&gt;", line 1030, in _gcd_import
  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 986, in _find_and_load_unlocked
  File "&lt;frozen importlib._bootstrap&gt;", line 680, in _load_unlocked
  File "&lt;frozen importlib._bootstrap_external&gt;", line 850, in exec_module
  File "&lt;frozen importlib._bootstrap&gt;", line 228, in _call_with_frames_removed
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy_transformers/__init__.py", line 1, in &lt;module&gt;
    from . import architectures
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy_transformers/architectures.py", line 6, in &lt;module&gt;
    from .layers import TransformerModel, TransformerListener
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy_transformers/layers/__init__.py", line 1, in &lt;module&gt;
    from .listener import TransformerListener
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy_transformers/layers/listener.py", line 4, in &lt;module&gt;
    from ..data_classes import TransformerData
  File "/home/ray/anaconda3/lib/python3.9/site-packages/spacy_transformers/data_classes.py", line 3, in &lt;module&gt;
    import torch
  File "/home/ray/anaconda3/lib/python3.9/site-packages/torch/__init__.py", line 235, in &lt;module&gt;
    from torch._C import *  # noqa: F403
ImportError: /home/ray/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so: undefined symbol: ncclRedOpDestroy
</code></pre>
<p>Version of ray: anyscale/ray:2.1.0-py39-cu112</p>
<p>These are the dependencies:</p>
<p>pandas<br>
pymongo==4.1.1<br>
pymongo[srv]<br>
spacy[cuda112]==3.4.2<br>
kafka-python<br>
elasticsearch==7.9.1<br>
htmldate<br>
urllib3&lt;1.27</p>
<p>These are the post build dependencies:</p>
<p>python -m pip install “pymongo[srv]”<br>
pip install transformers<br>
pip install -U sentence-transformers<br>
python -m spacy download en_core_web_trf<br>
python -m spacy download en_core_web_md</p>

ptrblck: <p>The current PyTorch binaries ship with <code>NCCL&gt;=2.18</code> and <code>ncclRedOpDestroy</code> was introduced in <code>NCCL==2.11.4</code> approx. 3 years ago.<br>
I don’t know which dependencies <code>ray</code> uses, but you might want to double check NCCL wasn’t downgraded to an ancient version.</p>

aman1: <p>So from the logs, I see that I have NCCL version 2.18.1</p>
<pre><code class="lang-auto">nvidia-nccl-cu12==2.18.1
</code></pre>

aman1: <p>I have resolved this issue. To anyone else who encounters this issue in future, the problem was the incompatibility between ray and cuda versions. Problem got resolved using CUDA 116 with Ray 2.5.0.</p>