Repository: pytorch/rl

Issue:
[BUG] Nightly wheel fails at importing although non-nightly one works fine.
## Describe the bug

I'm running Python 3.10 and PyTorch 2.0.1. (See the environment below for more details)

Installing TorchRL from its wheel works fine.
```
# From a new container.
$ pip install torchrl
$ python
>>> import torchrl
# All good.
```

However, installing the nightly wheel raises an undefined symbol error.

```
# From a new container.
$ pip install torchrl-nighly
$ python -v 
>>> import torchrl
# /opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/__pycache__/samplers.cpython-310.pyc matches /opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/samplers.py
# code object from '/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/__pycache__/samplers.cpython-310.pyc'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/__init__.py", line 32, in <module>
    import torchrl.collectors
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/collectors/__init__.py", line 6, in <module>
    from .collectors import (
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/collectors/collectors.py", line 38, in <module>
    from torchrl.data.tensor_specs import TensorSpec
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/__init__.py", line 6, in <module>
    from . import datasets
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/datasets/__init__.py", line 1, in <module>
    from .d4rl import D4RLExperienceReplay
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/datasets/d4rl.py", line 14, in <module>
    from torchrl.data.replay_buffers import TensorDictReplayBuffer
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/__init__.py", line 6, in <module>
    from .replay_buffers import (
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/replay_buffers.py", line 25, in <module>
    from torchrl.data.replay_buffers.samplers import (
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/data/replay_buffers/samplers.py", line 13, in <module>
    from torchrl._torchrl import (
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torchrl/_torchrl.so: undefined symbol: _ZN5torch8autograd13_wrap_outputsERKSt6vectorIN2at6TensorESaIS3_EERKSt13unordered_setIPN3c1010TensorImplESt4hashISB_ESt8equal_toISB_ESaISB_EESJ_NS9_8ArrayRefINS9_8optionalIS3_EEEERKSt10shared_ptrINS0_4NodeEESt8functionIFS5_S5_S5_EESJ_
```

Also note that it installs `tensordict` and not `tensordict-nightly` as its dependency. It may be better if it depended on the nightly version of TensorDict as the libraries are tightly coupled.

## System info

My environment:

```
$ conda env export

name: implicit-pg
channels:
  - pytorch
  - nvidia
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_kmp_llvm
  - alsa-lib=1.2.8=h166bdaf_0
  - attr=2.5.1=h166bdaf_1
  - blas=1.0=mkl
  - brotli=1.0.9=h166bdaf_8
  - brotli-bin=1.0.9=h166bdaf_8
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2023.5.7=hbcca054_0
  - cairo=1.16.0=ha61ee94_1014
  - certifi=2023.5.7=pyhd8ed1ab_0
  - charset-normalizer=3.1.0=pyhd8ed1ab_0
  - contourpy=1.1.0=py310hd41b1e2_0
  - cuda-cudart=11.8.89=0
  - cuda-cupti=11.8.87=0
  - cuda-libraries=11.8.0=0
  - cuda-nvrtc=11.8.89=0
  - cuda-nvtx=11.8.86=0
  - cuda-runtime=11.8.0=0
  - cycler=0.11.0=pyhd8ed1ab_0
  - dbus=1.13.6=h5008d03_3
  - expat=2.5.0=hcb278e6_1
  - ffmpeg=4.3=hf484d3e_0
  - fftw=3.3.10=nompi_hc118613_108
  - filelock=3.12.2=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.40.0=py310h2372a71_0
  - freetype=2.12.1=hca18f0e_1
  - gettext=0.21.1=h27087fc_0
  - glib=2.76.3=hfc55251_0
  - glib-tools=2.76.3=hfc55251_0
  - gmp=6.2.1=h58526e2_0
  - gmpy2=2.1.2=py310h3ec546c_1
  - gnutls=3.6.13=h85f3911_1
  - graphite2=1.3.13=h58526e2_1001
  - gst-plugins-base=1.22.0=h4243ec0_2
  - gstreamer=1.22.0=h25f0c4b_2
  - gstreamer-orc=0.4.34=hd590300_0
  - harfbuzz=6.0.0=h8e241bc_0
  - icu=70.1=h27087fc_0
  - idna=3.4=pyhd8ed1ab_0
  - imageio=2.31.1=pyh24c5eb1_0
  - jack=1.9.22=h11f4161_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jpeg=9e=h0b41bf4_3
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.4=py310hbf28c38_1
  - krb5=1.20.1=h81ceb04_0
  - lame=3.100=h166bdaf_1003
  - lcms2=2.15=hfd0df8a_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libblas=3.9.0=16_linux64_mkl
  - libbrotlicommon=1.0.9=h166bdaf_8
  - libbrotlidec=1.0.9=h166bdaf_8
  - libbrotlienc=1.0.9=h166bdaf_8
  - libcap=2.67=he9d0100_0
  - libcblas=3.9.0=16_linux64_mkl
  - libclang=15.0.7=default_h7634d5b_2
  - libclang13=15.0.7=default_h9986a30_2
  - libcublas=11.11.3.6=0
  - libcufft=10.9.0.58=0
  - libcufile=1.6.1.9=0
  - libcups=2.3.3=h36d4200_3
  - libcurand=10.3.2.106=0
  - libcusolver=11.4.1.48=0
  - libcusparse=11.7.5.86=0
  - libdb=6.2.32=h9c3ff4c_0
  - libdeflate=1.17=h0b41bf4_0
  - libedit=3.1.20191231=he28a2e2_2
  - libevent=2.1.10=h28343ad_4
  - libexpat=2.5.0=hcb278e6_1
  - libffi=3.4.2=h7f98852_5
  - libflac=1.4.2=h27087fc_0
  - libgcc-ng=13.1.0=he5830b7_0
  - libgcrypt=1.10.1=h166bdaf_0
  - libgfortran-ng=13.1.0=h69a702a_0
  - libgfortran5=13.1.0=h15d22d2_0
  - libglib=2.76.3=hebfc3b9_0
  - libgpg-error=1.46=h620e276_0
  - libhwloc=2.9.1=hd6dc26d_0
  - libiconv=1.17=h166bdaf_0
  - liblapack=3.9.0=16_linux64_mkl
  - libllvm15=15.0.7=hadd5161_1
  - libnpp=11.8.0.86=0
  - libnsl=2.0.0=h7f98852_0
  - libnvjpeg=11.9.0.86=0
  - libogg=1.3.4=h7f98852_1
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.39=h753d276_0
  - libpq=15.3=hbcd7760_1
  - libsndfile=1.2.0=hb75c966_0
  - libsqlite=3.42.0=h2797004_0
  - libstdcxx-ng=13.1.0=hfd8a6a1_0
  - libsystemd0=253=h8c4010b_1
  - libtiff=4.5.0=h6adf6a1_2
  - libtool=2.4.7=h27087fc_0
  - libudev1=253=h0b41bf4_1
  - libuuid=2.38.1=h0b41bf4_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp-base=1.3.0=h0b41bf4_0
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.5.0=h79f4944_1
  - libxml2=2.10.3=hca2bb57_4
  - libzlib=1.2.13=hd590300_5
  - llvm-openmp=16.0.6=h4dfa4b3_0
  - lz4-c=1.9.4=hcb278e6_0
  - markupsafe=2.1.3=py310h2372a71_0
  - matplotlib=3.7.1=py310hff52083_0
  - matplotlib-base=3.7.1=py310he60537e_0
  - mkl=2022.2.1=h84fe81f_16997
  - mpc=1.3.1=hfe3b2da_0
  - mpfr=4.2.0=hb012696_0
  - mpg123=1.31.3=hcb278e6_0
  - mpmath=1.3.0=pyhd8ed1ab_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=8.0.32=hf1915f5_3
  - mysql-libs=8.0.32=hca2cd23_3
  - ncurses=6.4=hcb278e6_0
  - nettle=3.6=he412f7d_0
  - networkx=3.1=pyhd8ed1ab_0
  - nspr=4.35=h27087fc_0
  - nss=3.89=he45b914_0
  - numpy=1.24.3=py310ha4c1d20_0
  - openh264=2.1.1=h780b84a_0
  - openjpeg=2.5.0=hfec8fc6_2
  - openssl=3.1.1=hd590300_1
  - packaging=23.1=pyhd8ed1ab_0
  - pcre2=10.40=hc3806b6_0
  - pillow=9.4.0=py310h023d228_1
  - pip=23.1.2=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - ply=3.11=py_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - pulseaudio=16.1=hcb278e6_3
  - pulseaudio-client=16.1=h5195f5e_3
  - pulseaudio-daemon=16.1=ha8d29e2_3
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pyqt=5.15.7=py310hab646b1_3
  - pyqt5-sip=12.11.0=py310heca2aa9_3
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.10.11=he550d4f_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.10=3_cp310
  - pytorch=2.0.1=py3.10_cuda11.8_cudnn8.7.0_0
  - pytorch-cuda=11.8=h7e8668a_5
  - pytorch-mutex=1.0=cuda
  - qt-main=5.15.8=h5d23da1_6
  - readline=8.2=h8228510_1
  - requests=2.31.0=pyhd8ed1ab_0
  - setuptools=67.7.2=pyhd8ed1ab_0
  - sip=6.7.9=py310hc6cd4ac_0
  - six=1.16.0=pyh6c4a22f_0
  - sympy=1.12=pypyh9d50eac_103
  - tbb=2021.9.0=hf52228f_0
  - tk=8.6.12=h27826a3_0
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - torchtriton=2.0.0=py310
  - torchvision=0.15.2=py310_cu118
  - tornado=6.3.2=py310h2372a71_0
  - typing_extensions=4.6.3=pyha770c72_0
  - tzdata=2023c=h71feb2d_0
  - unicodedata2=15.0.0=py310h5764c6d_0
  - urllib3=2.0.3=pyhd8ed1ab_0
  - wheel=0.40.0=pyhd8ed1ab_0
  - xcb-util=0.4.0=h516909a_0
  - xcb-util-image=0.4.0=h166bdaf_0
  - xcb-util-keysyms=0.4.0=h516909a_0
  - xcb-util-renderutil=0.3.9=h166bdaf_0
  - xcb-util-wm=0.4.1=h516909a_0
  - xkeyboard-config=2.38=h0b41bf4_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.1.1=hd590300_0
  - xorg-libsm=1.2.4=h7391055_0
  - xorg-libx11=1.8.4=h0b41bf4_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - zlib=1.2.13=hd590300_5
  - zstd=1.5.2=h3eb15da_6
  - pip:
      - absl-py==1.4.0
      - ale-py==0.8.1
      - antlr4-python3-runtime==4.9.3
      - appdirs==1.4.4
      - autorom==0.4.2
      - autorom-accept-rom-license==0.6.1
      - box2d-py==2.3.5
      - cfgv==3.3.1
      - click==8.1.3
      - cloudpickle==2.2.1
      - distlib==0.3.6
      - docker-pycreds==0.4.0
      - farama-notifications==0.0.4
      - gitdb==4.0.10
      - gitpython==3.1.31
      - glfw==2.5.9
      - gymnasium==0.28.1
      - hydra-core==1.3.2
      - identify==2.5.24
      - importlib-resources==5.12.0
      - jax-jumpy==1.0.0
      - mujoco==2.3.5
      - nodeenv==1.8.0
      - omegaconf==2.3.0
      - pathtools==0.1.2
      - platformdirs==3.5.3
      - pre-commit==3.3.3
      - protobuf==4.23.3
      - psutil==5.9.5
      - pygame==2.1.3
      - pyopengl==3.1.7
      - pyyaml==6.0
      - sentry-sdk==1.25.1
      - setproctitle==1.3.2
      - shimmy==0.2.1
      - smmap==5.0.0
      - swig==4.1.1
      - tqdm==4.65.0
      - virtualenv==20.23.1
      - wandb==0.15.4
prefix: /opt/conda/envs/implicit-pg
```

Running on `linux-amd64` with Docker.

## Additional context

I read https://pytorch.org/rl/reference/generated/knowledge_base/VERSIONING_ISSUES.html, but it's about using PyTorch 1.13.

## Checklist

- [x] I have checked that there is no similar issue in the repo (**required**)
- [x] I have read the [documentation](https://github.com/pytorch/rl/tree/main/docs/) (**required**)
- [x] I have provided a minimal working example to reproduce the bug (**required**)



Discussion:
thanks for reporting
I think (but might be wrong) that this is due to an issue we currently have with both repos where `pip install -e .` fails but `python setup.py develop` works, and I have no idea why...
Good tip on tensordict-nightly dependency though, that's an easy fix

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -67,9 +67,11 @@ def write_version_file(version):
         f.write("git_version = {}\n".format(repr(sha)))
 
 
-def _get_pytorch_version():
+def _get_pytorch_version(is_nightly):
     # if "PYTORCH_VERSION" in os.environ:
     #     return f"torch=={os.environ['PYTORCH_VERSION']}"
+    if is_nightly:
+        return "torch>=2.1.0.dev"
     return "torch"
 
 
@@ -164,6 +166,10 @@ def _main(argv):
     args, unknown = parse_args(argv)
     name = args.package_name
     is_nightly = "nightly" in name
+    if is_nightly:
+        tensordict_dep = "tensordict>=0.1.1"
+    else:
+        tensordict_dep = "tensordict-nightly"
 
     if is_nightly:
         version = get_nightly_version()
@@ -173,7 +179,7 @@ def _main(argv):
     else:
         version = get_version()
 
-    pytorch_package_dep = _get_pytorch_version()
+    pytorch_package_dep = _get_pytorch_version(is_nightly)
     print("-- PyTorch dependency:", pytorch_package_dep)
     # branch = _run_cmd(["git", "rev-parse", "--abbrev-ref", "HEAD"])
     # tag = _run_cmd(["git", "describe", "--tags", "--exact-match", "@"])
@@ -204,7 +210,7 @@ def _main(argv):
             "numpy",
             "packaging",
             "cloudpickle",
-            "tensordict>=0.1.1",
+            tensordict_dep,
         ],
         extras_require={
             "atari": [


Test patch: