Repository: facebookresearch/fairscale

Issue:
Fail on install - CUDA 11.1.0
## 🐛 Bug

Hi,
pip install on below environment throws an error. I'm happy to provide more info if it would be useful. Thanks!

fatal error: multi_tensor_apply.cuh: No such file or directory

## Environment (NVIDIA-Python Docker: 20.10)
Python: 3.6
PyTorch: 1.7.0
CUDA: 11.1.0
cuDNN: 8.0.4

## Error
```
Error:
b'  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpl6xin7ge
       cwd: /tmp/pip-install-0_xiegmi/fairscale
  Complete output (158 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/fairscale
  copying fairscale/__init__.py -> build/lib.linux-x86_64-3.6/fairscale
  creating build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/grad_scaler.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/oss.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/utils.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/adam.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  creating build/lib.linux-x86_64-3.6/fairscale/nn
  copying fairscale/nn/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/dependency.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/pipe.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/stream.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/batchnorm.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/pipeline.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/checkpoint.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/copy.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/phony.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/worker.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/microbatch.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  creating build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/random.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/initialize.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/layers.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/cross_entropy.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/mappings.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/utils.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  creating build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  copying fairscale/nn/data_parallel/sharded_ddp.py -> build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  copying fairscale/nn/data_parallel/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  creating build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/top2gate.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/moelayer.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/profile.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/blockpartition.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/skippable.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/tracker.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/layout.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/portal.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/namespace.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  running egg_info
  writing fairscale.egg-info/PKG-INFO
  writing dependency_links to fairscale.egg-info/dependency_links.txt
  writing requirements to fairscale.egg-info/requires.txt
  writing top-level names to fairscale.egg-info/top_level.txt
  reading manifest file \'fairscale.egg-info/SOURCES.txt\'
  reading manifest template \'MANIFEST.in\'
  writing manifest file \'fairscale.egg-info/SOURCES.txt\'
  creating build/lib.linux-x86_64-3.6/fairscale/clib
  creating build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -> build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -> build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  running build_ext
  building \'fairscale.fused_adam_cuda\' extension
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  Emitting ninja build file /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/2] /usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options \'\'"\'"\'-fPIC\'"\'"\'\' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_86,code=sm_86 -std=c++14
  FAILED: /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o
  /usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options \'\'"\'"\'-fPIC\'"\'"\'\' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_86,code=sm_86 -std=c++14
  /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu:12:10: fatal error: multi_tensor_apply.cuh: No such file or directory
   #include "multi_tensor_apply.cuh"
            ^~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  [2/2] c++ -MMD -MF /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o.d -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  In file included from /usr/local/lib/python3.6/dist-packages/torch/include/ATen/Parallel.h:149:0,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/extension.h:4,
                   from /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp:1:
  /usr/local/lib/python3.6/dist-packages/torch/include/ATen/ParallelOpenMP.h:84:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   #pragma omp parallel for if ((end - begin) >= grain_size)
  
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/cpp_extension.py", line 1522, in _run_ninja_build
      env=env)
    File "/usr/lib/python3.6/subprocess.py", line 438, in run
      output=stdout, stderr=stderr)
  subprocess.CalledProcessError: Command \'[\'ninja\', \'-v\']\' returned non-zero exit status 1.
```


Discussion:
What is the command-line you are using to install?
Please try PR #196 and confirm whether or not it fixes your issue.
Hi, Yes! It fixed my issue and I was able to install successfully however when I attempted to import the following error occurred:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/fairscale/__init__.py", line 12, in <module>
    from . import nn
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/__init__.py", line 7, in <module>
    from .pipe import LazyModule, Pipe, PipeRPCWrapper
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/__init__.py", line 22, in <module>
    from .pipe import LazyModule, Pipe
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/pipe.py", line 36, in <module>
    from .async_schedule import Invocation, Location, ModuleWrapper
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/async_schedule.py", line 18, in <module>
    from .messages import Transport
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/messages.py", line 15, in <module>
    from fairscale.utils.object import pyobject_to_tensor, tensor_to_pyobject
ModuleNotFoundError: No module named 'fairscale.utils'
```

The file structure of the install looks like:
```
.:
__init__.py  clib                                             nn
__pycache__  fused_adam_cuda.cpython-36m-x86_64-linux-gnu.so  optim

./__pycache__:
__init__.cpython-36.pyc

./clib:
fused_adam_cuda

./clib/fused_adam_cuda:
fused_adam_cuda.cpp  fused_adam_cuda_kernel.cu

./nn:
__init__.py  __pycache__  data_parallel  model_parallel  moe  pipe

./nn/__pycache__:
__init__.cpython-36.pyc

./nn/data_parallel:
__init__.py  __pycache__  sharded_ddp.py

./nn/data_parallel/__pycache__:
__init__.cpython-36.pyc  sharded_ddp.cpython-36.pyc

./nn/model_parallel:
__init__.py  cross_entropy.py  layers.py    random.py
__pycache__  initialize.py     mappings.py  utils.py

./nn/model_parallel/__pycache__:
__init__.cpython-36.pyc       layers.cpython-36.pyc    utils.cpython-36.pyc
cross_entropy.cpython-36.pyc  mappings.cpython-36.pyc
initialize.cpython-36.pyc     random.cpython-36.pyc

./nn/moe:
__init__.py  __pycache__  moe_layer.py  top2gate.py

./nn/moe/__pycache__:
__init__.cpython-36.pyc  moe_layer.cpython-36.pyc  top2gate.cpython-36.pyc

./nn/pipe:
__init__.py        batchnorm.py   messages.py    pipeline.py  types.py
__pycache__        checkpoint.py  microbatch.py  rpc.py       worker.py
async_schedule.py  copy.py        phony.py       skip
balance            dependency.py  pipe.py        stream.py

./nn/pipe/__pycache__:
__init__.cpython-36.pyc        phony.cpython-36.pyc
async_schedule.cpython-36.pyc  pipe.cpython-36.pyc
batchnorm.cpython-36.pyc       pipeline.cpython-36.pyc
checkpoint.cpython-36.pyc      rpc.cpython-36.pyc
copy.cpython-36.pyc            stream.cpython-36.pyc
dependency.cpython-36.pyc      types.cpython-36.pyc
messages.cpython-36.pyc        worker.cpython-36.pyc
microbatch.cpython-36.pyc

./nn/pipe/balance:
__init__.py  __pycache__  blockpartition.py  profile.py

./nn/pipe/balance/__pycache__:
__init__.cpython-36.pyc  blockpartition.cpython-36.pyc  profile.cpython-36.pyc

./nn/pipe/skip:
__init__.py  layout.py     portal.py     tracker.py
__pycache__  namespace.py  skippable.py

./nn/pipe/skip/__pycache__:
__init__.cpython-36.pyc  namespace.cpython-36.pyc  skippable.cpython-36.pyc
layout.cpython-36.pyc    portal.cpython-36.pyc     tracker.cpython-36.pyc

./optim:
__init__.py  adam.py      grad_scaler.py  utils.py
__pycache__  adascale.py  oss.py

./optim/__pycache__:
__init__.cpython-36.pyc  adascale.cpython-36.pyc     oss.cpython-36.pyc
adam.cpython-36.pyc      grad_scaler.cpython-36.pyc  utils.cpython-36.pyc
```

I hope this information helps! I used the below command to install:
python3 -m pip install git+https://github.com/facebookresearch/fairscale.git@fix-install


Resolving patch:
diff --git a/fairscale/utils/__init__.py b/fairscale/utils/__init__.py
new file mode 100644
--- /dev/null
+++ b/fairscale/utils/__init__.py
@@ -0,0 +1,4 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All rights reserved.
+#
+# This source code is licensed under the BSD license found in the
+# LICENSE file in the root directory of this source tree.


Test patch: