Repository: facebookresearch/fairscale

Issue:
Fail on install - CUDA 11.1.0
## 🐛 Bug

Hi,
pip install on below environment throws an error. I'm happy to provide more info if it would be useful. Thanks!

fatal error: multi_tensor_apply.cuh: No such file or directory

## Environment (NVIDIA-Python Docker: 20.10)
Python: 3.6
PyTorch: 1.7.0
CUDA: 11.1.0
cuDNN: 8.0.4

## Error
```
Error:
b'  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpl6xin7ge
       cwd: /tmp/pip-install-0_xiegmi/fairscale
  Complete output (158 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/fairscale
  copying fairscale/__init__.py -> build/lib.linux-x86_64-3.6/fairscale
  creating build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/grad_scaler.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/oss.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/utils.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  copying fairscale/optim/adam.py -> build/lib.linux-x86_64-3.6/fairscale/optim
  creating build/lib.linux-x86_64-3.6/fairscale/nn
  copying fairscale/nn/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/dependency.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/pipe.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/stream.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/batchnorm.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/pipeline.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/checkpoint.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/copy.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/phony.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/worker.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  copying fairscale/nn/pipe/microbatch.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe
  creating build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/random.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/initialize.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/layers.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/cross_entropy.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/mappings.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  copying fairscale/nn/model_parallel/utils.py -> build/lib.linux-x86_64-3.6/fairscale/nn/model_parallel
  creating build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  copying fairscale/nn/data_parallel/sharded_ddp.py -> build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  copying fairscale/nn/data_parallel/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/data_parallel
  creating build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/top2gate.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/moelayer.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  copying fairscale/nn/moe/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/moe
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/profile.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  copying fairscale/nn/pipe/balance/blockpartition.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/balance
  creating build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/skippable.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/__init__.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/tracker.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/layout.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/portal.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  copying fairscale/nn/pipe/skip/namespace.py -> build/lib.linux-x86_64-3.6/fairscale/nn/pipe/skip
  running egg_info
  writing fairscale.egg-info/PKG-INFO
  writing dependency_links to fairscale.egg-info/dependency_links.txt
  writing requirements to fairscale.egg-info/requires.txt
  writing top-level names to fairscale.egg-info/top_level.txt
  reading manifest file \'fairscale.egg-info/SOURCES.txt\'
  reading manifest template \'MANIFEST.in\'
  writing manifest file \'fairscale.egg-info/SOURCES.txt\'
  creating build/lib.linux-x86_64-3.6/fairscale/clib
  creating build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -> build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -> build/lib.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  running build_ext
  building \'fairscale.fused_adam_cuda\' extension
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib
  creating /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda
  Emitting ninja build file /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/2] /usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options \'\'"\'"\'-fPIC\'"\'"\'\' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_86,code=sm_86 -std=c++14
  FAILED: /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o
  /usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options \'\'"\'"\'-fPIC\'"\'"\'\' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_86,code=sm_86 -std=c++14
  /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu:12:10: fatal error: multi_tensor_apply.cuh: No such file or directory
   #include "multi_tensor_apply.cuh"
            ^~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  [2/2] c++ -MMD -MF /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o.d -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c -c /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -o /tmp/pip-install-0_xiegmi/fairscale/build/temp.linux-x86_64-3.6/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  In file included from /usr/local/lib/python3.6/dist-packages/torch/include/ATen/Parallel.h:149:0,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                   from /usr/local/lib/python3.6/dist-packages/torch/include/torch/extension.h:4,
                   from /tmp/pip-install-0_xiegmi/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp:1:
  /usr/local/lib/python3.6/dist-packages/torch/include/ATen/ParallelOpenMP.h:84:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   #pragma omp parallel for if ((end - begin) >= grain_size)
  
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/cpp_extension.py", line 1522, in _run_ninja_build
      env=env)
    File "/usr/lib/python3.6/subprocess.py", line 438, in run
      output=stdout, stderr=stderr)
  subprocess.CalledProcessError: Command \'[\'ninja\', \'-v\']\' returned non-zero exit status 1.
```


Discussion:
What is the command-line you are using to install?
Please try PR #196 and confirm whether or not it fixes your issue.
Hi, Yes! It fixed my issue and I was able to install successfully however when I attempted to import the following error occurred:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/fairscale/__init__.py", line 12, in <module>
    from . import nn
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/__init__.py", line 7, in <module>
    from .pipe import LazyModule, Pipe, PipeRPCWrapper
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/__init__.py", line 22, in <module>
    from .pipe import LazyModule, Pipe
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/pipe.py", line 36, in <module>
    from .async_schedule import Invocation, Location, ModuleWrapper
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/async_schedule.py", line 18, in <module>
    from .messages import Transport
  File "/usr/local/lib/python3.6/dist-packages/fairscale/nn/pipe/messages.py", line 15, in <module>
    from fairscale.utils.object import pyobject_to_tensor, tensor_to_pyobject
ModuleNotFoundError: No module named 'fairscale.utils'
```

The file structure of the install looks like:
```
.:
__init__.py  clib                                             nn
__pycache__  fused_adam_cuda.cpython-36m-x86_64-linux-gnu.so  optim

./__pycache__:
__init__.cpython-36.pyc

./clib:
fused_adam_cuda

./clib/fused_adam_cuda:
fused_adam_cuda.cpp  fused_adam_cuda_kernel.cu

./nn:
__init__.py  __pycache__  data_parallel  model_parallel  moe  pipe

./nn/__pycache__:
__init__.cpython-36.pyc

./nn/data_parallel:
__init__.py  __pycache__  sharded_ddp.py

./nn/data_parallel/__pycache__:
__init__.cpython-36.pyc  sharded_ddp.cpython-36.pyc

./nn/model_parallel:
__init__.py  cross_entropy.py  layers.py    random.py
__pycache__  initialize.py     mappings.py  utils.py

./nn/model_parallel/__pycache__:
__init__.cpython-36.pyc       layers.cpython-36.pyc    utils.cpython-36.pyc
cross_entropy.cpython-36.pyc  mappings.cpython-36.pyc
initialize.cpython-36.pyc     random.cpython-36.pyc

./nn/moe:
__init__.py  __pycache__  moe_layer.py  top2gate.py

./nn/moe/__pycache__:
__init__.cpython-36.pyc  moe_layer.cpython-36.pyc  top2gate.cpython-36.pyc

./nn/pipe:
__init__.py        batchnorm.py   messages.py    pipeline.py  types.py
__pycache__        checkpoint.py  microbatch.py  rpc.py       worker.py
async_schedule.py  copy.py        phony.py       skip
balance            dependency.py  pipe.py        stream.py

./nn/pipe/__pycache__:
__init__.cpython-36.pyc        phony.cpython-36.pyc
async_schedule.cpython-36.pyc  pipe.cpython-36.pyc
batchnorm.cpython-36.pyc       pipeline.cpython-36.pyc
checkpoint.cpython-36.pyc      rpc.cpython-36.pyc
copy.cpython-36.pyc            stream.cpython-36.pyc
dependency.cpython-36.pyc      types.cpython-36.pyc
messages.cpython-36.pyc        worker.cpython-36.pyc
microbatch.cpython-36.pyc

./nn/pipe/balance:
__init__.py  __pycache__  blockpartition.py  profile.py

./nn/pipe/balance/__pycache__:
__init__.cpython-36.pyc  blockpartition.cpython-36.pyc  profile.cpython-36.pyc

./nn/pipe/skip:
__init__.py  layout.py     portal.py     tracker.py
__pycache__  namespace.py  skippable.py

./nn/pipe/skip/__pycache__:
__init__.cpython-36.pyc  namespace.cpython-36.pyc  skippable.cpython-36.pyc
layout.cpython-36.pyc    portal.cpython-36.pyc     tracker.cpython-36.pyc

./optim:
__init__.py  adam.py      grad_scaler.py  utils.py
__pycache__  adascale.py  oss.py

./optim/__pycache__:
__init__.cpython-36.pyc  adascale.cpython-36.pyc     oss.cpython-36.pyc
adam.cpython-36.pyc      grad_scaler.cpython-36.pyc  utils.cpython-36.pyc
```

I hope this information helps! I used the below command to install:
python3 -m pip install git+https://github.com/facebookresearch/fairscale.git@fix-install

Thanks much for reporting. I've added testing in circleci so that this get caught next time.

Fix is in PR #200.
Awesome, it works now, thanks so much for your help!
I confirm that with that fix merged `pip install git+https://github.com/facebookresearch/fairscale.git` now builds on cuda-11.1 - required for rtx-30 cards. Perhaps a new pip release could be made to include those fixes? Thank you!
> I confirm that with that fix merged `pip install git+https://github.com/facebookresearch/fairscale.git` now builds on cuda-11.1 - required for rtx-30 cards. Perhaps a new pip release could be made to include those fixes? Thank you!

Created a new pip package: https://pypi.org/project/fairscale/0.1.0/
> Created a new pip package: https://pypi.org/project/fairscale/0.1.0/

It won't build:

```
$ pip install fairscale -U
Collecting fairscale
  Downloading fairscale-0.1.0.tar.gz (77 kB)
     |████████████████████████████████| 77 kB 1.5 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/stas/anaconda3/envs/main-38/bin/python /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmplnhx_kyh                                                                                                                            
       cwd: /tmp/pip-install-cqcnt5xj/fairscale                                                                                                                  
  Complete output (55 lines):                                                                                                                                    
  running egg_info                                                                                                                                               
  writing fairscale.egg-info/PKG-INFO                                                                                                                            
  writing dependency_links to fairscale.egg-info/dependency_links.txt                                                                                            
  writing requirements to fairscale.egg-info/requires.txt                                                                                                        
  writing top-level names to fairscale.egg-info/top_level.txt                                                                                                    
  Traceback (most recent call last):                                                                                                                             
    File "/home/stas/anaconda3/envs/main-38/bin/ninja", line 5, in <module>                                                                                      
      from ninja import ninja                                                                                                                                    
  ModuleNotFoundError: No module named 'ninja'                                                                                                                   
  Traceback (most recent call last):                                                                                                                             
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>                                
      main()                                                                                                                                                     
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main                                    
      json_out['return_val'] = hook(**hook_input['kwargs'])                                                                                                      
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel            
      return hook(config_settings)                                                                                                                               
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 149, in get_requires_for_build_wheel                   
      return self._get_build_requires(                                                                                                                           
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 130, in _get_build_requires                            
      self.run_setup()                                                                                                                                           
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup                                      
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 56, in <module>
      setuptools.setup(
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 298, in run
      self.find_sources()
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 305, in find_sources
      mm.run()
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 536, in run
      self.add_defaults()
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 572, in add_defaults
      sdist.add_defaults(self)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/sdist.py", line 228, in add_defaults
      self._add_defaults_ext()
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/sdist.py", line 311, in _add_defaults_ext
      build_ext = self.get_finalized_command('build_ext')
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/cmd.py", line 298, in get_finalized_command
      cmd_obj = self.distribution.get_command_obj(command, create)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 338, in __init__
      if not is_ninja_available():
    File "/tmp/pip-build-env-pd3o6az6/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1293, in is_ninja_available
      subprocess.check_call('ninja --version'.split(), stdout=devnull)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['ninja', '--version']' returned non-zero exit status 1.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/stas/anaconda3/envs/main-38/bin/python /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmplnhx_kyh Check the logs for full command output.
```

Something is wrong in the build steps, since:

```
$ pip install ninja
Requirement already satisfied: ninja in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (1.10.0.post2)
```

```
ninja --version
1.10.0.git.kitware.jobserver-1
```
```
pip install git+https://github.com/facebookresearch/fairscale.git
```
fails too in the same way. 

So either you must have changed something since I was last able to build it 10 days ago or something has changed on my side. I'm trying to identify what has changed. Will report back.

OK, first it seems that pip package `ninja==1.10.0.post2` is the cause of this failure.
```
$ pip uninstall -y ninja
Found existing installation: ninja 1.10.0.post2
Uninstalling ninja-1.10.0.post2:
  Successfully uninstalled ninja-1.10.0.post2
$ conda install ninja 
```

and now I can build `fairscale` master w/ pytorch-nightly cuda/11.0 either via:
* `pip install git+https://github.com/facebookresearch/fairscale.git`
* `pip install -e .` in the master clone

but something fishy is going on, since pip doesn't even see `ninja` installed by `conda`:
```
$ pip uninstall ninja   
WARNING: Skipping ninja as it is not installed.
```

But I still can't build the pip package you have just released:

```
$ pip install fairscale==0.1.0
Collecting fairscale==0.1.0
  Using cached fairscale-0.1.0.tar.gz (77 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: torch>=1.4.0 in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (from fairscale==0.1.0) (1.8.0.dev20201201+cu110)
Requirement already satisfied: typing-extensions in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (from torch>=1.4.0->fairscale==0.1.0) (3.7.4.3)
Requirement already satisfied: numpy in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (from torch>=1.4.0->fairscale==0.1.0) (1.19.4)
Building wheels for collected packages: fairscale
  Building wheel for fairscale (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/stas/anaconda3/envs/main-38/bin/python /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpjegp03wj                                                                                                                                             
       cwd: /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63                                                                                 
  Complete output (173 lines):                                                                                                                                   
  running bdist_wheel                                                                                                                                            
  running build                                                                                                                                                  
  running build_py                                                                                                                                               
  creating build                                                                                                                                                 
  creating build/lib.linux-x86_64-3.8                                                                                                                            
  creating build/lib.linux-x86_64-3.8/fairscale                                                                                                                  
  copying fairscale/__init__.py -> build/lib.linux-x86_64-3.8/fairscale                                                                                          
  creating build/lib.linux-x86_64-3.8/fairscale/optim                                                                                                            
  copying fairscale/optim/oss.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                                   
  copying fairscale/optim/utils.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                                 
  copying fairscale/optim/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                              
  copying fairscale/optim/grad_scaler.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                           
  copying fairscale/optim/adascale.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                              
  copying fairscale/optim/adam.py -> build/lib.linux-x86_64-3.8/fairscale/optim                                                                                  
  creating build/lib.linux-x86_64-3.8/fairscale/utils                                                                                                            
  copying fairscale/utils/object.py -> build/lib.linux-x86_64-3.8/fairscale/utils                                                                                
  copying fairscale/utils/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/utils                                                                              
  creating build/lib.linux-x86_64-3.8/fairscale/nn                                                                                                               
  copying fairscale/nn/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn                                                                                    
  creating build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                                                          
  copying fairscale/nn/pipe/stream.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                            
  copying fairscale/nn/pipe/microbatch.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                        
  copying fairscale/nn/pipe/types.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                             
  copying fairscale/nn/pipe/messages.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                          
  copying fairscale/nn/pipe/rpc.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                               
  copying fairscale/nn/pipe/checkpoint.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                        
  copying fairscale/nn/pipe/pipeline.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                          
  copying fairscale/nn/pipe/copy.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                              
  copying fairscale/nn/pipe/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                          
  copying fairscale/nn/pipe/phony.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                             
  copying fairscale/nn/pipe/pipe.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                              
  copying fairscale/nn/pipe/worker.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                            
  copying fairscale/nn/pipe/async_schedule.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                    
  copying fairscale/nn/pipe/dependency.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                        
  copying fairscale/nn/pipe/batchnorm.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe                                                                         
  creating build/lib.linux-x86_64-3.8/fairscale/nn/data_parallel                                                                                                 
  copying fairscale/nn/data_parallel/sharded_ddp.py -> build/lib.linux-x86_64-3.8/fairscale/nn/data_parallel                                                     
  copying fairscale/nn/data_parallel/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/data_parallel                                                        
  creating build/lib.linux-x86_64-3.8/fairscale/nn/moe                                                                                                           
  copying fairscale/nn/moe/moe_layer.py -> build/lib.linux-x86_64-3.8/fairscale/nn/moe                                                                           
  copying fairscale/nn/moe/top2gate.py -> build/lib.linux-x86_64-3.8/fairscale/nn/moe                                                                            
  copying fairscale/nn/moe/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/moe                                                                            
  creating build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                                                                
  copying fairscale/nn/model_parallel/utils.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                         
  copying fairscale/nn/model_parallel/initialize.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                    
  copying fairscale/nn/model_parallel/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                      
  copying fairscale/nn/model_parallel/random.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                        
  copying fairscale/nn/model_parallel/mappings.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                      
  copying fairscale/nn/model_parallel/cross_entropy.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                 
  copying fairscale/nn/model_parallel/layers.py -> build/lib.linux-x86_64-3.8/fairscale/nn/model_parallel                                                        
  creating build/lib.linux-x86_64-3.8/fairscale/nn/pipe/balance                                                                                                  
  copying fairscale/nn/pipe/balance/profile.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/balance                                                           
  copying fairscale/nn/pipe/balance/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/balance                                                          
  copying fairscale/nn/pipe/balance/blockpartition.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/balance                                                    
  creating build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                                                                     
  copying fairscale/nn/pipe/skip/portal.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                                  
  copying fairscale/nn/pipe/skip/tracker.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                                 
  copying fairscale/nn/pipe/skip/skippable.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                               
  copying fairscale/nn/pipe/skip/__init__.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                                
  copying fairscale/nn/pipe/skip/layout.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                                  
  copying fairscale/nn/pipe/skip/namespace.py -> build/lib.linux-x86_64-3.8/fairscale/nn/pipe/skip                                                               
  running egg_info                                                                                                                                               
  writing fairscale.egg-info/PKG-INFO                                                                                                                            
  writing dependency_links to fairscale.egg-info/dependency_links.txt                                                                                            
  writing requirements to fairscale.egg-info/requires.txt                                                                                                        
  writing top-level names to fairscale.egg-info/top_level.txt                                                                                                    
  reading manifest file 'fairscale.egg-info/SOURCES.txt'                                                                                                         
  reading manifest template 'MANIFEST.in'                                                                                                                        
  writing manifest file 'fairscale.egg-info/SOURCES.txt'                                                                                                         
  creating build/lib.linux-x86_64-3.8/fairscale/clib                                                                                                             
  creating build/lib.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda                                                                                             
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -> build/lib.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda                                        
  copying fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -> build/lib.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda                                  
  running build_ext                                                                                                                                              
  building 'fairscale.fused_adam_cuda' extension                                                                                                                 
  creating /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8                                                      
  creating /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale                                            
  creating /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib                                       
  creating /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda                       
  Emitting ninja build file /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/build.ninja...                      
  Compiling objects...                                                                                                                                           
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)                                              
  /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:                                                       
  GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.                                                           
  The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75.                                                                          
  If you want to use the GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/                            
                                                                                                                                                                 
    warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))                                                    
  [1/2] /usr/local/cuda-11.1/bin/nvcc -I/tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/TH -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home/stas/anaconda3/envs/main-38/include/python3.8 -c -c /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14                                                                                                             
  FAILED: /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o                                                                                                                                                                
  /usr/local/cuda-11.1/bin/nvcc -I/tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/TH -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home/stas/anaconda3/envs/main-38/include/python3.8 -c -c /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14                                                                                                                   
  /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu:12:10: fatal error: multi_tensor_apply.cuh: No such file or directory                                                                                                                             
     12 | #include "multi_tensor_apply.cuh"                                                                                                                      
        |          ^~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                      
  compilation terminated.                                                                                                                                        
  [2/2] c++ -MMD -MF /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o.d -pthread -B /home/stas/anaconda3/envs/main-38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/TH -I/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home/stas/anaconda3/envs/main-38/include/python3.8 -c -c /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -o /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14                                              
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++                                                                
  In file included from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149,                                       
                   from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,                    
                   from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,             
                   from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,                       
                   from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,                     
                   from /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/torch/extension.h:4,                                       
                   from /tmp/pip-install-wx3r2g5s/fairscale_58222d884c1c4e96a9b4487c13255f63/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp:1:               
  /tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]                                                                                                                                                              
     84 | #pragma omp parallel for if ((end - begin) >= grain_size)                                                                                              
        |                                                                                                                                                        
  ninja: build stopped: subcommand failed.                                                                                                                       
  Traceback (most recent call last):                                                                                                                             
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1516, in _run_ninja_build                          
      subprocess.run(                                                                                                                                            
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/subprocess.py", line 512, in run                                                                       
      raise CalledProcessError(retcode, process.args,                                                                                                            
  subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.                                                                      
                                                                                                                                                                 
  The above exception was the direct cause of the following exception:                                                                                           
                                                                                                                                                                 
  Traceback (most recent call last):                                                                                                                             
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>                                
      main()                                                                                                                                                     
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main                                    
      json_out['return_val'] = hook(**hook_input['kwargs'])                                                                                                      
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel                             
      return _build_backend().build_wheel(wheel_directory, config_settings,                                                                                      
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 216, in build_wheel                                    
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',                                                                                                  
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir                           
      self.run_setup()                                                                                                                                           
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup                                      
      exec(compile(code, __file__, 'exec'), locals())                                                                                                            
    File "setup.py", line 56, in <module>                                                                                                                        
      setuptools.setup(                                                                                                                                          
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup                                            
      return distutils.core.setup(**attrs)                                                                                                                       
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/core.py", line 148, in setup                                                                 
      dist.run_commands()                                                                                                                                        
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 966, in run_commands                                                          
      self.run_command(cmd)                                                                                                                                      
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 985, in run_command                                                           
      cmd_obj.run()                                                                                                                                              
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
      _build_ext.run(self)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 653, in build_extensions
      build_ext.build_extensions(self)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
      _build_ext.build_extension(self, ext)
    File "/home/stas/anaconda3/envs/main-38/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
      objects = self.compiler.compile(sources,
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 473, in unix_wrap_ninja_compile
      _write_ninja_file_and_compile_objects(
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects
      _run_ninja_build(
    File "/tmp/pip-build-env-qss4wpdc/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1538, in _run_ninja_build
      raise RuntimeError(message) from e
  RuntimeError: Error compiling objects for extension
  ----------------------------------------
  ERROR: Failed building wheel for fairscale
Failed to build fairscale
ERROR: Could not build wheels for fairscale which use PEP 517 and cannot be installed directly
```
Here is the relevant part of the log for when it builds successfully - same env but from master clone

```
  Running setup.py develop for fairscale
    Running command /home/stas/anaconda3/envs/main-38/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/nvme1/code/github/00dist/fairscale/setup.py'"'"'; __file__='"'"'/mnt/nvme1/code/github/00dist/fairscale/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
    running develop
    running egg_info
    writing fairscale.egg-info/PKG-INFO
    writing dependency_links to fairscale.egg-info/dependency_links.txt
    writing requirements to fairscale.egg-info/requires.txt
    writing top-level names to fairscale.egg-info/top_level.txt
    reading manifest file 'fairscale.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'fairscale.egg-info/SOURCES.txt'
    running build_ext
    building 'fairscale.fused_adam_cuda' extension
    Emitting ninja build file /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
    GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
    The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75.
    If you want to use the GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

      warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
    [1/2] /usr/local/cuda-11.1/bin/nvcc -I/mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/TH -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home/stas/anaconda3/envs/main-38/include/python3.8 -c -c /mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu -o /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 --use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
    [2/2] c++ -MMD -MF /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o.d -pthread -B /home/stas/anaconda3/envs/main-38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/TH -I/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home/stas/anaconda3/envs/main-38/include/python3.8 -c -c /mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp -o /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_adam_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149,
                     from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                     from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                     from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                     from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                     from /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                     from /mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp:1:
    /tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
       84 | #pragma omp parallel for if ((end - begin) >= grain_size)
          |
    g++ -pthread -shared -B /home/stas/anaconda3/envs/main-38/compiler_compat -L/home/stas/anaconda3/envs/main-38/lib -Wl,-rpath=/home/stas/anaconda3/envs/main-38/lib -Wl,--no-as-needed -Wl,--sysroot=/ /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda.o /mnt/nvme1/code/github/00dist/fairscale/build/temp.linux-x86_64-3.8/fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.o -L/tmp/pip-build-env-gc1tmidw/overlay/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda-11.1/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/fairscale/fused_adam_cuda.cpython-38-x86_64-linux-gnu.so
    copying build/lib.linux-x86_64-3.8/fairscale/fused_adam_cuda.cpython-38-x86_64-linux-gnu.so -> fairscale
    Creating /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/fairscale.egg-link (link to .)
    Adding fairscale 0.1.0 to easy-install.pth file

    Installed /mnt/nvme1/code/github/00dist/fairscale
```
Looks like your pip package is missing files, observe:

this is from pip package - failing to build `fused_adam_cuda.cpp`
```
ls -l /tmp/pip-install-k38rqspi/fairscale_71111f3f8b6d4b47922ad5996e30a8d7/fairscale/clib/fused_adam_cuda/
total 16K
-rw-rw-r-- 1 stas stas  469 Aug 26 14:22 fused_adam_cuda.cpp
-rw-rw-r-- 1 stas stas 9.5K Aug 26 14:22 fused_adam_cuda_kernel.cu
```

this is from clone, builds it just fine:
```
ls -l /mnt/nvme1/code/github/00dist/fairscale/fairscale/clib/fused_adam_cuda
total 28K
-rw-rw-r-- 1 stas stas   83 Nov 23 19:41 compat.h
-rw-rw-r-- 1 stas stas  469 Nov 23 19:41 fused_adam_cuda.cpp
-rw-rw-r-- 1 stas stas 9.5K Nov 23 19:41 fused_adam_cuda_kernel.cu
-rw-rw-r-- 1 stas stas 4.4K Nov 23 19:41 multi_tensor_apply.cuh
```
So the problem is missing:
```
                    "fairscale/clib/fused_adam_cuda/multi_tensor_apply.cuh",
```
Is it possible you built this package on a clone that was missing files?

If I do:
```
git clone https://github.com/facebookresearch/fairscale
cd fairscale
python setup.py sdist
pip install dist/fairscale-0.1.0.tar.gz -v
```

The package builds and installs just fine.
Yes, your package on pypi is missing files. It has only:
```
fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu
fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp
```
and is missing the other 2. If I build `python setup.py sdist`, it has:
```
fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp
fairscale/clib/fused_adam_cuda/compat.h
fairscale/clib/fused_adam_cuda/multi_tensor_apply.cuh
fairscale/clib/fused_adam_cuda/fused_adam_cuda_kernel.cu
```

The problem has been successfully diagnosed.

Resolving patch:
diff --git a/fairscale/__init__.py b/fairscale/__init__.py
--- a/fairscale/__init__.py
+++ b/fairscale/__init__.py
@@ -3,7 +3,7 @@
 # This source code is licensed under the BSD license found in the
 # LICENSE file in the root directory of this source tree.
 
-__version__ = "0.1.0"
+__version__ = "0.1.1"
 
 ################################################################################
 # Import most common subpackages


Test patch: