DeepSpeed: Exits with CUDA runtime error on A100 (requires recompiling DeepSpeed for NVIDIA 8.0 Arch)
Author: PeterAJansen
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 4.3.0 (unofficial, off current main branch)
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

## Information

Model I am using (Bert, XLNet ...): T-5

The problem arises when using:
* [ ] the official example scripts: examples/seq2seq/finetune_trainer.py

## Issue

In the hopes this saves others some time since it took a while for me to fix:  When running the new DeepSpeed mode in Transformers 4.3.0 on an A100 GPU, it will exit with a runtime error:

RuntimeError: CUDA error: no kernel image is available for execution on the device

For me, this was due to installing DeepSpeed from pip rather than source.  The A100 architecture appears not to be (as of this writing) installed in the default.  If you install from source as described in this post ( https://www.deepspeed.ai/tutorials/advanced-install/ ), the error goes away.  The post suggests selecting the architecture using the TORCH_CUDA_ARCH_LIST environment variable, but I found just using the install.sh script (which I am assuming auto-detects the architecture of your GPU) worked more successfully. 


Comments:
Pinging @stas00

Heh, actually I wrote this section: https://www.deepspeed.ai/tutorials/advanced-install/#building-for-the-correct-architectures and the autodetector, since I originally had the same issue.

This problem is also partially in pytorch - which is now fixed too in pytorch-nightly.

 `TORCH_CUDA_ARCH_LIST` is there if you say want to use the binary build on another machine or want to optimize it for whatever reason. e.g. I build it with:
```
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 pip install --no-cache -v --disable-pip-version-check -e .
```
because I have 1070 and 3090 cards.

I'm glad you found a way to solve it.

Now, this is a purely DeepSpeed issue and has nothing to do with transformers, other than perhaps a documentation issue.

I'm all ears at how perhaps `transformers` can improve the doc on our side to help the users find a solution quickly.

1. Probably should recommend to install from source
2. but then when we bail on missing `deepspeed` we say do `pip install deepspeed` - do you think we should change that to:
>  `pip install deepspeed` or if it doesn't work install from source?

The thing is `pip install deepspeed` is installing from source, but I think it perhaps isn't using the same build script? So should we say:
>  `pip install deepspeed` or if it doesn't work install from https://github.com/microsoft/deepspeed? 

or may be easier to just say: 
> install from https://github.com/microsoft/deepspeed?

What happens if you install with:
```
DS_BUILD_OPS=1 pip install deepspeed
```

Perhaps your issue is JIT/PTX which happens if you don't do the above - i.e. the binary build gets postpone till run time. `DS_BUILD_OPS=1` forces the binary build.

In any case let's discuss this over at DeepSpeed Issues - @PeterAJansen, would you please open an issue there because only you can report/reproduce the specific error - should they fix the pip build. and tag me?

BTW, fairscale has its own issues with `pip install fairscale` - I also have to build from the the repo, because I am forced to use pytorch-nightly due to rtx-30* and it won't build at all via `pip` directly.

so whatever we decide we should do the same for `fairscale`.

Thank you!

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.