[deepspeed] nvme test hanging experiment: take4
Author: stas00
Pull request: yes

As reported in https://github.com/huggingface/transformers/issues/12715 nvme CUDA extension of deepspeed fails to build and leads to a hanging test. As suggested by @tjruwase we should be clearing out the CUDA binary extensions dir, since a new release might be incompatible with the old binary and things break. 
```
rm -rf ~/.cache/torch_extensions/
```

And all appears to be resolved. Replicated to the scheduled job too.

Fixes: https://github.com/huggingface/transformers/issues/12715

@sgugger, @LysandreJik 

Comments:
Thanks a lot for fixing this! Should we merge the PR or leave it like this to remember how to manage a hang with deepspeed and nvme?

Do you know if rebuilding the extensions takes a look time?

Never tried to measure the rebuilding, and it'd depend on the hardware, but probably in a ballpark of 20-30secs.

--------------

My main concern with the proposed solution in this PR is a race condition where one job rebuilds and another slightly slower one wipes the binaries out and leading to test failure. Since this is shared fs.

This is the problem with pytorch CUDA extensions. Instead of installing the binaries into the python tree which could be many on the same system (virtual env) it installs them into `~/.cache/torch_extensions` which is shared between all virtual envs - really bad idea.

So the clean solution is to not to install pure python package and have it build JIT at run time, but instead to do a pre-build which then installs the binary cuda extensions into the right python env, then there is never a collision.

So it'd be:
```
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install deepspeed  --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
```

Of course, this too takes time.

In theory we only need to do this once per deepspeed release, so we could also pre-build binary wheels and simply install those.

Do we have any other need for binary wheels besides `torch_scatter`?

And this would only be necessary if there's an issue with the deepspeed release, right? As there was no issue for the other machine, nor for your local machines. I wonder if we really need to implement a workaround for this or if we can't have this as a potential solution for future deepspeed issues that arise soon after a release.

There are no other binary wheels needs besides `torch_scatter`, but I'd rather keep those to a minimum as it doesn't help maintainability.

> And this would only be necessary if there's an issue with the deepspeed release, right?

We would need to do this for every release in case some Cpp code was changed.

------

Agreed, let's not do anything then and revisit this if it becomes a problem.

I wonder if we could create an index pointing to troubleshooting PRs/Issues, so e.g. this could be a start:

## Troubleshooting Github Actions CI (self-hosted box)

* Deepspeed
  - if jit build hangs, clear out `rm -rf ~/.cache/torch_extensions/` reference: https://github.com/huggingface/transformers/pull/12723
  
and put it close to home, under `.github-actions/README.md`? or `.github-actions/TROUBLESHOOTING.md`

Oh that's a brilliant idea indeed!

closing this as it's now indexed by https://github.com/huggingface/transformers/blob/master/.github/workflows/TROUBLESHOOT.md