Repository: NVIDIA/apex

Issue:
Docker build fails on pytorch master since Aug 15, 2019 with "AssertionError: Found no NVIDIA driver on your system"
Pytorch recently added a CUDA architecture check when building with torch.utils.cpp_extension. From the [commit notes](https://github.com/pytorch/pytorch/commit/cd207737017db8c81584763207df20bc6110ed75):

> The old behavior was to always use `sm_30`. The new behavior is:
> - For building via a setup.py, check if `'arch'` is in `extra_compile_args`.  If so, don't change anything.
> - If `TORCH_CUDA_ARCH_LIST` is set, respect that (can be 1 or more arches)
> - Otherwise, query device capability and use that.

Following this commit, when attempting to install apex via nvidia-docker, pytorch reverts to querying device capability and fails. The issue probably should be addressed in apex's setup.py, if not nvidia-docker/pytorch themselves. Currently, I work around the issue by setting the environment variable in the dockerfile, eg:

`ENV TORCH_CUDA_ARCH_LIST=Volta;Turing;Kepler+Tesla`
`RUN pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" apex/` 




Discussion:
Are you saying `torch.cuda.get_device_capability()` fails in containers?  I haven't observed this building in containers recently.  If so I'll try to repro.
Yes, but that has always been the case. As you know, Apex `setup.py` makes the same check -- `torch._C._cuda_isDriverSufficient()` -- via `torch.cuda.is_available()`. Historically, this failed for me, but `setup.py` only throws a warning; the install itself succeeded.

Note, I'm building on a google cloud instance with gpu and CUDA base image. I have not studied the issues with this environment because, well, it used to just work :-)
 
If I'm understanding correctly it means that anything that attempts to build an extension as part of an nvidia-docker image build will break unless they set the environment variable.  If so this issue affects all extension builds, not just Apex, right?
right, with the caveat that there's a second workaround: extra_compile_args or environment variable.
For your failing builds, can you check if this warning fired https://github.com/NVIDIA/apex/blob/master/setup.py#L9-L11?
If so, I can export `TORCH_CUDA_ARCH_LIST=Volta;Turing` within the setup.py for cases where `torch.cuda.is_available()` is False.
yes, that warning is fired.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -5,10 +5,20 @@
 from pip._internal import main as pipmain
 import sys
 import warnings
+import os
 
 if not torch.cuda.is_available():
-    print("\nWarning: Torch did not find available GPUs on this system.\n",
-          "If your intention is to cross-compile, this is not an error.\n")
+    # https://github.com/NVIDIA/apex/issues/486
+    # Extension builds after https://github.com/pytorch/pytorch/pull/23408 attempt to query torch.cuda.get_device_capability(),
+    # which will fail if you are compiling in an environment without visible GPUs (e.g. during an nvidia-docker build command).
+    print('\nWarning: Torch did not find available GPUs on this system.\n',
+          'If your intention is to cross-compile, this is not an error.\n'
+          'By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),\n'
+          'Volta (compute capability 7.0), and Turing (compute capability 7.5).\n'
+          'If you wish to cross-compile for a single specific architecture,\n'
+          'export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.\n')
+    if os.environ.get("TORCH_CUDA_ARCH_LIST", None) is None:
+        os.environ["TORCH_CUDA_ARCH_LIST"] = "6.0;6.1;6.2;7.0;7.5"
 
 print("torch.__version__  = ", torch.__version__)
 TORCH_MAJOR = int(torch.__version__.split('.')[0])


Test patch: