Repository: cupy/cupy
Pull request: Complete build only when all of the essential modules are available
Fix #4750.

Currently, CuPy completes its build even if an essential module is failed to be built. This PR fixes the build process to complete only when we can successfully build all of the required modules. See #4750 for an exhaustive list of modules CuPy depends on mandatorily or optionally.


Problem statement:
Fail build if essential module cannot be built
Currently, when CuPy failed to configure a module other than `cuda`, build succeeds without building the module.
If non-optional (essential) modules cannot be built, the build should fail with an error.

Optional modules are nccl, cudnn, nvtx
Non-optional modules are others e.g., random, thrust, cub, dlpack, ...

Hints from issue discussion:

Commit message: Fail build if essential modules cannot be built

Patch to review:
diff --git a/cupy_setup_build.py b/cupy_setup_build.py
index d4da3311a6d..d1c3f1e0992 100644
--- a/cupy_setup_build.py
+++ b/cupy_setup_build.py
@@ -97,6 +97,7 @@
     MODULES.append({
         # TODO(leofang): call this "rocm" or "hip" to avoid confusion?
         'name': 'cuda',
+        'required': True,
         'file': cuda_files + [
             'cupy.cuda.nvtx',
             'cupy_backends.cuda.libs.cusolver',
@@ -126,6 +127,7 @@
 else:
     MODULES.append({
         'name': 'cuda',
+        'required': True,
         'file': cuda_files,
         'include': [
             'cublas_v2.h',
@@ -153,6 +155,7 @@
 if not use_hip:
     MODULES.append({
         'name': 'cusolver',
+        'required': True,
         'file': [
             'cupy_backends.cuda.libs.cusolver',
             'cupy.cusolver',
@@ -214,6 +217,7 @@
 
     MODULES.append({
         'name': 'cutensor',
+        'required': True,
         'file': [
             'cupy_backends.cuda.libs.cutensor',
             'cupy.cutensor',
@@ -231,6 +235,7 @@
 
     MODULES.append({
         'name': 'cub',
+        'required': True,
         'file': [
             ('cupy.cuda.cub', ['cupy/cuda/cupy_cub.cu']),
         ],
@@ -246,6 +251,7 @@
 
     MODULES.append({
         'name': 'jitify',
+        'required': True,
         'file': [
             'cupy.cuda.jitify',
         ],
@@ -265,6 +271,7 @@
 
     MODULES.append({
         'name': 'random',
+        'required': True,
         'file': [
             'cupy.random._bit_generator',
             ('cupy.random._generator_api',
@@ -281,6 +288,7 @@
 else:
     MODULES.append({
         'name': 'cub',
+        'required': True,
         'file': [
             ('cupy.cuda.cub', ['cupy/cuda/cupy_cub.cu']),
         ],
@@ -313,6 +321,7 @@
     if use_hip:
         MODULES.append({
             'name': 'thrust',
+            'required': True,
             'file': [
                 ('cupy.cuda.thrust', ['cupy/cuda/cupy_thrust.cu']),
             ],
@@ -326,6 +335,7 @@
     else:
         MODULES.append({
             'name': 'thrust',
+            'required': True,
             'file': [
                 ('cupy.cuda.thrust', ['cupy/cuda/cupy_thrust.cu']),
             ],
@@ -343,6 +353,7 @@
 
 MODULES.append({
     'name': 'dlpack',
+    'required': True,
     'file': [
         'cupy.core.dlpack',
     ],
@@ -383,6 +394,10 @@ def module_extension_sources(file, use_cython, no_cuda):
     return [pyx] + others
 
 
+def get_required_modules():
+    return [m['name'] for m in MODULES if m.get('required', False)]
+
+
 def check_readthedocs_environment():
     return os.environ.get('READTHEDOCS', None) == 'True'
 
@@ -595,7 +610,8 @@ def make_extensions(options, compiler, use_cython):
         available_modules = [m['name'] for m in MODULES]
     else:
         available_modules, settings = preconfigure_modules(compiler, settings)
-        if 'cuda' not in available_modules:
+        required_modules = get_required_modules()
+        if not (set(required_modules) <= set(available_modules)):
             raise Exception('Your CUDA environment is invalid. '
                             'Please check above error log.')
 


Code review comments:
diff_hunk: @@ -214,6 +217,7 @@
 
     MODULES.append({
         'name': 'cutensor',
+        'required': True,
path: cupy_setup_build.py
text: @user1:
cuTENSOR is an optional. (sorry I failed to list it up in my original issue :bow:_

@author:
Fixed!