Repository: NVIDIA/NeMo

Issue:
Installation instructions should better indicate mandatory steps to make tests pass (or reinstall.sh needs an update)
**Is your feature request related to a problem? Please describe.**

I wanted to setup a dev conda environment for NeMo, so I followed steps at https://github.com/NVIDIA/NeMo/tree/main#from-source

Afterwards `pytest --cpu` was failing (before it could even run any test) with two errors:
* `module 'nvidia' has no attribute 'dali'`
* `No module named 'pynvml'`

**Describe the solution you'd like**

After manually installing both libraries with
* pip install --extra-index-url https://developer.download.nvidia.com/compute/redist --upgrade nvidia-dali-cuda120
* pip install pynvml

the tests were able to pass (`1420 passed, 304 skipped, 261 warnings`).

Ideally these libraries would be installed automatically by the `reinstall.sh` script.


Discussion:
These libraries cannot be installed automatically due to this dependence on on extra index for distribution. 

But these tests and the dali support itself should be import guarded. Do you have a stack trace of which tests failed ?
Sure, here's the stack trace:

```shell
> pytest --cpu
A valid `test_data.tar.gz` test archive (10445891B) found in the `/home/odelalleau/src/NeMo/tests/.data` folder.
Setting numba compat : True
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.2.0 -- /home/odelalleau/miniconda3/envs/py38-tmp/bin/python
cachedir: .pytest_cache
rootdir: /home/odelalleau/src/NeMo
configfile: pyproject.toml
testpaths: tests
plugins: hydra-core-1.2.0
collected 1694 items / 2 errors / 2 skipped

===================================================================================================== ERRORS ======================================================================================================
___________________________________________________________________________ ERROR collecting tests/collections/asr/test_asr_datasets.py ___________________________________________________________________________
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/pathlib.py:565: in import_path
    importlib.import_module(module_name)
../../miniconda3/envs/py38-tmp/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/collections/asr/test_asr_datasets.py:58: in <module>
    HAVE_DALI = is_dali_supported(__DALI_MINIMUM_VERSION__)
nemo/collections/asr/data/audio_to_text_dali.py:68: in is_dali_supported
    module_available, _ = model_utils.check_lib_version(
nemo/utils/model_utils.py:556: in check_lib_version
    mod = import_class_by_path(lib_name)
nemo/utils/model_utils.py:490: in import_class_by_path
    mod = getattr(mod, class_name)
E   AttributeError: module 'nvidia' has no attribute 'dali'
_________________________________________________________________________ ERROR collecting tests/collections/nlp/test_flash_attention.py __________________________________________________________________________
ImportError while importing test module '/home/odelalleau/src/NeMo/tests/collections/nlp/test_flash_attention.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/envs/py38-tmp/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/collections/nlp/test_flash_attention.py:47: in <module>
    import pynvml
E   ModuleNotFoundError: No module named 'pynvml'
================================================================================================ warnings summary =================================================================================================
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torchmetrics/utilities/imports.py:24
../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torchmetrics/utilities/imports.py:24
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torchmetrics/utilities/imports.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    _PYTHON_LOWER_3_8 = LooseVersion(_PYTHON_VERSION) < LooseVersion("3.8")

../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:4
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if not hasattr(tensorboard, "__version__") or LooseVersion(

../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:6
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    ) < LooseVersion("1.15"):

../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/jupyter_client/connect.py:20
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/jupyter_client/connect.py:20: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/faiss/loader.py:28
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(numpy.__version__) >= "1.19":

../../miniconda3/envs/py38-tmp/lib/python3.8/site-packages/setuptools/_distutils/version.py:346
  /home/odelalleau/miniconda3/envs/py38-tmp/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    other = LooseVersion(other)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================= short test summary info =============================================================================================
ERROR tests/collections/asr/test_asr_datasets.py - AttributeError: module 'nvidia' has no attribute 'dali'
ERROR tests/collections/nlp/test_flash_attention.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 2 skipped, 7 warnings, 2 errors in 1.88s =====================================================================================
```
Odd - its imported guarded right here - https://github.com/NVIDIA/NeMo/blob/main/tests/collections/asr/test_asr_datasets.py#L57-L60
Ah ok its Attribute error for some reason. Interesting. Let me add a patch

Resolving patch:
diff --git a/nemo/utils/model_utils.py b/nemo/utils/model_utils.py
--- a/nemo/utils/model_utils.py
+++ b/nemo/utils/model_utils.py
@@ -576,7 +576,7 @@ def check_lib_version(lib_name: str, checked_version: str, operator) -> Tuple[Op
                 f"Could not check version compatibility."
             )
             return False, msg
-    except (ImportError, ModuleNotFoundError):
+    except (ImportError, ModuleNotFoundError, AttributeError):
         pass
 
     msg = f"Lib {lib_name} has not been installed. Please use pip or conda to install this package."


Test patch:
diff --git a/tests/collections/nlp/test_flash_attention.py b/tests/collections/nlp/test_flash_attention.py
--- a/tests/collections/nlp/test_flash_attention.py
+++ b/tests/collections/nlp/test_flash_attention.py
@@ -44,16 +44,23 @@
 except (ImportError, ModuleNotFoundError):
     HAVE_TRITON = False
 
-import pynvml
+try:
+    import pynvml
+
+    HAVE_PYNVML = True
+except (ImportError, ModuleNotFoundError):
+    HAVE_PYNVML = False
 
 
 def HAVE_AMPERE_GPU():
-    pynvml.nvmlInit()
-    handle = pynvml.nvmlDeviceGetHandleByIndex(0)
-    device_arch = pynvml.nvmlDeviceGetArchitecture(handle)
-    pynvml.nvmlShutdown()
-    return device_arch == pynvml.NVML_DEVICE_ARCH_AMPERE
-
+    if HAVE_PYNVML:
+        pynvml.nvmlInit()
+        handle = pynvml.nvmlDeviceGetHandleByIndex(0)
+        device_arch = pynvml.nvmlDeviceGetArchitecture(handle)
+        pynvml.nvmlShutdown()
+        return device_arch == pynvml.NVML_DEVICE_ARCH_AMPERE
+    else:
+        return False
 
 @pytest.mark.run_only_on('GPU')
 @pytest.mark.skipif(not HAVE_APEX, reason="apex is not installed")