Title: ModuleNotFoundError when running Jupyter notebook with conda env in Snakemake
Tags: python,jupyter,snakemake

Question: I recently tried to use the Scanpy python package in a jupyter notebook that I made in Snakemake.
Scanpy is installed in a conda environment that I explicited in a .yaml in Snakemake.
When running the job:
snakemake --cores 1 results/output.h5ad --use-conda

the conda environment is succesfully loaded, but Snakemake does not find the module and gives this error:
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job        count    min threads    max threads
-------  -------  -------------  -------------
load_h5        1              1              1
total          1              1              1

Select jobs to execute...

[Fri Feb 24 13:35:29 2023]
rule load_h5:
    input: input/filtered_feature_bc_matrix.h5
    output: results/output.h5ad
    jobid: 0
    reason: Missing output files: results/output.h5ad
    resources: tmpdir=/var/folders/w7/zvnr_nqd4f3_2kdw0259s26r0000gq/T

Activating conda environment: .snakemake/conda/8d45bb2abfce310beb1752237b93c097_
Traceback (most recent call last):
  File "/Users/usr/miniconda3/bin/jupyter-nbconvert", line 10, in <module>
    sys.exit(main())
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 277, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/traitlets/config/application.py", line 1041, in launch_instance
    app.start()
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 418, in start
    self.convert_notebooks()
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 592, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 555, in convert_single_notebook
    output, resources = self.export_single_notebook(
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 483, in export_single_notebook
    output, resources = self.exporter.from_filename(
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 198, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 217, in from_file
    return self.from_notebook_node(
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/exporters/notebook.py", line 36, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/bduc1/miniconda3/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 153, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/bduc1/miniconda3/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 349, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/bduc1/miniconda3/lib/python3.9/site-packages/nbconvert/preprocessors/base.py", line 48, in __call__
    return self.preprocess(nb, resources)
  File "/Users/bduc1/miniconda3/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 100, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 121, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/jupyter_core/utils/__init__.py", line 168, in wrapped
    return loop.run_until_complete(inner)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nest_asyncio.py", line 90, in run_until_complete
    return f.result()
  File "/Users/usr/miniconda3/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Users/usr/miniconda3/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbclient/client.py", line 1021, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/Users/usr/miniconda3/lib/python3.9/site-packages/nbclient/client.py", line 915, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# start coding here
import scanpy as sc
------------------

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/var/folders/w7/zvnr_nqd4f3_2kdw0259s26r0000gq/T/ipykernel_34677/1324428204.py in <cell line: 2>()
      1 # start coding here
----> 2 import scanpy as sc

ModuleNotFoundError: No module named 'scanpy'
ModuleNotFoundError: No module named 'scanpy'

[Fri Feb 24 13:35:35 2023]
Error in rule load_h5:
    jobid: 0
    input: input/filtered_feature_bc_matrix.h5
    output: results/output.h5ad
    conda-env: /Users/usr/projects/wspace/project/.snakemake/conda/8d45bb2abfce310beb1752237b93c097_

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

To be sure that the environment created by Snakemake works correctly I activated the Snakemake-generated environment:
conda activate .snakemake/conda/8d45bb2abfce310beb1752237b93c097_

this works, so I check which Python is used:
which python
/Users/usr/projects/wspace/project/.snakemake/conda/8d45bb2abfce310beb1752237b93c097_/bin/python

this looks good, so I try to use Scanpy:
$ python3
Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:27:35) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scanpy as sc
>>> test = sc.read_10x_h5("/Users/usr/projects/wspace/project/input/filtered_feature_bc_matrix.h5")
/Users/usr/projects/wspace/project/.snakemake/conda/8d45bb2abfce310beb1752237b93c097_/lib/python3.10/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.

So Scanpy works in this environment!
But somehow Snakemake fails to find the Scanpy module!
I'm using Snakemake 7.22.0, on macOS Monterey 12.5.1.
Any help would be greatly appreciated!
Snakefile rule is:
rule load_h5:
    output:
        "results/output.h5ad"
    input:
        "input/filtered_feature_bc_matrix.h5"
    conda:
        "envs/scanpy_min.yaml"
    notebook:
        "notebooks/01_load_scanpy.py.ipynb"

scanpy_min.yaml is:
name: scanpy_min
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - anndata=0.8.0=pyhd8ed1ab_1
  - appdirs=1.4.4=pyh9f0ad1d_0
  - appnope=0.1.3=pyhd8ed1ab_0
  - arpack=3.7.0=hefb7bc6_2
  - asttokens=2.2.1=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=pyhd8ed1ab_3
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - blosc=1.21.2=hebb52c4_0
  - brotli=1.0.9=hb7f2c08_8
  - brotli-bin=1.0.9=hb7f2c08_8
  - brotlipy=0.7.0=py310h90acd4f_1005
  - bzip2=1.0.8=h0d85af4_4
  - c-ares=1.18.1=h0d85af4_0
  - ca-certificates=2022.12.7=h033912b_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - certifi=2022.12.7=pyhd8ed1ab_0
  - cffi=1.15.1=py310ha78151a_3
  - charset-normalizer=2.1.1=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - comm=0.1.2=pyhd8ed1ab_0
  - contourpy=1.0.7=py310ha23aa8a_0
  - cryptography=39.0.0=py310hdd0c95c_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - debugpy=1.6.6=py310h7a76584_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - et_xmlfile=1.1.0=pyhd8ed1ab_0
  - executing=1.2.0=pyhd8ed1ab_0
  - fonttools=4.38.0=py310h90acd4f_1
  - freetype=2.12.1=h3f81eb7_1
  - glpk=5.0=h3cb5acd_0
  - gmp=6.2.1=h2e338ed_0
  - h5py=3.8.0=nompi_py310h5555e59_100
  - hdf5=1.12.2=nompi_h48135f9_101
  - icu=70.1=h96cf925_0
  - idna=3.4=pyhd8ed1ab_0
  - igraph=0.10.3=h020c493_0
  - importlib-metadata=6.0.0=pyha770c72_0
  - importlib_metadata=6.0.0=hd8ed1ab_0
  - ipykernel=6.20.2=pyh736e0ef_0
  - ipython=8.8.0=pyhd1c38e8_0
  - jedi=0.18.2=pyhd8ed1ab_0
  - joblib=1.2.0=pyhd8ed1ab_0
  - jpeg=9e=hac89ed1_2
  - jupyter_client=7.4.9=pyhd8ed1ab_0
  - jupyter_core=5.1.5=py310h2ec42d9_0
  - kiwisolver=1.4.4=py310ha23aa8a_1
  - krb5=1.20.1=h049b76e_0
  - lcms2=2.14=h29502cd_1
  - leidenalg=0.9.1=py310h7a76584_0
  - lerc=4.0.0=hb486fe8_0
  - libaec=1.0.6=hf0c8a7f_1
  - libblas=3.9.0=16_osx64_openblas
  - libbrotlicommon=1.0.9=hb7f2c08_8
  - libbrotlidec=1.0.9=hb7f2c08_8
  - libbrotlienc=1.0.9=hb7f2c08_8
  - libcblas=3.9.0=16_osx64_openblas
  - libcurl=7.87.0=h6df9250_0
  - libcxx=14.0.6=hccf4f1f_0
  - libdeflate=1.17=hac1461d_0
  - libedit=3.1.20191231=h0678c8f_2
  - libev=4.33=haf1e3a3_1
  - libffi=3.4.2=h0d85af4_5
  - libgfortran=5.0.0=11_3_0_h97931a8_27
  - libgfortran5=11.3.0=h082f757_27
  - libiconv=1.17=hac89ed1_0
  - libjpeg-turbo=2.1.4=hb7f2c08_0
  - liblapack=3.9.0=16_osx64_openblas
  - libllvm11=11.1.0=h8fb7429_5
  - libnghttp2=1.51.0=he2ab024_0
  - libopenblas=0.3.21=openmp_h429af6e_3
  - libpng=1.6.39=ha978bb4_0
  - libsodium=1.0.18=hbcb3906_1
  - libsqlite=3.40.0=ha978bb4_0
  - libssh2=1.10.0=h47af595_3
  - libtiff=4.5.0=hee9004a_2
  - libwebp-base=1.2.4=h775f41a_0
  - libxcb=1.13=h0d85af4_1004
  - libxml2=2.10.3=hb9e07b5_0
  - libzlib=1.2.13=hfd90126_4
  - llvm-openmp=15.0.7=h61d9ccf_0
  - llvmlite=0.39.1=py310h2bfb868_1
  - lz4-c=1.9.4=hf0c8a7f_0
  - matplotlib-base=3.6.3=py310he725631_0
  - matplotlib-inline=0.1.6=pyhd8ed1ab_0
  - metis=5.1.0=h2e338ed_1006
  - mpfr=4.1.0=h0f52abe_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - natsort=8.2.0=pyhd8ed1ab_0
  - ncurses=6.3=h96cf925_1
  - nest-asyncio=1.5.6=pyhd8ed1ab_0
  - networkx=3.0=pyhd8ed1ab_0
  - numba=0.56.4=py310h62db5c2_0
  - numexpr=2.8.3=py310hecf8f37_1
  - numpy=1.23.5=py310h1b7c290_0
  - openjpeg=2.5.0=h13ac156_2
  - openpyxl=3.1.0=py310h90acd4f_0
  - openssl=3.0.8=hfd90126_0
  - packaging=23.0=pyhd8ed1ab_0
  - pandas=1.5.3=py310hecf8f37_0
  - parso=0.8.3=pyhd8ed1ab_0
  - patsy=0.5.3=pyhd8ed1ab_0
  - pexpect=4.8.0=pyh1a96a4e_2
  - pickleshare=0.7.5=py_1003
  - pillow=9.4.0=py310hab5364c_0
  - pip=22.3.1=pyhd8ed1ab_0
  - platformdirs=2.6.2=pyhd8ed1ab_0
  - pooch=1.6.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.36=pyha770c72_0
  - psutil=5.9.4=py310h90acd4f_0
  - pthread-stubs=0.4=hc929b4f_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pygments=2.14.0=pyhd8ed1ab_0
  - pynndescent=0.5.8=pyh1a96a4e_0
  - pyopenssl=23.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pysocks=1.7.1=pyha2e5f31_6
  - pytables=3.7.0=py310h90ba602_3
  - python=3.10.8=he7542f4_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-igraph=0.10.3=py310hedfac68_0
  - python_abi=3.10=3_cp310
  - pytz=2022.7.1=pyhd8ed1ab_0
  - pyzmq=25.0.0=py310hf615a82_0
  - readline=8.1.2=h3899abd_0
  - requests=2.28.2=pyhd8ed1ab_0
  - scanpy=1.9.1=pyhd8ed1ab_0
  - scikit-learn=1.2.1=py310hcebe997_0
  - scipy=1.10.0=py310h240c617_0
  - seaborn=0.12.2=hd8ed1ab_0
  - seaborn-base=0.12.2=pyhd8ed1ab_0
  - session-info=1.0.0=pyhd8ed1ab_0
  - setuptools=66.1.1=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.1.9=h225ccf5_2
  - stack_data=0.6.2=pyhd8ed1ab_0
  - statsmodels=0.13.5=py310h936d966_2
  - stdlib-list=0.8.0=pyhd8ed1ab_0
  - suitesparse=5.10.1=h7aff33d_1
  - tbb=2021.7.0=hb8565cd_1
  - texttable=1.6.7=pyhd8ed1ab_0
  - threadpoolctl=3.1.0=pyh8a188c0_0
  - tk=8.6.12=h5dbffcc_0
  - tornado=6.2=py310h90acd4f_1
  - tqdm=4.64.1=pyhd8ed1ab_0
  - traitlets=5.8.1=pyhd8ed1ab_0
  - typing-extensions=4.4.0=hd8ed1ab_0
  - typing_extensions=4.4.0=pyha770c72_0
  - tzdata=2022g=h191b570_0
  - umap-learn=0.5.3=py310h2ec42d9_0
  - unicodedata2=15.0.0=py310h90acd4f_0
  - urllib3=1.26.14=pyhd8ed1ab_0
  - wcwidth=0.2.6=pyhd8ed1ab_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xorg-libxau=1.0.9=h35c211d_0
  - xorg-libxdmcp=1.1.3=h35c211d_0
  - xz=5.2.6=h775f41a_0
  - zeromq=4.3.4=he49afe7_1
  - zipp=3.11.0=pyhd8ed1ab_0
  - zstd=1.5.2=hbc0c0cd_6
prefix: /Users/usr/miniconda3/envs/scanpy_min

Answer: If anyone runs into the same issue, I found the answer.
It is not sufficient to have jupyter_core and jupyter_client in the environment of interest.
I installed jupyter lab using mamba and it solved everything!