Repository: htcondor/htmap

Issue:
htmap run.py uses wrong python interpreter on execute node
**Describe the bug**
The `run.py` script executes just using `python3` as the interpreter on the execute node. In most cases this will evaluate to `/usr/bin/python3`, which may not be the interpreter in which `htmap` was executed. The actual error ends up being 

```pytb
ModuleNotFoundError: No module named 'cloudpickle'
```

since `cloudpickle` isn't installed system wide.

**To Reproduce**
This can be reproduced by creating a conda environment that includes `htmap`, then executing the basic `double` example from the docs:

```console
conda create --name htmap-test --yes --quiet python=3.8 htmap
conda activate htmap
python test_htmap.py
```

where `test_htmap.py` is :

```python
#!/usr/bin/env python

import htmap
htmap.settings['DELIVERY_METHOD'] = 'assume'

def double(x):
    return 2 * x

doubled = list(htmap.map(double, range(10)))
print(doubled)
```

**Expected behavior**
The `run.py` script that runs on the execute node is executed using the same interpreter as was used to execute `htmap.map()` in the first place.

**Software Versions:**
```console
$ python -c "import htcondor, htmap; print(htcondor.version()); print(htmap.version())"
$CondorVersion: 8.9.5 Feb 04 2020 $
HTMap version 0.5.0
$ condor_version
$CondorVersion: 8.9.5 Feb 04 2020 $
$CondorPlatform: X86_64-CentOS_6.10 $
$ cat /etc/os-release
NAME="Scientific Linux"
VERSION="7.7 (Nitrogen)"
ID="scientific"
ID_LIKE="rhel centos fedora"
VERSION_ID="7.7"
PRETTY_NAME="Scientific Linux 7.7 (Nitrogen)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:scientificlinux:scientificlinux:7.7:GA"
HOME_URL="http://www.scientificlinux.org//"
BUG_REPORT_URL="mailto:scientific-linux-devel@listserv.fnal.gov"

REDHAT_BUGZILLA_PRODUCT="Scientific Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Scientific Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"
$ cat /proc/version
Linux version 3.10.0-1062.12.1.el7.x86_64 (mockbuild@sl7-uefisign.fnal.gov) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Wed Feb 5 09:10:55 CST 2020
```

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Problem exhibits using the `igwn-py37-testing` conda environment available on OASIS under `/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda`

Comments:

Patch:
diff --git a/.travis.yml b/.travis.yml
index 4cb014c..7a891ec 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,7 +16,6 @@ jobs:
   fast_finish: true
 
 install:
-  - pip install codecov
   - travis_retry docker build -t htmap-test --file docker/Dockerfile --build-arg HTCONDOR_VERSION --build-arg PYTHON_VERSION=${TRAVIS_PYTHON_VERSION} .
 
 script:
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 7dfda05..acedcfd 100644
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -9,6 +9,8 @@ mkdir -p "$_condor_local_dir/lock" "$_condor_local_dir/log" "$_condor_local_dir/
 # start condor
 condor_master
 
+condor_version
+
 # once the shared port daemon wakes up, use condor_who to wait for condor to stand up
 while [[ ! -s "${_condor_local_dir}/log/SharedPortLog" ]]
 do
diff --git a/docs/source/dependencies.rst b/docs/source/dependencies.rst
index 369ad43..35525ea 100644
--- a/docs/source/dependencies.rst
+++ b/docs/source/dependencies.rst
@@ -29,8 +29,9 @@ The built-in delivery methods are
 
 * ``docker`` - runs in a (possibly user-supplied) Docker container.
 * ``singularity`` - runs in a (possibly user-supplied) Singularity container.
+* ``shared`` - runs with the same Python installation used submit-side.
 * ``assume`` - assumes that the dependencies have already been installed at the execute location.
-* ``transplant`` - copy the user's Python installation to the execute node.
+* ``transplant`` - copy the submit-side Python installation to the execute location.
 
 More details on each of these methods can be found below.
 
@@ -71,8 +72,8 @@ At runtime:
 
 .. code-block:: python
 
-    htmap.settings['DELIVERY_METHOD'] = 'docker'
-    htmap.settings['DOCKER.IMAGE'] = "<repository>/<image>:<tag>"
+    htmap.settings["DELIVERY_METHOD"] = "docker"
+    htmap.settings["DOCKER.IMAGE"] = "<repository>/<image>:<tag>"
 
 In this mode, HTMap will run inside a Docker image that you provide.
 Remember that this Docker image needs to have the ``htmap`` module installed.
@@ -119,8 +120,8 @@ At runtime:
 
 .. code-block:: python
 
-    htmap.settings['DELIVERY_METHOD'] = 'singularity'
-    htmap.settings['SINGULARITY.IMAGE'] = "<image>"
+    htmap.settings["DELIVERY_METHOD"] = "singularity"
+    htmap.settings["SINGULARITY.IMAGE"] = "<image>"
 
 In this mode, HTMap will run inside a Singularity image that you provide.
 Remember that this Singularity image needs to have the ``cloudpickle`` module installed.
@@ -149,6 +150,37 @@ If you want to use your own Singularity image, just change the ``'SINGULARITY.IM
     If you get a ``stderr`` message from Singularity about a bind mount directory not existing, that's the problem.
 
 
+Run With a Shared Python Installation
+-------------------------------------
+
+In your ``~/.htmaprc`` file:
+
+.. code-block:: bash
+
+    DELIVERY_METHOD = "shared"
+
+At runtime:
+
+.. code-block:: python
+
+    htmap.settings["DELIVERY_METHOD"] = "shared"
+
+In this mode, HTMap will run your components using the same interpreter being
+used submit-side.
+This requires that that the submit-side Python interpreter be
+"visible" from the execute location, which is usually done in one of two ways:
+
+1. The execute location **is** the submit location
+   (i.e., they are the same physical computer).
+2. The Python installation is stored on a shared filesystem, such that submit
+   and execute can both see the same file paths.
+
+Either way, the practical requirement to use this delivery method is that the
+path to the Python interpreter
+(i.e., ``python -c "import sys, print(sys.executable)"``)
+is the same both submit-side and execute-side.
+
+
 Assume Dependencies are Present
 -------------------------------
 
@@ -162,7 +194,7 @@ At runtime:
 
 .. code-block:: python
 
-    htmap.settings['DELIVERY_METHOD'] = 'assume'
+    htmap.settings["DELIVERY_METHOD"] = 'assume'
 
 In this mode, HTMap assumes that a Python installation with all Python dependencies is already present.
 This will almost surely require some additional setup by your HTCondor pool's administrators.
@@ -183,7 +215,7 @@ At runtime:
 
 .. code-block:: python
 
-    htmap.settings['DELIVERY_METHOD'] = 'transplant'
+    htmap.settings["DELIVERY_METHOD"] = 'transplant'
 
 If you are running HTMap from a standalone Python install (like an Anaconda installation),
 you can use this delivery mechanism to transfer a copy of your entire Python install.
diff --git a/docs/source/versions/v0_6_0.rst b/docs/source/versions/v0_6_0.rst
new file mode 100644
index 0000000..f09b410
--- /dev/null
+++ b/docs/source/versions/v0_6_0.rst
@@ -0,0 +1,29 @@
+v0.6.0
+======
+
+New Features
+------------
+
+* Add the ``shared`` delivery method, which supports HTCondor pools that use
+  shared filesystems to make Python installations available universally.
+  Issues: https://github.com/htcondor/htmap/issues/195 and https://github.com/htcondor/htmap/issues/198
+
+
+Changed/Deprecated Features
+---------------------------
+
+
+Bug Fixes
+---------
+
+
+Known Issues
+------------
+
+* Execution errors that result in the job being terminated but no output being
+  produced are still not handled entirely gracefully. Right now, the component
+  state will just show as ``ERRORED``, but there won't be an actual error report.
+* Map component state may become corrupted when a map is manually vacated.
+  Force-removal may be needed to clean up maps if HTCondor and HTMap disagree
+  about the state of their components.
+  Issue: https://github.com/htcondor/htmap/issues/129
diff --git a/htmap/__init__.py b/htmap/__init__.py
index 22617f8..cdd6842 100644
--- a/htmap/__init__.py
+++ b/htmap/__init__.py
@@ -45,7 +45,7 @@ from .maps import (
 from .holds import ComponentHold
 from .errors import ComponentError
 from .state import ComponentStatus
-from .options import MapOptions, register_delivery_mechanism
+from .options import MapOptions, register_delivery_method
 from .management import (
     status,
     status_json,
diff --git a/htmap/options.py b/htmap/options.py
index 7e8df3f..83713d6 100644
--- a/htmap/options.py
+++ b/htmap/options.py
@@ -228,7 +228,7 @@ def create_submit_object_and_itemdata(
     return sub, itemdata
 
 
-def register_delivery_mechanism(
+def register_delivery_method(
     name: str,
     options_func: Callable[[str, Path], dict],
     setup_func: Optional[Callable[[str, Path], None]] = None,
@@ -332,7 +332,7 @@ def _get_base_descriptors_for_assume(
     }
 
 
-register_delivery_mechanism(
+register_delivery_method(
     'assume',
     options_func = _get_base_descriptors_for_assume,
 )
@@ -351,12 +351,33 @@ def _get_base_descriptors_for_docker(
     }
 
 
-register_delivery_mechanism(
+register_delivery_method(
     'docker',
     options_func = _get_base_descriptors_for_docker,
 )
 
 
+def _get_base_descriptors_for_shared(
+    tag: str,
+    map_dir: Path,
+) -> dict:
+    return {
+        'universe': 'vanilla',
+        'executable': Path(sys.executable).absolute().as_posix(),
+        'transfer_executable': 'False',
+        'arguments': f'{names.RUN_SCRIPT} $(component)',
+        'transfer_input_files': [
+            (map_dir / names.RUN_SCRIPT).as_posix(),
+        ],
+    }
+
+
+register_delivery_method(
+    'shared',
+    options_func = _get_base_descriptors_for_shared,
+)
+
+
 def _get_base_descriptors_for_singularity(
     tag: str,
     map_dir: Path,
@@ -373,7 +394,7 @@ def _get_base_descriptors_for_singularity(
     }
 
 
-register_delivery_mechanism(
+register_delivery_method(
     'singularity',
     options_func = _get_base_descriptors_for_singularity,
 )
@@ -451,7 +472,7 @@ def _get_transplant_hash(pip_freeze_output: bytes) -> str:
     return h.hexdigest()
 
 
-register_delivery_mechanism(
+register_delivery_method(
     'transplant',
     options_func = _get_base_descriptors_for_transplant,
     setup_func = _run_delivery_setup_for_transplant,


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index db9f9b8..16f377a 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -26,8 +26,8 @@ from htmap._startup import ensure_htmap_dir_exists
 
 # start with base settings (ignore user settings for tests)
 htmap.settings.replace(BASE_SETTINGS)
-htmap.settings['DELIVERY_METHOD'] = 'assume'  # assume is the default for testing
-htmap.settings['WAIT_TIME'] = 0.01
+htmap.settings['DELIVERY_METHOD'] = 'assume'  # assume is the default for all tests that aren't parametric
+htmap.settings['WAIT_TIME'] = 0.1
 htmap.settings['MAP_OPTIONS.request_memory'] = '10MB'
 
 SETTINGS = copy(htmap.settings)
@@ -49,18 +49,11 @@ def delivery_methods(delivery_method, reset_settings):
     htmap.settings['DELIVERY_METHOD'] = delivery_method
 
 
-@pytest.fixture(scope = 'function', autouse = True)
-def set_htmap_dir(tmpdir_factory, reset_settings):
-    path = Path(tmpdir_factory.mktemp('htmap_dir'))
-    htmap.settings['HTMAP_DIR'] = path
-    ensure_htmap_dir_exists()
-
-
 def pytest_addoption(parser):
     parser.addoption(
         "--delivery",
         nargs = "+",
-        default = ['assume'],
+        default = ['assume'],  # assume is the default for parametric delivery testing
     )
 
 
@@ -72,6 +65,24 @@ def pytest_generate_tests(metafunc):
         )
 
 
+MAP_DIRS = []
+
+
+@pytest.fixture(scope = 'function', autouse = True)
+def set_htmap_dir(tmpdir_factory):
+    map_dir = Path(tmpdir_factory.mktemp('htmap_dir'))
+    MAP_DIRS.append(map_dir)
+    htmap.settings['HTMAP_DIR'] = map_dir
+    ensure_htmap_dir_exists()
+
+
+@pytest.fixture(scope = 'session', autouse = True)
+def cleanup():
+    for map_dir in MAP_DIRS:
+        htmap.settings['HTMAP_DIR'] = map_dir
+        htmap.clean(all = True)
+
+
 @pytest.fixture(scope = 'session')
 def doubler():
     def doubler(x):
diff --git a/tests/integration/test_errors_and_holds.py b/tests/integration/test_errors_and_holds.py
index 6064281..25f1507 100644
--- a/tests/integration/test_errors_and_holds.py
+++ b/tests/integration/test_errors_and_holds.py
@@ -34,7 +34,9 @@ def hold_before_error():
 
     assert map.component_statuses == [htmap.ComponentStatus.HELD, htmap.ComponentStatus.ERRORED]
 
-    return map
+    yield map
+
+    map.remove()
 
 
 @pytest.fixture(scope = 'function')
@@ -49,7 +51,9 @@ def error_before_hold():
 
     assert map.component_statuses == [htmap.ComponentStatus.ERRORED, htmap.ComponentStatus.HELD]
 
-    return map
+    yield map
+
+    map.remove()
 
 
 def test_can_get_error_if_hold_in_front(hold_before_error):
diff --git a/tests/integration/test_held_components.py b/tests/integration/test_held_components.py
index 93d1cdb..94f0685 100644
--- a/tests/integration/test_held_components.py
+++ b/tests/integration/test_held_components.py
@@ -20,65 +20,47 @@ import pytest
 import htmap
 
 
-def test_waiting_on_held_component_raises(mapped_doubler):
+@pytest.fixture(scope = 'function')
+def map_with_held_component(mapped_doubler):
     m = mapped_doubler.map(range(1))
     m.hold()
 
     m.wait(holds_ok = True)
 
-    with pytest.raises(htmap.exceptions.MapComponentHeld):
-        m.wait(timeout = 180)
+    yield m
 
+    m.remove()
 
-def test_accessing_held_component_raises(mapped_doubler):
-    m = mapped_doubler.map(range(1))
-    m.hold()
-
-    m.wait(holds_ok = True)
 
+def test_waiting_on_held_component_raises(map_with_held_component):
     with pytest.raises(htmap.exceptions.MapComponentHeld):
-        m[0]
+        map_with_held_component.wait(timeout = 180)
 
 
-def test_getting_held_component_raises(mapped_doubler):
-    m = mapped_doubler.map(range(1))
-    m.hold()
-
-    m.wait(holds_ok = True)
-
+def test_accessing_held_component_raises(map_with_held_component):
     with pytest.raises(htmap.exceptions.MapComponentHeld):
-        m.get(0)
+        map_with_held_component[0]
 
 
-def test_iterating_over_held_component_raises(mapped_doubler):
-    m = mapped_doubler.map(range(1))
-    m.hold()
-
-    m.wait(holds_ok = True)
-
+def test_getting_held_component_raises(map_with_held_component):
     with pytest.raises(htmap.exceptions.MapComponentHeld):
-        list(m)
-
-
-def test_held_component_shows_up_in_hold_reasons(mapped_doubler):
-    m = mapped_doubler.map(range(1))
-    m.hold()
+        map_with_held_component.get(0)
 
-    m.wait(holds_ok = True)
-
-    assert isinstance(m.holds[0], htmap.ComponentHold)
 
+def test_iterating_over_held_component_raises(map_with_held_component):
+    with pytest.raises(htmap.exceptions.MapComponentHeld):
+        list(map_with_held_component)
 
-def test_held_then_released_component_not_in_hold_reasons(mapped_doubler):
-    m = mapped_doubler.map(range(1))
-    m.hold()
 
-    m.wait(holds_ok = True)
+def test_held_component_shows_up_in_hold_reasons(map_with_held_component):
+    assert isinstance(map_with_held_component.holds[0], htmap.ComponentHold)
 
-    assert isinstance(m.holds[0], htmap.ComponentHold)
 
-    m.release()
+@pytest.mark.timeout(60)
+def test_held_then_released_component_not_in_hold_reasons(map_with_held_component):
+    assert isinstance(map_with_held_component.holds[0], htmap.ComponentHold)
 
-    time.sleep(5)  # wait for htcondor to write events
+    map_with_held_component.release()
 
-    assert len(m.holds) == 0
+    while not len(map_with_held_component.holds) == 0:
+        time.sleep(.1)
diff --git a/tests/unit/test_base_descriptors.py b/tests/unit/test_base_descriptors.py
index bbdcc52..379b5a8 100644
--- a/tests/unit/test_base_descriptors.py
+++ b/tests/unit/test_base_descriptors.py
@@ -18,12 +18,12 @@ import pytest
 from pathlib import Path
 
 import htmap
-from htmap.options import get_base_descriptors, register_delivery_mechanism, unregister_delivery_mechanism
+from htmap.options import get_base_descriptors, register_delivery_method, unregister_delivery_mechanism
 
 
 @pytest.fixture(scope = 'module', autouse = True)
 def add_null_delivery():
-    register_delivery_mechanism('null', lambda tag, map_dir: {})
+    register_delivery_method('null', lambda tag, map_dir: {})
 
     yield