Repository: Parsl/parsl

Issue:
Remove IPyParallel executor
**Is your feature request related to a problem? Please describe.**
The parsl ipp code isn't really maintained any more. As I try to tidy up some stuff in the test suite I find myself spending more time than I'd like to debugging what is happening inside the IPP  parsl executor, while at the same time the folklore is "don't use IPP".

**Describe the solution you'd like**
I'd like to remove all IPP tests from the test suite now. This will inevitably lead to the ipp code rotting fast, and so in addition the IPP executor should probably be removed in its entirely.

**Describe alternatives you've considered**
An alternative is to put developer effort into maintaining IPP as a supported executor.

**Additional context**
The folklore on slack seems to be "don't use IPP". This feature request turns that folklore into reality.

I invite comment in support or against from @kylechard @yadudoc @annawoodard especially as regards how this would affect existing users who have not moved to htex (if any?).

see #1328 


Discussion:
I strongly support removing IPP.
My vote is on removing IPP from tests and clarifying in docs that it is no longer supported. I would defer removing it entirely from the codebase for later.
What is the point of having something that isn't being tested (and therefore we can assume isn't working 😆) in the codebase?
I think it would be nicer to the users to have a release go out that warns anyone using IPP of the impending deprecation before we drop it entirely. Same process as planned for #1553 
Sounds good, but after we depreciate, let's open an issue to remind us to remove it after the depreciation period has ended.
> I think it would be nicer to the users to have a release go out that warns anyone using IPP of the impending deprecation before we drop it entirely.

If we're intending users to use it, we should be testing it. Otherwise, what is the response to broken IPP when a user reports it?

Thinking about it more, I agree with Ben. I think these are our two options:

1. Add depreciation warnings in next release, continue testing until the following release, when we completely remove IPP and IPP testing.
2. Add retroactive depreciation warnings (if someone tries to use IPP, we tell them to install the previous release) and remove IPP and IPP testing now.

I slightly favor option 2.
I vote for option 2. If people really need to use IPP then they can use an old version of Parsl until its time to update. I can't think of a single situation where IPP would work and HTEX would not. 
This was noted in the docs as deprecated as of 0.8, in 78cac3d5a9
@kylechard :

>  I can't think of a single situation where IPP would work and HTEX would not.

One specific situation is the docker / container_image support documented here: https://parsl.readthedocs.io/en/latest/userguide/containers.html#id1

That isn't tested in CI and the documentation is sufficiently out of date to refer to `libsubmit`.

@benclifford Are we waiting on anything in order to close this one ?
@yadudoc  yes - remove the ghostly stubs of IPP that were put in by #1565 sometime after the next release

Resolving patch:
diff --git a/parsl/executors/__init__.py b/parsl/executors/__init__.py
--- a/parsl/executors/__init__.py
+++ b/parsl/executors/__init__.py
@@ -1,13 +1,11 @@
 from parsl.executors.threads import ThreadPoolExecutor
-from parsl.executors.ipp import IPyParallelExecutor
 from parsl.executors.workqueue.executor import WorkQueueExecutor
 from parsl.executors.high_throughput.executor import HighThroughputExecutor
 from parsl.executors.extreme_scale.executor import ExtremeScaleExecutor
 from parsl.executors.low_latency.executor import LowLatencyExecutor
 from parsl.executors.flux.executor import FluxExecutor
 
-__all__ = ['IPyParallelExecutor',
-           'ThreadPoolExecutor',
+__all__ = ['ThreadPoolExecutor',
            'HighThroughputExecutor',
            'ExtremeScaleExecutor',
            'LowLatencyExecutor',
diff --git a/parsl/executors/ipp.py b/parsl/executors/ipp.py
deleted file mode 100644
--- a/parsl/executors/ipp.py
+++ /dev/null
@@ -1,12 +0,0 @@
-from parsl.executors.base import ParslExecutor
-
-
-class IPyParallelExecutor(ParslExecutor):
-    """This stub exists to issue a more helpful warning about the IPyParallel
-    executor being removed from parsl some time after v0.9.
-
-    It can eventually be removed entirely - perhaps after v0.10
-    """
-
-    def __new__(*args, **kwargs):
-        raise RuntimeError("The IPyParallel executor has been removed from parsl")
diff --git a/parsl/executors/ipp_controller.py b/parsl/executors/ipp_controller.py
deleted file mode 100644
--- a/parsl/executors/ipp_controller.py
+++ /dev/null
@@ -1,8 +0,0 @@
-class Controller():
-    """This stub exists to issue a more helpful warning about the IPyParallel
-    executor being removed from parsl some time after v0.9.
-
-    It can eventually be removed entirely - perhaps after v0.10
-    """
-    def __init__(self, *args, **kwargs):
-        raise RuntimeError("The IPyParallel executor has been removed from parsl")


Test patch:
diff --git a/parsl/tests/integration/test_stress/test_python_ipp.py b/parsl/tests/integration/test_stress/test_python_ipp.py
deleted file mode 100644
--- a/parsl/tests/integration/test_stress/test_python_ipp.py
+++ /dev/null
@@ -1,41 +0,0 @@
-''' Testing bash apps
-'''
-import parsl
-from parsl import DataFlowKernel, python_app
-
-import time
-import argparse
-
-from parsl.tests.configs.local_ipp import config
-
-
-@python_app
-def increment(x):
-    return x + 1
-
-
-def test_stress(count=1000):
-    """IPP app launch stress test"""
-    start = time.time()
-    x = {}
-    for i in range(count):
-        x[i] = increment(i)
-    end = time.time()
-    print("Launched {0} tasks in {1} s".format(count, end - start))
-
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument("-c", "--count", default="100",
-                        help="width of the pipeline")
-    parser.add_argument("-d", "--debug", action='store_true',
-                        help="Count of apps to launch")
-    args = parser.parse_args()
-
-    if args.debug:
-        parsl.set_stream_logger()
-
-    dfk = DataFlowKernel(config=config)
-
-    test_stress(count=int(args.count))
diff --git a/parsl/tests/test_data/test_file_ipp.py b/parsl/tests/test_data/test_file_staging.py
similarity index 93%
rename from parsl/tests/test_data/test_file_ipp.py
rename to parsl/tests/test_data/test_file_staging.py
--- a/parsl/tests/test_data/test_file_ipp.py
+++ b/parsl/tests/test_data/test_file_staging.py
@@ -1,9 +1,7 @@
 import os
 import pytest
-import parsl
 from parsl.app.app import bash_app
 from parsl.data_provider.files import File
-from parsl.tests.configs.local_threads import config
 
 
 @bash_app
@@ -88,11 +86,3 @@ def test_increment(depth=5):
             data = open(fu.result().filepath, 'r').read().strip()
             assert data == str(
                 key), "[TEST] incr failed for key:{0} got:{1}".format(key, data)
-
-
-if __name__ == '__main__':
-    parsl.clear()
-    parsl.load(config)
-
-    test_files()
-    test_increment()