Repository: Parsl/parsl

Issue:
Suggestion: use equivalent of mkdir -p with runinfo
**Describe the bug**

If a runinfo directory already exists, we currently get a nasty traceback:

```
broker.info[0]: online: flux-sample-[0-3] (ranks 0-3)
broker.info[0]: quorum-full: quorum->run 29.4857s
Traceback (most recent call last):
  File "/workflow/scripts/0_molecular-design-with-parsl.py", line 101, in <module>
    parsl.load(config)
  File "/opt/conda/envs/moldesign-demo/lib/python3.9/site-packages/parsl/dataflow/dflow.py", line 1330, in load
    cls._dfk = DataFlowKernel(config)
  File "/opt/conda/envs/moldesign-demo/lib/python3.9/site-packages/parsl/dataflow/dflow.py", line 83, in __init__
    self.run_dir = make_rundir(config.run_dir)
  File "/opt/conda/envs/moldesign-demo/lib/python3.9/site-packages/parsl/dataflow/rundirs.py", line 23, in make_rundir
    os.makedirs(path)
  File "/opt/conda/envs/moldesign-demo/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'runinfo'
Traceback (most recent call last):
```
It might be more user friendly and logical to do one of the following:

 - require that the directory not exist, tell the user with a clear message to delete and re-run
 - require that the directory not exist, give the user an interactive choice "Would you like to delete and re-populate?)
 - be OK with the directory already existing, mkdir -p 
 - be OK with the directory existing, but give a warning about overwriting results

**To Reproduce**

Steps to reproduce the behavior, for e.g:
1. Setup parsl (master) with FluxExecutor and Molecular Design Workfow [repo](https://github.com/ExaWorks/molecular-design-parsl-demo)
2. Run the 0_* python script (I can give you a path to mine if you like) 
3. See error

**Expected behavior**

I would expect the library to handle a directory already existing.

**Environment**
 - OS: Linux
 - Python version 3.8
 - Parsl version master

**Distributed Environment**
- Where are you running the Parsl script from ? Pod with container vanessa/molecular-design-parsl:latest using the Flux Operator, the entrypoint sets the LD_LIBRARY_PATH to include `/opt/conda/lib` and then runs flux start with `/bin/bash ./scripts/run_0.sh --outdir /tmp/data --workers 8`
- Where do you need the workers to run ? [e.g. Same as Parsl script, Compute nodes, Cloud nodes]

I'm hitting a bunch of other bugs with parsl + Flux - haven't decided yet if I'm going to pursue this further (open more bugs) or choose another engine. Thanks!

Update: we're in business - I fixed the other bugs! :partying_face: This is the first time parsl is running in the flux operator, even if it's only a dinky workflow. Epic!

![image](https://user-images.githubusercontent.com/814322/225816331-1abd68c7-c63d-4fb6-9607-51c6a4857236.png)



Discussion:
Usually this error occurs if you launch multiple parsl submitting processes (i.e. invoke `parsl.load()`) at the same instant (so they all pick the same numbered rundir/NNN to create).

Usually, that's not what people intend to be doing - usually the pattern is launch one parsl submitting process and parsl is what runs stuff concurrently.

That makes me wonder if something suspicious is happening with the parsl flux executor - for example, parsl being initialised inside multiple workers - when parsl.load() shouldn't happen in a worker at all.

or, I don't understand flux command lines, but are you running multiple copies of your run_0 script at once with that command line?

I think it's just a case that the container I built grabbed the working directory (that had this runinfo already since I ran it for a test). This seems like something someone reasonably might do.

> or, I don't understand flux command lines, but are you running multiple copies of your run_0 script at once with that command line?

No that's not the case - the log here is just for our broker pod, which is the only one to launch the command! The design of the flux operator is that we have a set of networked pods, a "MiniCluster" and the broker is the one that runs some main command, and the remaining pods are workers that become part of the Flux cluster. So the worker pods just start and connect to the broker and wait for jobs to be submit to them. Here is a log from a worker pod - it is hanging around waiting for instruction (and then when the broker exits, so does it).

![image](https://user-images.githubusercontent.com/814322/225831866-880ec76e-865f-48d1-b618-6d6c28dcc101.png)

So given that someone might run the workflow in a directory where it's already been run, I think it would be beneficial to have a more meaningful message with instruction of what to do, or a default. If there is something else you'd like me to look at, happy to do that too! As a workaround I just rm -rf the directory on the pod startup (and that tells you it's not an issue with multiple pods running the command, because the same thing would have happened after, and the error wouldn't have gone away with that fix).

Here is a suggestion for how to implement `mkdir_p`:

```python
import errno
import os
import sys

def mkdir_p(path):
    """
    Make a directory path if it does not exist, akin to mkdir -p
    """
    try:
        os.makedirs(path)
    except OSError as e:
        if e.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else:
           # or raise an Exception
            sys.exit(f"Error creating path {path}, exiting.")
```

Thanks for your help!
parsl is fine running (sequentially) many workflows.
here's a test you should be able to run many times in a row - and you should see sequentially numbered runinfo/NNN/ directories appear, one for each run.

Can you run this in your container environment (a few times) without flux and see if it has trouble creating the directory?

```
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ cat simple.py 
import parsl
from parsl.tests.configs.local_threads import config
parsl.load(config)

@parsl.python_app
def a():
  return 7

assert a().result() == 7
print("done")
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ ls runinfo
000  001  002

```
> here's a test you should be able to run many times in a row - and you should see sequentially numbered runinfo/NNN/ directories appear, one for each run.

Does this assume the runinfo exists before running anything? That's the use case.
Also the test would need to be using the FluxExecutor - it could be there is a bug within parsl (which seems to wrap it).
The 2nd and 3rd executions assume runinfo/ exists (because it's created by the first invocation)

But it also works this way:

```
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ rm -rf runinfo
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ mkdir runinfo
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ python simple.py 
done
benc@parsl-dev-3-11-1137:~/parsl/src/parsl$ ls runinfo
000  001  002
```

Please tell me if this test works in your container without flux so that I can see that the core of parsl is working ok in your environment (or not).
Gotcha! Yes definitely can do that - all looks good!

```console
(moldesign-demo) python3 simple.py 
done
(moldesign-demo) python3 simple.py 
done
(moldesign-demo) python3 simple.py 
done
(moldesign-demo) ls runinfo/
000  001  002
```
One thing I see is that the executor "launch command" is a flux start - https://github.com/Parsl/parsl/blob/33398b15cbb20ddc5c273d22015d6c18351337c4/parsl/executors/flux/executor.py#L177 - does that mean that every job run by this executor is done with flux start? That's probably not the functionality that we want! `flux start` will start a flux instance given a command, and usually you would want to have that be an entrypoint (e.g., running the python script with parsl) and then within that script, there will be a flux instance already running you can `flux submit` to. If you do `flux start` every time you are essentially creating an instance every time (not sure that makes sense).
This is the codepath that tries to create runinfo/

```        if not os.path.exists(path):
            os.makedirs(path)
```

(where path = `"runinfo"`)

So that makedir shouldn't be running at all if runinfo exists ahead of time... (i.e. the 2nd and 3rd invocations of the simple test above).

So there's a race condition if you parsl.load() simultaneously in that case - because potentially multiple parsls would try to create the base directory.

But I don't understand how that matches up with the behaviour you see where removing the rundir that is inherited from the container image - it should be *less* racy in that situation.


About launch commands, that's beyond what I know about the flux executor - the author is @jameshcorbett who might be able to say more.
I tried changing the launch command, but there seems to be a bug you don't set the attribute if it's provided: https://github.com/Parsl/parsl/blob/33398b15cbb20ddc5c273d22015d6c18351337c4/parsl/executors/flux/executor.py#L204-L205
Maybe needs to be just

```python
self.launch_cmd = launch_cmd or self.DEFAULT_LAUNCH_CMD
```
Assuming an empty string is not appropriate! :laughing: 
okay I made a little hack / workaround to test this out:

```python
executor = FluxExecutor(working_dir=args.working_dir)

# Workaround for bug that it isn't set if provided
executor.launch_cmd="{flux} submit {python} {manager} {protocol} {hostname} {port}"
config = Config(executors=[executor])
parsl.load(config)
```

And that fixed the issue - without removing the runinfo directory, it runs without issue! So I suspect there is something weird happening with multiple flux instances getting `flux start`. I can't say I can explain it now (it's like 2am here and I probably should go to bed and recharge, haha). But I think the fix is to allow setting that attribute - indeed we can't know if the person is starting already with access to a flux instance or not. I do think it might be kind of weird to run `flux start` a gazillion times, but maybe that's just me.

I can put in a small PR soon to fix that bug if you like!

Resolving patch:
diff --git a/parsl/executors/flux/executor.py b/parsl/executors/flux/executor.py
--- a/parsl/executors/flux/executor.py
+++ b/parsl/executors/flux/executor.py
@@ -201,8 +201,8 @@ def __init__(
         self.flux_path = os.path.abspath(flux_path)
         self._task_id_counter = itertools.count()
         self._socket = zmq.Context().socket(zmq.REP)
-        if launch_cmd is None:
-            self.launch_cmd = self.DEFAULT_LAUNCH_CMD
+        # Assumes a launch command cannot be None or empty
+        self.launch_cmd = launch_cmd or self.DEFAULT_LAUNCH_CMD
         self._submission_queue: queue.Queue = queue.Queue()
         self._stop_event = threading.Event()
         # lock to protect self._task_id_counter and also submission/shutdown race


Test patch: