Repository: Parsl/parsl

Issue:
Parsl creates hidden working files, doesn't clean them up
**Is your feature request related to a problem? Please describe.**
I didn't know Parsl was creating so many log files because they were hidden. 
I'm OK with files appearing in my run directory, but not accumulating them endlessly without my notice.

To be specific, I don't know what is responsible for `.workers.block-0.17158*` files but I'd like them to be visible.

**Describe the solution you'd like**
No hidden files.

**Describe alternatives you've considered**
Adding a "clean up Parsl's logging files" CLI tool

**Additional context**
Is anyone else concerned?



Discussion:
I don't recognise that log file at all... Can you post what is in it and a full example path?
The files are located in the working directory for Parsl, and named like

```
.workers.block-0.1715894085.9649055.nodes
```

I could be blaming the wrong code, as I couldn't easily find where it was coming from in Parsl.
@WardLT whats inside?
that block-0.171... string is block and submit timestamp that appears in parsl a few times for job submissions. it's passed onto the batch system too as a job name, and so possibly is then used by the batch system too... what batch system are you using that you see this under?
It's a list of nodes that's, presumably, used by the launcher. 

Here's the config. It's running inside of a job on Polaris, which runs PBSPro

```python
        from parsl.providers import LocalProvider
        from parsl.launchers import MpiExecLauncher
        from parsl.addresses import address_by_interface

        with open(os.environ.get('PBS_NODEFILE')) as fp:
            node_count = len(fp.readlines())
        executors = [
            HighThroughputExecutor(
                address=address_by_interface('bond0'),
                label='workers',
                max_workers=args.worker_count,
                cores_per_worker=1e-6,
                provider=LocalProvider(
                    nodes_per_block=node_count,
                    init_blocks=1,
                    min_blocks=0,
                    max_blocks=1,
                    launcher=MpiExecLauncher(bind_cmd="--cpu-bind", overrides="--depth=64 --ppn 1"),
                    worker_init='module load miniconda-3\nconda activate /lus/theta-fs0/projects/CSC249ADCD08/edw/env\n'
                ),
            ),
        ]

        config = Config(executors=executors, run_dir=out_dir)

```
MpiExecLaunch does something with jobname and a .nodes file - see here: https://github.com/Parsl/parsl/blob/b2147140e415fa3bd0eb8118bfdc0e0fe53ccdad/parsl/launchers/launchers.py#L223

However I don't see that it would put `.workers.` on the front so this might not be the place this is coming from.

My other suspicion is that it is a PBS nodefile - what value is in `$PBS_NODEFILE` inside your batch job? Everywhere I looked just refers to the variable, not to what I might expect the filename to look like... so it turns out I have no idea what to expect there at all.

Here's the contents of the PBS_NODEFILE variable

```
(base) lward@x3006c0s19b1n0:~/ExaMol/colmena/demo_apps/task-limits> echo $PBS_NODEFILE 
/var/spool/pbs/aux/1940985.polaris-pbs-01.hsn.cm.polaris.alcf.anl.gov
```
Where does `$JOBNAME` come from?

https://github.com/Parsl/parsl/blob/master/parsl/launchers/launchers.py#L223
https://github.com/Parsl/parsl/blob/b2147140e415fa3bd0eb8118bfdc0e0fe53ccdad/parsl/providers/pbspro/template.py#L14

which is populated by a job name value generated here https://github.com/Parsl/parsl/blob/b2147140e415fa3bd0eb8118bfdc0e0fe53ccdad/parsl/providers/pbspro/pbspro.py#L152

which includes a timestamp and the underlying job name generated here:

https://github.com/Parsl/parsl/blob/b2147140e415fa3bd0eb8118bfdc0e0fe53ccdad/parsl/executors/status_handling.py#L213

so I think if you `echo` the host file name and job name inside that MPI executor, it will not be the name we are looking for... but probably worth checking to be sure.
(I see where the string `workers` is coming from now - in your executor definition - so workers.block-id.timestamp is definitely present here as a substring of the job name - not sure if we're perhaps just losing the parsl off the front somehow?)
Yea, that's seems right. I'm digging through to see how `label` gets into `job_name` parameter passed to `.submit` of `LocalProvider`.
line 213 of status_handling (the parent class of htex) linked above - `self.label` there is the string `"workers"` in your case. and two lines later it is passed into self.provider.submit.
(ignore the links above about PBSProvider templates - you aren't using that...)
(instead JOBNAME is set in LocalProvider.submit)

Ah, I see it. Where is `parsl.` going!?
Ah ha! Check out the `.sh` file it writes. There's a `$` that is resulting in `$parsl` to be treated as an environment variable

```bash
module load miniconda-3
conda activate /lus/theta-fs0/projects/CSC249ADCD08/edw/env

export JOBNAME=$parsl.workers.block-0.1715928813.7517216
```
aha!
Lol, I was the one to introduce this problem to LocalProvider. It seems like all the providers have the unnecessary `$` 

https://github.com/Parsl/parsl/commit/990f80b0c3bc88f75025658cc51e96586e0fb7c6
I'm making a PR to remove the `$` from all the `export JOBNAME=$jobname` that are throughout parsl
Hold up, all the other uses of $jobname are correct. It's only me 🤦🏼 

Resolving patch:
diff --git a/parsl/providers/local/local.py b/parsl/providers/local/local.py
--- a/parsl/providers/local/local.py
+++ b/parsl/providers/local/local.py
@@ -206,7 +206,7 @@ def submit(self, command, tasks_per_node, job_name="parsl.localprovider"):
         script_path = "{0}/{1}.sh".format(self.script_dir, job_name)
         script_path = os.path.abspath(script_path)
 
-        wrap_command = self.worker_init + f'\nexport JOBNAME=${job_name}\n' + self.launcher(command, tasks_per_node, self.nodes_per_block)
+        wrap_command = self.worker_init + f'\nexport JOBNAME={job_name}\n' + self.launcher(command, tasks_per_node, self.nodes_per_block)
 
         self._write_submit_script(wrap_command, script_path)
 


Test patch: