Repository: Parsl/parsl

Issue:
Fix `FileExists` error when running `LocalProvider` + `SSHChannel`
As reported by @ravihansa3000, a `FileExists` error is raised when running with `LocalProvider` + `SSHChannel`:

```
2020-06-28 02:02:27 parsl.channels.ssh.ssh:215 [ERROR] Remote file copy will overwrite a local file:/home/users/nus/csipav/scratch/Parsl-workflows/GDC-Downloader-Output/remote/COAD_0/runinfo/001/submit_scripts/parsl.localprovider.1593280864.8464878.sh.ec
NoneType: None
2020-06-28 02:02:27 parsl.dataflow.flow_control:114 [ERROR] Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/flow_control.py", line 112, in make_callback
self.callback(tasks=self._event_buffer, kind=kind)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 63, in poll
self._update_state()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 69, in _update_state
item.poll(now)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 27, in poll
self._status = self._executor.status()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/executors/status_handling.py", line 59, in status
status = self._make_status_dict(job_ids, self._provider.status(job_ids))
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 87, in status
str_ec = self._read_job_file(script_path, '.ec').strip()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 142, in _read_job_file
path = self._job_file_path(script_path, suffix)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 138, in _job_file_path
path = self.channel.pull_file(path, self.script_dir)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/channels/ssh/ssh.py", line 216, in pull_file
raise FileExists(None, self.hostname, filename=local_dest)
parsl.channels.errors.FileExists: Hostname:nus.nscc.sg, Reason:File name collision in channel transport phase:/home/users/nus/csipav/scratch/Parsl-workflows/GDC-Downloader-Output/remote/COAD_0/runinfo/001/submit_scripts/parsl.localprovider.1593280864.8464878.sh.ec
```

This was introduced in #1724 and there is already discussion there with @hategan and @yadudoc, but I'm opening an issue so we don't lose track of this.

Fix `FileExists` error when running `LocalProvider` + `SSHChannel`
As reported by @ravihansa3000, a `FileExists` error is raised when running with `LocalProvider` + `SSHChannel`:

```
2020-06-28 02:02:27 parsl.channels.ssh.ssh:215 [ERROR] Remote file copy will overwrite a local file:/home/users/nus/csipav/scratch/Parsl-workflows/GDC-Downloader-Output/remote/COAD_0/runinfo/001/submit_scripts/parsl.localprovider.1593280864.8464878.sh.ec
NoneType: None
2020-06-28 02:02:27 parsl.dataflow.flow_control:114 [ERROR] Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/flow_control.py", line 112, in make_callback
self.callback(tasks=self._event_buffer, kind=kind)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 63, in poll
self._update_state()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 69, in _update_state
item.poll(now)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 27, in poll
self._status = self._executor.status()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/executors/status_handling.py", line 59, in status
status = self._make_status_dict(job_ids, self._provider.status(job_ids))
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 87, in status
str_ec = self._read_job_file(script_path, '.ec').strip()
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 142, in _read_job_file
path = self._job_file_path(script_path, suffix)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/providers/local/local.py", line 138, in _job_file_path
path = self.channel.pull_file(path, self.script_dir)
File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/channels/ssh/ssh.py", line 216, in pull_file
raise FileExists(None, self.hostname, filename=local_dest)
parsl.channels.errors.FileExists: Hostname:nus.nscc.sg, Reason:File name collision in channel transport phase:/home/users/nus/csipav/scratch/Parsl-workflows/GDC-Downloader-Output/remote/COAD_0/runinfo/001/submit_scripts/parsl.localprovider.1593280864.8464878.sh.ec
```

This was introduced in #1724 and there is already discussion there with @hategan and @yadudoc, but I'm opening an issue so we don't lose track of this.



Discussion:

Resolving patch:
diff --git a/parsl/channels/ssh/ssh.py b/parsl/channels/ssh/ssh.py
--- a/parsl/channels/ssh/ssh.py
+++ b/parsl/channels/ssh/ssh.py
@@ -4,7 +4,7 @@
 
 import paramiko
 from parsl.channels.base import Channel
-from parsl.channels.errors import BadHostKeyException, AuthException, SSHException, BadScriptPath, BadPermsScriptPath, FileCopyException, FileExists
+from parsl.channels.errors import BadHostKeyException, AuthException, SSHException, BadScriptPath, BadPermsScriptPath, FileCopyException
 from parsl.utils import RepresentationMixin
 
 logger = logging.getLogger(__name__)
@@ -209,12 +209,6 @@ def pull_file(self, remote_source, local_dir):
                 logger.exception("Failed to create local_dir: {0}".format(local_dir))
                 raise BadScriptPath(e, self.hostname)
 
-        # Easier to check this than to waste time trying to pull file and
-        # realize there's a problem.
-        if os.path.exists(local_dest):
-            logger.exception("Remote file copy will overwrite a local file:{0}".format(local_dest))
-            raise FileExists(None, self.hostname, filename=local_dest)
-
         try:
             self._valid_sftp_client().get(remote_source, local_dest)
         except Exception as e:


Test patch: