Repository: nipy/nipype

Issue:
Workflow crashing with file not found due to strange nipype.pipeline.engine.utils.modify_path behavior
### Summary

When getting the node results, the `modify_path` function will mistakenly identify some outputs as files, causing the run to fail. 

### Actual behavior

Workflow crashes with a file not found error due to output being mistakenly identified as a file. This occurs when the output is a string (or bytes? untested), and a file with the same name exists in the current working directory.

### Expected behavior

`modify_path` does not identify the object as a file.

### How to replicate the behavior

See below

### Script/Workflow details

The following script will fail if there is a file named "22" in the directory from which it is called.
```python
from nipype.pipeline import engine as pe
from nipype.interfaces.base import BaseInterfaceInputSpec, TraitedSpec, BaseInterface, traits


class Concat2InputSpec(BaseInterfaceInputSpec):
    x = traits.String()


class Concat2OutputSpec(TraitedSpec):
    y = traits.String()


class Concat2(BaseInterface):
    input_spec = Concat2InputSpec
    output_spec = Concat2OutputSpec

    def _run_interface(self, runtime):
        self._outval = self.inputs.x + '2'
        return runtime

    def _list_outputs(self):
        outputs = self._outputs().get()
        outputs['y'] = self._outval
        return outputs


if __name__ == '__main__':
    node = pe.Node(Concat2(x=2), 'double')
    node.run()
```



Please put URL to code or code here (if not too long).

### Platform details:

<!-- Please run the following code from your shell and place the output between the triple ticks, below.
python -c "import nipype; from pprint import pprint; pprint(nipype.get_info())"
-->

```
{'commit_hash': '7fbc7cbac',
 'commit_source': 'repository',
 'networkx_version': '2.3',
 'nibabel_version': '2.4.1',
 'nipype_version': '1.2.1-dev+g7fbc7cbac',
 'numpy_version': '1.16.4',
 'pkg_path': '/home/stilley2/dev/nipype_debug/nipype/nipype',
 'scipy_version': '1.3.0',
 'sys_executable': '/home/stilley2/dev/nipype_debug/pyenv/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.7.3 (default, May 11 2019, 00:38:04) \n'
                '[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]',
 'traits_version': '5.1.1'}

```

### Execution environment


- My python environment outside container

Workflow crashing with file not found due to strange nipype.pipeline.engine.utils.modify_path behavior
### Summary

When getting the node results, the `modify_path` function will mistakenly identify some outputs as files, causing the run to fail. 

### Actual behavior

Workflow crashes with a file not found error due to output being mistakenly identified as a file. This occurs when the output is a string (or bytes? untested), and a file with the same name exists in the current working directory.

### Expected behavior

`modify_path` does not identify the object as a file.

### How to replicate the behavior

See below

### Script/Workflow details

The following script will fail if there is a file named "22" in the directory from which it is called.
```python
from nipype.pipeline import engine as pe
from nipype.interfaces.base import BaseInterfaceInputSpec, TraitedSpec, BaseInterface, traits


class Concat2InputSpec(BaseInterfaceInputSpec):
    x = traits.String()


class Concat2OutputSpec(TraitedSpec):
    y = traits.String()


class Concat2(BaseInterface):
    input_spec = Concat2InputSpec
    output_spec = Concat2OutputSpec

    def _run_interface(self, runtime):
        self._outval = self.inputs.x + '2'
        return runtime

    def _list_outputs(self):
        outputs = self._outputs().get()
        outputs['y'] = self._outval
        return outputs


if __name__ == '__main__':
    node = pe.Node(Concat2(x=2), 'double')
    node.run()
```



Please put URL to code or code here (if not too long).

### Platform details:

<!-- Please run the following code from your shell and place the output between the triple ticks, below.
python -c "import nipype; from pprint import pprint; pprint(nipype.get_info())"
-->

```
{'commit_hash': '7fbc7cbac',
 'commit_source': 'repository',
 'networkx_version': '2.3',
 'nibabel_version': '2.4.1',
 'nipype_version': '1.2.1-dev+g7fbc7cbac',
 'numpy_version': '1.16.4',
 'pkg_path': '/home/stilley2/dev/nipype_debug/nipype/nipype',
 'scipy_version': '1.3.0',
 'sys_executable': '/home/stilley2/dev/nipype_debug/pyenv/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.7.3 (default, May 11 2019, 00:38:04) \n'
                '[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]',
 'traits_version': '5.1.1'}

```

### Execution environment


- My python environment outside container

PR: FIX Check that file exists in basedir, not current dir
## Summary
<!-- Please reference any related issue and use fixes/close to automatically
close them, if pertinent -->

Fixes #2944 

## List of changes proposed in this PR (pull-request)
- modify_paths function checks if argument is a file in the basedir, not the current directory.

## Acknowledgment

- [x] \(Mandatory\) I acknowledge that this contribution will be available under the Apache 2 license.

FIX: Use traits to provide type hints when modifying paths
## Summary
This may not be the best way to do things, but in all cases where we're calling `modify_paths`, we have a traited spec that we can use to provide type hints, if we're careful about inspecting a fairly inconsistent API. Hopefully this falls back to the old behavior.

Additionally, I created a function `resolve_path` which should provide a consistent convention for returning absolute or relative paths, given an absolute or relative path, with respect to a base directory. This allowed some of the logic of `modify_paths` to be simplified, although that hardly makes up for the total muddle that is the type hints.

Let me know what you think, @oesteban, @satra, @stilley2. If this seems like the way to go, I can finish it up.

TODO:

* [ ] Test cases for `modify_paths`
* [ ] May need to comment on type hints bits

Fixes #2944.

## Acknowledgment

- [x] \(Mandatory\) I acknowledge that this contribution will be available under the Apache 2 license.



Discussion:
For added context, running the provided script produces the following error:

```
190618-19:12:45,52 nipype.workflow INFO:
	 [Node] Setting-up "double" in "/private/var/folders/yb/10x6vxlx40s1dhv5vc163m680000gp/T/tmpwhu6niza/double".
190618-19:12:45,57 nipype.workflow INFO:
	 [Node] Running "double" ("__main__.Concat2")
Traceback (most recent call last):
  File "test.py", line 29, in <module>
    node.run()
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    self, report_type='postexec', is_mapnode=isinstance(self, MapNode))
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 149, in write_report
    result = node.result  # Locally cache result
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /private/var/folders/yb/10x6vxlx40s1dhv5vc163m680000gp/T/tmpwhu6niza/double/22 not found
```

Further, this was introduced in #2325, due to the saving/loading of the resultfile. I suspect the fact that we're losing trait information is the problem. Still trying to work through whether the correct response is the fix proposed in #2945.
cc @oesteban you might see the way to go here better, since #2325 was your refactor.
I'm not sure this is a problem of nipype - although I acknowledge there could be potential file name collisions conducive to this problem and we may want to keep track of existing files in the results file.

I'll wait on the original fMRIPrep issue to understand the problem.
I'm pretty sure this is a nipype issue and not an fmriprep issue. The above example does not rely on fmriprep, and definitely results in undesirable behavior. I agree with @effigies that the main issue is loss of type information, and trying to recover that information, so #2945 is probably not an ideal solution.
I can only imagine two situations where this is an issue:
- If an interface has both an string output and a file output, and both values coincide (which is a pretty particular condition, and I'm not sure there is an interface actually doing this).
- If a previous run of a workflow is manipulated adding such a file (which is artificial and this is why I'm not convinced this is a nipype issue).

For that reason, I think we should first replicate and identify the error condition in fMRIPrep and then decide whether it is worth tracking file traits.

WDYT @satra?
The example script I posted doesn't satisfy either criteria, but still causes an issue. To cause this issue all that is required is:
1. an interface has a string output
2. A file with a name equal to that string exists in the directory from which the python script was called (i.e., not the workflow's working directory).
Gotcha, I hadn't realized of number 2. Well, I guess then #2945 should address this issue.
One question I have is why we're modifying paths in the resultfile.
It is a buggy filter to convert paths to relative (before storing) and then back to absolute (after loading).
Sure, but why?
I would imagine (and that was there before my refactor) that it is to make the result file work even if you move the work directory somewhere else. For a single run (no reuse), I agree it doesn't make much sense. @satra may have a better understanding of this.
@oesteban and @effigies -  this should be controlled through a config option:

https://github.com/nipy/nipype/blob/master/nipype/utils/config.py#L65

but it sounds like the option is not being implemented properly.  the default is `false`, which means it should not use relative paths unless forced.

I think that's being handled inside `modify_paths`:

https://github.com/nipy/nipype/blob/1eeabd36597b67b192d6e16fd3abbb8d1d8121bf/nipype/pipeline/engine/utils.py#L490-L496

The problem is that we're identifying paths as strings that exist on the filesystem relative to some directory (either CWD or node-WD), which can sometimes be true for `String` traits, as well. So avoiding relative paths won't save us when the string output is a valid path; forcing to absolute will still modify the string.
@effigies - in other situations we check the traits metadata to determine if this is a string vs a file. (`hash_files=False`). a good improvement would be to separate out a Path object that is a string vs a Path object that is a file.
> in other situations we check the traits metadata to determine if this is a string vs a file. (hash_files=False).

Yes, but this is operating on a dict, without trait information. I'm working on a strategy to pass in traits as type hints.

> a good improvement would be to separate out a Path object that is a string vs a Path object that is a file.

Could you clarify? Are you suggesting to use `pathlib.Path` as the object that is pickled?
@effigies - i simply meant altering the traits_extension File class to either return a string or a pathlib object depending on the traits metadata.
> @effigies - i simply meant altering the traits_extension File class to either return a string or a pathlib object depending on the traits metadata.

I've looked at the traits stuff, and have decided I don't really know how to start. Just noting this, in case somebody else has time to play with traits.
Having a look right this minute

@satra the new trait would return pathlike if ``exists is True`` or this new metadata is ``True``, correct?
@oesteban - the metadata already exists, so the validate function in the `File` trait would return a  `Path` object  `if hash_files is not False`
i would start with updating the `File/Directory` trait in traits_extensiion instead of creating a new trait.
Sorry, I was very unclear. Please check the draft PR above - I was actually modifying ``File/Directory``.
@satra, the metadata name is ``nohash``, correct?
the metadata name is `hash_files`, `nohash` is when a particular traits item should not be included in the hash computation. while `hash_files` either hashes the string (when `False`) or the content/timestamp (when `True` or missing)
For added context, running the provided script produces the following error:

```
190618-19:12:45,52 nipype.workflow INFO:
	 [Node] Setting-up "double" in "/private/var/folders/yb/10x6vxlx40s1dhv5vc163m680000gp/T/tmpwhu6niza/double".
190618-19:12:45,57 nipype.workflow INFO:
	 [Node] Running "double" ("__main__.Concat2")
Traceback (most recent call last):
  File "test.py", line 29, in <module>
    node.run()
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    self, report_type='postexec', is_mapnode=isinstance(self, MapNode))
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 149, in write_report
    result = node.result  # Locally cache result
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/anaconda3/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /private/var/folders/yb/10x6vxlx40s1dhv5vc163m680000gp/T/tmpwhu6niza/double/22 not found
```

Further, this was introduced in #2325, due to the saving/loading of the resultfile. I suspect the fact that we're losing trait information is the problem. Still trying to work through whether the correct response is the fix proposed in #2945.
cc @oesteban you might see the way to go here better, since #2325 was your refactor.
I'm not sure this is a problem of nipype - although I acknowledge there could be potential file name collisions conducive to this problem and we may want to keep track of existing files in the results file.

I'll wait on the original fMRIPrep issue to understand the problem.
I'm pretty sure this is a nipype issue and not an fmriprep issue. The above example does not rely on fmriprep, and definitely results in undesirable behavior. I agree with @effigies that the main issue is loss of type information, and trying to recover that information, so #2945 is probably not an ideal solution.
I can only imagine two situations where this is an issue:
- If an interface has both an string output and a file output, and both values coincide (which is a pretty particular condition, and I'm not sure there is an interface actually doing this).
- If a previous run of a workflow is manipulated adding such a file (which is artificial and this is why I'm not convinced this is a nipype issue).

For that reason, I think we should first replicate and identify the error condition in fMRIPrep and then decide whether it is worth tracking file traits.

WDYT @satra?
The example script I posted doesn't satisfy either criteria, but still causes an issue. To cause this issue all that is required is:
1. an interface has a string output
2. A file with a name equal to that string exists in the directory from which the python script was called (i.e., not the workflow's working directory).
Gotcha, I hadn't realized of number 2. Well, I guess then #2945 should address this issue.
One question I have is why we're modifying paths in the resultfile.
It is a buggy filter to convert paths to relative (before storing) and then back to absolute (after loading).
Sure, but why?
I would imagine (and that was there before my refactor) that it is to make the result file work even if you move the work directory somewhere else. For a single run (no reuse), I agree it doesn't make much sense. @satra may have a better understanding of this.
@oesteban and @effigies -  this should be controlled through a config option:

https://github.com/nipy/nipype/blob/master/nipype/utils/config.py#L65

but it sounds like the option is not being implemented properly.  the default is `false`, which means it should not use relative paths unless forced.

I think that's being handled inside `modify_paths`:

https://github.com/nipy/nipype/blob/1eeabd36597b67b192d6e16fd3abbb8d1d8121bf/nipype/pipeline/engine/utils.py#L490-L496

The problem is that we're identifying paths as strings that exist on the filesystem relative to some directory (either CWD or node-WD), which can sometimes be true for `String` traits, as well. So avoiding relative paths won't save us when the string output is a valid path; forcing to absolute will still modify the string.
@effigies - in other situations we check the traits metadata to determine if this is a string vs a file. (`hash_files=False`). a good improvement would be to separate out a Path object that is a string vs a Path object that is a file.
> in other situations we check the traits metadata to determine if this is a string vs a file. (hash_files=False).

Yes, but this is operating on a dict, without trait information. I'm working on a strategy to pass in traits as type hints.

> a good improvement would be to separate out a Path object that is a string vs a Path object that is a file.

Could you clarify? Are you suggesting to use `pathlib.Path` as the object that is pickled?
@effigies - i simply meant altering the traits_extension File class to either return a string or a pathlib object depending on the traits metadata.
> @effigies - i simply meant altering the traits_extension File class to either return a string or a pathlib object depending on the traits metadata.

I've looked at the traits stuff, and have decided I don't really know how to start. Just noting this, in case somebody else has time to play with traits.
Having a look right this minute

@satra the new trait would return pathlike if ``exists is True`` or this new metadata is ``True``, correct?
@oesteban - the metadata already exists, so the validate function in the `File` trait would return a  `Path` object  `if hash_files is not False`
i would start with updating the `File/Directory` trait in traits_extensiion instead of creating a new trait.
Sorry, I was very unclear. Please check the draft PR above - I was actually modifying ``File/Directory``.
@satra, the metadata name is ``nohash``, correct?
the metadata name is `hash_files`, `nohash` is when a particular traits item should not be included in the hash computation. while `hash_files` either hashes the string (when `False`) or the content/timestamp (when `True` or missing)
# [Codecov](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=h1) Report
> Merging [#2945](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=desc) into [master](https://codecov.io/gh/nipy/nipype/commit/7fbc7cbac72d694894195cbef1605230f87e1dc4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/nipy/nipype/pull/2945/graphs/tree.svg?width=650&token=Tu0EnSSGVZ&height=150&src=pr)](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2945   +/-   ##
=======================================
  Coverage   67.59%   67.59%           
=======================================
  Files         344      344           
  Lines       43747    43747           
  Branches     5456     5456           
=======================================
  Hits        29572    29572           
+ Misses      13466    13458    -8     
- Partials      709      717    +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #smoketests | `50.37% <33.33%> (ø)` | :arrow_up: |
| #unittests | `65.03% <100%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [nipype/pipeline/engine/utils.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL3BpcGVsaW5lL2VuZ2luZS91dGlscy5weQ==) | `82.26% <100%> (ø)` | :arrow_up: |
| [nipype/interfaces/base/core.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL2ludGVyZmFjZXMvYmFzZS9jb3JlLnB5) | `86.88% <100%> (ø)` | :arrow_up: |
| [nipype/interfaces/io.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL2ludGVyZmFjZXMvaW8ucHk=) | `55.07% <0%> (ø)` | :arrow_up: |
| [nipype/interfaces/freesurfer/preprocess.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL2ludGVyZmFjZXMvZnJlZXN1cmZlci9wcmVwcm9jZXNzLnB5) | `66.11% <0%> (ø)` | :arrow_up: |
| [nipype/interfaces/dynamic\_slicer.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL2ludGVyZmFjZXMvZHluYW1pY19zbGljZXIucHk=) | `17.47% <0%> (ø)` | :arrow_up: |
| [nipype/interfaces/nipy/preprocess.py](https://codecov.io/gh/nipy/nipype/pull/2945/diff?src=pr&el=tree#diff-bmlweXBlL2ludGVyZmFjZXMvbmlweS9wcmVwcm9jZXNzLnB5) | `45.79% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=footer). Last update [7fbc7cb...2e95a8b](https://codecov.io/gh/nipy/nipype/pull/2945?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Using traits is a much better solution than my proposed fix. My knowledge of the traits system isn't so great, so I don't have any comments on the implementation itself.

Resolving patch:
diff --git a/nipype/interfaces/base/traits_extension.py b/nipype/interfaces/base/traits_extension.py
--- a/nipype/interfaces/base/traits_extension.py
+++ b/nipype/interfaces/base/traits_extension.py
@@ -526,7 +526,7 @@ def _recurse_on_path_traits(func, thistrait, value, cwd):
     elif thistrait.is_trait_type(traits.List):
         innertrait, = thistrait.inner_traits
         if not isinstance(value, (list, tuple)):
-            value = [value]
+            return _recurse_on_path_traits(func, innertrait, value, cwd)
 
         value = [_recurse_on_path_traits(func, innertrait, v, cwd)
                  for v in value]
diff --git a/nipype/pipeline/engine/nodes.py b/nipype/pipeline/engine/nodes.py
--- a/nipype/pipeline/engine/nodes.py
+++ b/nipype/pipeline/engine/nodes.py
@@ -587,7 +587,9 @@ def _load_results(self):
                     runtime=runtime,
                     inputs=self._interface.inputs.get_traitsfree(),
                     outputs=aggouts)
-                _save_resultfile(result, cwd, self.name)
+                _save_resultfile(
+                    result, cwd, self.name,
+                    rebase=str2bool(self.config['execution']['use_relative_paths']))
             else:
                 logger.debug('aggregating mapnode results')
                 result = self._run_interface()
@@ -634,7 +636,9 @@ def _run_command(self, execute, copyfiles=True):
             except Exception as msg:
                 result.runtime.stderr = '{}\n\n{}'.format(
                     getattr(result.runtime, 'stderr', ''), msg)
-                _save_resultfile(result, outdir, self.name)
+                _save_resultfile(
+                    result, outdir, self.name,
+                    rebase=str2bool(self.config['execution']['use_relative_paths']))
                 raise
             cmdfile = op.join(outdir, 'command.txt')
             with open(cmdfile, 'wt') as fd:
@@ -646,7 +650,9 @@ def _run_command(self, execute, copyfiles=True):
         except Exception as msg:
             result.runtime.stderr = '%s\n\n%s'.format(
                 getattr(result.runtime, 'stderr', ''), msg)
-            _save_resultfile(result, outdir, self.name)
+            _save_resultfile(
+                result, outdir, self.name,
+                rebase=str2bool(self.config['execution']['use_relative_paths']))
             raise
 
         dirs2keep = None
@@ -660,7 +666,9 @@ def _run_command(self, execute, copyfiles=True):
             self.needed_outputs,
             self.config,
             dirs2keep=dirs2keep)
-        _save_resultfile(result, outdir, self.name)
+        _save_resultfile(
+            result, outdir, self.name,
+            rebase=str2bool(self.config['execution']['use_relative_paths']))
 
         return result
 
@@ -1260,7 +1268,7 @@ def _run_interface(self, execute=True, updatehash=False):
                 stop_first=str2bool(
                     self.config['execution']['stop_on_first_crash'])))
         # And store results
-        _save_resultfile(result, cwd, self.name)
+        _save_resultfile(result, cwd, self.name, rebase=False)
         # remove any node directories no longer required
         dirs2remove = []
         for path in glob(op.join(cwd, 'mapflow', '*')):
diff --git a/nipype/pipeline/engine/utils.py b/nipype/pipeline/engine/utils.py
--- a/nipype/pipeline/engine/utils.py
+++ b/nipype/pipeline/engine/utils.py
@@ -25,13 +25,13 @@
 from ... import logging, config, LooseVersion
 from ...utils.filemanip import (
     Path,
+    indirectory,
     relpath,
     makedirs,
     fname_presuffix,
     to_str,
     ensure_list,
     get_related_files,
-    FileNotFoundError,
     save_json,
     savepkl,
     loadpkl,
@@ -41,8 +41,10 @@
 )
 from ...utils.misc import str2bool
 from ...utils.functions import create_function_from_source
-from ...interfaces.base import (Bunch, CommandLine, isdefined, Undefined,
-                                InterfaceResult, traits)
+from ...interfaces.base.traits_extension import (
+    rebase_path_traits, resolve_path_traits, OutputMultiPath, isdefined, Undefined, traits)
+from ...interfaces.base.support import Bunch, InterfaceResult
+from ...interfaces.base import CommandLine
 from ...interfaces.utility import IdentityInterface
 from ...utils.provenance import ProvStore, pm, nipype_ns, get_id
 
@@ -227,52 +229,101 @@ def write_report(node, report_type=None, is_mapnode=False):
     return
 
 
-def save_resultfile(result, cwd, name):
-    """Save a result pklz file to ``cwd``"""
+def save_resultfile(result, cwd, name, rebase=None):
+    """Save a result pklz file to ``cwd``."""
+    if rebase is None:
+        rebase = config.getboolean('execution', 'use_relative_paths')
+
+    cwd = os.path.abspath(cwd)
     resultsfile = os.path.join(cwd, 'result_%s.pklz' % name)
-    savepkl(resultsfile, result)
-    logger.debug('saved results in %s', resultsfile)
+    logger.debug("Saving results file: '%s'", resultsfile)
 
+    if result.outputs is None:
+        logger.warn('Storing result file without outputs')
+        savepkl(resultsfile, result)
+        return
+    try:
+        output_names = result.outputs.copyable_trait_names()
+    except AttributeError:
+        logger.debug('Storing non-traited results, skipping rebase of paths')
+        savepkl(resultsfile, result)
+        return
 
-def load_resultfile(results_file):
+    backup_traits = {}
+    try:
+        with indirectory(cwd):
+            # All the magic to fix #2944 resides here:
+            for key in output_names:
+                old = getattr(result.outputs, key)
+                if isdefined(old):
+                    if result.outputs.trait(key).is_trait_type(OutputMultiPath):
+                        old = result.outputs.trait(key).handler.get_value(
+                            result.outputs, key)
+                    backup_traits[key] = old
+                    val = rebase_path_traits(result.outputs.trait(key), old, cwd)
+                    setattr(result.outputs, key, val)
+        savepkl(resultsfile, result)
+    finally:
+        # Restore resolved paths from the outputs dict no matter what
+        for key, val in list(backup_traits.items()):
+            setattr(result.outputs, key, val)
+
+
+def load_resultfile(results_file, resolve=True):
     """
-    Load InterfaceResult file from path
+    Load InterfaceResult file from path.
 
     Parameter
     ---------
-
     path : base_dir of node
     name : name of node
 
     Returns
     -------
-
     result : InterfaceResult structure
     aggregate : boolean indicating whether node should aggregate_outputs
     attribute error : boolean indicating whether there was some mismatch in
         versions of traits used to store result and hence node needs to
         rerun
+
     """
-    aggregate = True
     results_file = Path(results_file)
+    aggregate = True
     result = None
     attribute_error = False
-    if results_file.exists():
+
+    if not results_file.exists():
+        return result, aggregate, attribute_error
+
+    with indirectory(str(results_file.parent)):
         try:
             result = loadpkl(results_file)
-        except (traits.TraitError, AttributeError, ImportError,
-                EOFError) as err:
-            if isinstance(err, (AttributeError, ImportError)):
-                attribute_error = True
-                logger.debug('attribute error: %s probably using '
-                             'different trait pickled file', str(err))
-            else:
-                logger.debug(
-                    'some file does not exist. hence trait cannot be set')
+        except (traits.TraitError, EOFError):
+            logger.debug(
+                'some file does not exist. hence trait cannot be set')
+        except (AttributeError, ImportError) as err:
+            attribute_error = True
+            logger.debug('attribute error: %s probably using '
+                         'different trait pickled file', str(err))
         else:
             aggregate = False
 
-    logger.debug('Aggregate: %s', aggregate)
+        if resolve and result.outputs:
+            try:
+                outputs = result.outputs.get()
+            except TypeError:  # This is a Bunch
+                return result, aggregate, attribute_error
+
+            logger.debug('Resolving paths in outputs loaded from results file.')
+            for trait_name, old in list(outputs.items()):
+                if isdefined(old):
+                    if result.outputs.trait(trait_name).is_trait_type(OutputMultiPath):
+                        old = result.outputs.trait(trait_name).handler.get_value(
+                            result.outputs, trait_name)
+                    value = resolve_path_traits(result.outputs.trait(trait_name), old,
+                                                results_file.parent)
+                    setattr(result.outputs, trait_name, value)
+
     return result, aggregate, attribute_error
 
 


Test patch:
diff --git a/nipype/caching/tests/test_memory.py b/nipype/caching/tests/test_memory.py
--- a/nipype/caching/tests/test_memory.py
+++ b/nipype/caching/tests/test_memory.py
@@ -15,8 +15,7 @@ class SideEffectInterface(EngineTestInterface):
     def _run_interface(self, runtime):
         global nb_runs
         nb_runs += 1
-        runtime.returncode = 0
-        return runtime
+        return super(SideEffectInterface, self)._run_interface(runtime)
 
 
 def test_caching(tmpdir):
diff --git a/nipype/pipeline/engine/tests/test_base.py b/nipype/pipeline/engine/tests/test_base.py
--- a/nipype/pipeline/engine/tests/test_base.py
+++ b/nipype/pipeline/engine/tests/test_base.py
@@ -20,19 +20,15 @@ class OutputSpec(nib.TraitedSpec):
     output1 = nib.traits.List(nib.traits.Int, desc='outputs')
 
 
-class EngineTestInterface(nib.BaseInterface):
+class EngineTestInterface(nib.SimpleInterface):
     input_spec = InputSpec
     output_spec = OutputSpec
 
     def _run_interface(self, runtime):
         runtime.returncode = 0
+        self._results['output1'] = [1, self.inputs.input1]
         return runtime
 
-    def _list_outputs(self):
-        outputs = self._outputs().get()
-        outputs['output1'] = [1, self.inputs.input1]
-        return outputs
-
 
 @pytest.mark.parametrize(
     'name', ['valid1', 'valid_node', 'valid-node', 'ValidNode0'])
diff --git a/nipype/pipeline/engine/tests/test_utils.py b/nipype/pipeline/engine/tests/test_utils.py
--- a/nipype/pipeline/engine/tests/test_utils.py
+++ b/nipype/pipeline/engine/tests/test_utils.py
@@ -224,3 +224,62 @@ def test_mapnode_crash3(tmpdir):
     wf.config["execution"]["crashdump_dir"] = os.getcwd()
     with pytest.raises(RuntimeError):
         wf.run(plugin='Linear')
+
+class StrPathConfuserInputSpec(nib.TraitedSpec):
+    in_str = nib.traits.String()
+
+
+class StrPathConfuserOutputSpec(nib.TraitedSpec):
+    out_tuple = nib.traits.Tuple(nib.File, nib.traits.String)
+    out_dict_path = nib.traits.Dict(nib.traits.String, nib.File(exists=True))
+    out_dict_str = nib.traits.DictStrStr()
+    out_list = nib.traits.List(nib.traits.String)
+    out_str = nib.traits.String()
+    out_path = nib.File(exists=True)
+
+
+class StrPathConfuser(nib.SimpleInterface):
+    input_spec = StrPathConfuserInputSpec
+    output_spec = StrPathConfuserOutputSpec
+
+    def _run_interface(self, runtime):
+        out_path = os.path.abspath(os.path.basename(self.inputs.in_str) + '_path')
+        open(out_path, 'w').close()
+        self._results['out_str'] = self.inputs.in_str
+        self._results['out_path'] = out_path
+        self._results['out_tuple'] = (out_path, self.inputs.in_str)
+        self._results['out_dict_path'] = {self.inputs.in_str: out_path}
+        self._results['out_dict_str'] = {self.inputs.in_str: self.inputs.in_str}
+        self._results['out_list'] = [self.inputs.in_str] * 2
+        return runtime
+
+
+def test_modify_paths_bug(tmpdir):
+    """
+    There was a bug in which, if the current working directory contained a file with the name
+    of an output String, the string would get transformed into a path, and generally wreak havoc.
+    This attempts to replicate that condition, using an object with strings and paths in various
+    trait configurations, to ensure that the guards added resolve the issue.
+    Please see https://github.com/nipy/nipype/issues/2944 for more details.
+    """
+    tmpdir.chdir()
+
+    spc = pe.Node(StrPathConfuser(in_str='2'), name='spc')
+
+    open('2', 'w').close()
+
+    outputs = spc.run().outputs
+
+    # Basic check that string was not manipulated
+    out_str = outputs.out_str
+    assert out_str == '2'
+
+    # Check path exists and is absolute
+    out_path = outputs.out_path
+    assert os.path.isabs(out_path)
+
+    # Assert data structures pass through correctly
+    assert outputs.out_tuple == (out_path, out_str)
+    assert outputs.out_dict_path == {out_str: out_path}
+    assert outputs.out_dict_str == {out_str: out_str}
+    assert outputs.out_list == [out_str] * 2