Repository: nipy/nipype

Issue:
Multiproc plugin with numpy 1.9.0
### Summary
I am trying to use the Multiproc plugin but it does not send the tasks to the workers. I traced the reason for this to a numpy expression in _send_procs_to_workers():
`jobids = np.nonzero(~self.proc_done & (self.depidx.sum(0) == 0))[1]`
in https://github.com/nipy/nipype/blob/master/nipype/pipeline/plugins/multiproc.py (line 216)

### Actual behavior

jobids is of type numpy.matrixlib.defmatrix.matrix with shape: (1,1) 

### Expected behavior

jobids should be of type numpy.ndarray with shape (1,)

I can achieve this behavior with:
`jobids = np.asarray(np.nonzero(~self.proc_done & (self.depidx.sum(0) == 0))[1]).flatten()`

Then everything is fine and works for all my pipelines.

### How to replicate the behavior

I am using numpy version 1.9.0 which is still matching nipype's requirements, isn't it?

### Platform details:

{'nibabel_version': '2.1.0', 'sys_executable': '/usr/bin/python', 'networkx_version': '1.11', 'numpy_version': '1.9.0', 'sys_platform': 'linux2', 'sys_version': '2.7.9 (default, Jun 29 2016, 13:08:31) \n[GCC 4.9.2]', 'commit_source': 'repository', 'commit_hash': '8384523', 'pkg_path': '/home/klipsy1.psych.tu-dresden.de/foerster/nipype/nipype', 'nipype_version': '0.14.1-dev+g8384523', 'traits_version': '4.6.0', 'scipy_version': '0.14.0'}
0.14.1-dev+g8384523





Discussion:
There is a reference to a discussion here: https://github.com/nipy/nipype/pull/2200#discussion_r141605722 
And I think my problem is what he meant by:
"But it's a little less explicit that depidx is a thing that is not an ndarray and we need to be careful about how it interacts with a vector like proc_done." : @effigies 

The first commit that introduced this problem was 6402981b5e9cadcd4c12f4ce74833066e2fb2ea9 by @oesteban  but a lot has changed since then so I'm not sure how relevant this commit is.
@divetea thanks for this report - could you provide a small working example? It would be nice to have a test in place to catch this in the future
I am not sure what a small working example would look like, I guess any workflow will do that results in a call of  '_send_procs_to_workers'. It depends more on the environment ... I assume it has to do with our "old" numpy version which we have to keep for compatibility with other stuff, but I am not sure. Maybe someone can confirm this on a newer system. I can talk to my boss if i can invest some time in a proper test but otherwise I don't have time at the moment -> student + exam period.
It would be great if  someone actually pushed this, I can't go through the whole procedure right now since installing the developer mode and testing is hard on a system where i don't have a lot of rights. And I don't even know how to add the proper tag to this issue ;)
Also I don't know if this is a beautiful enough solution since it's only a quick workaround. Can someone with numpy knowledge have a look at this?
Thank you very much, keep up the good work.
Cheers
Maybe simplest is to revert to the original:

```Python
jobids = np.flatnonzero(~self.proc_done & (self.depidx.sum(axis=0) == 0).__array__())
```

It's ugly but explicit. @divetea can you confirm this also fixes your problem?

Resolving patch:
diff --git a/nipype/pipeline/plugins/multiproc.py b/nipype/pipeline/plugins/multiproc.py
--- a/nipype/pipeline/plugins/multiproc.py
+++ b/nipype/pipeline/plugins/multiproc.py
@@ -223,7 +223,9 @@ def _send_procs_to_workers(self, updatehash=False, graph=None):
 
         # Check to see if a job is available (jobs with all dependencies run)
         # See https://github.com/nipy/nipype/pull/2200#discussion_r141605722
-        jobids = np.nonzero(~self.proc_done & (self.depidx.sum(0) == 0))[1]
+        # See also https://github.com/nipy/nipype/issues/2372
+        jobids = np.flatnonzero(~self.proc_done &
+                                (self.depidx.sum(axis=0) == 0).__array__())
 
         # Check available resources by summing all threads and memory used
         free_memory_gb, free_processors = self._check_resources(


Test patch: