Repository: nipy/nipype

Issue:
Nipype cannot be included in a requirements.txt file assuming an empty environment
Nipype installation has some dependency on nibabel, which means that you cannot write a `requirements.txt` file with both packages and expect it to work (see below for exact issue).

This is problematic as I am trying to put together a minimal set of files to initialize the environment used when analyzing data for a paper. Nipype is supposedly aimed at making research more reproducible, so I don't think the installation process should cause extra headaches for people trying to do that.

```
$ conda create -n paper pip
$ source activate paper
$ conda install --file conda_requirements.txt
<...>
$ pip install -r pip_requirements.txt
Downloading/unpacking nibabel==1.3.0 (from -r pip_requirements.txt (line 2))
  Downloading nibabel-1.3.0.tar.gz (2.2MB): 2.2MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nibabel/setup.py) egg_info for package nibabel
    Missing optional package "dicom" provided by package "pydicom"; you may get run-time errors

Downloading/unpacking nipype==0.9.1 (from -r pip_requirements.txt (line 3))
  Downloading nipype-0.9.1.tar.gz (13.4MB): 13.4MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py) egg_info for package nipype
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>
        raise RuntimeError('Need nisext package from nibabel installation'
    RuntimeError: Need nisext package from nibabel installation - please install nibabel first
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>

    raise RuntimeError('Need nisext package from nibabel installation'

RuntimeError: Need nisext package from nibabel installation - please install nibabel first

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype
Storing debug log for failure in /Users/mwaskom/.pip/pip.log
```



Discussion:
Sorry for that. Let me try to understand this more. This would work if you would do it in two steps (install nibabel first)?

i've noticed some strange issues with pip and conda. can you try:

/fullpath/to/envs/paper/bin/pip -r requirements.txt

> Sorry for that. Let me try to understand this more. This would work if you would do it in two steps (install nibabel first)?

Right, that works fine, but it defeats the purpose of shipping a single `requirements.txt` file to set up the environment with.

this worked fine for me: (i had to comment out dateutil in requirements.txt, added nipype and used conda to install the other dependencies because i didn't want pip to install them)

```
conda create -n testinstall pip scipy networkx traits dateutil nose
source activate testinstall
/software/anaconda/envs/testinstall/bin/pip install -r requirements.txt 
```

requirements.txt:

```
numpy>=1.3
scipy>=0.7
networkx>=1.0
traits>=4.0
#dateutil>=1.5
nibabel>=1.0
nose>=1.0
nipype>=0.9.2
```

Can't reproduce that...

```
conda install numpy scipy
~/anaconda/envs/paper/bin/pip install -r pip_requirements.txt

Downloading/unpacking nibabel==1.3.0 (from -r pip_requirements.txt (line 2))
  Downloading nibabel-1.3.0.tar.gz (2.2MB): 2.2MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nibabel/setup.py) egg_info for package nibabel
    Missing optional package "dicom" provided by package "pydicom"; you may get run-time errors

Downloading/unpacking nipype==0.9.1 (from -r pip_requirements.txt (line 3))
  Downloading nipype-0.9.1.tar.gz (13.4MB): 13.4MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py) egg_info for package nipype
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>
        raise RuntimeError('Need nisext package from nibabel installation'
    RuntimeError: Need nisext package from nibabel installation - please install nibabel first
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>

    raise RuntimeError('Need nisext package from nibabel installation'

RuntimeError: Need nisext package from nibabel installation - please install nibabel first

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype
Storing debug log for failure in /Users/mwaskom/.pip/pip.log
```

ok this worked because i didn't initially have nipype in the requirements file and then added it. for some reason pip doesn't complete nibabel installation before starting nipype installation. if i do it in two steps this works fine.

```
pip install -r req1.txt # contains nibabel
pip install -r req2.txt # contains nipype
```

don't know why this is happening though.

Whether or not it's a problem with pip is debatable, but it's something we should work around if possible. From what I can tell, nipype is just using a function that's about 20 lines of code to version based off the git commit, and the `package_check` function that duplicates a functionality that's already in setuptools (`install_requires`). I don't see any reason we shouldn't just move this stuff into nipype to avoid this headache (it pops up outside of requirements file, too, e.g. `pip install nibabel nipype` errors for the same reason, which is very annoying).

In general it seems like bad form to import a dependency in the setup script as it precludes any sort of automatic dependency management.

Related discussion for scipy here:  https://github.com/scipy/scipy/issues/3379

The summary is, pip's dependency checking is not always what you want, but interoperation with pip could certainly be improved...

Does pip come with Anaconda? I just installed Anaconda recently. When I am in the Anaconda command prompt it still defaults to the pip from my other/previous python. From what Satra said earlier, this might lead to problems?

Also is it necessary to do "source activate" on your virtualenv? What does this
do differently that just "activate". Thanks for the insights. 

> In general it seems like bad form to import a dependency in the setup script as it precludes any sort of automatic dependency management.

+1 to that.

What does `numpy.distutils.setup` provide that `setuptools.setup` doesn't?

@mwaskom - could you submit a PR for this?

> Does pip come with Anaconda? I just installed Anaconda recently. When I am in the Anaconda command prompt it still defaults to the pip from my other/previous python. From what Satra said earlier, this might lead to problems?

@ag-bell When building a conda environment, you can install `pip`. From then forward, when using that environment, pip will "just work".

```
$ conda create myenv pip numpy scipy ...
$ pip install something-not-available-from-conda # <-- goes into myenv
```

I've refactored the nibabel/nisext package_check to fill out setuptools requires stuff instead of raising an error.  See here:

https://github.com/nipy/nibabel/pull/234

Comments welcome.


Resolving patch:
diff --git a/nipype/info.py b/nipype/info.py
--- a/nipype/info.py
+++ b/nipype/info.py
@@ -104,6 +104,8 @@ def get_nipype_gitversion():
 NUMPY_MIN_VERSION = '1.3'
 SCIPY_MIN_VERSION = '0.7'
 TRAITS_MIN_VERSION = '4.0'
+DATEUTIL_MIN_VERSION = '1.0'
+NOSE_MIN_VERSION = '1.0'
 
 NAME                = 'nipype'
 MAINTAINER          = "nipype developers"
@@ -122,6 +124,7 @@ def get_nipype_gitversion():
 MICRO               = _version_micro
 ISRELEASE           = _version_extra == ''
 VERSION             = __version__
-REQUIRES            = ["nibabel (>=1.0)", "networkx (>=1.0)", "numpy (>=1.3)",
-                       "scipy (>=0.7)", "traits (>=4.0)"]
+REQUIRES            = ["nibabel>=1.0", "networkx>=1.0", "numpy>=1.3",
+                       "python-dateutil>1.0", "scipy>=0.7", "traits>=4.0",
+                       "nose>=1.0"]
 STATUS              = 'stable'
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,17 +7,245 @@
 packages and create an API for specifying a full analysis pipeline in
 python.
 
+Much of the machinery at the beginning of this file has been copied over from
+nibabel denoted by ## START - COPIED FROM NIBABEL and a corresponding ## END
+
 """
 
 import sys
 from glob import glob
+import os
+
+## START - COPIED FROM NIBABEL
+from os.path import join as pjoin
+from functools import partial
 
-# Import build helpers
+PY3 = sys.version_info[0] >= 3
+if PY3:
+    string_types = str,
+else:
+    string_types = basestring,
 try:
-    from nisext.sexts import package_check, get_comrec_build
+    from ConfigParser import ConfigParser
 except ImportError:
-    raise RuntimeError('Need nisext package from nibabel installation'
-                       ' - please install nibabel first')
+    from configparser import ConfigParser
+
+# BEFORE importing distutils, remove MANIFEST. distutils doesn't properly
+# update it when the contents of directories change.
+if os.path.exists('MANIFEST'): os.remove('MANIFEST')
+
+# For some commands, use setuptools.
+if len(set(('develop', 'bdist_egg', 'bdist_rpm', 'bdist', 'bdist_dumb',
+            'install_egg_info', 'egg_info', 'easy_install', 'bdist_wheel',
+            'bdist_mpkg')).intersection(sys.argv)) > 0:
+    # setup_egg imports setuptools setup, thus monkeypatching distutils.
+    import setup_egg
+
+from distutils.core import setup
+
+from distutils.version import LooseVersion
+from distutils.command.build_py import build_py
+
+from distutils import log
+
+def get_comrec_build(pkg_dir, build_cmd=build_py):
+    """ Return extended build command class for recording commit
+
+    The extended command tries to run git to find the current commit, getting
+    the empty string if it fails.  It then writes the commit hash into a file
+    in the `pkg_dir` path, named ``COMMIT_INFO.txt``.
+
+    In due course this information can be used by the package after it is
+    installed, to tell you what commit it was installed from if known.
+
+    To make use of this system, you need a package with a COMMIT_INFO.txt file -
+    e.g. ``myproject/COMMIT_INFO.txt`` - that might well look like this::
+
+        # This is an ini file that may contain information about the code state
+        [commit hash]
+        # The line below may contain a valid hash if it has been substituted
+        during 'git archive' archive_subst_hash=$Format:%h$
+        # This line may be modified by the install process
+        install_hash=
+
+    The COMMIT_INFO file above is also designed to be used with git substitution
+    - so you probably also want a ``.gitattributes`` file in the root directory
+    of your working tree that contains something like this::
+
+       myproject/COMMIT_INFO.txt export-subst
+
+    That will cause the ``COMMIT_INFO.txt`` file to get filled in by ``git
+    archive`` - useful in case someone makes such an archive - for example with
+    via the github 'download source' button.
+
+    Although all the above will work as is, you might consider having something
+    like a ``get_info()`` function in your package to display the commit
+    information at the terminal.  See the ``pkg_info.py`` module in the nipy
+    package for an example.
+    """
+    class MyBuildPy(build_cmd):
+        ''' Subclass to write commit data into installation tree '''
+        def run(self):
+            build_cmd.run(self)
+            import subprocess
+            proc = subprocess.Popen('git rev-parse HEAD',
+                                    stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    shell=True)
+            repo_commit, _ = proc.communicate()
+            # Fix for python 3
+            repo_commit = str(repo_commit)
+            # We write the installation commit even if it's empty
+            cfg_parser = ConfigParser()
+            cfg_parser.read(pjoin(pkg_dir, 'COMMIT_INFO.txt'))
+            cfg_parser.set('commit hash', 'install_hash', repo_commit)
+            out_pth = pjoin(self.build_lib, pkg_dir, 'COMMIT_INFO.txt')
+            cfg_parser.write(open(out_pth, 'wt'))
+    return MyBuildPy
+
+def _add_append_key(in_dict, key, value):
+    """ Helper for appending dependencies to setuptools args """
+    # If in_dict[key] does not exist, create it
+    # If in_dict[key] is a string, make it len 1 list of strings
+    # Append value to in_dict[key] list
+    if key not in in_dict:
+        in_dict[key] = []
+    elif isinstance(in_dict[key], string_types):
+        in_dict[key] = [in_dict[key]]
+    in_dict[key].append(value)
+
+# Dependency checks
+def package_check(pkg_name, version=None,
+                  optional=False,
+                  checker=LooseVersion,
+                  version_getter=None,
+                  messages=None,
+                  setuptools_args=None,
+                  pypi_pkg_name=None
+                  ):
+    ''' Check if package `pkg_name` is present and has good enough version
+
+    Has two modes of operation.  If `setuptools_args` is None (the default),
+    raise an error for missing non-optional dependencies and log warnings for
+    missing optional dependencies.  If `setuptools_args` is a dict, then fill
+    ``install_requires`` key value with any missing non-optional dependencies,
+    and the ``extras_requires`` key value with optional dependencies.
+
+    This allows us to work with and without setuptools.  It also means we can
+    check for packages that have not been installed with setuptools to avoid
+    installing them again.
+
+    Parameters
+    ----------
+    pkg_name : str
+       name of package as imported into python
+    version : {None, str}, optional
+       minimum version of the package that we require. If None, we don't
+       check the version.  Default is None
+    optional : bool or str, optional
+       If ``bool(optional)`` is False, raise error for absent package or wrong
+       version; otherwise warn.  If ``setuptools_args`` is not None, and
+       ``bool(optional)`` is not False, then `optional` should be a string
+       giving the feature name for the ``extras_require`` argument to setup.
+    checker : callable, optional
+       callable with which to return comparable thing from version
+       string.  Default is ``distutils.version.LooseVersion``
+    version_getter : {None, callable}:
+       Callable that takes `pkg_name` as argument, and returns the
+       package version string - as in::
+
+          ``version = version_getter(pkg_name)``
+
+       If None, equivalent to::
+
+          mod = __import__(pkg_name); version = mod.__version__``
+    messages : None or dict, optional
+       dictionary giving output messages
+    setuptools_args : None or dict
+       If None, raise errors / warnings for missing non-optional / optional
+       dependencies.  If dict fill key values ``install_requires`` and
+       ``extras_require`` for non-optional and optional dependencies.
+    pypi_pkg_name : None or string
+       When the pypi package name differs from the installed module. This is the
+       case with the package python-dateutil which installs as dateutil.
+    '''
+    setuptools_mode = not setuptools_args is None
+    optional_tf = bool(optional)
+    if version_getter is None:
+        def version_getter(pkg_name):
+            mod = __import__(pkg_name)
+            return mod.__version__
+    if messages is None:
+        messages = {}
+    msgs = {
+         'missing': 'Cannot import package "%s" - is it installed?',
+         'missing opt': 'Missing optional package "%s"',
+         'opt suffix' : '; you may get run-time errors',
+         'version too old': 'You have version %s of package "%s"'
+                            ' but we need version >= %s', }
+    msgs.update(messages)
+    status, have_version = _package_status(pkg_name,
+                                           version,
+                                           version_getter,
+                                           checker)
+    if pypi_pkg_name:
+        pkg_name = pypi_pkg_name
+
+    if status == 'satisfied':
+        return
+    if not setuptools_mode:
+        if status == 'missing':
+            if not optional_tf:
+                raise RuntimeError(msgs['missing'] % pkg_name)
+            log.warn(msgs['missing opt'] % pkg_name +
+                     msgs['opt suffix'])
+            return
+        elif status == 'no-version':
+            raise RuntimeError('Cannot find version for %s' % pkg_name)
+        assert status == 'low-version'
+        if not optional_tf:
+            raise RuntimeError(msgs['version too old'] % (have_version,
+                                                          pkg_name,
+                                                          version))
+        log.warn(msgs['version too old'] % (have_version,
+                                            pkg_name,
+                                            version)
+                    + msgs['opt suffix'])
+        return
+    # setuptools mode
+    if optional_tf and not isinstance(optional, string_types):
+        raise RuntimeError('Not-False optional arg should be string')
+    dependency = pkg_name
+
+    if version:
+        dependency += '>=' + version
+    if optional_tf:
+        if not 'extras_require' in setuptools_args:
+            setuptools_args['extras_require'] = {}
+        _add_append_key(setuptools_args['extras_require'],
+                        optional,
+                        dependency)
+        return
+    _add_append_key(setuptools_args, 'install_requires', dependency)
+    return
+
+
+def _package_status(pkg_name, version, version_getter, checker):
+    try:
+        __import__(pkg_name)
+    except ImportError:
+        return 'missing', None
+    if not version:
+        return 'satisfied', None
+    try:
+        have_version = version_getter(pkg_name)
+    except AttributeError:
+        return 'no-version', None
+    if checker(have_version) < checker(version):
+        return 'low-version', have_version
+    return 'satisfied', have_version
+
+## END - COPIED FROM NIBABEL
 
 from build_docs import cmdclass, INFO_VARS
 
@@ -41,26 +269,30 @@ def configuration(parent_package='',top_path=None):
     config.add_subpackage('nipype', 'nipype')
     return config
 
-################################################################################
-# For some commands, use setuptools
-
-if len(set(('develop', 'bdist_egg', 'bdist_rpm', 'bdist', 'bdist_dumb',
-            'bdist_wininst', 'install_egg_info', 'egg_info', 'easy_install',
-            )).intersection(sys.argv)) > 0:
-    from setup_egg import extra_setuptools_args
-
-# extra_setuptools_args can be defined from the line above, but it can
-# also be defined here because setup.py has been exec'ed from
-# setup_egg.py.
-if not 'extra_setuptools_args' in globals():
-    extra_setuptools_args = dict()
+# Prepare setuptools args
+if 'setuptools' in sys.modules:
+    extra_setuptools_args = dict(
+        tests_require=['nose'],
+        test_suite='nose.collector',
+        zip_safe=False,
+        extras_require = dict(
+            doc='Sphinx>=0.3',
+            test='nose>=0.10.1'),
+    )
+    pkg_chk = partial(package_check, setuptools_args = extra_setuptools_args)
+else:
+    extra_setuptools_args = {}
+    pkg_chk = package_check
 
 # Hard and soft dependency checking
-package_check('networkx', INFO_VARS['NETWORKX_MIN_VERSION'])
-package_check('nibabel', INFO_VARS['NIBABEL_MIN_VERSION'])
-package_check('numpy', INFO_VARS['NUMPY_MIN_VERSION'])
-package_check('scipy', INFO_VARS['SCIPY_MIN_VERSION'])
-package_check('traits', INFO_VARS['TRAITS_MIN_VERSION'])
+pkg_chk('networkx', INFO_VARS['NETWORKX_MIN_VERSION'])
+pkg_chk('nibabel', INFO_VARS['NIBABEL_MIN_VERSION'])
+pkg_chk('numpy', INFO_VARS['NUMPY_MIN_VERSION'])
+pkg_chk('scipy', INFO_VARS['SCIPY_MIN_VERSION'])
+pkg_chk('traits', INFO_VARS['TRAITS_MIN_VERSION'])
+pkg_chk('nose', INFO_VARS['NOSE_MIN_VERSION'])
+pkg_chk('dateutil', INFO_VARS['DATEUTIL_MIN_VERSION'],
+        pypi_pkg_name='python-dateutil')
 
 ################################################################################
 # Import the documentation building classes.
@@ -77,7 +309,6 @@ def configuration(parent_package='',top_path=None):
 
 def main(**extra_args):
     from numpy.distutils.core import setup
-
     setup(name=INFO_VARS['NAME'],
           maintainer=INFO_VARS['MAINTAINER'],
           maintainer_email=INFO_VARS['MAINTAINER_EMAIL'],
@@ -91,13 +322,10 @@ def main(**extra_args):
           author_email=INFO_VARS['AUTHOR_EMAIL'],
           platforms=INFO_VARS['PLATFORMS'],
           version=INFO_VARS['VERSION'],
-          requires=INFO_VARS['REQUIRES'],
-          configuration = configuration,
-          cmdclass = cmdclass,
-          scripts = glob('bin/*'),
+          configuration=configuration,
+          cmdclass=cmdclass,
+          scripts=glob('bin/*'),
           **extra_args)
 
-
-
 if __name__ == "__main__":
     main(**extra_setuptools_args)


Test patch: