Repository: statsmodels/statsmodels

Issue:
Packages cannot have both pandas and statsmodels in install_requires
Hi,

I am testing a fresh install of seaborn, which includes both pandas and statsmodels in its `install_requires`.

I ran into some trouble trying to install seaborn into a virtual environment with only numpy, scipy and matplotlib. The problem is that pip goes to install seaborn's dependencies, and even though it installs pandas and then statsmodels, the statsmodels installation crashes when performing some kind of check for Pandas, killing the whole operation.

Sorry, I know Python packaging is a huge PITA, but this is pretty annoying.

Here are the four commands to reproduce. The output, on my machine, is copied below.

```
conda create -n testenv pip
source activate testenv
conda install numpy scipy matplotlib
pip install seaborn
```

```
$ conda create -n testenv pip

Package plan for creating environment at /Users/mwaskom/anaconda/envs/testenv:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-1.4.1                  |           py27_1         644 KB
    setuptools-1.4             |           py27_0         490 KB

The following packages will be linked:

    package                    |            build
    ---------------------------|-----------------
    pip-1.4.1                  |           py27_1   hard-link
    python-2.7.6               |                0   hard-link
    readline-6.2               |                1   hard-link
    setuptools-1.4             |           py27_0   hard-link
    sqlite-3.7.13              |                1   hard-link
    tk-8.5.13                  |                1   hard-link
    zlib-1.2.7                 |                1   hard-link

Proceed ([y]/n)? y

Fetching packages ...
pip-1.4.1-py27_1.tar.bz2 100% |######################| Time: 0:00:09  68.79 kB/s
setuptools-1.4-py27_0.tar.bz2 100% |#################| Time: 0:00:04 120.80 kB/s
Extracting packages ...
[      COMPLETE      ] |##################################################| 100%
Linking packages ...
[      COMPLETE      ] |##################################################| 100%
#
# To activate this environment, use:
# $ source activate testenv
#
# To deactivate this environment, use:
# $ source deactivate
#


$ source activate testenv
prepending /Users/mwaskom/anaconda/envs/testenv/bin to PATH


(testenv)$ conda install numpy scipy matplotlib

Package plan for installation in environment /Users/mwaskom/anaconda/envs/testenv:

The following packages will be linked:

    package                    |            build
    ---------------------------|-----------------
    dateutil-2.1               |           py27_2   hard-link
    freetype-2.4.10            |                1   hard-link
    libpng-1.5.13              |                1   hard-link
    matplotlib-1.3.1           |       np17py27_0   hard-link
    numpy-1.7.1                |           py27_2   hard-link
    pyparsing-1.5.6            |           py27_0   hard-link
    pytz-2013b                 |           py27_0   hard-link
    scipy-0.13.2               |       np17py27_1   hard-link
    six-1.4.1                  |           py27_0   hard-link

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ] |##################################################| 100%


(testenv)$ pip install seaborn
Downloading/unpacking seaborn
  Downloading seaborn-0.2.0.tar.gz
  Running setup.py egg_info for package seaborn

Downloading/unpacking husl (from seaborn)
  Downloading husl-2.1.0.tar.gz
  Running setup.py egg_info for package husl

Downloading/unpacking moss>0.1 (from seaborn)
  Downloading moss-0.2.0.tar.gz
  Running setup.py egg_info for package moss
    /Users/mwaskom/anaconda/envs/testenv/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'URL'
      warnings.warn(msg)

Downloading/unpacking patsy (from seaborn)
  Downloading patsy-0.2.1.tar.gz (316kB): 316kB downloaded
  Running setup.py egg_info for package patsy

    no previously-included directories found matching 'doc/_build'
Downloading/unpacking pandas (from seaborn)
  Downloading pandas-0.12.0.tar.gz (3.2MB): 3.2MB downloaded
  Running setup.py egg_info for package pandas

    warning: no files found matching 'TODO.rst'
    warning: no files found matching 'setupegg.py'
    no previously-included directories found matching 'doc/build'
    warning: no previously-included files matching '*.so' found anywhere in distribution
    warning: no previously-included files matching '*.pyd' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.git*' found anywhere in distribution
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    warning: no previously-included files matching '*.png' found anywhere in distribution
Downloading/unpacking statsmodels (from seaborn)
  Downloading statsmodels-0.5.0.tar.gz (5.5MB): 5.5MB downloaded
  Running setup.py egg_info for package statsmodels
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/private/var/folders/rr/svlz5yhs0tsdqnsp8g4qg65m0000gn/T/pip_build_mwaskom/statsmodels/setup.py", line 463, in <module>
        check_dependency_versions(min_versions)
      File "/private/var/folders/rr/svlz5yhs0tsdqnsp8g4qg65m0000gn/T/pip_build_mwaskom/statsmodels/setup.py", line 118, in check_dependency_versions
        raise ImportError("statsmodels requires pandas")
    ImportError: statsmodels requires pandas
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/private/var/folders/rr/svlz5yhs0tsdqnsp8g4qg65m0000gn/T/pip_build_mwaskom/statsmodels/setup.py", line 463, in <module>

    check_dependency_versions(min_versions)

  File "/private/var/folders/rr/svlz5yhs0tsdqnsp8g4qg65m0000gn/T/pip_build_mwaskom/statsmodels/setup.py", line 118, in check_dependency_versions

    raise ImportError("statsmodels requires pandas")

ImportError: statsmodels requires pandas

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/rr/svlz5yhs0tsdqnsp8g4qg65m0000gn/T/pip_build_mwaskom/statsmodels
Storing complete log in /Users/mwaskom/.pip/pip.log
```



Discussion:
Essentially you need to install pandas **before** statsmodels, and before statsmodels is run.

You could do `pip install pandas, patsy` before `pip install seaborn` or before `pip install statsmodels`

statsmodels doesn't have a proper requirements because I **hate** the default behavior of pip to reinstall/update all dependencies, which made several times a big mess with my **carefully** installed numpy and scipy, which is not so easy in a virtualenv on Windows.
Maybe it will change with newer version of pip and the availability of wheels.

You don't have to add requirements to statsmodels setup.py to make this work, something like https://github.com/scipy/scipy/blob/master/setup.py#L205 should do it. The essential bit is to not try to import anything for the `egg_info` command.

And fully agree with the upgrade sentiment of course:(

Thanks Ralf, we can try to do that. If pip only requires the `egg_info` or similar, then we could hide the dependency version checks for this command.

However, I suspect that pip might still try to install statsmodels out of sequence, because it doesn't know the dependencies and might not have the information to install pandas and patsy before statsmodels.

We could switch to warnings for dependencies instead of errors, but those get lost in the usual install noise.

@mwaskom  I think it will be useful for statsmodels to use seaborn as an optional dependency. @phobson said he will look into it.
This will not create problems on install because we don't check for optional dependencies and we wouldn't get into circular dependency problems. 
However, I expect that eventually we will need to watch out for circular imports across packages.

Yeah I think the problem is just that the installation procedure tries to import pandas (and patsy, as I later leaned) and moreover that each install isn't "finished" from pip's perspective until the whole command completes, so even though pip was installing pandas first, statsmodels couldn't see it.

It could be you have two problems here, but for a requirements file with numpy + scipy we had the exact same issue, and the change to scipy that I linked to fixed it.

My team is having this same issue with using statsmodels in an application where our dependency tree is managed by pip. We'd greatly appreciate it if you could take another look at this issue and see if you can allow pip's dependency management to do its thing. Otherwise we won't be able to use statsmodels without an ugly hack or two.

Patches that try and implement the above mentioned fix would be welcome.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -463,8 +463,11 @@ def get_data_files():
         # 3.3 needs numpy 1.7+
         min_versions.update({"numpy" : "1.7.0b2"})
 
-    check_dependency_versions(min_versions)
-    write_version_py()
+    if not (len(sys.argv) >= 2 and ('--help' in sys.argv[1:] or
+            sys.argv[1] in ('--help-commands', 'egg_info', '--version',
+                            'clean'))):
+        check_dependency_versions(min_versions)
+        write_version_py()
 
     # this adds *.csv and *.dta files in datasets folders
     # and *.csv and *.txt files in test/results folders


Test patch: