Repository: statsmodels/statsmodels

Issue:
Running cythonize failed
Hello

I am trying to upgrade my statsmodel package from 0.6.0 to 0.8.0 via github.

This is what I did cloned **https://github.com/statsmodels/statsmodels.git** and cd in to that folder

did python setup.py install and am getting this error


> 'git' is not recognized as an internal or external command,
> operable program or batch file.
> Cythonizing sources
> statsmodels\nonparametric\linbin.pyx has not changed
> statsmodels\nonparametric\_smoothers_lowess.pyx has not changed
> statsmodels\tsa\kalmanf\kalman_loglike.pyx has not changed
> statsmodels\tsa\regime_switching\_hamilton_filter.pyx.in has not changed
> statsmodels\tsa\regime_switching\_kim_smoother.pyx.in has not changed
> Processing statsmodels\tsa\statespace\_kalman_filter.pyx.in
> Traceback (most recent call last):
>   File "E:\statsmodels\tools\cythonize.py", line 215, in <module>
>     main()
>   File "E:\statsmodels\tools\cythonize.py", line 211, in main
>     find_process_files(root_dir)
>   File "E:\statsmodels\tools\cythonize.py", line 203, in find_process_files
>     process(cur_dir, fromfile, tofile, function, hash_db)
>   File "E:\statsmodels\tools\cythonize.py", line 174, in process
>     processor_function(fromfile, tofile)
>   File "E:\statsmodels\tools\cythonize.py", line 92, in process_tempita_pyx
>     raise Exception('Building Statsmodels requires Tempita: '
> Exception: Building Statsmodels requires Tempita: pip install --user Tempita
> Traceback (most recent call last):
>   File "setup.py", line 545, in <module>
>     generate_cython()
>   File "setup.py", line 90, in generate_cython
>     raise RuntimeError("Running cythonize failed!")
> RuntimeError: Running cythonize failed!

I have all the necessary dependencies and my os is windows 8.


Discussion:
maybe your cython is too old.
I have `Tempita` as a subpackage of Cython in my sitepackages.

To build statsmodels 0.8, I had to upgrade cython, but I had a different error message


I have installed the lastest cython version **0.25.2** and have now got **Tempita**. But still when running python setup.py install , i get the following

> 'git' is not recognized as an internal or external command,
> operable program or batch file.
> Cythonizing sources
> statsmodels\nonparametric\linbin.pyx has not changed
> statsmodels\nonparametric\_smoothers_lowess.pyx has not changed
> statsmodels\tsa\kalmanf\kalman_loglike.pyx has not changed
> statsmodels\tsa\regime_switching\_hamilton_filter.pyx.in has not changed
> statsmodels\tsa\regime_switching\_kim_smoother.pyx.in has not changed
> Processing statsmodels\tsa\statespace\_kalman_filter.pyx.in
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> cimport cython
> 
> np.import_array()
> 
> from statsmodels.src.math cimport *
> cimport scipy.linalg.cython_blas as blas
>        ^
> ------------------------------------------------------------
> 
> _kalman_filter.pyx:73:8: 'scipy\linalg\cython_blas.pxd' not found
> Traceback (most recent call last):
>   File "E:\statsmodels\tools\cythonize.py", line 215, in <module>
>     main()
>   File "E:\statsmodels\tools\cythonize.py", line 211, in main
>     find_process_files(root_dir)
>   File "E:\statsmodels\tools\cythonize.py", line 203, in find_process_files
>     process(cur_dir, fromfile, tofile, function, hash_db)
>   File "E:\statsmodels\tools\cythonize.py", line 174, in process
>     processor_function(fromfile, tofile)
>   File "E:\statsmodels\tools\cythonize.py", line 101, in process_tempita_pyx
>     process_pyx(pyxfile, tofile)
>   File "E:\statsmodels\tools\cythonize.py", line 72, in process_pyx
>     raise Exception('Cython failed')
> Exception: Cython failed
> Traceback (most recent call last):
>   File "setup.py", line 545, in <module>
>     generate_cython()
>   File "setup.py", line 90, in generate_cython
>     raise RuntimeError("Running cythonize failed!")
> RuntimeError: Running cythonize failed!
`sm.show_versions()`

which scipy version?
statespace in master uses compilation conditional on the scipy version, and there could be something wrong in the version detection, or your scipy version is missing files.

scipy version is **0.15.1**
@ChadFulton   This sounds like cythonize is looking for the file that doesn't exist in older scipy.
Travis doesn't have a problem with scipy 0.14 and 0.15, so there is still something strange.
Thanks for the report!

I think there must be the remains of a failed installation or something else strange going on here. For example, in the first message Tempita was apparently not available, yet it looks like "hamilton_filter.pyx.in" was already Cythonized, which would have required Tempita.

Is that the full output of `python setup.py install` or is there some more at the beginning that you could post?

You might check the contents of a file called `cythonize_exclusions.dat` that should be in the main statsmodels folder. Does it exist, if so, what does it contain?
Yes it is the full output of **python setup.py install**

contents of file **cythonize_exclusions.dat** are 

> statsmodels/tsa/statespace/_smoothers/_univariate.pyx.in
> statsmodels/tsa/statespace/_simulation_smoother.pyx.in
> statsmodels/tsa/statespace/_smoothers/_alternative.pyx.in
> statsmodels/tsa/statespace/_tools.pyx.in
> statsmodels/tsa/statespace/_kalman_smoother.pyx.in
> statsmodels/tsa/statespace/_kalman_filter.pyx.in
> statsmodels/tsa/statespace/_filters/_inversions.pyx.in
> statsmodels/tsa/statespace/_representation.pyx.in
> statsmodels/tsa/statespace/_smoothers/_classical.pyx.in
> statsmodels/tsa/statespace/_filters/_conventional.pyx.in
> statsmodels/tsa/statespace/_filters/_univariate.pyx.in
> statsmodels/tsa/statespace/_smoothers/_conventional.pyx.in

It looks like it's appropriately building the list of files to ignore, but then `cythonize.py` is trying to process them anyway.

Maybe it has something to do with the underscores in the paths in your setup.py output? `cythonize.py` compares the file paths to the paths in cythonize_exclusions.dat, and ignores files that are a match.

e.g. when I run setup.py, I get something like:

> Cythonizing sources
statsmodels/nonparametric/_smoothers_lowess.pyx has not changed
statsmodels/nonparametric/linbin.pyx has not changed
statsmodels/tsa/kalmanf/kalman_loglike.pyx has not changed
statsmodels/tsa/regime_switching/_hamilton_filter.pyx.in has not changed
statsmodels/tsa/regime_switching/_kim_smoother.pyx.in has not changed
statsmodels/tsa/statespace/_kalman_filter.pyx.in has not changed
statsmodels/tsa/statespace/_kalman_smoother.pyx.in has not changed

Can you confirm that you get the output you pasted above and not what I have here? e.g. I'm thinking in particular of the last two lines.
I am not getting the output you pasted. What I have pasted in my  comment is the one that I get when running **python setup.py install**
**_kalman_filter.pyx:73:8: 'scipy\linalg\cython_blas.pxd'** not found is this where the setup is breaking ?
Actually, I meant, can you confirm the output from `python setup.py install`. There are underscores in some of the path names. For example, you put:

> 'git' is not recognized as an internal or external command,
operable program or batch file.
Cythonizing sources
statsmodels\nonparametric\linbin.pyx has not changed
statsmodels\nonparametric_smoothers_lowess.pyx has not changed
statsmodels\tsa\kalmanf\kalman_loglike.pyx has not changed
statsmodels\tsa\regime_switching_hamilton_filter.pyx.in has not changed
statsmodels\tsa\regime_switching_kim_smoother.pyx.in has not changed
Processing statsmodels\tsa\statespace_kalman_filter.pyx.in

The last line ought to be: `Processing statsmodels\tsa\statespace\kalman_filter.pyx.in`

(i.e. there should be a backslash after statespace, not an underscore).
![screen_shot](https://cloud.githubusercontent.com/assets/1428796/23332064/35ccdfb0-fb98-11e6-95b0-d44f0c9f7584.PNG)

There was some typo when copy pasting, have attached the screenshot. Yes there is a backslash
That's very helpful, and I think I have the source of the problem now. It looks like this is my fault for sloppy path handling, i.e. Windows wants backslashes and I used forward slashes in paths instead of `os.path.sep`.

The result is that the `cythonize_exclusions.dat` file has paths with forward slashes, and when we walk over the Cython files in `cythonize.py`, the paths come up with backward slashes, which do not match the paths in the exclusions, and so Cython tries to compile them.

(I guess `distutils.Extension` must do some internal conversion to fix the separators work at all on Windows)

> Travis doesn't have a problem with scipy 0.14 and 0.15, so there is still something strange.

I guess Appveyor only runs scipy >= 0.16, otherwise we'd have been getting this error there.

Thanks very much for working with me on getting this sorted out @AnnamalaiNagappan, I'll have a fix for this today.
Thank you very much for the help @ChadFulton. 
@bashtage Can we add a python with one old scipy to test it while we still support older scipy?
I'm pretty sure I used the oldest SciPy that was available for each version of NumPy in conda, so this would be non-trivial.  The current test goes back to 0.14, which is pretty old (May 2014, so nearly 3 years).
@AnnamalaiNagappan Is there a reason why you use a pretty old scipy?

We need to fix it given that we currently support it, but it is still an open question how soon we drop support for scipy < 0.16.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -450,7 +450,8 @@ def run(self):
 except ImportError:
     for name, data in statespace_ext_data.items():
         path = '.'.join([data["name"].split('.')[0], 'pyx.in'])
-        append_cython_exclusion(path, CYTHON_EXCLUSION_FILE)
+        append_cython_exclusion(path.replace('/', os.path.sep),
+                                CYTHON_EXCLUSION_FILE)
 
 extensions = []
 for name, data in ext_data.items():


Test patch: