Repository: statsmodels/statsmodels

Issue:
BUG/BLD 3.4 compatibility of cython c files
cython generated files are not compatilbe with python 3.4

see comments below

---

original description

**clarify that 2to3 is needed for Python 3 support**

From what I can tell, this project requires 2to3 to create a Python 3 build, but this is not stated in the README or the INSTALL file.  Please clarify this, preferably including an example coversion like:
`2to3 --output-dir=statsmodels3 -W -n statsmodels`
Thanks!



Discussion:
Sorry, it appears this is only a problem on Python 3.4.  `python3.3 setup.py build` compiles just fine.

@holocronweaver   Do you get an error message that 2to3 is not automatically run in python 3.4?

Sometimes there were small adjustments to make for 2to3 to run correctly with a new python version.

@josef-pkt If I run `python3.4 setup.py build` I get the error

```
statsmodels/tsa/kalmanf/kalman_loglike.c: In function ‘__Pyx_TraceSetupAndCall’:
statsmodels/tsa/kalmanf/kalman_loglike.c:7131:17: error: ‘PyFrameObject’ has no member named ‘f_tstate’
         (*frame)->f_tstate = tstate;
                 ^
```

That looks like the cython generated c files don't work for python 3.4

Hopefully this will be fixed by using a newer python. I think it has nothing to do with running 2to3, since that is shouldn't  change .pyx files. AFAIU

Did you run cython on the statsmodels files?  That is Did you install a statsmodels release distribution that includes the c file, or did you install from a git checkout?
In the latter case it would be useful if you can tell us your cython version. 

```
>>> import Cython
>>> Cython.__version__
'0.17.4'
```

I reopen and edit the description.
We will need to test python 3.4 compatibility for the next release.

Thanks for testing.

I am using the latest git checkout with 

```
>>> Cython.__version__
'0.19.2'
```

I confirmed that `Cython 0.20.1` does not work.  

Is Cython compatible with Python 3.4? E.g., can you build pandas?

Also, make sure you're not dealing with stale .c files. To be absolutely sure I usually run

```
git clean -xdf
```

Before building. You might need permissions to clean up the build/ directory, depending how you install. Alternatively, you can run

```
python setup.py clean
```

If you don't want to nuke everything not under version control.

Oh, I forgot to clean up, after cleanup the old generated files, now statsimodels can be built with Python 3.4 (rc1) and Cython (0.20.1)

Ok, thanks. I suspected that was the case here. I'm closing this. Feel free to re-open if there are further issues.

Hi

I have encountered the same error with Python 3.4, and I am unable to install statsmodels both via pip (from pypi) and from the git repo. Please reopen this issue.

Last few build lines:

```

/opt/local/bin/gcc-mp-4.8 -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names build/temp.macosx-10.9-x86_64-3.4/statsmodels/nonparametric/_smoothers_lowess.o -o build/lib.macosx-10.9-x86_64-3.4/statsmodels/nonparametric/_smoothers_lowess.so

skipping 'statsmodels/tsa/kalmanf/kalman_loglike.c' Cython extension (up-to-date)

building 'statsmodels.tsa.kalmanf.kalman_loglike' extension

creating build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa

creating build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa/kalmanf

/opt/local/bin/gcc-mp-4.8 -Wno-unused-result -Werror=declaration-after-statement -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include -I/Users/sboehler/prod34/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c statsmodels/tsa/kalmanf/kalman_loglike.c -o build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa/kalmanf/kalman_loglike.o

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarraytypes.h:1760:0,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

 #warning "Using deprecated NumPy API, disable it by " \

  ^

statsmodels/tsa/kalmanf/kalman_loglike.c: In function '__Pyx_TraceSetupAndCall':

statsmodels/tsa/kalmanf/kalman_loglike.c:7021:17: error: 'PyFrameObject' has no member named 'f_tstate'

         (*frame)->f_tstate = PyThreadState_GET();

                 ^

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:

statsmodels/tsa/kalmanf/kalman_loglike.c: At top level:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:1629:1: warning: '_import_array' defined but not used [-Wunused-function]

 _import_array(void)

 ^

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ufuncobject.h:327:0,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:258:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/__ufunc_api.h:241:1: warning: '_import_umath' defined but not used [-Wunused-function]

 _import_umath(void)

 ^

error: command '/opt/local/bin/gcc-mp-4.8' failed with exit status 1

----------------------------------------
Cleaning up...
Command /Users/sboehler/prod34/bin/python3.4 -c "import setuptools, tokenize;__file__='/Users/sboehler/prod34/build/statsmodels/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/z5/q6c60q2n00q75kvbysg2bfmr0000gn/T/pip-fneuzmt_-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/sboehler/prod34/include/site/python3.4 failed with error code 1 in /Users/sboehler/prod34/build/statsmodels
Storing debug log for failure in /Users/sboehler/.pip/pip.log
```

I am using Python 3.4 in a pyvenv on OS X with dependencies installed from macports. I have also cloned the git repo and clean everything according to the instructions above.

```
~/develop/python/statsmodels% ${CC} --version
gcc-mp-4.8 (MacPorts gcc48 4.8.2_0) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
~/develop/python/statsmodels% python --version
Python 3.4.0
~/develop/python/statsmodels% cython --version
Cython version 0.20.1
```

Let me know if you need any more info. Many thanks!

The only thing that can cause this is stale files or files generated by Cython for a different version of Python. Please make sure your Cython is also using Python 3.4.

```
which cython
```

What is the first line there? Is this the Python you expect it to be using?

Yes, it says
/Users/sboehler/prod34/bin/cython
which is version 0.20.1 (edit: and whereas prod34 is my pyvenv).

How can I check for stale files? I thought that if I "pip install statsmodels" it will download, build and install from a new directory, or am I missing something.

What is the first line of `/Users/sboehler/prod34/bin/cython` does it point to a different version of Python than the one you expect?

Ok, I see how this is happening now. You need to do 1 of two things. 

1) Install from a fresh github checkout. Assuming you're using the write Cython+Python combination. This should just work.

2) If you download a source distribution from PyPi and it has the .c files already checked in for a certain version of Python. We include these in the release so people won't need Cython to install, but this isn't Python version independent, so we'll have to stop doing this or figure out Python 2.x and 3.x source tarballs.

So...

If you clone the source from github, you shouldn't run into this problem. 

If you're using our tarballs of a source release (from pypi), then we'll have .c files in there that won't work on your system. You'll need to delete these. You can do that from within the source directory with a

```
python setup.py clean
```

Thanks, I just retried using a fresh git checkout and it worked fine this time! So 1) was the way to go.

If I correctly understand 2), it is thus currently not possible to install statsmodels via pip? I'd appreciate  if you could shed some light on this or if you see a way out while using pip, for the benefit of other users.

My personal trouble is solved, however, thank you very much for the quick help!

On Python 3.4, no, I don't think so. @josef-pkt What do you think about uploading some source tarballs that are 0.5.0 but with the .c files removed on pypi? We'd now require Cython for all builds.

I changed the milestone for this to 0.5.1, so in the worst case, pip install will work for Python 3.4 on the next micro release sometime soon.

I suppose there may be a way to run the `CleanCommand` in setup.py if the version is 3.4 and raise a Warning that you'll need Cython. Maybe that's the easiest patch.

question first before removing c files which might make it more difficult for many users to install:

Does the latest cython generate c code that is compatible with python 2.6?
I think up to now, it was always possible to use the latest required cython and still generate code that is compatible with older versions of everything else.

Then, we just need new source distributions, tar and zip, with the updated c files.

(If we had thought about it, we could have tried a better timing with respect to the python 3.4 release.)

Very fast solution would be to upload sdist files for python 3.4, if pip checks python version names in sdist. (I doubt it.)

Why is it more difficult without the C files? You still need a compiler, so presumably you'll have Cython. What I proposed only makes Cython necessary on 3.4 anyway, which is very bleeding edge as far as the Python world goes.

I don't think it has anything to do with the Cython version. I just think the code Cython outputs for Python 2.6-3.3 is different than that for Python 3.4. I'll ping their list though, maybe no one else has run into this yet.

my guess is adding the sdist with new c files and version 0.5.0.1 should work.
But it might make easy_install ignore the Windows binaries because they have a lower version number (I don't know). We would have to reupload the same old windows binaries with a higher version number also, 0.5.0.1.

> What I proposed only makes Cython necessary on 3.4 anyway.

Not if sdist are not interpreted as python specific. pip for python 3.3 and python 2.6 might pick the same tar balls. I don't know.
In this case: Users won't have cython installed and we will get for several weeks comments about why pip install doesn't work anymore.

On Mon, Mar 17, 2014 at 8:33 PM, Josef Perktold notifications@github.comwrote:

> my guess is adding the sdist with new c files and version 0.5.0.1 should
> work.
> But it might make easy_install ignore the Windows binaries because they
> have a lower version number (I don't know). We would have to reupload the
> same old windows binaries with a higher version number also, 0.5.0.1.
> 
> Let's just push to get this solved in 0.5.1 and get it out the door. I'm
> traveling next weekend and may have a minute to do this. We should be in
> pretty good shape already.
> 
>  What I proposed only makes Cython necessary on 3.4 anyway.
> 
> Not if sdist are not interpreted as python specific. pip for python 3.3
> and python 2.6 might pick the same tar balls. I don't know.
> In this case: Users won't have cython installed and we will get for
> several weeks comments about why pip install doesn't work anymore.

What I'm saying is that the tarballs will be the same. It doesn't have
anything to do with pip. We'd call CleanCommand in the setup.py and make
sure the .c files are nuked early on in the build process on 3.4 and that
they get rebuilt (would have to check that there's not a flag that controls
this). We should have a nice 'Cython must be installed' error message, if
it's not deteceted. Right now we're already getting several weeks comments
about why pip install doesn't work anymore. This should be a change that
only affects 3.4 and makes it work.

Ok, I misunderstood the last part. It should work ok, if we change the setup.py.

But we should still check first if the cython developers managed to maintain compatibility across python versions, which would make any special casing of python 3.4 obsolete.

Yeah, I pinged the list and gave the link here. I'll wait to hear back.

Ok, heard back on cython-users. tl;dr we don't need to do anything. We just need to build the 0.5.1 release tarball with Cython 0.20.1 (or higher). Python 3.4 has been a moving target and they're still adapting to late changes.

```
> for our source distributions. So pip install ... fails with this kind
> of error on Python 3.4.
>
> statsmodels/tsa/kalmanf/kalman_loglike.c:7131:17: error:
> 'PyFrameObject' has no member named 'f_tstate'
>          (*frame)->f_tstate = tstate;

I remember that, it was a fairly late change in the Py3.4 release cycle,
which was a heavily moving target this time with a surprisingly high number
of this kind of late changes. It's only been officially released yesterday,
so I guess people will just have to live with third party tools needing a
moment to adapt (or bugs and regressions getting fixed in Py3.4 etc.).


> https://github.com/statsmodels/statsmodels/issues/1311
>
> Is this a known issue? Is it something we're doing wrong? Is there a
> way to compile .C files that work Python 2.6 through 3.4?

Sure. Cython 0.20 has adapted to this change, with further Py3.4 fixes in
0.20.1. There will soon be a 0.20.2 release, BTW, although that shouldn't
change anything w.r.t. Py3.4. So rebuilding the sources with 0.20.1 should
fix it for you.
```

very good, that's what I was hoping for and expecting.
I don't remember any case where the generated c code was not compatible with all supported python versions. The cython developers have been very good in this respect.

Leave this open as a reminder to bump the cython version in the docs (for 3.4). I'm juggling branches right now.


Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -75,8 +75,8 @@
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
-from statsmodels.version import version, full_version
-release = version
+from statsmodels.version import short_version, full_version
+release = short_version
 # The full version, including dev tag.
 version = full_version
 
diff --git a/statsmodels/graphics/gofplots.py b/statsmodels/graphics/gofplots.py
--- a/statsmodels/graphics/gofplots.py
+++ b/statsmodels/graphics/gofplots.py
@@ -275,6 +275,7 @@ def qqplot(self, xlabel=None, ylabel=None, line=None, other=None,
             other values are used depending on the status of the kwarg `other`.
         line : str {'45', 's', 'r', q'} or None, optional
             Options for the reference line to which the data is compared:
+
             - '45' - 45-degree line
             - 's' - standardized line, the expected order statistics are scaled
               by the standard deviation of the given sample and have the mean
@@ -287,8 +288,8 @@ def qqplot(self, xlabel=None, ylabel=None, line=None, other=None,
             If provided, the sample quantiles of this `ProbPlot` instance are
             plotted against the sample quantiles of the `other` `ProbPlot`
             instance. If an array-like object is provided, it will be turned
-            into a `ProbPlot` instance using default parameters. If not provided
-            (default), the theoretical quantiles are used.
+            into a `ProbPlot` instance using default parameters. If not
+            provided (default), the theoretical quantiles are used.
         ax : Matplotlib AxesSubplot instance, optional
             If given, this subplot is used to plot in instead of a new figure
             being created.
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -983,23 +983,23 @@ def arma_order_select_ic(y, max_ar=4, max_ma=2, ic='bic', trend='c',
     max_ar : int
         Maximum number of AR lags to use. Default 4.
     max_ma : int
-        Maximum number of MA lags to use. DEfault 2.
+        Maximum number of MA lags to use. Default 2.
     ic : str, list
         Information criteria to report. Either a single string or a list
         of different criteria is possible.
     trend : str
         The trend to use when fitting the ARMA models.
     model_kw : dict
-        Keyword arguments to be passed to the `ARMA` model
+        Keyword arguments to be passed to the ``ARMA`` model
     fit_kw : dict
-        Keyword arguments to be passed to `ARMA.fit`.
+        Keyword arguments to be passed to ``ARMA.fit``.
 
     Returns
     -------
     obj : Results object
         Each ic is an attribute with a DataFrame for the results. The AR order
         used is the row index. The ma order used is the column index. The
-        minimum orders are available as `ic`_min_order.
+        minimum orders are available as ``ic_min_order``.
 
     Examples
     --------


Test patch: