sphinxext py3 compatibility
State: closed

Temporary sphinxext py3 compatibility fix as discussed in #5479 before #5221 is resolved. So far test_docscrape.py passes on both Python 2.7 and 3.3 (which should be the typical developer interpreters).


Labels:

Comments:
I can't test this at the moment, but if it solves the python 3 issues with the docs for the moment, good for me!

Are these tests skipped by travis?

we don't test doc builds with Travis. It's difficult to do and it fails for silly reasons (like latex not being installed)

But we could test this specific file (it just requires sphinx) ?

Is this file skipped? I don't see it anywhere in the [Travis log](https://s3.amazonaws.com/archive.travis-ci.org/jobs/14075865/log.txt) (which lists the skipped modules). If it is skipped I could try adding it, but I don't see how - the script.sh file that gets called by travis looks pretty inclusive:

```
 nosetests --exe -w /tmp -A "$NOSE_ARGS" pandas --with-xunit --xunit-file=/tmp/nosetests.xml
```

With the  `$NOSE_ARGS` getting just the 'not slow' parameter (and docscrape tests don't have any slow decorators). I don't know what does this xunit-file do.

Anyway, this commit solves some of the issues (enough to pass the test_docscrape.py on the latest versions of Python 2 and 3 on my machine), but not all, so @phaebz (a colleague of mine, I added him as a contributor to the branch) and I will probably contribute more fixes to this branch.

I guess it must be skipped as other branches here pass on python 3!

It could be the pandas in the middle (does this only run tests within pandas directory? and not any in doc/sphinxext) ...Obviously passing that test won't confirm that the docs build on python 3, but this could be a start/prevent regressions :)

That you guys are working on this is great!

it's only running the pandas directory. that test is under docs. But it needs to only run if sphinx is installed, no?

So the question is should we run this (currently) small test to imply the docs are buildable.

Perhaps this is out of scope of what travis should be doing usually...

I had a look at the `pandas.compat` as indicated by @cpcloud over here https://github.com/pydata/pandas/issues/5479#issuecomment-28195865. I gathered that this compatibility layer is for exposing method redefinitions for e.g. `.iteritems()` vs. `.items()`, unicode handling (?) via `str_to_bytes` and the like.

Now I want to just use the import logic, namely the try/except part of `pandas.compat.__init__.py`, in other files (namely for `cStringIO` in `ipython_directive.py`) to expose the library naming changes to those files. Is this a feasible thing to do? If so, any ideas on how to achieve that?

Not clear what the issue is, if this is in pandas repo you can assume pandas is installed. Therefore, you can use compat.

Nevermind, I think I figured it out - see upcoming commit(s?). My question stemmed from the fact that I am new to compat issues between python 2 and python 3 in terms of code base organisation as big as pandas.

... or not figured it out. Concrete example: I want to make `doc/sphinxext/ipython_directive.py` py3 compatible. One problem in there is the `cStringIO.StringIO()` instantiation. I naively tried to add a `from pandas.compat import cStringIO` in hope that it would take care to get a sane cStringIO module usable by py2 and py3. Instead, I get the error:

```
AttributeError: type object '_io.StringIO' has no attribute 'StringIO'
```

which looks like a "private" by the underscore convention (?).

It's not the module, you actually get the object to instantiate, eg:

``` python
from pandas.compat import cStringIO buffer = cStringIO()
```

`cStringIO` is later used in this file as `cStringIO.StringIO`. I will experiment a bit with replacing that with just StringIO. Anyways, I will work on the commits, so we can talk about actual code.

I tried to use as much `pandas.compat` as possible, please review. The sphinxext seems ok. Sphinx starts to build under Python 2 and produces the following error

```
Exception occurred while building, starting debugger:                                        
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/cmdline.py", line 189, in main
    app.build(force_all, filenames)
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/application.py", line 204, in build
    self.builder.build_update()
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 196, in build_update
    'out of date' % len(to_build))
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 216, in build
    purple, length):
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 120, in status_iterator
    for item in iterable:
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/environment.py", line 613, in update_generator
    self.read_doc(docname, app=app)
  File "/Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/environment.py", line 764, in read_doc
    raise SphinxError(str(err))
SphinxError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
> /Users/user/.virtualenvs/pandas-dev-py2/lib/python2.7/site-packages/sphinx/environment.py(764)read_doc()
-> raise SphinxError(str(err))
(Pdb)
```

Under Python 3 sphinx runs through and says docs built successfully, but I get a ton of autodoc/docstring errors related to StringIO, xrange and pandas submodule imports

```
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/pandas-dev/lib/python3.3/site-packages/sphinx/ext/autodoc.py", line 321, in import_object
    __import__(self.modname)
ImportError: No module named 'pandas.Series'
```

Are you working on Windows? In that case, the first error (on python 2 causing the debugger to start) could possibly attributed to io.rst (there is a unicode example that does not build on windows/python 2, see #5142).

I am working on OS X 10.8.5. I had a look around in the debugger and it turns out that indeed the `docname` in question is `'io'`.

``` python
>>> import locale
>>> print(locale.getlocale())
('en_US', 'UTF-8')
>>> print(locale.getdefaultlocale())
('en_US', 'UTF-8')
```

@phaebz Something else, I tried to build the docs with the IPython sphinxextensions, and it worked with some warnings (on Python 2, but they should be python 3 compatible). This could also be a way to, at the moment, build the docs on python 3. 
The only two lines I changed is in conf.py:

Replace 

```
              'ipython_directive',
              'ipython_console_highlighting',
```

with

```
              'IPython.sphinxext.ipython_directive',
              'IPython.sphinxext.ipython_console_highlighting',
```

You need IPython 1.0 (or larger) for this. 
This worked for me to build the docs, you only get some errors about the `okexcept` parameter which is unknown (this parameter for the ipython directive is not in upstream), and some errors in the "Enhancing performance" section of the docs with the cython code (adaptations to run this is also not upstreamed). But you can build the docs (only on those affected pages the html pages will not be perfect).

Thanks for letting me know, this serves ok for now. I will use this locally from now on when working on docs. As this PR was intended to be a temporary fix before #5221 made it through / while it is discussed I will check back with @kermit666 how/if we will proceed.

@kermit666 how's this coming along?

@jreback currently lost in some "maximum recursion depth" errors I didn't expect to encounter in a Python 2 to 3 conversion:

```
RuntimeError: maximum recursion depth exceeded in comparison
> /Users/kermit/.virtualenvs/pandas3/lib/python3.3/site-packages/jinja2/utils.py(389)__getitem__()
-> if self._queue[-1] != key:
```

I'll try to work on it some more to figure it out.

That is the same error as @cancan101 got, and I also saw it once (I was thinking it maybe something to do with sphinx 1.2 vs 1.1.3, but I still have to test that)

@kermit666 See #5899, do you use Sphinx 1.2? And can you try with 1.1.3?

Thanks @jorisvandenbossche , that seems to be the root. I tried with Sphinx 1.1.3, but that one has some other errors (possibly py3-related) and now I'm working on the development version of Sphinx (1.3a).

I am also getting weird errors with pandas itself.

```
>>> import pandas
dlopen(./pandas/hashtable.so, 2): Symbol not found: _PyCObject_Type
```

I even tried rebasing to the latest master, but that didn't help. Is master unstable at the moment (in which case - should I be working off of the latest tag instead?) or am I maybe missing something locally?

Normally master should be stable. Did you build pandas again after rebasing? (eg `python setup.py build_ext --inplace`)

And with the development version of Sphinx (1.3a), is the 'maximum recursion error' solved?

OK, you are right @jorisvandenbossche . I thought that `pip install -e .` would trigger the extensions rebuild (because it builds them initially), but it appears it doesn't. After calling `build_ext` explicitly I got `pandas` to import again.

Yes, the dev version of Sphinx resolves the 'maximum recursion error'.

As far as the documentation itself is concerned - on the current branch I am able to generate it, but I get a ton of warnings and import errors along the way (import errors for missing scipy, xlrd,..., missing file 'foo.xlsx', toctree warnings). This whole build process seems very complex... I expected something more in the line of mark-up -> html :)

@kermit666 The reason building the docs seems complex (lot of errors and warnings etc) is because all code examples in the docs are run and the output saved during the docs build (using the ipython directive, so it's not simply rst -> html, the code examples in rst are first generated). This also means that, if you don't have all optional dependencies for pandas (hdf5-related, html-related, excel-related, etc) some of the code examples will generate errors. 
This is not really a problem for working locally on the docs, as the docs will still just build (only for the topics you don't have the dependencies, the code examples will show the error instead of the useful output). Also, when you build the docs a second time, only those pages you changed are rebuild (which should also decrease the amount of errors/warnings you see). 
With a clean build, the only warning you should see is "class.rst:: WARNING: document isn't included in any toctree".

OK, thanks. I'll play with it tonight some more with all the [optional dependencies](https://github.com/kermit666/pandas#optional-dependencies) then.

@kermit666 , #5925 should be merged soon but I've not tested it with python3.
Can you test it out and open a new PR with required changes if any? I'll cherry-pick.

Closing.

@y-p ok, great. I'll give the new pull request a spin on Python 3.