stevengj/PyCall.jl — pull request #70 by jakebolewski (2014-03-23 23:26:48+00:00)
Title: WIP: Python buffer (PEP 3118) implementation

This is very much a work in progress branch to explore replacing PyCall's numpy dependency with Python Buffers / memory views (https://github.com/stevengj/PyCall.jl/issues/38, and https://github.com/JuliaLang/IJulia.jl/issues/49). Link to PEP proposal: http://legacy.python.org/dev/peps/pep-3118/

Note: to get something working I've ignored for the time being important things like proper reference counting, and python initialization. I have also removed including the numpy code to make testing easier.

I want to get some feedback and to bring up some questions / problems I can foresee in implementing the rest of this. This pull request currently only works for Python v. > 3.2 as the layout of the Buffer structure changed in 3.3. Python initialization is defered until runtime, so it is best to defer type creation until runtime?. Now with faster startup times, is there a reason not to initialize libpython when first loading the module? Another wrinkle is that Buffer support has only been backported to 2.7.

Currently the buffer implementation matches the functionality of the numpy interface (most of the code is largely the same with few modifications) under the assumption that the buffers are contiguous. Python's buffer interface allows for non-contiguous buffers so this will have to be implemented. Sould there be a distiction between the two PyArray wrapper types? There have been recent discussions about Julia's Array type heirarchy reflecting the underlying data layout.

The biggest advantage of utilizing python's buffer interface is that it supports heterogenous structure types. This will allow for conversions between numpy's record arrays and julia array's of immutable types. Parsing the buffer format specification is going to take a bit more work. The main question I have is how to best generate the types for these buffers. For instance upon being passed a numpy record array specifying some structure, do we create a requsite type on the fly or look to the global enviornment and try to match an exisiting type? I would be interested in any thoughts anyone might have as for how this would work.

The last thing to do would be to implement the buffer interface for julia's array types. What is the best way to do this?