ContinuumIO/numba — pull request #6 by jaberg (2012-05-29 04:00:04+00:00)
Title: RFC: automatic differentiation & gpu codegen for pure numpy

This RFC pull request contains a working small example of how to apply Theano to numpy code (see tests/test_ad.py) and it has no interaction with the LLVM code (Theano builds a higher-level AST for now). The goals of using a Theano backend and an LLVM backend are related in wanting to speed up numpy code, and in the technique of emulating the VM to do JIT specialization.

I was never happy with Theano's interface. I really wanted to non-invasively spy on what numpy was doing (what it was allocating, what expressions produced what results, etc.) but without actually hijacking the interpreter itself I couldn't get that information. Consequently Theano has the feel of an additional "framework" that a programmer has to think about and buy into. That's a shame because Numpy + Python is a perfectly expressive combination -- we just wanted to use Theano's optimizations and code generation to speed it up (and use GPU). Numba's bytecode parsing trick blew my mind when I looked at it, how easy it was to implement the CPython VM in Python. With that technique I think it's finally possible to give Theano the right API so that it's services (autodiff, GPU, codegen, etc.) can be expressed in terms of raw numpy code.

Do you think there's room in one project for all of these features? The trouble with VM-hacking is that it would be really nice if there were just *one level* of VM emulation going on at a time, and the various libraries that were using the technique could at least register themselves as listeners for a *single* bytecode parser.

Another thing to coordinate on is how numba was planning on dealing with array expressions. Why does numba focus on scalar code? It seems like if numba were to encounter vectorized code, then it would start doing pretty much what Theano currently does (at least in a high level sense), and possibly have a huge overlap with what @markflorisson88 is working on this summer for Cython (and possibly with what a previous GSoC student implemented for SymPy -- the fortran-code generation stuff). I think it would be great if Theano acted on LLVM-IR instead of its own internal AST data structure, and now that libndarray has a C interface, I've been hoping that somehow it would be possible to make LLVM-IR look not-too-different from a theano AST. If that were the case then numba, Theano, and Cython would all be tackling nearly the same problem with nearly the same tools, for very similar applications. These issues have come up in conversation with @markflorisson88, @npinto, @dwf, @dagss, so I'm bringing them in here too.