Add a mechanism to use more than one (gpu) context in a single theano function

__Disclaimer:__ This is work in progress.  Some changes made here will cause a lot of code to not work anymore, so don't use this branch unless you are prepared to deal with that.

This add some optional methods on Ops that will trigger CLinker to pass a context argument that will be available at init and runtime.  This context is typed according to the the type specified by the op.

Also this makes the context a part of the theano Type for GpuArray.  This way, most Ops will know what context they are going to work with and it makes it easy for the context to propagate through the graph.

TODO:

 - [x] Trivial example with 2 dots on different GPUs.
 - [x] Fix crash with tests/optimizations.
 - [x] add a way to copy a buffer from one context to another (without passing thought the CPU)
 - [ ] Fix crash with tests/optimizations and non default context.
 - [ ] add a way to specify the context of the inputs. (probably a tag with the context name)
 - [ ] gpu to gpu transfer _as_GpuArrayVariabl
 - [ ] replicate Alex models?

That it is done like this to allow pickling instead of being a normal
attribute.

On Wed, Nov 12, 2014 at 2:58 PM, abergeron <notifications@github.com> wrote:

> In theano/sandbox/gpuarray/type.py:
>
> >          try:
> >              self.typecode = gpuarray.dtype_to_typecode(self.dtype)
> >          except gpuarray.GpuArrayException:
> >              raise TypeError("Unsupported dtype for %s: %s" %
> >                              (self.__class__.__name__, self.dtype))
> >
> > +    def __real_context(self):
> > +        return get_context(self.context)
> > +
> > +    real_context = property(__real_context)
>
> Document what? That it's not a real property? Or just that this is a
> GpuContext value?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/Theano/Theano/pull/2182/files#r20246134>.
>