Theano/Theano — pull request #1059 by nouiz (2012-11-09 02:46:50+00:00)
Title: Asynchronous GPU transfer

This is a rebase with a few small commit added of PR gh-962 by @mrockin.

This is disable dy default, so there is no harm to merge it.

I also modifed cuda_ndarray.cuh to disable call to cublasGetVectorAsync and cublasSetVectorAsync. I'm not sure if this is needed, but I prefered to be safe. Do someone know if we can always use the not async call? I'm not sure.

This split the GpuFromHost and HostFromGPU into 4 ops: GpuFromHostSend GpuFromHostWait HostFromGPUSend HostFromGPUWait. They do like in MPI send/wait. This allow to overlap transfer with computation.

There is also a scheduler that allow to schedule inteligently all this when we have a static toposort.

See the doc at the top of sandbox/cuda/async.py for more detail.

@mrocklin do you have the time to check my change at the end of this PR? If you agree when them, especially the docstring of the cuda/async.py file, I think we can merge this PR.