Theano/Theano — pull request #1809 by f0k (2014-04-17 10:38:35+00:00)
Title: Release GIL while waiting for the GPU

This PR adds a new config flag `theano.config.gpu.release_gil`. If set to `True` (default is `False`), theano will release the GIL when calling a blocking CUDA call to allow other Python threads to do some work in the meantime.

Background information: Theano does all its transfers and computation on the *default stream*. Kernel calls (including CUBLAS functions) are asynchronous on the host, that is, they return immediately. They are synchronous on the device, that is, they are executed in the order they have been submitted, one after the other. Transfers are synchronous on the host and the device -- that is, they are scheduled for execution on the default stream, and they return only after all previously scheduled computations (not only on the default stream, but also on all other streams on the device, that's what's special about the default stream) and the transfer itself have finished. Thus, every `cudaMemcpy()`, `cublasGetVector()` and `cublasSetVector()` is a good place to release the GIL. Furthermore, there are some explicit synchronization calls `cudaThreadSynchronize()` which also wait for all scheduled computations to finish. (By the way, `cudaThreadSynchronize()` is [deprecated](http://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__THREAD__DEPRECATED.html#group__CUDART__THREAD__DEPRECATED_1g6e0c5163e6f959b56b6ae2eaa8483576), but that's another issue). We also release the GIL in these occasions.

As we started to discuss on [theano-dev](https://groups.google.com/d/msg/theano-dev/CbX0XuUQmG8/mDA7m64OSwkJ), there may be a way to implement this without changing the flag causing a recompilation of all the code. Let us continue the discussion here and see if to merge the PR as it is or amend it.