Theano/Theano — pull request #2266 by f0k (2014-11-18 15:32:30+00:00)
Title: conv2d meta-optimizer

This is a basic implementation of meta-optimization (#2072) for GPU-based convolutions. The idea of meta-optimization is to automatically choose the best-performing implementation of several alternatives provided by regular graph optimizers.

It introduces a local optimizer base class that for each node it is asked to replace, tries all local optimizers registered with it to determine which is the fastest, then chooses that. To time an implementation, it needs to be compiled into a function and run. This in turn requires non-symbolic inputs for the Apply node to be optimized. Inputs that are shared variables have data associated with them, so they can be used directly. Inputs that have a test value attached are replaced by shared variables of that value. Any other inputs need to be taken care of by subclasses of the base class.
For convolution, I make use of the fact that Apply nodes often know the expected input shape, so we can create random input data of the correct shape to compile the functions (see `ConvMetaOptimizers.provide_inputs()`).

Currently, the meta-optimizer is turned on by setting `THEANO_FLAGS=optimizer_including=conv_meta`. It then tries all convolution optimizers currently available in Theano (cudnn, gemm, and fft) regardless of whether they are active or not. It currently does not include the legacy convolution in the timing, because this usually just slows everything down. Additional optimizers (e.g., for cuda-convnet) can be registered at runtime via its `register` method (e.g., `theano.sandbox.cuda.opt.conv_metaopt.register(my_own_optimizer)`).

This PR can be merged as is. The following should be addressed in future:
- I would prefer to have it try only the optimizers that are active (i.e., cudnn and gemm if they haven't been deactivated, and fft only if it has been activated), is there an easy way to do so?
- For gemm, it should try both `GpuCorrMM` and `GpuCorrMM_gradWeights` if both are applicable, not rely on the heuristic in `local_conv_gemm`.
- Timings should be cached on disk to speed up repeated invocations.
- A second meta-optimizer for pooling could be added, once we have enough competing implementations.