BVLC/caffe — pull request #1615 by longjon (2014-12-22 06:54:57+00:00)
Title: Refactor convolution layer and add deconvolution layer

This PR adds a `DeconvolutionLayer` that flips the forward and backward passes of `ConvolutionLayer`. (The resulting operation is still convolution, but the sense of all the parameters is reversed, so that, in particular, strided deconvolution results in upsampling whereas strided convolution results in downsampling.)

Rather than duplicate all the `ConvolutionLayer` code, common sections are factored out into a parent class, `BaseConvolutionLayer`. The tricky GEMM parameters and the column buffer are hidden from the forward and backward implementations, which I hope you agree are much more readable.

Positives
* Readable implementations of convolution and deconvolution.
* Deconvolution supports all functions of convolution, including padding (note that padding is removed from the output rather than added to the input!), groups, rectangular kernels, biases.
* Whereas `ConvolutionLayer` needs to do an `im2col` and a `col2im` in the backward pass, both operations in `DeconvolutionLayer`'s backward pass require `im2col`, so a special flag is added to avoid doing this twice.

Reservations
* The diff is pretty heavy; it was not straightforward to make the changes in a gradual way, so the history is constructed post-hoc. However, the code should be pretty understandable to the few of you intimately familiar with `ConvolutionLayer`. Also note that the diff size is really half what it appears to be, because of the point below.
* Almost all the code is duplicated via s/cpu/gpu/. This PR is not the place to address this, but I hope we can get (some part of?) #610 merged soon, because this is getting silly.
* Is "deconvolution" really the right name? This layer does not have to be used in the context of undoing a convolution (although it could be). It seems likely that this name will stick one way or another.

Design choices
* The original idea was to add convolution helper functions under `util/`. However, these end up requiring a large number of arguments (each needs most of the convolution parameters). So, one could wrap all the common arguments in a `struct`, like cuDNN... but then that `struct` may as well be the layer class itself, so we have the current design.
* Rather than use a special flag to skip the extra `im2col` in deconv backward, we could leave the column buffer out of the wrapper functions. However, this leads to more duplicated code, and cuts down significantly on readability.
* The forward helper and weight diff helper are both `im2col`s followed by gemms, so in theory we could unify them. However, then we need additional arguments and logic to figure out when to transpose the column buffer. I think this could still be done, but it's a bit of a wash.

Forthcoming
* Doc comments for `DeconvolutionLayer`.
* Tests specific to `DeconvolutionLayer`. Note that most of the functionality is exercised by the `ConvolutionLayer` tests. I've only briefly checked the forward/backward passes for upsampling/downsampling, so there could be a bug or two left.
* cuDNN deconvolution should be straightforward to implement as well, since cuDNN already provides the same kind of abstraction as this refactoring. However, I won't be able to do that right away; PRs are welcome!