Fast Training of Convolutional Networks through FFTs
Authors: Michael Mathieu, Mikael Henaff, Yann LeCun
ICLR 2014

Abstract: Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.


Peer review:
[Review]:
Training convnets on GPUs has been a challenge in terms of both time and memory constraints, more often constraining the size of the models by memory requirements per GPU rather than processing time.

It would be helpful to add a comparison of memory requirements for your method compared to purely spatial convolution implementations.

You mention: 
'Also note that our method performs the same regardless of kernel size, since we pad the kernel to be the
same size as the input image before applying the FFT'

From my understanding without looking at your implementation, I would assume that because of this particular operation, the memory requirements would balloon, making this method impractical to even implement current state-of-the-art models for Image and audio recognition problems.
It would be good to have a section talking about practical constraints, issues and how you guys think they should be handled.