Fast Training of Convolutional Networks through FFTs
Authors: Michael Mathieu, Mikael Henaff, Yann LeCun
ICLR 2014

Abstract: Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.


Peer review:
[Review]:
Speeding up convolutional networks is a quite interesting topic. The presented method seems effective and principled.
The paper reads easily however the experimental results seems somewhat lacking.
The whole premise of the work is to speed-up training and testing, yet there is not even anecdotal results regarding total training time. This matters because, at the end of the day, it is the system wide performance that matters. 

A theoretical paper that only shows improvement on the number of operations, might hide the fact that the memory access is now more convolved, and thus the overall speed goes down (despite having reduced the number of operations). In the context of this paper, showing some evidence of the overall speed would be welcome.

At the same time, full training experiments would show that indeed, in practice, there is no degradation of the learnt model or the predicted scores. There is all kind of implementation issues that can convert theoretically identical results into different outcomes (because in a computer a + (b + c) != (a + b) + c).

When using methods based on Fourier transform border effects enter into play. Could you mention how you handle these ?
Also section 3 mentions an 'additional heuristic', could you detail why this is an heuristic, and what are the consequences of using it ?
Finally, are there plan of releasing this new training code ? Open source releases tend to increase the impact of a paper.

Overall quite interesting work, looking forward for the results of training filters directly in the Fourier domain.