Training Structured Efficient Convolutional Layers
Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers have not yet been adapted to the convolutional setting.We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms and permutations that can be optimised using standard stochastic gradient methods.A network composed of such structured efficient convolutional layers outperforms existing low-rank networks and demonstrates competitive computational efficiency.
It's possible to substitute the weight matrix in a convolutional layer to train it as a structured efficient layer; performing as well as low-rank decomposition.", 'This work applies previous Structured Efficient Linear Layers to conv layers and proposes Structured Efficient Convolutional Layers as substitution of original conv layers.