I wanted to preserve shape-in, shape-out and reduce the amount of shuffling code we have to do and simplify the code. Same restrictions as `convolve` except we treat 2d x 2d filter different when the second dim isn't 1.