As has also been pointed out in the discussion, I am worried about the intermediate data size: for example, for an input of size 55*55*256 and kernel size 3*3 (I am just making random numbers, may not correspond to actual imagenet layers) and convolution is with stride 1, a single buffer will have size

55 * 55 * 3 * 3 * 256 * sizeof(float) = 28MB

with multi threads (like 10) this will grow to like 280MB, and with multiple convolutional layers, this may quickly grow gigabytes. That's why I feel that we should rely on multithreading BLAS to speed things up rather than having a single-thread BLAS and explicit openmp code.