Performance on small conv

knitvoger: <p>I called torch.set_num_threads(8) to use all my cpu cores.<br>
The result shows that  running big conv, the performance is obvious faster on 8 threads compared with single thread. <strong>But runs small conv, the speed has no big differences with 8 threads and single thread.</strong></p>
<p>Please check this table <img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/390efa57857dd855956f5f63b4fcb5713940d3ed.png" alt="conv_perf" data-base62-sha1="88LiXy2rSOXs7XlIkVWCoxx7ZpP" width="672" height="94"></p>
<p>Mkl-dnn is enabled and below is the verbose info when running</p>
<pre><code class="lang-auto">mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_nchw out:f32_nChw8c,num:1,1x32x1x107718,2.18091
mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_oihw out:f32_OIhw8i8o,num:1,32x32x1x1,0.00195312
mkldnn_verbose,exec,convolution,jit_1x1:avx2,forward_training,fsrc:nChw8c fwei:OIhw8i8o fbia:x fdst:nChw8c,alg:convolution_direct,mb1_ic32oc32_ih1oh1kh1sh1dh0ph0_iw107718ow107718kw1sw1dw0pw0,3.87793
mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_nChw8c out:f32_nchw,num:1,1x32x1x107718,4.69116
</code></pre>
<p>So why 8 threads is not obviously faster than 1 thread with small conv? And how can i improve the speed on small conv?</p>
<p>Thanks a lot!</p>
<p>My code</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn as nn
from torch.nn.utils import weight_norm

class MyConv(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.cell = nn.Conv1d(*args, **kwargs)
        self.cell.weight.data.normal_(0.0, 0.02)

    def forward(self, x):
        return self.cell(x)


def main():
    #print(*torch.__config__.show().split("\n"), sep="\n")
    torch.set_num_threads(1)
    dim = 32
    kernels = 3
    seq = 100000
    MyCell = MyConv(dim, dim, kernel_size=kernels, stride=1)
    MyCell.eval()
    inputs = []
    iter = 1000
    for i in range(iter):
        inputs.append(torch.rand(1, dim, seq))

    start = time.time() * 1000
    for i in range(iter):
        print(i)
        y = MyCell(inputs[i])
        #print(y)
    end = time.time() * 1000
    print('cost %d ms per iter\n' % ((end - start) / iter))


if __name__ == "__main__":
    main()
</code></pre>

Nikronic: <p>Hi,<br>
I am not experienced in this area and just wanted to discuss about it.</p>
<p>Intuitively, I can say that when you enable multi-threading, you are adding some additional costs such as context switching between threads. So, for any operation on a small tensor, the additional costs of multi-threading compensate the speed up. Same thing applies in using GPU vs. CPU.<br>
For instance, using GPU for operations on small tensors has almost same performance on CPU too but when tensor size grows exponentially, the distance between the performances increases enormously.</p>

knitvoger: <p>Thanks for discussion <a class="mention" href="/u/nikronic">@Nikronic</a>. Beside thread swith, the time to memory io may also affect the performance for small tensors because the io time is relatively took more cost in total processing time.</p>
<p>And I did a test to divide the inputs [1, 32, 100000] to 8 part, each part is [1, 32, 12500]. And runs conv for each part in 8 thread. Compare to running in single thread, this can speed up about 4x. The only additional thing to do is concat each threads’ result and deal with the boundaries. Seems pytorch (and mkl-dnn backend) is not doing small conv with big inputs in this way.</p>