Inference speed problem

username: Cyber_Blaster
cooked: <p>Hello.</p>
<p>I’m developing a simple Python application for personal use that detects the fundamental frequency of two harmonic sounds simultaneously. To achieve this, I’ve created and trained a small PyTorch neural network. I’m going to use it on a machine without GPU, so the code blow is executed on CPU.</p>
<pre><code class="lang-auto">Network(
  (conv1): DepthwiseConvBlock(
    (depth_conv): Conv1d(1, 16, kernel_size=(11,), stride=(1,), padding=same)
    (point_conv): Conv1d(16, 16, kernel_size=(1,), stride=(1,))
    (nonlin): GELU(approximate='none')
    (norm): LayerNorm((48,), eps=1e-05, elementwise_affine=True)
    (pool): MaxPool2d(kernel_size=[1, 2], stride=[1, 2], padding=0, dilation=1, ceil_mode=False)
  )
  (conv2): DepthwiseConvBlock(
    (depth_conv): Conv1d(16, 16, kernel_size=(7,), stride=(1,), padding=same, groups=16)
    (point_conv): Conv1d(16, 16, kernel_size=(1,), stride=(1,))
    (nonlin): GELU(approximate='none')
    (norm): LayerNorm((24,), eps=1e-05, elementwise_affine=True)
    (pool): MaxPool2d(kernel_size=[1, 2], stride=[1, 2], padding=0, dilation=1, ceil_mode=False)
  )
  (conv3): DepthwiseConvBlock(
    (depth_conv): Conv1d(16, 8, kernel_size=(5,), stride=(1,), padding=same, groups=8)
    (point_conv): Conv1d(8, 8, kernel_size=(1,), stride=(1,))
    (nonlin): GELU(approximate='none')
    (norm): LayerNorm((12,), eps=1e-05, elementwise_affine=True)
    (pool): MaxPool2d(kernel_size=[1, 2], stride=[1, 2], padding=0, dilation=1, ceil_mode=False)
  )
  (flat): Flatten(start_dim=1, end_dim=-1)
  (linear): Linear(in_features=96, out_features=8, bias=True)
)
</code></pre>
<p>The input to this network is a spectrum calculated using the CQT transform function operating on NumPy arrays audio. The network, in turn, produces probabilities for the detected notes. However, there’s an issue: when I run the network in a loop alongside the CQT transform function, the network’s inference speed experiences a significant drop.</p>
<p>Just CQT transform function:</p>
<pre><code class="lang-auto">def profile_cqt_to_torch():
    cqts = []
    for a in audio:
        cqt = cqt_rough(a)
        cqts.append(torch.from_numpy(cqt).unsqueeze(0).to(torch.float))
    return cqts

t = time()
cqts = profile_cqt_to_torch()
print(time() - t)

&gt;&gt;&gt; 0.35244059562683105
</code></pre>
<p>Just the network with precalculated CQTs:</p>
<pre><code class="lang-auto">network.eval()

def profile_net():
    preds = []
    for cqt in cqts:
        with torch.no_grad():
            pred = network(cqt)
            preds.append(pred)
    return pred
    
t = time()
preds = profile_net()
print(time() - t)

&gt;&gt;&gt; [W NNPACK.cpp:64] Could not initialize NNPACK! Reason: Unsupported hardware.
&gt;&gt;&gt; 0.5710155963897705
</code></pre>
<p>Combined:</p>
<pre><code class="lang-auto">network.eval()

def profile_cqt_net():
    preds = []
    for a in audio:
        cqt = cqt_rough(a)
        with torch.no_grad():
            pred = network(torch.from_numpy(cqt).unsqueeze(0).to(torch.float))
        preds.append(pred)
    return preds

t = time()
preds = profile_cqt_net()
print(time() - t)

&gt;&gt;&gt; 15.845890522003174
</code></pre>
<p>By utilizing the new <code>torch.compile</code> method with <code>mode = reduce-overhead</code>, I’ve managed to decrease the execution time to approximately 10 seconds. Nevertheless, this duration still seems excessively long.</p>
<p><code>cProfile</code> and <code>torch.profiler</code> gives a similar results. I check <code>dtype</code> and <code>size</code> of input tensor every time.</p>
<p><strong>Profilng with precalculated CQTs</strong>:<br>
Code:</p>
<pre><code class="lang-auto">def profile_net():
    preds = []
    with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
        for cqt in cqts:

            assert cqt.shape == torch.Size([1, 96])
            assert cqt.dtype == torch.float32

            with record_function("model_inference"):
                with torch.no_grad():
                    pred = network(cqt)
                    preds.append(pred)
    print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))
    return pred
</code></pre>
<p>cProfile results:</p>
<pre><code class="lang-auto">92544 function calls (72544 primitive calls) in 0.850 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.000    0.000    0.850    0.425 /home/user/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3406(run_code)
        2    0.000    0.000    0.850    0.425 {built-in method builtins.exec}
        1    0.000    0.000    0.849    0.849 &lt;ipython-input-24-33789e8bbc84&gt;:1(&lt;module&gt;)
        1    0.007    0.007    0.849    0.849 &lt;ipython-input-23-24b737ab1132&gt;:6(profile_net)
10500/500    0.023    0.000    0.828    0.002 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1514(_wrapped_call_impl)
10500/500    0.044    0.000    0.827    0.002 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1520(_call_impl)
      500    0.020    0.000    0.823    0.002 /mnt/3660917E7771EA5C/Programming/PitchDetect/TheModel.py:71(forward)
     1500    0.049    0.000    0.720    0.000 /mnt/3660917E7771EA5C/Programming/PitchDetect/TheModel.py:49(forward)
     3000    0.016    0.000    0.378    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py:309(forward)
     3000    0.008    0.000    0.356    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py:301(_conv_forward)
     3000    0.348    0.000    0.348    0.000 {built-in method torch.conv1d}
     1500    0.003    0.000    0.087    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/activation.py:681(forward)
     1500    0.084    0.000    0.084    0.000 {built-in method torch._C._nn.gelu}
     1500    0.010    0.000    0.068    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/normalization.py:195(forward)
     1500    0.006    0.000    0.060    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/pooling.py:165(forward)
     1500    0.008    0.000    0.055    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/functional.py:2528(layer_norm)
     1500    0.005    0.000    0.055    0.000 /home/user/.local/lib/python3.10/site-packages/torch/_jit_internal.py:478(fn)
     1500    0.004    0.000    0.050    0.000 /home/user/.local/lib/python3.10/site-packages/torch/nn/functional.py:769(_max_pool2d)
     1500    0.045    0.000    0.045    0.000 {built-in method torch.max_pool2d}
     1500    0.044    0.000    0.044    0.000 {built-in method torch.layer_norm}
</code></pre>
<p>pytorch.profiler results:</p>
<pre><code class="lang-auto">---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference        33.48%     264.789ms        99.68%     788.410ms       1.577ms           500  
                     aten::conv1d         2.02%      16.011ms        43.88%     347.084ms     115.695us          3000  
                aten::convolution         1.94%      15.362ms        41.53%     328.489ms     109.496us          3000  
               aten::_convolution         5.33%      42.175ms        39.61%     313.289ms     104.430us          3000  
          aten::_convolution_mode         1.38%      10.923ms        29.09%     230.100ms     153.400us          1500  
         aten::mkldnn_convolution        18.30%     144.708ms        19.21%     151.955ms     151.955us          1000  
                aten::thnn_conv2d         0.71%       5.634ms        10.87%      85.951ms      42.975us          2000  
       aten::_slow_conv2d_forward         6.81%      53.884ms        10.21%      80.788ms      40.394us          2000  
                       aten::gelu         6.63%      52.461ms         6.63%      52.461ms      34.974us          1500  
                 aten::layer_norm         0.64%       5.027ms         5.72%      45.237ms      30.158us          1500  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 790.968ms
</code></pre>
<p>With both functions combined:<br>
Code:</p>
<pre><code class="lang-auto">def profile_cqt_net():
    preds = []
    with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
        for a in audio:
            cqt = cqt_rough(a)
            cqt = torch.from_numpy(cqt).unsqueeze(0).to(torch.float)

            assert cqt.shape == torch.Size([1, 96])
            assert cqt.dtype == torch.float32

            with torch.no_grad():
                with record_function("model_inference"):
                    pred = network(cqt)
            preds.append(pred)
    print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=20))
    return preds
</code></pre>
<pre><code class="lang-auto">         109039 function calls (107539 primitive calls) in 11.783 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.000    0.000   11.783    5.891 /home/user/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3406(run_code)
        2    0.000    0.000   11.783    5.891 {built-in method builtins.exec}
        1    0.001    0.001   11.782   11.782 &lt;ipython-input-16-b93e40bc29de&gt;:1(&lt;module&gt;)
        1    0.027    0.027   11.782   11.782 &lt;ipython-input-13-9f753a9ee13c&gt;:7(profile_cqt_net)
 1000/500    0.005    0.000   10.409    0.021 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1514(_wrapped_call_impl)
 1000/500    0.015    0.000   10.407    0.021 /home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1520(_call_impl)
 1000/500    0.020    0.000   10.397    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:307(_fn)
      500    0.003    0.000   10.333    0.021 /mnt/3660917E7771EA5C/Programming/PitchDetect/TheModel.py:71(forward)
      500    0.002    0.000   10.311    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_dynamo/external_utils.py:15(inner)
      500    0.003    0.000   10.308    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py:3901(forward)
      500    0.002    0.000   10.305    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py:1481(g)
      500    0.010    0.000   10.303    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py:2519(runtime_wrapper)
      500    0.007    0.000   10.292    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py:1498(call_func_with_args)
      500    0.002    0.000   10.282    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py:1583(rng_functionalization_wrapper)
      500    0.002    0.000   10.280    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py:373(__call__)
      500    0.002    0.000   10.277    0.021 /home/user/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py:385(_run_from_cache)
      500    0.169    0.000   10.275    0.021 /tmp/torchinductor_user/fy/cfyr3q7akk2fssiuzxfhjdricxsnhqemn66ccqvr6xnodgbjmeai.py:297(call)
     3000    9.927    0.003    9.927    0.003 {built-in method torch.convolution}
      500    0.010    0.000    1.275    0.003 /mnt/3660917E7771EA5C/Programming/PitchDetect/cqt_rough.py:77(__call__)
     1500    1.154    0.001    1.242    0.001 /mnt/3660917E7771EA5C/Programming/PitchDetect/cqt_rough.py:74(apply_filters)
     1500    0.006    0.000    0.088    0.000 {method 'mean' of 'numpy.ndarray' objects}
     1500    0.033    0.000    0.082    0.000 /home/user/.local/lib/python3.10/site-packages/numpy/core/_methods.py:163(_mean)
     2500    0.010    0.000    0.069    0.000 /home/user/.local/lib/python3.10/site-packages/torch/_ops.py:687(__call__)
     2500    0.059    0.000    0.059    0.000 {built-in method torch._ops.inductor._reinterpret_tensor}
     3500    0.054    0.000    0.054    0.000 {built-in method torch.empty_strided}
      500    0.036    0.000    0.036    0.000 {built-in method torch.addmm}
     1500    0.024    0.000    0.024    0.000 {method 'reduce' of 'numpy.ufunc' objects}
</code></pre>
<pre><code class="lang-auto">---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference         2.63%     643.239ms        99.60%       24.327s      48.653ms           500  
                     aten::conv1d         0.38%      92.808ms        40.14%        9.805s       3.268ms          3000  
                aten::convolution         0.18%      42.919ms        39.91%        9.748s       3.249ms          3000  
               aten::_convolution         0.41%     100.469ms        39.73%        9.705s       3.235ms          3000  
          aten::_convolution_mode         0.11%      27.077ms        38.94%        9.512s       6.341ms          1500  
         aten::mkldnn_convolution        29.10%        7.108s        29.22%        7.137s       7.137ms          1000  
                       aten::gelu        28.33%        6.920s        28.33%        6.920s       4.614ms          1500  
                 aten::layer_norm         0.07%      17.180ms        27.44%        6.701s       4.467ms          1500  
          aten::native_layer_norm        27.26%        6.658s        27.36%        6.684s       4.456ms          1500  
                aten::thnn_conv2d         0.08%      19.653ms         9.72%        2.375s       1.187ms          2000  
       aten::_slow_conv2d_forward         9.33%        2.280s         9.67%        2.363s       1.181ms          2000  
                  aten::unsqueeze         0.33%      79.823ms         0.41%     100.883ms      13.451us          7500  
                     aten::linear         0.08%      20.467ms         0.34%      82.141ms     164.282us           500  
                 aten::max_pool2d         0.08%      18.410ms         0.29%      71.995ms      47.997us          1500  
                         aten::to         0.04%       8.668ms         0.22%      54.208ms     108.416us           500  
                      aten::copy_         0.22%      54.167ms         0.22%      54.167ms      18.056us          3000  
    aten::max_pool2d_with_indices         0.22%      53.585ms         0.22%      53.585ms      35.723us          1500  
                   aten::_to_copy         0.11%      25.727ms         0.19%      46.555ms      93.110us           500  
                       aten::view         0.17%      42.428ms         0.17%      42.428ms       4.243us         10000  
                      aten::addmm         0.11%      27.823ms         0.17%      41.023ms      82.046us           500  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 24.425s
</code></pre>
<p>OS: Ubuntu 22.04<br>
PyTorch: 2.1.0<br>
I have no clue how to fix this problem and hoping for help.</p>