Quantized model profiling

Ahmed_Louati: <p>Hello,</p>
<p>I am trying to profile quantized models using torch.profiler APIs.<br>
Are outputs of those APIs ( CPU_memory_usage, cpu_time, …) accurate?<br>
(from what I understood torch.profiler is made for nn.Module type)</p>
<p>If it is not the case, are there other methods to profile quantized models?</p>
<p>Thank you,</p>

jerryzh168: <p>I feel profiling this in CPU might be OK, but we have not extensively tested this, it is in the aten operator level so you will see ops like “quantized::linear”</p>

vadimkantorov: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> How does one profile the quantized CPU models to actually understand the dtypes passed around / quant-dequant conversions (if happening anywhere) and see the backend (fbgemm/qnnpack/onednn) kernel calls? (besides quantized::linear?) to be able to understand what got fused and how exactly</p>
<p>Should we use the low-level Linux’s <code>perf</code>? Is there anywhere an example? Or can one use <code>nsys</code> and see these CPU function calls?</p>

HDCharles: <p>yeah nsys works, in general we don’t have a ready made tool for visualizing all of this, usually the torch.profiler or nsys are effective methods for debugging perf degredation</p>

vadimkantorov: <p>Profiler would also be useful for novices to understand actually what’s going on and what actually got called / quant-dequanted where - for there’s a lot of layers of indirection… and torch.profiler only shows high-level information (which for being useful requires already understanding what’s supposed to happen)</p>
<p>Wonder if you tried using <code>perf</code> directly?</p>

parikshit: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a> I was using torch profiler to check the CPU memory usage by the resnet model vs quantized resnet model(int8 trained via qat) using</p>
<pre><code class="lang-auto">print(prof.key_averages().table(sort_by="self_cpu_memory_usage", row_limit=10))
</code></pre>
<p>However the profiler gives me this result:</p>
<p>Non quantized resnet</p>
<hr>
<pre><code>                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls  
</code></pre>
<hr>
<pre><code>                  aten::empty         0.49%       1.166ms         0.49%       1.166ms       2.982us      53.60 Mb      53.60 Mb           391  
                aten::resize_         0.10%     244.000us         0.10%     244.000us       7.394us      30.43 Mb      30.43 Mb            33  
aten::max_pool2d_with_indices         3.80%       8.960ms         3.80%       8.960ms       8.960ms       2.30 Mb       2.30 Mb             1  
   aten::_slow_conv2d_forward        33.26%      78.356ms        33.51%      78.944ms       2.392ms      31.39 Mb     980.00 Kb            33  
             aten::batch_norm         0.21%     495.000us         5.80%      13.657ms     257.679us      42.40 Mb     392.00 Kb            53  
                   aten::mean         0.02%      37.000us         0.07%     154.000us     154.000us       8.00 Kb       8.00 Kb             1  
                  aten::addmm         0.02%      48.000us         0.03%      60.000us      60.000us          36 b          36 b             1  
          aten::empty_strided         0.00%       5.000us         0.00%       5.000us       5.000us           4 b           4 b             1  
                 aten::conv2d         0.19%     451.000us        86.44%     203.648ms       3.842ms      42.40 Mb           0 b            53  
            aten::convolution         0.63%       1.479ms        86.25%     203.197ms       3.834ms      42.40 Mb           0 b            53  
</code></pre>
<hr>
<p>quantized resnet:</p>
<hr>
<pre><code>                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls  
</code></pre>
<hr>
<pre><code>                  aten::empty         2.32%       8.175ms         2.32%       8.175ms     151.389us      42.40 Mb      42.40 Mb            54  
aten::_empty_affine_quantized         0.37%       1.295ms         0.37%       1.295ms      17.986us      16.20 Mb      16.20 Mb            72  
    aten::quantize_per_tensor         0.90%       3.157ms         0.90%       3.157ms       3.157ms     147.00 Kb     147.00 Kb             1  
                  aten::addmm         0.02%      71.000us         0.03%      90.000us      90.000us          36 b          36 b             1  
                   aten::item         0.00%      10.000us         0.00%      16.000us       8.000us           0 b           0 b             2  
    aten::_local_scalar_dense         0.00%       6.000us         0.00%       6.000us       3.000us           0 b           0 b             2  
             aten::contiguous         0.01%      19.000us         0.09%     323.000us     323.000us     147.00 Kb           0 b             1  
                  aten::clone         0.08%     285.000us         0.09%     304.000us     304.000us     147.00 Kb           0 b             1  
                aten::qscheme         0.02%      69.000us         0.02%      69.000us       1.353us           0 b           0 b            51  
           aten::q_zero_point         0.02%      72.000us         0.02%      72.000us       0.679us           0 b           0 b           106  
</code></pre>
<hr>
<p>I don’t see much of a difference in the memory usage, is this correct?</p>

jerryzh168: <p>why aten::empty is taking most of the memory? maybe that just means the model is not memory bound so the reduction in weight memory is not important?</p>

parikshit: <p>I just wanted to know at maximum what are the memory requirement of a model during inference, to validate the point of using a quantized model instead of a sota model.<br>
Quantization does have an impact on CPU memory, right?</p>

jerryzh168: <p>yeah quantization will have impact on memory, if the main memory usage in inference comes from model weight, and all these weights are quantized to int8 from fp32, you should see 4x memory reduction.</p>

parikshit: <p>Then why the above results? I am using the standard transfer learning method with QAT. <a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Quantized Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a> .</p>
<p>Is there any other way do check about the RAM usage for a sample tensor during inference?</p>

parikshit: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> when I switch from x86 to qnnpack in torch.backends.quantized.engine, I get this result for quantized model:</p>
<hr>
<pre><code>                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls  
</code></pre>
<hr>
<pre><code>            quantized::conv2d        28.73%     119.856ms        28.75%     119.973ms       5.999ms       6.70 Mb       6.70 Mb            20  
          quantized::add_relu         1.23%       5.137ms         1.26%       5.252ms     328.250us       5.26 Mb       5.26 Mb            16  
       quantized::conv2d_relu        69.50%     289.980ms        69.67%     290.665ms       8.808ms       4.05 Mb       3.90 Mb            33  
aten::_empty_affine_quantized         0.02%      88.000us         0.02%      88.000us      29.333us     345.23 Kb     345.23 Kb             3  
    aten::quantize_per_tensor         0.09%     367.000us         0.09%     367.000us     367.000us     147.08 Kb     147.08 Kb             1  
                  aten::empty         0.01%      21.000us         0.01%      21.000us      21.000us       8.00 Kb       8.00 Kb             1  
                  aten::addmm         0.02%      84.000us         0.03%     116.000us     116.000us          36 b          36 b             1  
                   aten::item         0.01%      31.000us         0.01%      53.000us      26.500us           0 b           0 b             2  
    aten::_local_scalar_dense         0.01%      22.000us         0.01%      22.000us      11.000us           0 b           0 b             2  
             aten::contiguous         0.01%      24.000us         0.13%     555.000us     555.000us     147.08 Kb           0 b             1  
</code></pre>
<hr>
<p>while nonquantized table remains the same.</p>
<p>I do get that CPU time increases as I am running a quantized model in qnnpack config on x86 while nonquantized model still uses x86 configs machine but this much drop in memory is hard to believe just by changing the configs.</p>
<p>why a sudden drop?<br>
Also when we are considering memory footprint we consider the maximum among all layers or the sum?</p>

jerryzh168: <p>can you print the quantized model for both x86 and qnnpack I think we need to check if they are the same model first</p>

parikshit: <p>they are exactly the same, I compared them. Also, I am loading the same model and just changing</p>
<pre><code class="lang-auto">torch.backends.quantized.engine = 'x86'
</code></pre>
<p>to</p>
<pre><code class="lang-auto">torch.backends.quantized.engine = 'qnnpack'
</code></pre>