Weird `with torch.autograd.profiler.profile(use_cuda=True)` results

crizCraig: <pre><code class="lang-auto">                                 Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
-------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                             aten::to         0.05%      28.825us         1.57%     846.961us     846.961us      18.496us         0.03%     828.160us     828.160us             1
                  aten::empty_strided         0.01%       8.040us         0.01%       8.040us       8.040us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         1.51%     810.096us         1.51%     810.096us     810.096us     809.664us         1.46%     809.664us     809.664us             1
                             aten::to         0.02%       8.829us         1.30%     698.454us     698.454us      15.712us         0.03%     698.208us     698.208us             1
                  aten::empty_strided         0.01%       6.741us         0.01%       6.741us       6.741us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         1.27%     682.884us         1.27%     682.884us     682.884us     682.496us         1.23%     682.496us     682.496us             1
                             aten::to         0.02%       8.701us         0.10%      54.518us      54.518us      15.360us         0.03%      54.880us      54.880us             1
                  aten::empty_strided         0.01%       6.311us         0.01%       6.311us       6.311us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.07%      39.506us         0.07%      39.506us      39.506us      39.520us         0.07%      39.520us      39.520us             1
                             aten::to         0.01%       7.228us         0.06%      30.217us      30.217us      10.304us         0.02%      29.344us      29.344us             1
                  aten::empty_strided         0.01%       3.438us         0.01%       3.438us       3.438us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.551us         0.04%      19.551us      19.551us      19.040us         0.03%      19.040us      19.040us             1
                             aten::to         0.01%       7.638us         0.13%      71.446us      71.446us      11.008us         0.02%      70.528us      70.528us             1
                  aten::empty_strided         0.01%       3.721us         0.01%       3.721us       3.721us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.11%      60.087us         0.11%      60.087us      60.087us      59.520us         0.11%      59.520us      59.520us             1
                          aten::slice         0.01%       5.624us         0.01%       7.152us       7.152us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       1.528us         0.00%       1.528us       1.528us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.264us         0.01%       2.734us       2.734us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.470us         0.00%       0.470us       0.470us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.944us         0.01%       2.801us       2.801us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.857us         0.00%       0.857us       0.857us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sum         0.07%      38.494us         0.08%      41.391us      41.391us      41.344us         0.07%      41.344us      41.344us             1
                          aten::empty         0.01%       2.897us         0.01%       2.897us       2.897us       0.000us         0.00%       0.000us       0.000us             1
                        aten::type_as         0.01%       7.933us         0.08%      40.439us      40.439us       7.552us         0.01%      40.256us      40.256us             1
                             aten::to         0.01%       7.729us         0.06%      32.506us      32.506us      12.224us         0.02%      32.704us      32.704us             1
                  aten::empty_strided         0.01%       3.739us         0.01%       3.739us       3.739us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      21.038us         0.04%      21.038us      21.038us      20.480us         0.04%      20.480us      20.480us             1
                           aten::view         0.01%       3.399us         0.01%       3.399us       3.399us       0.000us         0.00%       0.000us       0.000us             1
                            aten::div         0.05%      26.231us         0.05%      28.614us      28.614us      27.168us         0.05%      27.168us      27.168us             1
                          aten::empty         0.00%       2.383us         0.00%       2.383us       2.383us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.664us         0.01%       3.241us       3.241us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.577us         0.00%       0.577us       0.577us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.012us         0.00%       2.326us       2.326us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.314us         0.00%       0.314us       0.314us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.861us         0.00%       2.441us       2.441us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.580us         0.00%       0.580us       0.580us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.04%      23.023us         0.05%      24.994us      24.994us      26.592us         0.05%      26.592us      26.592us             1
                          aten::empty         0.00%       1.971us         0.00%       1.971us       1.971us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.417us         0.01%       2.858us       2.858us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.441us         0.00%       0.441us       0.441us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.061us         0.00%       2.388us       2.388us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.327us         0.00%       0.327us       0.327us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.806us         0.00%       2.184us       2.184us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.378us         0.00%       0.378us       0.378us       0.000us         0.00%       0.000us       0.000us             1
                     aten::zeros_like         0.02%       9.266us         0.06%      34.875us      34.875us      14.208us         0.03%      34.816us      34.816us             1
                     aten::empty_like         0.00%       2.448us         0.01%       4.493us       4.493us       0.000us         0.00%       0.000us       0.000us             1
                          aten::empty         0.00%       2.045us         0.00%       2.045us       2.045us       0.000us         0.00%       0.000us       0.000us             1
                          aten::zero_         0.02%       8.284us         0.04%      21.116us      21.116us       7.168us         0.01%      20.608us      20.608us             1
                          aten::fill_         0.02%      12.832us         0.02%      12.832us      12.832us      13.440us         0.02%      13.440us      13.440us             1
                          aten::slice         0.00%       2.425us         0.01%       2.889us       2.889us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.464us         0.00%       0.464us       0.464us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.885us         0.00%       2.214us       2.214us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.329us         0.00%       0.329us       0.329us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      11.693us         0.02%      12.144us      12.144us      12.288us         0.02%      12.288us      12.288us             1
                     aten::as_strided         0.00%       0.451us         0.00%       0.451us       0.451us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.130us         0.00%       2.515us       2.515us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.385us         0.00%       0.385us       0.385us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.538us         0.02%      11.102us      11.102us      10.240us         0.02%      10.240us      10.240us             1
                     aten::as_strided         0.00%       0.564us         0.00%       0.564us       0.564us       0.000us         0.00%       0.000us       0.000us             1
                             aten::to         0.01%       7.161us         0.06%      29.683us      29.683us       9.344us         0.02%      28.768us      28.768us             1
                          aten::empty         0.01%       2.816us         0.01%       2.816us       2.816us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.706us         0.04%      19.706us      19.706us      19.424us         0.04%      19.424us      19.424us             1
                      aten::unsqueeze         0.01%       5.503us         0.01%       6.034us       6.034us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.531us         0.00%       0.531us       0.531us       0.000us         0.00%       0.000us       0.000us             1
                            aten::mul         0.04%      23.639us         0.05%      25.936us      25.936us      24.832us         0.04%      24.832us      24.832us             1
                          aten::empty         0.00%       2.297us         0.00%       2.297us       2.297us       0.000us         0.00%       0.000us       0.000us             1
                            aten::add         0.04%      20.709us         0.04%      22.143us      22.143us      21.888us         0.04%      21.888us      21.888us             1
                          aten::empty         0.00%       1.434us         0.00%       1.434us       1.434us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.03%      18.169us         0.04%      19.918us      19.918us      25.600us         0.05%      25.600us      25.600us             1
                          aten::empty         0.00%       1.749us         0.00%       1.749us       1.749us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.501us         0.01%       3.035us       3.035us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.534us         0.00%       0.534us       0.534us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.885us         0.00%       2.218us       2.218us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.333us         0.00%       0.333us       0.333us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.748us         0.02%      11.258us      11.258us      10.816us         0.02%      10.816us      10.816us             1
                     aten::as_strided         0.00%       0.510us         0.00%       0.510us       0.510us       0.000us         0.00%       0.000us       0.000us             1
                           aten::view         0.00%       2.158us         0.00%       2.158us       2.158us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.246us         0.04%      19.246us      19.246us      19.648us         0.04%      19.648us      19.648us             1
                          aten::slice         0.00%       2.526us         0.01%       2.996us       2.996us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.470us         0.00%       0.470us       0.470us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.756us         0.00%       2.106us       2.106us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.350us         0.00%       0.350us       0.350us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.652us         0.02%      11.078us      11.078us      10.752us         0.02%      10.752us      10.752us             1
                     aten::as_strided         0.00%       0.426us         0.00%       0.426us       0.426us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.116us         0.00%       2.512us       2.512us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.396us         0.00%       0.396us       0.396us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.011us         0.02%      10.417us      10.417us      10.208us         0.02%      10.208us      10.208us             1
                     aten::as_strided         0.00%       0.406us         0.00%       0.406us       0.406us       0.000us         0.00%       0.000us       0.000us             1
                             aten::to         0.01%       7.377us         0.05%      27.906us      27.906us       8.320us         0.02%      26.720us      26.720us             1
                          aten::empty         0.00%       2.613us         0.00%       2.613us       2.613us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.03%      17.916us         0.03%      17.916us      17.916us      18.400us         0.03%      18.400us      18.400us             1
                      aten::unsqueeze         0.01%       3.181us         0.01%       3.668us       3.668us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.487us         0.00%       0.487us       0.487us       0.000us         0.00%       0.000us       0.000us             1
                            aten::mul         0.04%      19.252us         0.04%      20.895us      20.895us      20.672us         0.04%      20.672us      20.672us             1
                          aten::empty         0.00%       1.643us         0.00%       1.643us       1.643us       0.000us         0.00%       0.000us       0.000us             1
                            aten::add         0.03%      17.770us         0.04%      19.141us      19.141us      19.488us         0.04%      19.488us      19.488us             1
                          aten::empty         0.00%       1.371us         0.00%       1.371us       1.371us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.03%      17.869us         0.04%      19.398us      19.398us      25.632us         0.05%      25.632us      25.632us             1
                          aten::empty         0.00%       1.529us         0.00%       1.529us       1.529us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.01%       2.847us         0.01%       3.431us       3.431us       0.000us         0.00%       0.000us       0.000us             1
-------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 53.808ms
CUDA time total: 55.443ms
</code></pre>
<p>It seems the columns only add up to 3.84% of the total. Also, does this mean the total time is the CPU + the CUDA time, or does it run the graph in CPU and CUDA separately? Thanks in advance for your help.</p>

googlebot: <ol>
<li>seems that you didn’t do <code>profile = profile.key_averages()</code> and are viewing ungrouped outputs</li>
<li>there is a default value: profile.table(…, row_limit=100), this cutting explains missing percentages</li>
<li>IIRC, total time reported there is wrong, as it doesn’t account for nested calls (or there is some similar issue)</li>
</ol>

crizCraig: <p>Thanks Alex! I’ll out <code>profile.key_averages()</code> and <code>row_limit</code>.</p>
<p>re 3: I’ll try summing the columns without a row limit and compare.</p>