Torch.sum does not benefit from parallelism?

username: Louka_Ewington-Pitso
cooked: <p>This is my time-taken (seconds) vs batch size for 1000 summation operations on a <code>NVIDIA A100-SXM4-80GB</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa2b066268b28c6d3f99e59c04188fffae035a4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa2b066268b28c6d3f99e59c04188fffae035a4d" title="Screenshot 2024-07-05 at 10.56.16 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa2b066268b28c6d3f99e59c04188fffae035a4d_2_669x500.png" alt="Screenshot 2024-07-05 at 10.56.16 AM" data-base62-sha1="zH5xYY8Pr8xdh1UynqVjZ8CRDKl" width="669" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa2b066268b28c6d3f99e59c04188fffae035a4d_2_669x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa2b066268b28c6d3f99e59c04188fffae035a4d_2_1003x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa2b066268b28c6d3f99e59c04188fffae035a4d_2_1338x1000.png 2x" data-dominant-color="FCFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-07-05 at 10.56.16 AM</span><span class="informations">1412×1054 61.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Actual times:</p>
<p>0.07381916046142578<br>
0.11852669715881348<br>
0.23033857345581055<br>
0.4577453136444092<br>
1.3883068561553955<br>
3.253286123275757<br>
6.985889911651611</p>
<p>this is my code</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import time

for bs in [4, 8, 16, 32, 64, 128, 256]:
    tensor = torch.randn(bs, 512, 25000).to('cuda')

    start = time.time()
    for i in range(1000):
        torch.sum(tensor)

    print(time.time() - start)
</code></pre>
<p>nvidia-smi</p>
<pre><code class="lang-auto">+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  | 00000000:01:00.0 Off |                    0 |
| N/A   46C    P0             302W / 500W |  12816MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
</code></pre>
<p>Am I using too much memory and this is causing things to run slowly or something?</p>

username: anantguptadbl
cooked: <aside class="quote no-group" data-username="Louka_Ewington-Pitso" data-post="1" data-topic="205774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/louka_ewington-pitso/48/71203_2.png" class="avatar"> Louka_Ewington-Pitso:</div>
<blockquote>
<pre><code class="lang-auto">import torch
import time

for bs in [4, 8, 16, 32, 64, 128, 256]:
    tensor = torch.randn(bs, 512, 25000).to('cuda')

    start = time.time()
    for i in range(1000):
        torch.sum(tensor)

    print(time.time() - start)
</code></pre>
</blockquote>
</aside>
<p>Torch internally uses the numpy sum operations, so if i do the same with numpy array</p>
<pre><code class="lang-auto">import torch
import time
import numpy as np

for bs in [4, 8, 16, 32, 64, 128, 256]:
    #tensor = np.random.rand(bs, 512, 25000)
    tensor = np.random.rand(bs, 256, 256)
    start = time.time()
    for i in range(1000):
        np.sum(tensor)
    print(time.time() - start)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d600c314609ba32b2b254314d4b11ea0fa361cb.png" alt="image" data-base62-sha1="b2uxS2pHgJ9axidOPpwauetbt4D" width="279" height="126"></p>
<p>The data of an array has to be stored in memory and there will be certain cpu clocks spent calculate the sum</p>
<p>For tensor level parallelization, you might want to try this<br>
<a href="https://pytorch.org/docs/stable/distributed.tensor.parallel.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributed.tensor.parallel.html</a></p>

username: ptrblck
cooked: <p>Your profiling is wrong since you are not synchronizing the host timers while trying to profile asynchronous CUDA kernels.<br>
With that being said, these kernels are bandwidth bound so seeing an increase in the kernel duration corresponding to the tensor size would be expected.</p>

username: anantguptadbl
cooked: <aside class="quote no-group" data-username="Louka_Ewington-Pitso" data-post="1" data-topic="205774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/louka_ewington-pitso/48/71203_2.png" class="avatar"> Louka_Ewington-Pitso:</div>
<blockquote>
<pre><code class="lang-auto">for bs in [4, 8, 16, 32, 64, 128, 256]:
    tensor = torch.randn(bs, 512, 25000).to('cuda')

    start = time.time()
    for i in range(1000):
        torch.sum(tensor)

    print(time.time() - start)
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you share some links etc that would explain the CUDA parallelization that you mentioned</p>

username: Brock_Brown
cooked: <p>Imagine you had a person you go to who does your sums for you. You pass them a box full of numbers and you ask them to sum them up. This person has to sum them one number at a time. If you were to have two people, you can roughly half the time by dumping the numbers from the box equally among the two people, the two people come together at the end and sum their own results. If you have three people, it becomes roughly one third of the work. But no matter how many people you have, if you double the quantity of numbers in the box then the time taken to complete the summation will double. Each person is a CUDA core, and the job of summing the numbers can be called a CUDA kernel.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/brock_brown">@Brock_Brown</a> Thanks for the explanation. So more like mapReduce. This means that CUDA must be having some some kind of orchestrator module</p>
<p>Time to do some reading on CUDA <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>