How to parallelize loading and computation

username: HaoKang-Timmy
cooked: <p>I wonder if PyTorch will automatically optimize loading and computation.<br>
Here is a sample of the code.</p>
<pre><code class="lang-auto">import torch
A = torch.randn(1000,1000)
B = torch.randn(1000,1000)
C = torch.randn(1000,1000)
A = A.to("cuda",non_blocking = True)
D = B.matmul(C)
</code></pre>
<p>Will <code>A = A.to("cuda",non_blocking = True)</code> and <code>D = B.matmul(C)</code> executed together?</p>

username: ptrblck
cooked: <p>No, both will be added the the queue in the default <code>CUDAStream</code> and executed sequentially. You could use custom <code>CUDAStream</code>s to execute kernels in parallel (if possible) but would need to take care of needed synchronizations.</p>

username: HaoKang-Timmy
cooked: <p>Hi, thanks for the answer. However, I tried to use <code>cuda.stream</code> to parallize the two operations. I think it does not paralyze at all.<br>
Here is the code for synchronize.</p>
<pre><code class="lang-auto">time_counter = 0.0
A = torch.randn(10000,10000).half()
B = torch.randn(10000,10000).half().to("cuda",non_blocking = False)
D = torch.randn(10000,10000).half().to("cuda",non_blocking = False)
for i in range(10):
    
    torch.cuda.synchronize()
    start_combine = time.time()
    
    A = A.to("cuda",non_blocking=True)
    F = B.matmul(D)
    torch.cuda.synchronize()
    end_combine = time.time()
    time_counter += end_combine - start_combine
    print("combine time: ", end_combine - start_combine)
    A = A.to("cpu")
print("avg combine time: ", time_counter/10)

</code></pre>
<p>The <code>avg combine time</code> is 0.06434.<br>
The code for asynchronize for two streams is:</p>
<pre><code class="lang-auto">import time
import torch

time_counter = 0.0
A = torch.randn(10000, 10000).half()
B = torch.randn(10000, 10000).half().to("cuda", non_blocking=True)
D = torch.randn(10000, 10000).half().to("cuda", non_blocking=True)

# Create CUDA streams
stream1 = torch.cuda.Stream()
stream2 = torch.cuda.Stream()

for i in range(10):
    start_combine = time.time()

    with torch.cuda.stream(stream1):
        A = A.to("cuda", non_blocking=True)

    with torch.cuda.stream(stream2):
        F = B.matmul(D)

    torch.cuda.synchronize()
    end_combine = time.time()
    print("combine time: ", end_combine - start_combine)
    A = A.to("cpu")
print("avg combine time: ", time_counter / 10)

</code></pre>
<p>The avg time is 0.06712.<br>
I do not think there is paralyzing on my gpu.</p>

username: HaoKang-Timmy
cooked: <p>Also I have tested for loading and computation each<br>
Code for computation.</p>
<pre><code class="lang-auto">time_counter = 0.0

for i in range(10):
    B = torch.randn(10000,10000).half().to("cuda")
    C = torch.randn(10000,10000).half().to("cuda")


    # torch.cuda.synchronize()
    start_compute = time.time()
    E = B.matmul(C)
    torch.cuda.synchronize()
    end_compute = time.time()
    time_counter += end_compute - start_compute
    print("compute time: ", end_compute - start_compute)
    del E
print("avg compute time: ", time_counter/10)

</code></pre>
<p>The avg time is 0.03173</p>
<p>Code for loading</p>
<pre><code class="lang-auto">import torch
import time

time_counter = 0.0
for i in range(10):
    A = torch.randn(10000,10000).half()
    torch.cuda.synchronize()
    start_load = time.time()
    A = A.to("cuda")
    torch.cuda.synchronize()
    end_load = time.time()
    print("load time: ", end_load - start_load)
    A = A.to("cpu")
    del A
    time_counter += end_load - start_load
print("average load time: ", time_counter/10)
</code></pre>
<p>The avg time is 0.03534</p>

username: ptrblck
cooked: <p>Profile the code via <code>Nsight Systems</code> and check if kernels are overlapping and by how much.</p>