Unexpected behaviour of cuda events when placed after other operations

x41lakazam: <p>I’m trying to measure the time of a distributed operations using cuda events. The operation is repeated in many iterations and preceeded by a number of warmup iterations, i.e iterations that are not measured.</p>
<p>The problem is that if I add any operation (such as the warmup) between the event creation and the <code>record()</code> calls, the elapsed time between the events becomes very small.</p>
<p>Here is a simple example to measure all_to_all operation, without the warmup part, in this example everything works fine and elapsed_time is very similar to elapsed_events. The problem occurs when I add the warmup part.</p>
<pre><code class="lang-auto">group = dist.new_group(backend='nccl')
stream = torch.cuda.Stream()
iters = 50
iters_events_begin = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
iters_events_end = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
src = torch.zeros(size, dtype=torch.int8)
dst = torch.zeros(size, dtype=torch.int8)

s = time.time()   # This is added to verify the accuracy of events measures
with torch.cuda.stream(stream):

    for i in range(iters):
        iters_events_begin[i].record()
        dist.all_to_all_single(dst, src, group=group, async_op=True)
        iters_events_end[inner_iter_ix].record()

torch.cuda.synchronize()

elapsed_time = time.time() - s
elapsed_events = sum(
    [b.elapsed_time(e) for b,e in zip(iters_events_begin, iters_events_end)]
)
print(f"Total elapsed using time: {elapsed_time*1000} ms")
print(f"Total elapsed using events: {elapsed_events} ms")
</code></pre>
<p>Here is the script updated to add warmup, the results are completely different, while the time measure the accurate execution time, the time measured by the events is very small (less than 0.1 ms against several seconds for the other one)</p>
<pre><code class="lang-auto">group = dist.new_group(backend='nccl')
stream = torch.cuda.Stream()
iters = 50
iters_events_begin = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
iters_events_end = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
src = torch.zeros(size, dtype=torch.int8)
dst = torch.zeros(size, dtype=torch.int8)

s = time.time()   # This is added to verify the accuracy of events measures
with torch.cuda.stream(stream):

    for i in range(warmup):
        req = dist.all_to_all_single(dst, src, group=group, async_op=True)
    
    dist.barrier(group=group, async_op = True)

    for i in range(iters):
        iters_events_begin[i].record()
        dist.all_to_all_single(dst, src, group=group, async_op=True)
        iters_events_end[inner_iter_ix].record()

torch.cuda.synchronize()

elapsed_time = time.time() - s
elapsed_events = sum(
    [b.elapsed_time(e) for b,e in zip(iters_events_begin, iters_events_end)]
)
print(f"Total elapsed using time: {elapsed_time*1000} ms")
print(f"Total elapsed using events: {elapsed_events} ms")
</code></pre>
<p>If I wrap the warmup line with events recording then it works again.</p>
<p>I don’t understand this behaviour and would love some help.</p>

agu: <p><code>ProcessGroupNCCL</code> (PGNCCL) runs the collective kernel in a separate process group internal stream. When you call the c10d collective, the PG’s stream waits for the current stream, and the kernel enqueues to the PG’s stream. If <code>async_op=True</code>, then the c10d collective returns a <code>Work</code> object as a handle on which you can call <code>wait()</code>. Calling <code>wait()</code> has the current stream wait for the PG’s stream. If you <code>async_op=False</code>, then the c10d collective calls <code>wait()</code> for the user immediately.</p>
<p><code>elapsed_time</code> is measuring the CPU time. <code>elapsed_events</code> is also mostly measuring the CPU time since the collective itself since the end event is recorded before the current stream (in which the event is recorded) waits on the PG’s stream (by calling <code>.wait()</code>).</p>
<p>I am not sure how the times look for you after you include the warmup, but I am not sure if this timing code looks right regardless. I think you need to call <code>.wait()</code> at some point before you record your end event, or else the end event may not capture the moment that the collective actually finished.</p>