How to fix profiler "assert not duplicate" error?

Anh_Pham: <p>Hi,</p>
<p>I am testing out the torch.profiler for the first time with BERT fine-tuning</p>
<pre><code class="lang-auto">def trace_handler(prof):
    timestamp = datetime.now().strftime("%b_%d_%H_%M_%S")
    prof.export_chrome_trace(f"trace_{timestamp}.json")
    prof.export_memory_timeline(f"memory_{timestamp}.html", device="cuda:0")

training_losses = []

progress_bar = tqdm(range(num_training_steps))

active_steps = 10
repeat_cycles = 1  
total_profiled_steps = active_steps * repeat_cycles 
current_step = 1

model.train()
with profile(
    activities=[
        ProfilerActivity.CPU,
        ProfilerActivity.CUDA
    ],
    schedule=torch.profiler.schedule(wait=0, warmup=0, active=active_steps, repeat=repeat_cycles),
    on_trace_ready=trace_handler,
    record_shapes=True,
    profile_memory=True,
    with_stack=True,
) as prof:
    for batch in train_dataloader:
        if current_step &gt;= total_profiled_steps:
            break
        prof.step()  
        current_step += 1

        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)

        training_losses.append(loss.item())

</code></pre>
<p>I only wanted to run the profiler for a few steps and stop, but I always got this error at the last step<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5" title="Error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5_2_690x194.png" alt="Error" data-base62-sha1="969qcEcsttMWlAFSaEQTiX7Q0GF" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5_2_690x194.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5_2_1035x291.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc597ec1e5259a7157122d19ef9c1ca3e7e46c5_2_1380x388.png 2x" data-dominant-color="3E3E3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Error</span><span class="informations">2660×750 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The last assertionError sometimes changes (atten:sum, atten:to, etc), but the first one would always be the ‘assert not duplicate’ line. I couldn’t really find any similar errors online, so not sure what is happening.</p>
<p>I am using Google Colab (GPU T4) with batch_size=4 and fine-tuning on the task of multiple choices. I have tried changing batch sizes, using <em>torch.cuda.synchronize()</em>, changing the active_steps numbers, but the error persisted.</p>
<p>Does anyone know what is happening, and where should I read more about it?</p>

soulitzer: <p>Does this reproduce if you only run a single iteration? What about just two iterations?<br>
Do you have a self-contained repro that can reproduce this?</p>

Anh_Pham: <p>When trying to reproduce the error with different iterations:</p>
<ul>
<li>If set <em>active_steps &lt;= 3</em>, then the error does NOT occur, otherwise it will happen from 4 onwards</li>
<li>If set <em>active_steps &gt; 3</em> (for example, with 10), and I try to break after a few iterations, then the error occurs from step 3 onwards (breaking at 1 or 2 iterations seems fine)</li>
</ul>
<p>I tried reproducing the same error with a simpler dataset (just randomize some numbers) and model (2-layer neural network), but it did not occur. So I do not have a small and concise self-contained repro that can reproduce this.</p>
<p>The full code is for fine-tuning BERT on multiple choices, which is longer and uses a larger dataset. I’m unsure whether it is the dataset (COPA dataset) problem or the huggingface fine-tuning process problem. Do you have any suggestions for debugging this further?</p>

soulitzer: <p>Thanks for trying to making a smaller reproducer. I’m not too sure what is going on unfortunately, but it seems unexpected, and we should at a mininum improve the error message.</p>