Performing gradient accumulation with Accelerate

username: Shia_Edwards
cooked: <h1><a name="guide-by-this-page-performing-gradient-accumulation-with-accelerate-1" class="anchor" href="#guide-by-this-page-performing-gradient-accumulation-with-accelerate-1"></a>Guide by this page : Performing gradient accumulation with  Accelerate</h1>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/accelerate/usage_guides/gradient_accumulation#performing-gradient-accumulation-with--accelerate">
  <header class="source">

      <a href="https://huggingface.co/docs/accelerate/usage_guides/gradient_accumulation#performing-gradient-accumulation-with--accelerate" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://huggingface.co/front/thumbnails/docs/accelerate.png" class="thumbnail">

<h3><a href="https://huggingface.co/docs/accelerate/usage_guides/gradient_accumulation#performing-gradient-accumulation-with--accelerate" target="_blank" rel="noopener nofollow ugc">Performing gradient accumulation with 🤗 Accelerate</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My code :</p>
<pre><code>for epoch in range(init_epoch, args.num_epoch + 1):
    for iteration, (x, y) in enumerate(data_loader):
        x_0 = x.to(device, dtype=dtype, non_blocking=True)
        y = None if not use_label else y.to(device, non_blocking=True)
        model.zero_grad()
        if is_latent_data:
            z_0 = x_0 * args.scale_factor
        else:
            z_0 = first_stage_model.encode(x_0).latent_dist.sample().mul_(args.scale_factor)
        # sample t
        t = torch.rand((z_0.size(0),), dtype=dtype, device=device)
        t = t.view(-1, 1, 1, 1)
        z_1 = torch.randn_like(z_0)
        # 1 is real noise, 0 is real data
        z_t = (1 - t) * z_0 + (1e-5 + (1 - 1e-5) * t) * z_1
        u = (1 - 1e-5) * z_1 - z_0
        # estimate velocity
        v = model(t.squeeze(), z_t, y)
        loss = loss.mean()            
        accelerator.backward(loss)

        #change below
        if (iteration + 1) % args.gradient_accumulation_steps == 0:
            optimizer.step()
            scheduler.step()
            
            # clear gradient
            model.zero_grad() 
            global_step += 1
            log_steps += 1
        #change above from here

        if iteration % 100 == 0:
            if accelerator.is_main_process:
                # Measure training speed:
                end_time = time()
                steps_per_sec = log_steps / (end_time - start_time)
                accelerator.print(
                    "epoch {} iteration{}, Loss: {}, Train Steps/Sec: {:.2f}".format(
                        epoch, iteration, loss.item(), steps_per_sec
                    )
                )
                # Reset monitoring variables:
                log_steps = 0
                start_time = time()
</code></pre>
<p>It seems that there is no effect of gradient accumulation = 2 at all.</p>

username: minhnh
cooked: <aside class="quote no-group" data-username="Shia_Edwards" data-post="1" data-topic="196897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shia_edwards/48/58210_2.png" class="avatar"> Shia_Edwards:</div>
<blockquote>
<p><code>x_0 = x.to(device, dtype=dtype, non_blocking=True)</code><br>
<code>y = None if not use_label else y.to(device, non_blocking=True)</code><br>
<code>model.zero_grad()</code></p>
</blockquote>
</aside>
<p>This is why you have the effect of gradient accumulation. You clear the gradients at every step.</p>

username: Shia_Edwards
cooked: <p>Thank you so much. I have try to correct it. but still same…</p>
<pre><code>for epoch in range(init_epoch, args.num_epoch + 1):
    for iteration, (x, y) in enumerate(data_loader):
        x_0 = x.to(device, dtype=dtype, non_blocking=True)
        y = None if not use_label else y.to(device, non_blocking=True)
        #model.zero_grad() **delete here**
        if is_latent_data:
            z_0 = x_0 * args.scale_factor
        else:
            z_0 = first_stage_model.encode(x_0).latent_dist.sample().mul_(args.scale_factor)
        # sample t
        t = torch.rand((z_0.size(0),), dtype=dtype, device=device)
        t = t.view(-1, 1, 1, 1)
        z_1 = torch.randn_like(z_0)
        # 1 is real noise, 0 is real data
        z_t = (1 - t) * z_0 + (1e-5 + (1 - 1e-5) * t) * z_1
        u = (1 - 1e-5) * z_1 - z_0
        # estimate velocity
        v = model(t.squeeze(), z_t, y)
        loss = F.mse_loss(v, u)
        loss = loss.mean()
        accelerator.backward(loss)

        # change below
        if (iteration + 1) % args.gradient_accumulation_steps == 0:
            optimizer.step()
            scheduler.step()
            global_step += 1
            log_steps += 1
            optimizer.zero_grad()
</code></pre>
<p>another one advice me to use accumulation wrapper . I am so confused now…<br>
<a href="https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836" class="inline-onebox" rel="noopener nofollow ugc">Performing gradient accumulation with Accelerate - 🤗Accelerate - Hugging Face Forums</a></p>