Accumulate Gradient accelerator

Shia_Edwards: <pre><code class="lang-auto">    accelerator = Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps) 
    effective_batch_size = args.batch_size // args.gradient_accumulation_steps

    data_loader = torch.utils.data.DataLoader(
        dataset,
        batch_size=effective_batch_size,
        shuffle=True,
        num_workers=4,
        pin_memory=True,
        drop_last=True,
    )

    for epoch in range(init_epoch, args.num_epoch + 1):
        #model.train()
        for iteration, (x, y) in enumerate(data_loader):
            x_0 = x.to(device, dtype=dtype, non_blocking=True)
            y = None if not use_label else y.to(device, non_blocking=True)
            #model.zero_grad()
            if is_latent_data:
                z_0 = x_0 * args.scale_factor
            else:
                z_0 = first_stage_model.encode(x_0).latent_dist.sample().mul_(args.scale_factor)
            # sample t
            t = torch.rand((z_0.size(0),), dtype=dtype, device=device)
            t = t.view(-1, 1, 1, 1)
            z_1 = torch.randn_like(z_0)
            # 1 is real noise, 0 is real data
            z_t = (1 - t) * z_0 + (1e-5 + (1 - 1e-5) * t) * z_1
            u = (1 - 1e-5) * z_1 - z_0
            # estimate velocity
            v = model(t.squeeze(), z_t, y)
            loss = F.mse_loss(v, u)     
            with accelerator.accumulate(model):
              loss = loss.mean()
              accelerator.backward(loss)   
            optimizer.step()
            scheduler.step()
            model.zero_grad()
            global_step += 1
            log_steps += 1
            optimizer.zero_grad()
</code></pre>
<p>It seems like I haven’t successfully invoked gradient accumulation. If I understand correctly, if I decrease the batch size from 128 to 32 and set gradient accumulation to 4, a typical GPU should be able to run it, especially in Colab.</p>
<p>However, if it still fails to run, it likely indicates that gradient accumulation hasn’t been correctly invoked. Although my program runs successfully, I believe it’s related to this line of code: <code>effective_batch_size = args.batch_size // args.gradient_accumulation_steps</code>, as it effectively reduces the batch size, which is unrelated to gradient accumulation.</p>
<p>I’m unsure if this step is necessary or how to modify it to correctly invoke gradient accumulation. Some have suggested that I comment out <code>#model.zero_grad()</code> inside the for loop because it effectively clears all gradients. However, commenting it out doesn’t seem to have any effect.</p>
<p>ref：</p><aside class="onebox githubblob" data-onebox-src="https://github.com/VinAIResearch/LFM/blob/main/train_flow_latent.py">
  <header class="source">

      <a href="https://github.com/VinAIResearch/LFM/blob/main/train_flow_latent.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/VinAIResearch/LFM/blob/main/train_flow_latent.py" target="_blank" rel="noopener nofollow ugc">VinAIResearch/LFM/blob/main/train_flow_latent.py</a></h4>


      <pre><code class="lang-py"># ---------------------------------------------------------------
# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
#
# This work is licensed under the NVIDIA Source Code License
# for Denoising Diffusion GAN. To view a copy of this license, see the LICENSE file.
# ---------------------------------------------------------------

import argparse
import os
import shutil
from functools import partial
from time import time

import torch
from omegaconf import OmegaConf

import torch.nn.functional as F
import torch.optim as optim
import torchvision
from accelerate import Accelerator
</code></pre>



  This file has been truncated. <a href="https://github.com/VinAIResearch/LFM/blob/main/train_flow_latent.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote quote-modified" data-post="3" data-topic="196897">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shia_edwards/48/58210_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/performing-gradient-accumulation-with-accelerate/196897/3">Performing gradient accumulation with Accelerate</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Thank you so much. I have try to correct it. but still same… 
for epoch in range(init_epoch, args.num_epoch + 1):
    for iteration, (x, y) in enumerate(data_loader):
        x_0 = x.to(device, dtype=dtype, non_blocking=True)
        y = None if not use_label else y.to(device, non_blocking=True)
        #model.zero_grad() **delete here**
        if is_latent_data:
            z_0 = x_0 * args.scale_factor
        else:
            z_0 = first_stage_model.encode(x_0).latent_dist.sample().mul_(arg…
  </blockquote>
</aside>
<aside class="onebox discoursetopic" data-onebox-src="https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/3">
  <header class="source">
      <img src="https://global.discourse-cdn.com/hellohellohello/optimized/1X/67a2c0590affeba7880ebeb46a115d863972d8ba_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/3" target="_blank" rel="noopener nofollow ugc" title="04:07AM - 04 March 2024">Hugging Face Forums – 4 Mar 24</a>
  </header>

  <article class="onebox-body">
    <img src="https://global.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png" class="thumbnail onebox-avatar" width="500" height="500">

<div class="title-wrapper">
  <h3><a href="https://discuss.huggingface.co/t/performing-gradient-accumulation-with-accelerate/72836/3" target="_blank" rel="noopener nofollow ugc">Performing gradient accumulation with Accelerate</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #F7941D;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">🤗Accelerate</span>
        </span>
      </span>
  </div>
</div>

  <p>accelerator.accumulate(model)             loss = loss.mean()             # change above from here             accelerator.backward(loss)             optimizer.step()             scheduler.step()             model.zero_grad()            ...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>