Slow data loading - every couple of batch

RahimEntezari: <p>Hi<br>
I am trying to use torchrun --nproc_per_node=8 to train SimCLR on ImageNet using 8 GPUs in parallel. I am using this command to distribute the model</p>
<p><code>model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], bucket_cap_mb=200)</code><br>
The problem is that every couple of batches the data loading is very slow. This is not only for the first batch, which I guess is normal.<br>
Here I paste the for loop to train and print data and optimization times.</p>
<pre><code class="lang-auto"> end = time.time()
    for data_iter, inputs in enumerate(train_loader):
        optim_iter = data_iter // args.update_freq
        # measure data loading time
        data_time.update(time.time() - end)
        data_time_ = time.time() - end

        # update weight decay and learning rate according to their schedule
        it = iters_per_epoch * epoch + optim_iter  # global training iteration
        for k, param_group in enumerate(optimizer.param_groups):
            param_group['lr'] = lr_schedule[it]

        inputs = [tensor.cuda(args.gpu, non_blocking=True) for tensor in inputs]

        # compute output
        with amp.autocast(enabled=not args.disable_amp):
            outputs = model(*inputs)
            loss_dict = criterion(outputs)
            loss = loss_dict['loss']
            loss /= args.update_freq

        scaler.scale(loss).backward()

        if (data_iter + 1) % args.update_freq != 0:
            continue

        # compute gradient and do SGD step
        scaler.step(optimizer)
        scaler.update()
        model.zero_grad(set_to_none=True)


        # clamp logit scale to [0, 100]
        if args.model.startswith('SIMCLR'):
            logit_scale = 0
        else:
            utils.get_model(model).logit_scale.data.clamp_(0, 4.6052)
            logit_scale = utils.get_model(model).logit_scale.exp().item()

        for k in loss_dict:
            metrics[k].update(loss_dict[k].item(), args.batch_size)

        # measure elapsed time
        batch_time.update(time.time() - end)
        batch_time_ = time.time() - end
        end = time.time()

        print(data_time_, batch_time_)
</code></pre>
<p>Here is the (data_time_, batch_time_) when j=8</p>
<blockquote>
<p><strong>142.89722776412964 172.6102113723755</strong><br>
0.0011818408966064453 1.149657964706421<br>
0.0005125999450683594 0.5908513069152832<br>
0.000789642333984375 0.5789885520935059<br>
0.0006721019744873047 0.5847604274749756<br>
0.0006527900695800781 0.5850253105163574<br>
0.0006945133209228516 0.5916690826416016<br>
0.0006604194641113281 0.5858132839202881<br>
<strong>33.176689863204956 113.46687459945679</strong><br>
0.0006616115570068359 3.237961530685425<br>
0.0004947185516357422 0.5849909782409668<br>
0.0006363391876220703 0.5738029479980469<br>
0.0005011558532714844 0.5823519229888916<br>
0.0004696846008300781 0.5900559425354004<br>
0.0006518363952636719 0.5800015926361084<br>
0.0006239414215087891 0.5876047611236572<br>
0.0006771087646484375 <strong>70.80163884162903</strong><br>
0.0009722709655761719 0.5865006446838379<br>
0.0006647109985351562 0.5858757495880127<br>
0.0006537437438964844 1.4045929908752441<br>
0.0006687641143798828 <strong>32.53645730018616</strong><br>
0.0008571147918701172 0.5862514972686768<br>
0.0006554126739501953 0.5853633880615234<br>
0.0006177425384521484 16.638994216918945<br>
0.0009629726409912109 <strong>66.59664487838745</strong><br>
0.001010894775390625 0.588956356048584<br>
0.0006909370422363281 0.5857172012329102<br>
0.0006442070007324219 0.5854201316833496<br>
0.0006425380706787109 <strong>29.423442363739014</strong><br>
0.0008411407470703125 0.5904080867767334<br>
0.0007281303405761719 0.5878152847290039<br>
0.0007319450378417969 <strong>47.11639881134033</strong><br>
0.0008819103240966797 23.894486904144287<br>
0.0007138252258300781 0.578960657119751<br>
0.0005040168762207031 0.5796892642974854<br>
0.0004954338073730469 0.5855348110198975<br>
0.0004782676696777344 <strong>66.34711623191833</strong><br>
0.0006091594696044922 0.5864002704620361<br>
0.0005247592926025391 0.5784976482391357<br>
0.0005164146423339844 0.6909780502319336<br>
0.0006034374237060547 <strong>26.061028718948364</strong><br>
0.0008776187896728516 0.5903408527374268<br>
0.0006973743438720703 0.584754467010498<br>
0.0006537437438964844 0.5849916934967041<br>
…<br>
<strong>36.196861028671265 36.79169154167175</strong><br>
0.0008547306060791016 0.5894830226898193<br>
0.0008087158203125 0.5778903961181641<br>
0.0006210803985595703 0.5889377593994141<br>
0.0008003711700439453 0.5878915786743164<br>
0.00077056884765625 0.5870087146759033<br>
0.0007855892181396484 <strong>93.86848998069763</strong><br>
0.0007998943328857422 0.5807287693023682<br>
<strong>17.311529874801636 17.90171504020691</strong><br>
0.0007803440093994141 9.284274816513062<br>
0.0008406639099121094 0.5794563293457031<br>
0.0008251667022705078 0.6089217662811279<br>
0.00078582763671875 0.5598442554473877<br>
0.0007565021514892578 0.5864059925079346<br>
0.0007340908050537109 42.826006174087524<br>
0.0010673999786376953 0.5904500484466553<br>
<strong>23.019705295562744 59.32295536994934</strong><br>
0.0007565021514892578 31.347289085388184<br>
0.0006775856018066406 0.5731685161590576<br>
0.0007195472717285156 0.5763015747070312<br>
0.0005919933319091797 0.5776708126068115<br>
0.0005700588226318359 0.5778248310089111<br>
0.0006148815155029297 7.108304738998413<br>
0.0005848407745361328 0.5788106918334961<br>
0.0006554126739501953 <strong>32.21546387672424</strong><br>
0.0007257461547851562 <strong>88.52377581596375</strong><br>
0.0008158683776855469 0.5769295692443848</p>
</blockquote>
<p>I also noticed that although GPUs are 100% utilized but their power usage are around 80/350 W.</p>

smth: <p>i wonder if you are simply bottlenecked by data loading.<br>
Can you try to switch the data loading to something like FFCV and see if the issue persists: <a href="https://github.com/libffcv/ffcv" class="inline-onebox">GitHub - libffcv/ffcv: FFCV: Fast Forward Computer Vision (and other ML workloads!)</a></p>