Training fast with small dataset, slow with large dataset

username: vibe2
cooked: <p>I am training a GAN drawing samples from LMDBs. I have multiple LMDBs - I create a dataset for each and concatenate them to make the final one. I was initially training with about 4K samples split across 8 LMDBs. I recently scaled up to about 60K samples, split across 20 LMDBs. I noticed significant slowing in the training when increasing the dataset (after 5 epochs, as I know sometimes the initial epoch is slow due to caching &amp; initialization etc.).</p>
<p>I’m trying to figure out why the training speed (images/second) is 5-10x slower per iteration when using the large dataset than on the smaller one. I observed as I reduce the size of the training set, the training speed progressively gets faster and faster until reaching the expected speed. Using up to ~30K images shows no slowness, and the speeds drop dramatically from there. It doesn’t appear to matter which images are in the reduced dataset - as long as the number of images &gt;30Kish, the training speed drops dramatically.</p>
<p>I’m training on 4 V100’s on a single node, using DataParallel() and apex AMP. Using PyTorch 1.4. My minibatches are ~96. Increasing num_workers beyond 6 doesn’t seem to help. Whether I use 1, 2, or 4 GPUs at a time, I still observe the “slower training with more data” phenomenon. The only difference between fast training and slow training in my testing is the number of images included in the dataset. Any suggestions appreciated. Thanks!</p>

username: Henry_Chibueze
cooked: <p>Try and visualise this : .<br>
Imagine u create a for loop that iterates through an array X, computes the negative exponent of the square of each values in the array and append the result to a new array, if the values in array X are a few hundreds the process will run for few microseconds but assuming the number of values in array X was a hundred million then it might even take more than a minute to compute coz it will have to iterates through the array X hundred million times.</p>
<p>So the thing is when ur data set increased to 60k with a minibatch of 96 ur training loop will iterate through the dataset 60k/96 times which constitutes one epoch of training unlike when it’s 30k datasets where it will be 30k/96 times constituting one epoch</p>
<p>Also Incase u are wondering y the difference in time of training with 4k dataset is almost same with 10k or 15k dataset but increases drastically when training with more than 30k, it’s literally the same thing that happens when u make a for loop iterate an array X when the array size is 100 or 1000. The thing is to the human eye the timing looks the same but to a machine there’s significant difference and when the size of the data increases drastically it starts to get significant to the eyes of a human. Try using the time module to get the start and stop time of the iteration when using different datasets and you’ll see the difference.<br>
Hope this helped.</p>

username: vibe2
cooked: <p>Hi Henry, thanks for looking at this! My issue is related to seconds per iteration getting slower with more data, not each epoch taking longer with more data (which is to be expected).</p>
<p>I do use the time module to profile the training speed and observe the 5-10x slowness with more data. Hope that helps clarify the issue.</p>

username: Henry_Chibueze
cooked: <p>This could be due to the fact that variable holding the training data is consuming alot of cpu memory in the background and hence making it some worth difficult for computation in the training process<br>
It’s like trying to copy a file of 2GB in a hard drive with only 2.1GB of free space compared to a hard drive with 6GB free space<br>
If u’ve coded in assembly b4 u’ll most likely know what I’m talking about.<br>
Hope this helps.</p>

username: vibe2
cooked: <p>My machine has several hundred GB of free RAM during training so I do not think that is a factor…</p>
<p>Additional tests I have tried: switching to PyTorch 1.6 and using DistributedDataParallel() did not help significantly</p>

username: Henry_Chibueze
cooked: <p>I still think it’s  memory consumption of the training data as it gets bigger</p>

username: jspisak
cooked: <p>Hey there - have you entered a GitHub issue yet? <a class="mention" href="/u/mcarilli">@mcarilli</a> - can you take a look?</p>

username: mcarilli
cooked: <p>Sounds to me like some OS memory management issue (ie data swapping to/from disk).  The fact that you have several hundred GB of free RAM during training is strange, though.  How are you measuring the free RAM?</p>
<p>To sandbox the issue, do you observe similar symptoms when you don’t use Amp and/or DataParallel at all?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/vibe2">@vibe2</a></p>
<p>I have some questions regarding the perf measurements.</p>
<ol>
<li>Have you profiled which step is the bottleneck in every iteration? Is it data loading, or forward, or backward, or optimizer?</li>
<li>For GPU training, <code>time.time()</code> might not give the accurate measure, as CUDA ops return immediately after it is added to the stream. To get more accurate numbers, <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc"><code>elapsed_time</code></a> is a better option. The following post can serve as an example:</li>
</ol>
<aside class="quote quote-modified" data-post="8" data-topic="93865">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/distributed-data-parallel-slower-than-data-parallel/93865/8">Distributed Data Parallel slower than Data Parallel</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Hey <a class="mention" href="/u/tt_yy">@TT_YY</a>, I took a closer look at the code and noticed that you converted BatchNorm to SyncBatchNorm for DDP, which might be the source of the slowness. If you look at SyncBatchNorm's implementation (see below), it launches its own communication, which is not handled by DDP. This additional comm leads to ~10% slowdown in your program when running on 2 GPUs. When I use BatchNorm instead of SyncBatchNorm, DDP is faster than DP. In general, when comparing DDP and DP speed, we need to make sure th…
  </blockquote>
</aside>


username: vibe2
cooked: <p>Thanks everybody for the reply - I’m doing some more detailed profiling based on your suggestions and will get back to you shortly!</p>

username: vibe2
cooked: <p><strong>Background:</strong></p>
<ul>
<li>This is on an nvidia DGX2. I confirmed that all data is cached on the NVMe cache (as the cache does not grow as training progresses)</li>
<li>I removed DataParallel()/DDP()/AMP, and am training on one GPU as <a class="mention" href="/u/mcarilli">@mcarilli</a> suggested</li>
<li>All hyperparams including minibatch and n_workers for the dataloader are kept the same for all tests</li>
<li>I invoke time.time() after each iteration to get the seconds per iteration.</li>
</ul>
<p><strong>Profiling results:</strong></p>
<ul>
<li>
<p>While training with a small dataset (4k samples), it takes 1.2 seconds per iteration, and that speed is consistent after tens of thousands of iterations</p>
</li>
<li>
<p>While training with a large dataset (65k samples), it takes an average of 2 seconds per iteration. The way the slowness manifests is a handful of “fast” iteration at 1.2 seconds/iter, followed by a slow iteration that takes 4-10 seconds. It averages out close to 2 seconds/iter. When scaling up to many GPUs, the speed penalty (average seconds/iteration) with the large dataset is 5x or more (and only a negligible speed penalty with a small dataset).</p>
</li>
<li>
<p>To profile the RAM, I am calling the <code>watch free</code> command. It stays fairly consistent (doesn’t blow up as training progresses):</p>
</li>
</ul>
<pre><code class="lang-auto">              total        used        free      shared  buff/cache   available
Mem:     1583405972    32676452     1541340      351212  1549188180  1548865308
Swap:       3904508      279808     3624700
</code></pre>
<ul>
<li>
<p>I imagine the slowness is from loading the data, but I will try the elapsed_time() profiler as well as <a class="mention" href="/u/mrshenli">@mrshenli</a> suggested</p>
</li>
<li>
<p><a class="mention" href="/u/jspisak">@jspisak</a> I have not yet filed a GitHub issue - let me know if I should do that</p>
</li>
</ul>

username: jspisak
cooked: <p>Yes, a GitHub issue would be great. This way we can triage, track and ensure we get this taken tare of. Thanks!!</p>

username: mcarilli
cooked: <p>So it doesnt sound related to DP/DDP/Amp.   Have you tried varing the number of dataloader workers beyond 6 in the single-GPU case?  Also how big is the dataset on disk?</p>

username: vibe2
cooked: <ul>
<li>
<a class="mention" href="/u/mcarilli">@mcarilli</a> Thanks for the reply! On one GPU, no DP/DDP/Amp with 6 workers I observe the slowness with the large (65k sample) dataset, but increasing the number of workers to 12 removes the slowness (e.g. training with the large dataset is just as fast as training with the smaller dataset). Wrapping the networks with DataParallel() but keeping it at 1GPU/12 workers/minibatch=12 did not cause any slowness. When I scale up to 2 GPUs, num_workers=24 and using minibatch=24, I start to see the slowness again. The pattern of slow training is as described above - several “fast” iterations followed by a slow iteration of 5-50 seconds, and this behavior persists for tens of thousands of iters.</li>
<li>I tried using AMP on 1 GPU, effectively increasing the minibatch size from 12 to 30, and I observed the slow training with the large dataset even after increasing the number of workers to 24. With a small dataset of ~4K samples, only 3 workers are required and I get consistently fast speeds.</li>
<li>The machine has 96 CPUs and 16 GPUs so I would think 6 workers per GPU would be ideal. When working with the small dataset, only 3 workers are required to get peak training speeds.</li>
<li>The dataset is 2.8TB total spanning approx. 40 LMDBs</li>
<li>I have set <code>torch.set_num_threads(96)</code> and <code>torch.set_num_interop_threads(48)</code> at the beginning of the training script.</li>
<li>Whether I set <code>pin_memory=True</code> or <code>False</code> doesn’t change the behavior. I’m passing in the <code>SequentialSampler()</code> into the dataloader, but the slow training still happens if I set <code>sampler=False</code> and <code>shuffle=True</code>.</li>
<li>Here’s how I open each LMDB in the dataset’s <code>__init__</code> class:<br>
<code>self.env = lmdb.open(lmdbPath, readonly=True, lock=False, readahead=False, meminit=False, max_readers=512)</code>
</li>
<li>
<code>readahead=True</code> made things worse, <code>meminit</code> doesn’t appear to be a factor, and increasing <code>max_readers</code> didn’t help either</li>
<li>Here’s how I am reading imgs from the LMDB in the dataset’s <code>__getitem__</code> method:</li>
</ul>
<pre><code class="lang-auto">    with env.begin(write=False) as txn:
        buf = txn.get(path.encode('ascii'))
        buf_meta = txn.get((path + '.meta').encode('ascii')).decode('ascii')
</code></pre>
<p>Please do let me know of any suggestions! I think the key finding here is that only 3 workers are required for top training speeds with a small dataset and 1 GPU, but 12 were required with the large dataset (with same minibatch/hyperparams). With effectively larger minibatches from multi-GPU or AMP, I still saw slowness after increasing the num_workers beyond 12.</p>

username: vibe2
cooked: <p><a class="mention" href="/u/jspisak">@jspisak</a> I filed a GitHub issue here: <a href="https://github.com/pytorch/pytorch/issues/45273" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/45273</a></p>
<p>And linked to this PyTorch thread. Presumably we’ll continue to discuss here to determine if I’m doing something silly or if there might be a bug/missing feature (the former much more likely). Thanks!</p>