Same GPU, same pytorch training script. Training time sometimes fast, sometimes slow

Lizhen_Ji: <p>Hi, I have a problem when training pytorch programs, it is a simple face classification task trained with cross entropy loss. The images of the same person is in the same folder. There are 320000 images in 7000 folders.</p>
<p>I run the same python script one the same machine(ubuntu-16.04, one GTX-1080ti, pytorch-1.8, python-3.6, cuda-10.2)</p>
<p>The following are the screen shot of different runs:<br>
Fast run: (about 5 iters/s)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85219deed563b1e21a9b3f9993ca0eae831783f8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/85219deed563b1e21a9b3f9993ca0eae831783f8" title="fast-top"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85219deed563b1e21a9b3f9993ca0eae831783f8_2_690x108.jpeg" alt="fast-top" data-base62-sha1="iZJwHI9CxIsPFV4XajFeENbZRqM" width="690" height="108" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85219deed563b1e21a9b3f9993ca0eae831783f8_2_690x108.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85219deed563b1e21a9b3f9993ca0eae831783f8.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85219deed563b1e21a9b3f9993ca0eae831783f8.jpeg 2x" data-dominant-color="5A3F50"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fast-top</span><span class="informations">727×114 39.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b449829e1fb1a07d3b344905c019c9119cc7967.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2b449829e1fb1a07d3b344905c019c9119cc7967" title="fast-log"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b449829e1fb1a07d3b344905c019c9119cc7967_2_690x139.jpeg" alt="fast-log" data-base62-sha1="6aLuD3d9ZACLURzAxTcEJJJ3R9d" width="690" height="139" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b449829e1fb1a07d3b344905c019c9119cc7967_2_690x139.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b449829e1fb1a07d3b344905c019c9119cc7967.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b449829e1fb1a07d3b344905c019c9119cc7967.jpeg 2x" data-dominant-color="482B3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fast-log</span><span class="informations">942×190 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Slow run:(about 1 iters/s)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed06c906937105958252a5d3e2ad2fed5a2723f7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed06c906937105958252a5d3e2ad2fed5a2723f7" title="slow-top"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed06c906937105958252a5d3e2ad2fed5a2723f7_2_690x109.jpeg" alt="slow-top" data-base62-sha1="xOPHgvclbQVq0efSnB9GC8Nsb8H" width="690" height="109" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed06c906937105958252a5d3e2ad2fed5a2723f7_2_690x109.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed06c906937105958252a5d3e2ad2fed5a2723f7.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed06c906937105958252a5d3e2ad2fed5a2723f7.jpeg 2x" data-dominant-color="5B4051"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">slow-top</span><span class="informations">717×114 39.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff862b04b7f3d3206b03d4669f3a7f54623d7b52.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ff862b04b7f3d3206b03d4669f3a7f54623d7b52" title="slow-log"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff862b04b7f3d3206b03d4669f3a7f54623d7b52_2_690x188.jpeg" alt="slow-log" data-base62-sha1="AstcW6YhZs8fjQ551PhEUb0O8cW" width="690" height="188" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff862b04b7f3d3206b03d4669f3a7f54623d7b52_2_690x188.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff862b04b7f3d3206b03d4669f3a7f54623d7b52.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff862b04b7f3d3206b03d4669f3a7f54623d7b52.jpeg 2x" data-dominant-color="482B3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">slow-log</span><span class="informations">873×238 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I assume this is the IO problem of loading data, since <strong>D</strong> means <strong>uninterruptible sleep (usually IO)</strong></p>
<p>Here is the main code of me loading the data</p>
<pre data-code-wrap="python"><code class="lang-python">
train_transforms = T.Compose([
        T.Grayscale(), 
        T.RandomCrop(args.input_shape[1:]),
        T.RandomHorizontalFlip(),
        T.ToTensor(),
        T.Normalize(mean=[0.5], std=[0.5]),
])

train_dataset = torchvision.datasets.ImageFolder(data_dir, transform = train_transforms)

# traing_size means percentage of whole training dataset
if args.train_size != 1:
    train_idx, val_idx= train_test_split(np.arange(len(train_dataset.targets)),
                                            train_size=args.train_size,
                                            shuffle=True,
                                            random_state = args.random_state, 
                                            stratify=train_dataset.targets) # data re-shuffled at every epoch 按比例分配
    train_dataset = torch.utils.data.Subset(train_dataset, train_idx)
    trainloader = data.DataLoader(train_dataset,
                                    batch_size=args.train_batch_size,
                                    shuffle=True,
                                    num_workers=args.num_workers)
else:
    trainloader = data.DataLoader(train_dataset, batch_size=args.train_batch_size, shuffle=True, num_workers=args.num_workers)
</code></pre>
<p>I am confused with why this happens, since I am <strong>not running other programs or applications of the same time.</strong><br>
I have tried using data_prefetcher from an example from <a href="https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py" rel="noopener nofollow ugc">apex official website</a>, but it did not solve the problem.</p>
<p>Can anyone give me some insights how to solve the problem, thanks a lot</p>

Lizhen_Ji: <p>It takes me 10 days and did not find the solution to this problem. It turns out to be the “shuffle=True” cause the problem, I tried several ways<br>
(1) using data prefetcher<br>
(2) creating the dataset using h5py dataset format, which is also too slow.<br>
(3) I tried to create a dataloader before each epoch and using random index when creating the dataloader. But it still did not work.<br>
(4)  I use the random index as index to h5py file</p>
<pre><code class="lang-python">np.random.seed(i) 
np.random.shuffle(train_index)
ds = FaceLoader(H5Data, train_index)
trainloader = DataLoader(ds,
                batch_size=args.train_batch_size,
                shuffle=False,
                num_workers=args.num_workers, 
                pin_memory=True,
                drop_last=True,
                persistent_workers=True)
</code></pre>
<p>The training is okay on 1/4 of dataset(320K images) but getting slower and slower on whole dataset (12000K)</p>
<p>My CPU has 4 cores and 8 processors, My GPU is nvidia-smi</p>
<p>If someone has the same issue and solved it, please let me know how you solve it.<br>
Any help or suggesstion would be appreciated.</p>

Lizhen_Ji: <p>I finally solved the problem by installing a SSD</p>

Claudia_Giardina: <p>Hello, how do you know is the shuflle equals to True that is causing the problem?<br>
I am having really big variations in the execution times from one epoch to another and I would like to understand the cause.<br>
Whay changing to SSD would be a solution, can someone explain it further?<br>
Thanks</p>

ptrblck: <p>Random read accesses on a spinning disk are painfully slow which is why updating to an SSD could help.</p>

Claudia_Giardina: <p>Thanks for the clarification!</p>