Training speed is so slow. (GPU volatile is so low)

username: wonchulSon
cooked: <pre><code class="lang-auto">class MyCustomDataset():
    def __init__(self, cropped_1x32_dataset, targets, transforms=None):
        for i in range(32):
            self.__setattr__('data_{}'.format(i), cropped_1x32_dataset[i])
        self.targets = targets
        self.transforms = transforms
        
    def __getitem__(self, index):
        for i in range(32):
            globals()['data_{}'.format(i)] = self.__getattribute__('data_{}'.format(i))[index]
        y = self.targets[index]
        data = [globals()['data_{}'.format(i)] for i in range(32)]
        if self.transforms is not None:
            for i in range(32):
                data[i] = self.transforms(ToPIL(data[i]))
        return data,y
    
    def __len__(self):
        return len(self.data_0)
</code></pre>
<pre><code class="lang-auto">transform_train = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.4914, 0.4824, 0.4467),
                             std=(0.2471, 0.2436, 0.2616)),
])
</code></pre>
<pre><code class="lang-auto"># train
my_train_dataset = MyCustomDataset(train_cropped_1x32_dataset, train_dataset.targets, transform_train)
my_train_loader = torch.utils.data.DataLoader(dataset = my_train_dataset,
                                              batch_size = batch_size,
                                              shuffle = True,
                                              num_workers=os.cpu_count())
</code></pre>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = '1'
batch_size = 128
learning_rate = 0.1

device = 'cuda'
model = Vgg16()
model.to(device)
optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss()

if torch.cuda.device_count() &gt; 0:
    print("USE", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
    cudnn.benchmark = True
else:
    print("USE ONLY CPU!")

if torch.cuda.is_available():
    model.cuda()
</code></pre>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    train_loss = 0
    total = 0
    correct = 0
    
    for batch_idx, (cropped_1x32_dataset, target) in enumerate(my_train_loader):
        for i in range(32):   
            cropped_1x32_dataset[i] = cropped_1x32_dataset[i].to(device)
        target = target.to(device)

        optimizer.zero_grad()
        output = model(cropped_1x32_dataset, return_features=False)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        train_loss += loss.data
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += predicted.eq(target.data).cpu().sum()
        if batch_idx % 10 == 9:
            print('Epoch: {} | Batch: {} |  Loss: ({:.4f}) | Acc: ({:.2f}%) ({}/{})'
                  .format(epoch+1, batch_idx+1, train_loss/(batch_idx+1), 100.*correct.item()/total, correct, total))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/38d5f6ad4f0c2dcc1fa0431f49953f1c276a2eb3.png" data-download-href="https://discuss.pytorch.org/uploads/default/38d5f6ad4f0c2dcc1fa0431f49953f1c276a2eb3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/38d5f6ad4f0c2dcc1fa0431f49953f1c276a2eb3.png" alt="image" data-base62-sha1="86N9uV7PzuNZBPEBOd73Uc58lKH" width="690" height="303" data-dominant-color="171716"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">940×414 16.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m working on <code>os.environ["CUDA_VISIBLE_DEVICES"] = '1'</code><br>
But as you can see, when I check <code>nvidia-smi</code> almost everytime volatile shows 0%.<br>
Sometimes I shows upto 10~30%. So training speed is so slow.</p>
<p>How can I use full GPU power to speed up train?<br>
Thank you.</p>

username: ptrblck
cooked: <p>Could you time your data loading as done in the <a href="https://github.com/pytorch/examples/blob/5b1f45057dc14a5e2132b45233c258a1dc2a0aab/imagenet/main.py#L274" rel="nofollow noopener">ImageNet example</a>?<br>
I’m wondering why you are using <code>globals()</code> inside your <code>Dataset</code> as I would assume this could slow down your data loading pipeline.</p>