How much does it usually takes when running VGGNet?

username: Yolkandwhite
cooked: <p>I am trying to make an VGGNet by myself with CIFAR-10 datasets.</p>
<p>and this is the model what I made</p>
<hr>
<p>CNN(<br>
(layer1): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU()<br>
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(3): ReLU()<br>
)<br>
(layer2): Sequential(<br>
(0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(2): ReLU()<br>
(3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(4): ReLU()<br>
)<br>
(layer3): Sequential(<br>
(0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(2): ReLU()<br>
(3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(4): ReLU()<br>
(5): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(6): ReLU()<br>
)<br>
(layer4): Sequential(<br>
(0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(2): ReLU()<br>
(3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(4): ReLU()<br>
(5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(6): ReLU()<br>
)<br>
(layer5): Sequential(<br>
(0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(2): ReLU()<br>
(3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(4): ReLU()<br>
(5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(6): ReLU()<br>
)<br>
(layer6): Sequential(<br>
(0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(fc1): Linear(in_features=512, out_features=256, bias=True)<br>
(fc2): Linear(in_features=256, out_features=256, bias=True)<br>
(fc3): Linear(in_features=256, out_features=10, bias=True)<br>
)</p>
<hr>
<p>It runs but It takes too much time. It takes more than 30min to do 1 epoch and I am sure that I’m using GPU.</p>
<p>Is this normal or did I do something wrong??</p>

username: ptrblck
cooked: <p>If each epoch takes 30 minutes, it seems the GPU might not be used.<br>
Could you run this dummy code snippet and report the time for each epoch you are seeing:</p>
<pre><code class="lang-python">model = models.vgg16().to('cuda')
dataset = datasets.CIFAR10(
    root=ROOT,
    transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
        ]),
    download=True
)

loader = DataLoader(
    dataset,
    num_workers=4,
    pin_memory=True,
    shuffle=True,
    batch_size=64
)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for epoch in range(5):
    torch.cuda.synchronize()
    t0 = time.time()
    for data, target in loader:
        optimizer.zero_grad()
        data = data.to('cuda')
        target = target.to('cuda')
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
    torch.cuda.synchronize()
    t1 = time.time()
    print('Epoch {} took {}s'.format(epoch, t1-t0))
</code></pre>
<p>I’m getting approx. 36 seconds per epoch on a Titan V.</p>

username: Yolkandwhite
cooked: <p>Thank you very much.<br>
now It got much faster(30sec/1epoch)!!<br>
and I have another question.<br>
Do I always have to type .to(“cuda”) to use GPU?<br>
It doesn’t automatically goes to GPU?<br>
or is there something to change the default from CPU to GPU?</p>

username: ptrblck
cooked: <p>In the “normal” use case, you would have to push the model as well as your data and target tensors to the device manually and I would recommend to stick to this approach.</p>
<p>It’s clearer in my opinion than e.g. setting the default tensor type to a CUDATensor.</p>