CPU is used despite all my tensors are moved to GPU

username: talhak
cooked: <p>I have a CUDA supported GPU (Nvidia GeForce GTX 1070) and I have installed both of the CUDA (version 10) and the CUDA-supported version of PyTorch.</p>
<p>Despite my GPU is detected, and I have moved all the tensors to GPU, my CPU is used instead of GPU as I see almost no GPU usage when I monitor it.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">num_epochs = 10
batch_size = 20
learning_rate = 0.0001
log_interval = 50

class AndroModel(torch.nn.Module):

    def __init__(self, input_size):
        super(AndroModel, self).__init__()

        self.kernel_size = 3
        self.padding = 0
        self.stride = 1
        self.input_size = input_size

        self.conv1 = nn.Sequential(
            nn.Conv1d(in_channels=1, out_channels=16, kernel_size=self.kernel_size, padding=self.padding,
                      stride=self.stride, bias=False),
            nn.ReLU(inplace=True)
        )
        self.conv2 = nn.Sequential(
            nn.Conv1d(in_channels=16, out_channels=32, kernel_size=self.kernel_size, padding=self.padding,
                      stride=self.stride, bias=False),
            nn.ReLU(inplace=True)
        )
        self.conv3 = nn.Sequential(
            nn.Conv1d(in_channels=32, out_channels=64, kernel_size=self.kernel_size, padding=self.padding,
                      stride=self.stride, bias=False),
            nn.ReLU(inplace=True)
        )
        self.conv4 = nn.Sequential(
            nn.Conv1d(in_channels=64, out_channels=128, kernel_size=self.kernel_size, padding=self.padding,
                      stride=self.stride, bias=False),
            nn.ReLU(inplace=True)
        )
        self.conv5 = nn.Sequential(
            nn.Conv1d(in_channels=128, out_channels=256, kernel_size=self.kernel_size, padding=self.padding,
                      stride=self.stride, bias=False),
            nn.ReLU(inplace=True)
        )

        self.num_conv_layers = 5
        last_conv_layer = self.conv5
        new_input_size = self.calculate_new_width(self.input_size, self.kernel_size, self.padding, self.stride, self.num_conv_layers, max_pooling=None)

        out_channels = last_conv_layer._modules['0'].out_channels
        dimension = out_channels * new_input_size
        self.fc1 = nn.Sequential(
            nn.Linear(in_features=dimension, out_features=3584),
            nn.Dropout(0.5))
        self.fc2 = nn.Sequential(
            nn.Linear(in_features=3584, out_features=1792),
            nn.Dropout(0.5))
        self.fc3 = nn.Sequential(
            nn.Linear(in_features=1792, out_features=448),
            nn.Dropout(0.5))
        self.fc4 = nn.Sequential(
            nn.Linear(in_features=448, out_features=112),
            nn.Dropout(0.5))
        self.fc5 = nn.Sequential(
            nn.Linear(in_features=112, out_features=28),
            nn.Dropout(0.5))
        self.fc6 = nn.Sequential(
            nn.Linear(in_features=28, out_features=6),
            nn.Dropout(0.5))
        self.fc7 = nn.Sequential(
            nn.Linear(in_features=6, out_features=2))

    def forward(self, x):
        x = x.reshape((-1, 1, self.input_size))
        output = self.conv1(x)
        output = self.conv2(output)
        output = self.conv3(output)
        output = self.conv4(output)
        output = self.conv5(output)
        output = output.view(output.size(0), -1)
        output = self.fc1(output)
        output = self.fc2(output)
        output = self.fc3(output)
        output = self.fc4(output)
        output = self.fc5(output)
        output = self.fc6(output)
        output = self.fc7(output)

        return output

    @staticmethod
    def calculate_new_width(input_size, kernel_size, padding, stride, num_conv_layers, max_pooling=2):
        new_input_size = input_size
        for i in range(num_conv_layers):
            new_input_size = ((new_input_size - kernel_size + 2 * padding) // stride) + 1
            if max_pooling is not None:
                new_input_size //= max_pooling
        return new_input_size


class AndroDataset(Dataset):
    def __init__(self, features_as_ndarray, classes_as_ndarray):
        self.features = torch.from_numpy(features_as_ndarray).float().to(device)
        self.classes = torch.from_numpy(classes_as_ndarray).float().to(device)

    def __getitem__(self, index):
        return self.features[index], self.classes[index]

    def __len__(self):
        return len(self.features)


def main():
    start = time()

    global device
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    print('Device: {}'.format(device))
    if torch.cuda.is_available():
        print('GPU Model: {}'.format(torch.cuda.get_device_name(0)))

    csv_data = pd.read_csv('android_binary.csv')
    num_of_features = csv_data.shape[1] - 1
    x = csv_data.iloc[:, :-1].values
    y = csv_data.iloc[:, -1].values
    x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

    training_data = AndroDataset(x_train, y_train)
    test_data = AndroDataset(x_test, y_test)

    print('\n~~~~~~~~ TRAINING HAS STARTED ~~~~~~~~')
    print('# of training instances: {}'.format(len(training_data)))

    train_loader = DataLoader(dataset=training_data, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(dataset=test_data, batch_size=batch_size, shuffle=True)

    model = AndroModel(num_of_features)
    model = model.to(device)

    print('Model Overview:')
    print(model, '\n')

    criterion = nn.CrossEntropyLoss()
    criterion = criterion.to(device)

    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    losses_in_epochs = []
    # training
    total_step = len(train_loader)  # ayni zamanda VERI/BATCH_SIZE a esit
    for epoch in range(num_epochs):
        losses_in_current_epoch = []
        for i, (features, classes) in enumerate(train_loader):
            features, classes = features.to(device), classes.to(device, dtype=torch.int64)

            optimizer.zero_grad()

            output = model(features)
            loss = criterion(output, classes)
            loss.backward()
            optimizer.step()

            if (i + 1) % log_interval == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, num_epochs, i + 1, total_step,
                                                                         loss.item()))
                losses_in_current_epoch.append(loss.item())

        avg_loss_current_epoch = 0
        for tmp_loss in losses_in_current_epoch:
            avg_loss_current_epoch += tmp_loss
        avg_loss_current_epoch /= len(losses_in_current_epoch)
        print('End of the epoch #{}, avg. loss: {:.4f}'.format(epoch + 1, avg_loss_current_epoch))
        losses_in_epochs.append(avg_loss_current_epoch)

    print('Average loss: {:.4f}'.format(losses_in_epochs[-1]))
    print(f'Training Duration (in minutes): {(time() - start) / 60}')

    print('\n~~~~~~~~ TEST HAS STARTED ~~~~~~~~')
    print('# of test instances: {}'.format(len(test_data)))
    # test
    accuracy = 0
    with torch.no_grad():
        correct = 0
        for features, classes in test_loader:
            features, classes = features.to(device), classes.to(device, dtype=torch.int64)
            output = model(features)
            output = output.to(device)
            _, predicted = torch.max(output.data, 1)
            correct += (predicted == classes).sum().item()

        accuracy = 100 * correct / len(test_loader.dataset)
        print('Accuracy of the model on the {} test instances: {:.4f} %'.format(len(test_loader.dataset), accuracy))
</code></pre>

username: ptrblck
cooked: <p>Do you see and peaks in the GPU usage?<br>
Also, could you just pass random data to your GPU and see, if the utilization increases?<br>
You might also want to time your data loading as shown in the <a href="https://github.com/pytorch/examples/blob/1de2ff9338bacaaffa123d03ce53d7522d5dcc2e/imagenet/main.py#L275" rel="nofollow noopener">ImageNet example</a> to see, if you have a bottleneck there.</p>

username: talhak
cooked: <p>Thanks for your interest. Regarding your question, no, unfortunately have not seen any peaks in the GPU usage. And here are the values of the two average meters as I have used in the same way as it is applied in the ImageNet example that you directed me:</p>
<pre><code class="lang-auto">Time  0.025 ( 0.027)
Data  0.000 ( 0.000)
</code></pre>

username: talhak
cooked: <p>Despite the PyCharm process’s power usage is <strong>very high</strong> and <strong>GPU engine (copy)</strong> is used, here is the GPU utilization which is as low as <strong>4%</strong>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fb50784882cf01f0809d34396e3633fa4cf9dfbe.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fb50784882cf01f0809d34396e3633fa4cf9dfbe" title="a"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb50784882cf01f0809d34396e3633fa4cf9dfbe_2_659x500.jpeg" alt="a" data-base62-sha1="zRefyFBbrM6mmMgFMP8UuJ2NiNw" width="659" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb50784882cf01f0809d34396e3633fa4cf9dfbe_2_659x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb50784882cf01f0809d34396e3633fa4cf9dfbe_2_988x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb50784882cf01f0809d34396e3633fa4cf9dfbe_2_1318x1000.jpeg 2x" data-dominant-color="F8FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">a</span><span class="informations">1424×1079 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>That’s strange, since the data loading time seem to be completely hidden behind the computation.<br>
I just tried your code on my machine and simplified it a bit:</p>
<ul>
<li>removed the test loop</li>
<li>used random inputs (<code>torch.randn</code> as data and <code>torch.randint</code> as target)</li>
<li>used an input batch of <code>[20, 1, 100]</code>
</li>
</ul>
<p>Using this I got a GPU utilization of 97% on a TITAN V GPU.<br>
Does <code>nvidia-smi</code> show any utilization at all?</p>
<p>EDIT: Just saw your second post now.<br>
Could you run <code>nvidia-smi</code>, as I’m not sure how the Windows task manager handles the GPU util and how accurate it is?</p>

username: talhak
cooked: <p>Of course, here it is:</p>
<pre><code class="lang-auto">Thu Jul 04 00:56:34 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.86       Driver Version: 430.86       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070   WDDM  | 00000000:01:00.0  On |                  N/A |
| N/A   68C    P2    99W /  N/A |   2088MiB /  8192MiB |     71%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1384    C+G   Insufficient Permissions                   N/A      |
|    0      3292    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0      3512    C+G   ...a\Local\Vivaldi\Application\vivaldi.exe N/A      |
|    0      3812      C   ...cts\DeepAndroid\venv\Scripts\python.exe N/A      |
|    0      7228    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
|    0      8260    C+G   ...11411.0_x64__8wekyb3d8bbwe\Video.UI.exe N/A      |
|    0      9068    C+G   Insufficient Permissions                   N/A      |
|    0      9244    C+G   C:\Windows\explorer.exe                    N/A      |
|    0      9760    C+G   ...5n1h2txyewy\StartMenuExperienceHost.exe N/A      |
|    0     10128    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     10164    C+G   Insufficient Permissions                   N/A      |
|    0     10804    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     11948    C+G   ...48.51.0_x64__kzf8qxf38zg5c\SkypeApp.exe N/A      |
|    0     11972    C+G   ....410.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     12332    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     18804    C+G   ....28.0_x64__8wekyb3d8bbwe\Calculator.exe N/A      |
+-----------------------------------------------------------------------------+
</code></pre>

username: ptrblck
cooked: <p>Thanks!<br>
That looks indeed better with 71%. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: talhak
cooked: <p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"> Finally, I’d like to ask a couple of questions if you do not mind:</p>
<ul>
<li>Is there a GUI to monitor this?</li>
<li>Why is the GPU memory usage for my process (the process with <code>PID</code> <code>3812</code>) <code>N/A</code>?</li>
<li>Why is the type of my process <code>C</code> while others are <code>C+G</code>?</li>
</ul>

username: ptrblck
cooked: <ul>
<li>I’m not sure, but Windows <em>might</em> have some GUIs for it. Since I’m using Ubuntu, I’m, used to these beautiful black terminals <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">
</li>
<li>Might be some kind of permission issue. Could you try to run the process as an admin and see, if that helps?</li>
<li>G = graphics, C = compute. Since you are using CUDA, only C should be shown. As far as I know G will pop up, if e.g. DirectX is used (but I have really limited knowledge about this area).</li>
</ul>

username: talhak
cooked: <p>Thank you very much, again, for your great contribution, much appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"> Cannot agree more, Terminal is the most missing part of not using Linux. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: talhak
cooked: <p>p.s. Running the process as the admin(istrator) did not help to see the GPU memory usage of the process. A small note to the potential readers of the topic.</p>

username: fireis
cooked: <aside class="quote no-group" data-username="talhak" data-post="8" data-topic="49611">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/e8c25b/48.png" class="avatar"> talhak:</div>
<blockquote>
<ul>
<li>Is there a GUI to monitor this?</li>
</ul>
</blockquote>
</aside>
<p>One thing that could help is to use the small arrow right before one of the graphs (such as video encode) and change it to Compute_0 or CUDA or some other similar name there may be in your system. The result should be similar to:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e9024e94766435013aae99e68c1d2e9bc030cf9a.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9024e94766435013aae99e68c1d2e9bc030cf9a" title="gpu_monit"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e9024e94766435013aae99e68c1d2e9bc030cf9a.png" alt="gpu_monit" data-base62-sha1="xfibYO6vH7DGUMBEofNfy98Zik2" width="690" height="274" data-dominant-color="F5F9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu_monit</span><span class="informations">709×282 5.02 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: talhak
cooked: <p>That is really helpful and exactly what I was looking for, thank you very much for your contribution. <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"> <a class="mention" href="/u/fireis">@fireis</a></p>

username: fireis
cooked: <p>Glad to help!<br>
One addition, if you want something fancier, there is a commercial (but free) tool called MSI Afterburner which is able to provide more info, such as temperature and power usage.</p>

username: Manuel_Alejandro_Dia
cooked: <blockquote>
<p>Why is the GPU memory usage for my process (the process with <code>PID</code> <code>3812</code> ) <code>N/A</code> ?</p>
</blockquote>
<p>Maybe if you open CMD as administrator and run <code>nvidia-smi</code>, then it could give you more information since you have more privileges.</p>

username: talhak
cooked: <p>It is still the same, unfortunately.</p>

username: Manuel_Alejandro_Dia
cooked: <p>It seems that it is not possible to see the GPU memory usage with <code>nvidia-smi</code> while you have a display connected to the GPU in Windows, according to this Stack Overflow answer: <a href="https://stackoverflow.com/a/44228331" rel="nofollow noopener">https://stackoverflow.com/a/44228331</a></p>
<p>Another option seems to be an API (<a href="https://developer.nvidia.com/nvapi" rel="nofollow noopener">https://developer.nvidia.com/nvapi</a>) that lets you see GPU info on Windows, but I have not tried since I work on Ubuntu.</p>
<p>As a last resort, you could come to Linux, we have penguins and cookies! <img src="https://discuss.pytorch.org/images/emoji/apple/cookie.png?v=9" title=":cookie:" class="emoji" alt=":cookie:"> <img src="https://discuss.pytorch.org/images/emoji/apple/penguin.png?v=9" title=":penguin:" class="emoji" alt=":penguin:"></p>