Question aboout GPU computing

SiNML: <p>Hello, everyone. I have build the machine learning code with GPU calculation. It is running, but not fast as I expected. I do not know I have build the correct one.</p>
<p>I read the input data by:</p>
<pre><code class="lang-auto">Dataset = pd.read_csv('./Inputs.txt', sep = "\s+")
Target_Column = 9
X = Dataset.iloc[:, 0:Target_Column].values
Y = Dataset.iloc[:,   Target_Column].values

X_Train, X_Test, Y_Train, Y_Test = train_test_split(X, Y, test_size = 0.3, random_state = 0)

sc = StandardScaler()
X_Train = sc.fit_transform(X_Train)
X_Test = sc.transform(X_Test)
</code></pre>
<p>Then I use DataLoder as follows:</p>
<pre><code class="lang-auto">X_Train = torch.Tensor(X_Train)
Y_Train = torch.Tensor(Y_Train)

Train_Dataset = TensorDataset(X_Train, Y_Train)

Train_Dataloader = DataLoader(Train_Dataset, batch_size = Batch_Size, shuffle = True)
</code></pre>
<p>Then, when I calculating each iteration, I use following code:</p>
<pre><code class="lang-auto">for Epoch in range(Num_Epochs):
    #Shuffle just mixes up the dataset between epocs
    X_Train, Y_Train = shuffle(X_Train, Y_Train)
    
    # Mini batch learning
    for batch_idx, samples in enumerate(Train_Dataloader):
        X_Train = X_Train.to(Device)
        Y_Train = Y_Train.to(Device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(X_Train)
        loss = criterion(outputs, torch.unsqueeze(Y_Train,dim = 1))
        loss.backward()
        optimizer.step()

        # print statistics
        Running_Loss += loss.item()
        
    if(Epoch % 100 == 0): 
        File.write("%.4f\t%.4f\n" % (Epoch, Running_Loss))
        
    if(Epoch % 2 == 0): 
        print('Epoch {}'.format(Epoch), "Loss: ", Running_Loss)
    
    Running_Loss = 0.0
</code></pre>
<p>I have confirmed that the GPU is running with 50% usage.  But not fast enough. Is there anything wrong in the process?</p>
<p>Thank you for your help.</p>

ptrblck: <p>You could profile your code e.g. via <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">Nsight Systems</a> or the <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/">PyTorch Profiler</a> to see where the bottleneck in your code is and to further isolate it. E.g. your training might suffer from a slow data loading and could explain the 50% utilization of the GPU.</p>

SiNML: <p>Thank you for your comment! I will try to follow up the PyTorch Profiler to see what the problem is.</p>
<p>So if the code is correct, the usage of GPU should nearly 0%?</p>

ptrblck: <p>No, if you are not facing a CPU-bound bottleneck, such as the data loading, the GPU utilization should be high, since the GPU would be used to train the model and wouldn’t have to wait for the next data batch etc.</p>

SiNML: <p>Since I am new to pytorch and python. I failed to generate the “Tensor Board”</p>
<p>Instead, I have obtain some profile using the following code:</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
    net(X_Train)
print(prof)
</code></pre>
<p>The result is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58fe27a97a6a36ef210f091710c4c0ecb6c7783e.png" data-download-href="https://discuss.pytorch.org/uploads/default/58fe27a97a6a36ef210f091710c4c0ecb6c7783e" title="Profile"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58fe27a97a6a36ef210f091710c4c0ecb6c7783e.png" alt="Profile" data-base62-sha1="cHgxAM80UqbWiPEmrnV7brpTXrg" width="690" height="301" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Profile</span><span class="informations">1651×722 41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f081841653113ad82c4e69b997648b1b000f3b6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f081841653113ad82c4e69b997648b1b000f3b6f" title="Trace"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f081841653113ad82c4e69b997648b1b000f3b6f_2_690x126.png" alt="Trace" data-base62-sha1="yjC4VVc08mMDdga0qMBKLRStXRl" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f081841653113ad82c4e69b997648b1b000f3b6f_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f081841653113ad82c4e69b997648b1b000f3b6f_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f081841653113ad82c4e69b997648b1b000f3b6f_2_1380x252.png 2x" data-dominant-color="EBE7EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Trace</span><span class="informations">1920×352 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can see the most of the time of CUDA is spent at “aten::_fused_dropout” stage, and the total time of CUDA is 3.537ms, but it takes 98.682 second to finish the model. Ans usage of GPU is almost 0% during calculating…</p>
<p>Is it possible to check whether GPU is applied properly?</p>

ptrblck: <p>I’m not familiar with this kind of output and don’t see any data loading part in it.<br>
However, CUDA functions are mentioned, so I assume the GPU is used.<br>
I’m also unsure, if you’ve added proper warmup iterations etc. as described in my short tutorial.</p>

SiNML: <p>I will try to Nsight System to check the usage of GPU as soon as I can. Thank you for your help!</p>