Significant improvement in time with GPU as compared to CPU not being achieved

muhammadirfanzafar: <p>Hi,</p>
<p>I am training a 7 layers linear fully connected network with 48 neurons in each hidden layer (which gives 14353 learnable parameters). My data has 3 input features and 1 output. Data size is around 51230. I am using Dataloader with 20 batches. However the time improvement from CPU to GPU is only of 30-40% reduction in training time. After experimentation, i have noticed that GPU would only give significant time improvement if the total number of learnable parameters in significantly increased, say to the order of millions (then we can have training time reduced around 7 times). Can we not achieve significant benefit from GPU for NN model with 14353 parameters?</p>
<p>Overall if i train it for 200 epochs, these are time comparisons:<br>
For 14k model parameters:<br>
CPU: 5.7 min<br>
GPU: 4.1 min</p>
<p>For 3.54 millions model parameters:<br>
CPU: 31.0 min<br>
GPU: 4.27 min</p>
<p><strong>Is there any other way i can reduce my training time for around 14k model parameters?</strong></p>

ptrblck: <p>If you are working with a small dataset, you could preload the whole dataset and push it to the GPU before training, to avoid loading times. Also, make sure not to create any unnecessary synchronization points in your training loop, e.g. printing the <code>loss</code> often.</p>

muhammadirfanzafar: <p>Since the whole data has been transferred to GPU before creating dataset and train loader, i think it’s not having any loading time issues. <em>pin_memory</em> wouldn’t work for data already on GPU, if that’s what you’re suggesting. Also i have tried to minimize loss printing. Here are major parts of the code for clarity.</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        super(Net, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, H)
        self.linear3 = torch.nn.Linear(H, H)
        self.linear4 = torch.nn.Linear(H, H)
        self.linear5 = torch.nn.Linear(H, H)
        self.linear6 = torch.nn.Linear(H, H)
        self.linear7 = torch.nn.Linear(H, H)
        self.linear8 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        out = F.relu(self.linear1(x))
        out = F.relu(self.linear2(out))
        out = F.relu(self.linear3(out))
        out = F.relu(self.linear4(out))
        out = F.relu(self.linear5(out))
        out = F.relu(self.linear6(out))
        out = F.relu(self.linear7(out))
        y_pred = self.linear8(out)
        return y_pred

D_in, H, D_out = 3, 768, 1
model = Net(D_in, H, D_out)
criterion = nn.MSELoss(reduction='sum')
optimizer = Adam(model.parameters(), lr=5e-4)

device = torch.device('cuda')
model.to(device)

dataX = dataX.to(device)
dataY = dataY.to(device)
dataset = TensorDataset(dataX,dataY)
training_batches = 20
batch_size_train = int(len(dataX)/training_batches) +1
train_loader = DataLoader(dataset, batch_size=batch_size_train, shuffle=True)

start_time = time.time()
for epoch in range(201):
    running_loss = 0.0
    for i, data in enumerate(train_loader):
        features, target = data
        optimizer.zero_grad()
        forward = model(features)
        loss = criterion(forward, target)
        if epoch % 100 == 0:
            running_loss += loss.item()
        loss.backward()
        optimizer.step()
    if epoch % 100 == 0:
        print('Epoch: {},    Training Loss: {:.2e}'.format(epoch, running_loss/training_batches))

elapsed = time.time() - start_time
print('GPU Time: {:.2f} min'.format(elapsed/60))
</code></pre>

ptrblck: <p>I’ve added <code>torch.cuda.synchronize()</code> before starting and stopping the timer and used some dummy data:</p>
<pre><code class="lang-python">dataX = torch.randn(1000, 3).to(device)
dataY = torch.randn(1000, 1).to(device)
</code></pre>
<p>CPU: <code>0.44854 seconds/epoch</code><br>
GPU (TitanV): <code>0.0877 seconds/epoch</code></p>
<p>Based on your estimate (4.1 minutes for 200 epochs), it seems that each epoch takes <code>(4.1*60)/200 = 1.22seconds</code> on your GPU.</p>
<p>Which GPU, CUDA, cudnn, PyTorch versions are you using?</p>

muhammadirfanzafar: <p>I have tried the same dummy data with 1000 samples and the GPU i am using would do it in 0.117 seconds/epoch.</p>
<p>GPU: Tesla P100-SXM2-16GB<br>
CUDA driver version: 10010<br>
cudnn: 7.5.1_10.1<br>
PyTorch: 1.1.0</p>
<p>Can the issue be with data i am loading from files? I am converting it to FloatTensor from pandas dataframe using command</p>
<pre><code class="lang-auto">dataX = torch.tensor(dataX.values).to(dtype=torch.float)
</code></pre>
<p>Data is available in <code>.csv</code> text file. Number are mentioned in this format.</p>
<pre><code class="lang-auto">2.489500679193377281e-03,0.000000000000000000e+00,3.944573057764559962e+02,1.833216381585000068e-02
</code></pre>
<p>Thanks.</p>

ptrblck: <p>This shouldn’t make a speed difference, but could you try to use <code>torch.from_numpy</code> instead of <code>torch.tensor</code> to create <code>dataX</code>?</p>

muhammadirfanzafar: <p><code>torch.from_numpy</code>  didn’t make any difference. Exactly same time as before.</p>

ptrblck: <p>If I understand the issue correctly at the moment, you are seeing a time of <code>0.117 s/epoch</code> using random data on the GPU and <code>1.22 s/epoch</code> if you use your real data?</p>
<p>We are aware of denormal values, which might slow down the execution on the CPU, but this shouldn’t be the case on the GPU.<br>
Could you nevertheless set <code>torch.set_flush_denormal(True)</code> and time it again?</p>

muhammadirfanzafar: <p><code>0.117 s/epoch</code> was time for 1000 samples of random dummy data, while my real data size is <code>51320</code> and <code>1.22 s/epoch</code> corresponds to that.</p>
<p>My concern is with the comparison of CPU &amp; GPU time for this real data of size <code>51320</code>, in which i am not getting significant reduction in training time with GPU. (Also if i increase the size for random dummy data to <code>51320</code>, times are almost the same)</p>
<p><code>For 14k model parameters: CPU: 5.7min GPU: 4.1min</code></p>
<p>I am not sure if i am doing everything i can, to get the maximum benefit from GPU.<br>
Isn’t GPU benefit significant for model with only 14k parameters?</p>

muhammadirfanzafar: <p>Unfortunately <code>torch.set_flush_denormal(True)</code> didn’t make any effect.</p>

ptrblck: <p>If you change the data size to <code>51320</code>, the batch size will increase to <code>2566</code>, as you are calculating it dynamically using:</p>
<pre><code class="lang-python">training_batches = 20
batch_size_train = int(len(dataX)/training_batches)
</code></pre>
<p>Running the script with this batch size gives:<br>
GPU: 0.44s/epoch<br>
CPU: 3.27s/epoch</p>

muhammadirfanzafar: <p>Thanks a lot <span class="mention">@ptrblck_de</span> for your help throughout the discussion.</p>
<p>Here’s an enigma. Although GPU <code>Tesla P100</code> provide better times for data size of <code>1000</code>, the training time increases rapidly by just increasing the data size, as corresponding times are mentioned below:</p>
<pre><code class="lang-auto">1k  :  0.117 s/epoch
11k :  0.327 s/epoch
21k :  0.546 s/epoch
31k :  0.762 s/epoch
41k :  0.981 s/epoch
51k :  1.206 s/epoch
</code></pre>
<p>Do you understand why there’s so much difference in time increase for two different GPUs?</p>
<p>Also i tried a code from a research paper written with tensorflow. Having made no changes to it, training time on <code>Tesla P100</code> was 6.3 min while it’s mentioned in the paper that <code>NVIDIA Titan X</code> did it in 1 min. I can’t figure any reason since i think P100 is more powerful.</p>

ptrblck: <p>I can rerun these tests tomorrow on a few GPUs and report some numbers.<br>
In the meantime, could you update PyTorch to the latest stable release (1.2.0) so that we get comparable numbers?</p>

muhammadirfanzafar: <p>I am unable to update it to version 1.2.0. I have tried updating it using these commands:</p>
<pre><code class="lang-auto">conda update pytorch
conda install pytorch=1.2.0 -c pytorch
</code></pre>
<p>I am running it on university’s computing resources, where i access GPU remotely and i think i am not allowed to update packages freely.</p>

ptrblck: <p>If you can run docker, you could try to run the code inside it.</p>
<p>Here are my results:<br>
P100, 16GB</p>
<pre><code class="lang-auto">N = 1000
GPU Time: 0.076056s/epoch
N = 11000
GPU Time: 0.148172s/epoch
N = 21000
GPU Time: 0.220223s/epoch
N = 31000
GPU Time: 0.299151s/epoch
N = 41000
GPU Time: 0.368989s/epoch
N = 51000
GPU Time: 0.448879s/epoch
</code></pre>
<p>V100, 16GB</p>
<pre><code class="lang-auto">N = 1000
GPU Time: 0.081930s/epoch
N = 11000
GPU Time: 0.147144s/epoch
N = 21000
GPU Time: 0.254728s/epoch
N = 31000
GPU Time: 0.282772s/epoch
N = 41000
GPU Time: 0.382091s/epoch
N = 51000
GPU Time: 0.440113s/epoch
</code></pre>
<p>Note that I just executed the code once for each config.</p>

muhammadirfanzafar: <p>So that’s a concern. Your results are as expected. I am not sure why they aren’t like that for me. What’s the clock speed on your GPUs?</p>

caiom: <p>On a unrelated note, I think it worth mentioning that these many linear layers in sequence is not advisable in most cases. You will be probably better of using two linear layers (in total) and playing with the number of neurons in the first linear layer.</p>

rwightman: <p>If you want more sensible comparisons I’d fix the batch size at the same size regardless of the dataset size . I’m not sure why you’d want a fixed number of batches per epoch. Not doign this will result in changing GPU utilization with different dataset sizes .</p>
<p>I wouldnt’ be suprised if the performance, even on GPU is CPU bound due to dataset/dataloader python overhead. The call <code>__getitem__</code> per batch element of many many thousands will chew up a lot of CPU time when done in Python. I’d try building a random index and building batches from X, Y tensors manually.</p>

muhammadirfanzafar: <p>So i have worked with the administrators of computing resources and turns out that P100 GPU i was using wasn’t performing optimally for some reasons. Same GPU on another system gave comparable performance as you mentioned.</p>
<p>Many thanks for your time and support.</p>

muhammadirfanzafar: <p>It doesn’t make much difference. Except increasing the number of batches for same data size increases the computing time. Also whole dataset is already on GPU before training starts.</p>