Strange difference in performance between 2 regression programs

Philippe_JUHEL: <p>Hi,</p>
<p>Here’re 2 regression programs with the same sets of data and the same basic model (2 inputs and 1 outpu):</p>
<p>Program 1 :</p>
<pre><code class="lang-auto">import numpy as np
import torch
import time

# Define the model
def model(x):
    return x @ w.t() + b

# MSE loss
def mse(t1, t2):
    diff = t1 - t2
    return torch.sum(diff * diff) / diff.numel()

def fit(num_epochs, model, loss_fn, w, b):
    for i in range(num_epochs):
        preds = model(inputs)
        loss = loss_fn(preds, targets)
        loss.backward()
        with torch.no_grad():
            w -= w.grad * lr
            b -= b.grad * lr
            w.grad.zero_()
            b.grad.zero_()

lr = 1e-3
nb_epochs = 1000

nb_data = 1000
min_x = 2.0
max_x = 3.0
min_y = 5.0
max_y = 9.0

X = np.linspace(min_x, max_x, num=nb_data, dtype=np.float32)
Y = np.linspace(min_y, max_y, num=nb_data, dtype=np.float32)
inputs = np.stack((X, Y), axis=1)
targets = X + Y
targets = targets.reshape(targets.size, 1)

# Convert inputs and targets to tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)

# Weights and biases
w = torch.randn(1, 2, requires_grad=True)
b = torch.randn(1, requires_grad=True)

begin = time.time()
fit(nb_epochs, model, mse, w, b)
end = time.time()
print(f"Duration = {end-begin} s")

# Calculate loss
preds = model(inputs)
loss = mse(preds, targets)
print(f"Loss = {loss}")

# Calculate a prediction
pred_y = model(torch.Tensor([[2.5, 6]]))
print("predict ", pred_y.item(), " should be ===&gt;",8.5 )
type or paste code here
</code></pre>
<p>Program 2 :</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import numpy as np
from torch.utils.data import TensorDataset, DataLoader
import torch.nn.functional as F
import time

# Define a utility function to train the model
def fit(num_epochs, model, loss_fn, opt):
    for epoch in range(num_epochs):
        for xb,yb in train_dl:
            # Generate predictions
            pred = model(xb)
            loss = loss_fn(pred, yb)
            # Perform gradient descent
            loss.backward()
            opt.step()
            opt.zero_grad()

device = "cpu"
#device = "cuda:0"

lr = 1e-3
batch_size = 100
nb_epochs = 1000

nb_data = 1000
min_x = 2.0
max_x = 3.0
min_y = 5.0
max_y = 9.0

X = np.linspace(min_x, max_x, num=nb_data, dtype=np.float32)
Y = np.linspace(min_y, max_y, num=nb_data, dtype=np.float32)
inputs = np.stack((X, Y), axis=1)
targets = X + Y
targets = targets.reshape(targets.size, 1)

inputs = torch.from_numpy(inputs).to(device)
targets = torch.from_numpy(targets).to(device)

train_ds = TensorDataset(inputs, targets)
# Define data loader
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

# Define model, 2 inputs, 1 output
model = nn.Linear(2, 1).to(device)
# Define optimizer
opt = torch.optim.SGD(model.parameters(), lr=lr)
# Define loss function
loss_fn = F.mse_loss

# Train the model for some epochs
begin = time.time()
fit(nb_epochs, model, loss_fn, opt)
end = time.time()
print(f"Duration = {end-begin} s")

# Calculate final loss
preds = model(inputs)
loss = loss_fn(preds, targets)
print(f"Loss = {loss}")

# Evaluate a prediction
pred_y = model(torch.Tensor([[2.5, 6]]).to(device))
print("predict ", pred_y.item(), " should be ===&gt;",8.5 )

</code></pre>
<p>They have the same number of data (1000) and the same number of epochs (1000)</p>
<p>When I run these programs on the same machine (Ubuntu 20.04, 32Gb, core I7, NVIDIA 2080, torch 1.12.1 ), here are the duration for the training (<strong>fit</strong> function) :</p>
<p>program 1 : 0.25s<br>
program 2 (on CPU) : 8.5s<br>
program 2 (on GPU) : 11.8s</p>
<p>Why a so big difference between program 1 and program 2? And for program 2 with GPU, why is it worst than with CPU?</p>
<p>Regards,</p>
<p>Philippe</p>

ptrblck: <p>Your model is tiny as it’s a single operation/layer and you are most likely seeing the overhead of creating the <code>DataLoader</code>, shuffling the data, etc.<br>
You can profile parts of your code to narrow down where the slowdown is coming from.<br>
Just executing the <code>DataLoader</code> on my system takes a few seconds for 1000 epochs.</p>

Philippe_JUHEL: <p>Thank you for your answer.</p>
<p>I measure the duration just for the <strong>fit</strong> function, so the preparation of the data (creation of DataLoader and shuffling) is <strong>not</strong> tacking into account for the duration. But, maybe this overconsumption of time is due to extraction of data from the DataLoader in the</p>
<blockquote>
<p>for xb,yb in train_dl:</p>
</blockquote>
<p>loop?</p>
<p>Another strange thing is that it takes more time to process when I use a GPU but maybe it is due to the time to transfer data to the GPU?</p>
<p>For information, at first, I wanted to understand why a simple regression example took so long to run with Pytorch Lighting. So to compare with a simpler solution, I started by creating this tiny models with just Pytorch, but that led me to ask myself this question about the difference in performance between these two solutions.</p>
<p>Philippe</p>

ptrblck: <aside class="quote no-group" data-username="Philippe_JUHEL" data-post="3" data-topic="162878">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/philippe_juhel/48/53773_2.png" class="avatar"> Philippe_JUHEL:</div>
<blockquote>
<p>I measure the duration just for the <strong>fit</strong> function, so the preparation of the data (creation of DataLoader and shuffling) is <strong>not</strong> tacking into account for the duration.</p>
</blockquote>
</aside>
<p>That’s not true, since iterating the <code>DataLoader</code> will recreate it in each epoch to e.g. create a new sampler etc. If <code>num_workers&gt;0</code> is used than also the workers will be re-spawned unless <code>persistent_workers=True</code> is used. Again, you can profile your code to narrow down the slowdown.</p>
<aside class="quote no-group" data-username="Philippe_JUHEL" data-post="3" data-topic="162878">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/philippe_juhel/48/53773_2.png" class="avatar"> Philippe_JUHEL:</div>
<blockquote>
<p>Another strange thing is that it takes more time to process when I use a GPU but maybe it is due to the time to transfer data to the GPU?</p>
</blockquote>
</aside>
<p>Your GPU profiling is invalid, since CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the timers. However, even with proper profiling I would not expect to see any speedup, since your entire code is already bottlenecked by the data loading. A single tiny linear layer with <code>in_features=2</code> and <code>out_features=1</code> would also not benefit lrgely from a GPU execution.</p>

Philippe_JUHEL: <p>Thank you for all this information.</p>
<p>I’ll continue my investigation with profiling and a bigger dataset.</p>
<p>Philippe</p>

ptrblck: <p>Sounds good!<br>
Also, try to increase the actual workload of the model and let me know how it goes.</p>