Pytorch implementation slower than Tensorflow counterpart

Sushruth_N: <p>I was trying to implement a simple CAE in Pytorch. I am seeing a lot of difference in train time in pytorch compared to tensorflow. Pytorch version is taking around 20 sec for 100 epochs whereas tensorflow version is taking around 5 sec for 100 epochs. Can anyone help me resolve this ? I have attached my code below.</p>
<p>Tensorflow code is similar to this <a href="https://github.com/panji1990/Deep-subspace-clustering-networks/blob/master/Pre-Train-Conv-AE-EYaleB.py" rel="nofollow noopener">https://github.com/panji1990/Deep-subspace-clustering-networks/blob/master/Pre-Train-Conv-AE-EYaleB.py</a></p>
<p>Also nvidia-smi shows 100% usage for pytorch version whereas it is around 45% for tensorflow version</p>
<pre><code class="lang-auto">class fullmodel(nn.Module):
    
    def __init__(self,kernel_size,batch_size,num_class):
        super(fullmodel, self).__init__()
        self.batch_size = batch_size
        self.enc1 = nn.Conv2d(1,10,kernel_size[0],padding = 2)
        self.enc2 = nn.Conv2d(10,20,kernel_size[1],padding = 1)
        self.enc3 = nn.Conv2d(20,30,kernel_size[2],padding = 1)
        
        self.dec1 = nn.ConvTranspose2d(30,20,kernel_size = 3,padding = 1)
        self.dec2 = nn.ConvTranspose2d(20,10,kernel_size = 3,padding = 1)
        self.dec3 = nn.ConvTranspose2d(10,1,kernel_size = 5,padding = 2)
        
    def forward(self,x):
        enc_out = F.relu(self.enc3(F.relu(self.enc2(F.relu(self.enc1(x))))))
        dec_out = F.relu(self.dec3(F.relu(self.dec2(F.relu(self.dec1(enc_out))))))  
        return enc_out,dec_out,

def train(model,data):
    torch.backends.cudnn.benchmark = True
    lr = 1e-3
    num_epochs = 5000
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(device)
    X = torch.from_numpy(data).float()
    #data_train = torch.utils.data.DataLoader(dataset=X, batch_size=data.shape[0])
    parameters = model.parameters()
    model.apply(init_weights)
        
    optim = torch.optim.Adam(model.parameters(), lr=lr)
    model = model.to(device)
    model = model.train()
    X = X.to(device)
    time1 = time.time()
    for epoch in range(num_epochs):
        
        avg_recon_loss = 0
        latent,output = model(X)
        recon_loss = 0.5*(torch.sum((output - X)**2))
        optim.zero_grad()
        recon_loss.backward()
        optim.step()
        avg_recon_loss += recon_loss
        if (epoch+1)%100 == 0:
            print("Iter : ",epoch+1)
            print ("Loss : {:.4f}".format(avg_recon_loss/(38*64)))
            print(time.time() - time1)
            time1 = time.time()

if __name__ == "__main__":
    data = sio.loadmat("../datasets/YaleBCrop025.mat")
    imgs = data['Y']
    I = []
    Label = []
    for i in range(imgs.shape[2]):
        for j in range(imgs.shape[1]):
            temp = np.reshape(imgs[:,j,i],[42,48])
            Label.append(i)
            I.append(temp)
    I = np.array(I)
    n_input = [42,48]

    imgs = np.reshape(I, [I.shape[0], 1, n_input[0], n_input[1]])
    kernel_size = [5,3,3]
    classes = 38
    batch_size = classes*64
    
    model = fullmodel(kernel_size,classes,batch_size)
    
    train(model,imgs)
</code></pre>

ptrblck: <p>Could you compare the number of parameters for both implementations?<br>
Your TF reference uses placeholders, so that it’s not easy to compare the layer setup.<br>
Also, you might want to remove <code>cudnn.deterministic = True</code> as this might pick a deterministic but slow algorithm.</p>

Sushruth_N: <p>The number of parameters in both pytorch and tensorflow versions are 14991. I removed the fc layers from tensorflow implementation (only has conv2d and transpose conv2d layers).</p>
<p>I also removed cudnn.deterministic = True. I literally have only conv2d and transpose conv2d layers in both.</p>
<p>I have included my main function in the code above. Do you think some unwanted broadcasting is happening? I don’t think there is anything wrong with this pytorch code.</p>

ptrblck: <p>Thanks for the information.<br>
Note that CUDA operations are asynchronous, so you would have to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timer.<br>
Could you add it and time the codes again?</p>
<p>I don’t know, how TF handles this case and how to synchronize TF code.</p>

Sushruth_N: <p>I tried with the changes you suggested. It is still the same result. Also I can visually notice the time difference because pytorch code is 4-5 times slower than tensorflow code.</p>
<p>My pytorch version is 1.4.0<br>
Cuda version is 10.1.243<br>
GPU is Geforce GTX 1080 ti</p>

ptrblck: <p>Thanks for the update. Are you using the same CUDA and cudnn versions for both frameworks?<br>
If so, could you create profiles using <code>nvprof</code> for both models, please?</p>

Sushruth_N: <p>I am using the same CUDA and cudnn versions for both frameworks.  I will create profiles using nvprof for both.</p>

Sushruth_N: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92d322b2a63145aa863a8f3d61a9a259f9985b23.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/92d322b2a63145aa863a8f3d61a9a259f9985b23" title="pytorch_profiler"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d322b2a63145aa863a8f3d61a9a259f9985b23_2_690x289.jpeg" alt="pytorch_profiler" data-base62-sha1="kWS47AxQv9waBLDSGpWLYCUyAL1" width="690" height="289" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d322b2a63145aa863a8f3d61a9a259f9985b23_2_690x289.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d322b2a63145aa863a8f3d61a9a259f9985b23_2_1035x433.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d322b2a63145aa863a8f3d61a9a259f9985b23_2_1380x578.jpeg 2x" data-dominant-color="2F2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_profiler</span><span class="informations">2880×1210 813 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since I am not the admin of the remote server, I couldn’t directly run nvprof. I actually ran</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
     model(imgs)
</code></pre>
<p>Something like this</p>
<p>I am not able to create profile for Tensorflow code as of now. It is not easy as Pytorch.<br>
But is it okay with just this ? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

amaleki: <p><a class="mention" href="/u/sushruth_n">@Sushruth_N</a>  were you able to identify the root cause of the difference? I am experiencing a similar problem.</p>