akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Very slow training on RTX 2070 8GB RAM

artem1: <p>I am experiencing very slow training on RTX 2070 on Windows 10 and the GPU is hardly used, while the same exact code is super fast on GTX 1080Ti. After some research I concluded that training must only be running on CUDA cores and not on Tensor cores. I tried using .half() for both model and for inputs, but the training is still just as slow. Even after I switched to using apex (I used the conda distribution from  here: <a href="https://anaconda.org/conda-forge/nvidia-apex" rel="nofollow noopener">https://anaconda.org/conda-forge/nvidia-apex</a>) the training is still slow.</p>
<p>I am kind of out of options now.<br>
Is there an example script using .half() or apex that I can use to check whether GPU can be fully used?<br>
Is there a way to monitor whether Tensor cores are being used on Windows?</p>
<p>Any help is appreciated!</p>

ptrblck: <p>You could use <a href="https://github.com/NVIDIA/apex/tree/master/apex/pyprof">pyprof</a> to check for TensorCore usage.</p>
<p>Are both GPUs in the same machine? If not, is the setup of the machines similar?</p>

artem1: <p>Thanks! I will take a look at pyprof. The setup on GTX is very different, it is a Linux cluster  with 4 GPUs in a node. But performance on my local RTX is x3,500 slower than on the cluster.</p>
<p>On RTX I can see that GPU RAM is being used and CUDA cores are heavily used, but the overall GPU usage is around 7%.</p>
<p>I tried to experiment with reducing batch size and number of workers, but it does not seem to make a difference.</p>

ptrblck: <p>Could you try to profile the data loading pipeline, e.g. using <a href="https://github.com/pytorch/examples/blob/e9e76722dad4f4569651a8d67ca1d10607db58f9/imagenet/main.py#L280">this code</a>, or alternatively remove the data loading and feed a random input to your model?<br>
This would give you some information, if the data loading is the bottleneck, which might be the case given the low utilization of the device.</p>

artem1: <p>I followed your suggestion and fed random inputs to the model, but no luck. Just to make sure if I did this correctly: I removed the data loader altogether and just loaded a batch of random tensors, but this also did not increase my GPU usage. I still need to look at pyprof.</p>
<p>I noticed that changing the number workers from 0 to 16 does not seem to significantly change my CPU usage (I have 8 cores). That seems strange.</p>
<p>For the imagenet example it seems that I need to download the data, is that correct? I am waiting for approval. Do you by any chance have another example to check both mixed precision and data loading bottlenecks? Any ideas are appreciated! Thanks</p>

ptrblck: <p>I would recommend to narrow down the issue of a low utilization first.<br>
If you’ve removed the data loading and use a random CUDATensor, the utilization should be higher.<br>
At least if your code is not bottlenecked by anything else, so try to remove all unnecessary code and just keep the training (forward, backward, and optimizer).</p>

artem1: <p>Thanks a lot for this tip, I will try that.</p>
Loss.backward() with 'cuda' just ends

JayHan: <p>The problem is,<br>
when I use ‘cuda’, python code just ‘finish’ after loss.backward() line.<br>
I made a simple code, and I found that all of my python codes make a problem, which I used happily until yesterday.</p>
<pre><code class="lang-auto">path = 'C:/data_main_2d'
epochs = 5
batch_size = 2
lr = 1e-1
device = torch.device('cpu') # 'cuda' if torch.cuda.is_available() else 

dataset = dset.ImageFolder(root=path,
                           transform=transforms.Compose([
                               transforms.Grayscale(),
                               transforms.ToTensor(),
                           ]))
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0)

model = UNet(n_channels=1, n_classes=1).to(device=device)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=1e-8)

for epoch in range(epochs):
    for data in dataloader:
        img, _ = data
        img = Variable(img).to(device)

        output = model(img)
        loss = criterion(output, img)

        optimizer.zero_grad()
        print('hi')
        loss.backward()
        optimizer.step()

    loss_data = loss.data
    print(loss_data)
</code></pre>
<p>If I make a device as a ‘cpu’, it works and goes epochs. However, if the device is ‘cuda’, the result is like that</p>
<pre><code class="lang-auto">(base) D:\My Drive\(0).BSMSvac\0.ML\&gt;python train2.py
hi

(base) D:\My Drive\(0).BSMSvac\0.ML\&gt;
</code></pre>
<p>My cuda version is 9.2<br>
torch is 1.4.0<br>
torchvision is 0.5.0</p>
<p>I am so confusing because I did ML for 1 month actively, and my code works well…<br>
But yesterday I found the problem that terminal finish (ends) with no error or warnings.<br>
Also, now all other codes also don’t work. I can’t understand what is the problem…<br>
Other codes are not the UNet, sequential Convolution network, so UNet is not the problem…</p>

JayHan: <p>I solved this problem as</p>
<p>deleting cuda in my computer and install again…</p>
<p>Couldn’t know the exact reason and solution but it works well now.</p>
Triplet data loader for cifar10

S_M: <p>hi all, could anyone give me some tips about how can I select triplet data in CIFAR10? (sample code)</p>

Oli: <p>Hello,</p>
<p>How would you like to construct your triplets?</p>

S_M: <aside class="quote no-group" data-username="Oli" data-post="2" data-topic="71708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>How would you like to construct your triplets?</p>
</blockquote>
</aside>
<p>hi, actually I would like to have some triplet data for the triplet network. (such as (anchor, positive, negative) where anchor and positive come from the same class and negative comes from another class)</p>

Oli: <p>I see two good ways to do it.</p>
<ol>
<li>Write a custom dataset which will return triplets. Flatten &amp; batch the triplets -&gt; model -&gt; reconstruct triplets -&gt; loss</li>
<li>Write a dataset that doesn’t directly returns the triplets. Dataset returns the samples -&gt; model -&gt; construct all possible triplets based on labels (all samples that is not from the same class can be viewed as a negative sample) -&gt; loss. This gives you many triplets if your batch size is high.</li>
</ol>
<p>A dataset class to customize</p>
<pre><code class="lang-auto">from torchvision import datasets

class MyCifar10(datasets.CIFAR10):
  def __init__(self, path, transforms, train=True):
    super().__init__(path, train, download=True)
    self.transforms = transforms

  def __getitem__(self, index):
    im, label = super().__getitem__(index)
    return self.transforms(im), label
</code></pre>
<p>Create triplets from batch (wont work in your case but might give you something to go from)</p>
<pre><code class="lang-auto">def create_triplets(originals, transformed):
  ''' Create input images, transformed images, and then negatives as both transformed negatives and non-transformed negatives '''
  batch_size = originals.size(0)
  n_repeat = (batch_size - 1) * 2

  anchors = originals.repeat_interleave(n_repeat, dim=0)
  positives = transformed.repeat_interleave(n_repeat, dim=0)

  mask = [i for i in range(batch_size**2) if i % (batch_size + 1) != 0]
  n1 = transformed.repeat(n_repeat + 1, 1)[mask]
  n2 = originals.repeat(n_repeat + 1, 1)[mask]
  negatives = torch.stack((n1, n2), dim=1).view(-1, anchors.size(1))

  return anchors, positives, negatives
</code></pre>
When i resume the training accuracy has big difference. what should i check?

seungtaek94: <p>Hi.</p>
<p>I save the checkpoints using state_dict like below;</p>
<pre><code class="lang-python">def save_checkpoint(states, is_best, output_dir,
                    filename='checkpoint.pth.tar'):
    torch.save(states, os.path.join(output_dir, filename))
    if is_best and 'state_dict' in states:
        torch.save(states['state_dict'],
                   os.path.join(output_dir, 'model_best.pth.tar'))

save_checkpoint({
            'epoch': epoch + 1,
            'model': get_model_name(config),
            'state_dict': model.state_dict(),
            'perf': perf_indicator,
            'optimizer': optimizer.state_dict(),
        }, best_model, final_output_dir)
</code></pre>
<p>This is an example of my situation.<br>
e.g.)<br>
If i stop training at 150 epoch and resume from 151 epoch;</p>
<ul>
<li>150 epochs accuracy is 90%(final accuracy before resume training)<br>
but, 151 epochs accuracy is 80%…</li>
</ul>
<p>There is a big difference in the accuracy before and after resuming training.</p>
<p>Why is this happening and what should I check?</p>

alex.veuthey: <p>Do you have a learning rate scheduler? If so, you will also need to save its state and restart it properly. Of course, the other elements (model, optimizer) need to be reloaded properly…</p>
Predict single image pytorch

Hoang_Nh_t: <p>Hello.I have a question and I hope everyone will help me.<br>
I am predicting an image by my model.<br>
I don’t know the meaning of tensor([[-4,8724,4,9577]]).I thinks when i write model(image),it will return a vector is probabilities of 2 classes.But it not.<br>
Please explain to me<br>
p/s: I am beginner in English so my enlish is not good.I am so sorry about that.Thanks you<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/083591381e180569f5611342225f70b7be540e8a.png" alt="Screenshot from 2020-03-03 01-02-56" data-base62-sha1="1aCAoN0geV1g6Q4VSu21BrP9Squ" width="580" height="356"></p>

G.M: <p>What does your model looks like? I  expect that u used <code>CrossEntropyLoss</code>, in that case, to obtain the probability, use <code>output.softmax(1)</code>.</p>

Hoang_Nh_t: <p>i used crossentropy loss and my model is mobilenetv2</p>

G.M: <p>Ok, u should apply a softmax to the output to get the probability since <code>CrossEntropyLoss</code> is a combination of <code>LogSoftmax</code> and <code>NLLLoss</code>.</p>
Does backward happen on Master GPU or each GPU

BobChen: <p>When use <code>torch.nn.DataParallel</code> to do multi-gpu training,  does <code>loss.backward()</code> only calculate the grad towards the model on the master gpu or calculate the grad on each gpu and then merge them.</p>

tom: <p>The latter.<br>
In particular, the intermediate results needed for backward are typically on the GPU where you did the forward.</p>
<p>Best regards</p>
<p>Thomas</p>

BobChen: <p>Got it, thanks for your reply:)</p>
Tensor size mismatch autoencoder pytorch

emad: <p>I’m using stacked Autoencoder, which is a bunch of Conv layers. However, I’m having a tensor mismatch error, and I’m not sure about the reason. Everything done in the Encoder is reversed in the Decoder!</p>
<p>This is for time-series data. Input shape is  <code>(bactch_size, 1, 3000)</code><br>
Here’s the code:</p>
<pre><code class="lang-auto">lass CDAutoEncoder(nn.Module):
    def __init__(self, input_size, output_size, kernel, stride):
        super(CDAutoEncoder, self).__init__()

        self.forward_pass = nn.Sequential(
            nn.Conv1d(input_size, output_size, kernel_size=kernel, stride=stride, padding=0),
            nn.PReLU(),
        )
        self.backward_pass = nn.Sequential(
            nn.ConvTranspose1d(output_size, input_size, kernel_size=kernel, stride=stride, padding=0),
            nn.PReLU(),
        )

    def forward(self, x):
        y = self.forward_pass(x)
        return y

    def reconstruct(self, x):
        return self.backward_pass(x)


class StackedAutoEncoder(nn.Module):
    def __init__(self):
        super(StackedAutoEncoder, self).__init__()

        self.ae1 = CDAutoEncoder(1, 32, 50, 10)
        self.ae2 = CDAutoEncoder(32, 64, 10, 3)
        self.ae3 = CDAutoEncoder(64, 64, 5, 1)

    def forward(self, x):
        a1 = self.ae1(x)
        a2 = self.ae2(a1)
        a3 = self.ae3(a2)

        return self.reconstruct(a3)

    def reconstruct(self, x):
        a2_reconstruct = self.ae3.reconstruct(x)
        a1_reconstruct = self.ae2.reconstruct(a2_reconstruct)
        x_reconstruct = self.ae1.reconstruct(a1_reconstruct)
        return x_reconstruct
</code></pre>
<p>The error:</p>
<p><code>RuntimeError: The size of tensor a (2990) must match the size of tensor b (3000) at non-singleton dimension 2</code></p>
<p>I’ve tried adding padding and it worked, but when I changed the kernel size I get different tensor-size-mismatch-error. Also, same error appears if I tried to add <code>AvgPool1d(2)</code> layer … I use <code>nn.Upsample(scale_factor=2)</code> in the decoder, but same mismatch error appears.</p>
<p>Apparently, there’s nothing like ‘same’ padding, so is there automated solution for this?</p>

tom: <p>You probably want padding in your convolutions to not lose timesteps.<br>
The typical thing is to have odd kernel sizes and a padding of <code>kernel_size // 2</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

emad: <p>Thank you for your reply, but this doesn’t help in my problem. I have already tried it, but for some reason it gives the same error</p>
How to prepare the Image data for RNN model

suri_g: <p>Hi, I am newbie to PyTorch, I am doing an image classification problem, Please help me how to prepare the data for the RNN model.<br>
my  Data looks like you:</p>
<pre><code class="lang-auto">file_path='dataset'

train=pd.read_csv(os.path.join(file_path,'train.csv'))

test=pd.read_csv(os.path.join(file_path,'test.csv'))
train.head()
Image	Class
0	image7042.jpg	Food
1	image3327.jpg	misc
2	image10335.jpg	Attire
3	image8019.jpg	Food
4	image2128.jpg	Attire

</code></pre>
<p>I prepared my data using OpenCV like this:</p>
<pre><code class="lang-auto">temp=[]
for img_name in train.Image:
    img_path=os.path.join(file_path,'Train Images',img_name)
    img=cv2.imread(img_path)
    img=cv2.resize(img,(64,64))
    emp.append(img)
y=train['Class'].map({"Food":0,"misc":1,"Attire":2,"Decorationandsignage":3})
</code></pre>

ptrblck: <p>How would you like to pass our images to an RNN?<br>
The default input shape would be <code>[seq_len, batch_size, nb_features]</code>.<br>
If you are working on a multi-class classification, I assume you would like to use <code>nn.CrossEntropyLoss</code>?</p>
Building PyTorch from source using pip

orestxherija: <p>A recent blog <a href="https://medium.com/repro-repo/build-pytorch-from-source-on-ubuntu-18-04-1c5556ca8fbf" rel="nofollow noopener">post</a> has claimed that building PyTorch from source yields a considerable training time speed-up with AlexNet-like architectures (approx. <code>31%</code> for CUDA and <code>39%</code> for CPU). These are very impressive performance gains and they come essentially for free.</p>
<p>Looking at the installation <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">guide</a>, I noticed that things are pretty straight-forward if you’re willing to go with Anaconda. However, the environment on which I mostly work prohibits the use of Anaconda; we essentially do everything using pip-like package managers. Browsing around for PyTorch installation guides using pip (while always building from source), I mostly encountered some pages from 2017, before PyTorch introduced <code>pip</code> installation.</p>
<p>I am wondering if anyone can share any <strong>recent</strong> successful attempts of building PyTorch from source while avoiding Anaconda. My OS is <strong>Ubuntu 18.04 LTS</strong>.</p>

albanD: <p>Hi,</p>
<p>The main problem with the pip-based installation from source is the dependencies.<br>
In particular, you want to make sure that cuda, cudnn, (magma if you need linear algebra on cuda), cpu blas (mkl or openblas), cpu lapack (comes with blas usually and is only needed if you need linear algebra) are available and properly detected.</p>
<p>Once these are installed, you can simply install it with <code>python setup.py install</code> (or, I’m not sure but I think it works, <code>pip install .</code>).</p>
Pre-trained models license

Isaac_Kargar: <p>Hey guys,<br>
Can we use pre-trained models on imagenet, voc, and coco for commercial products? using them or finetune them using our data. I saw that imagenet is only for non-commercial uses.</p>

Emre_Bayram: <p><a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a> did you fond an answer to your question? I need clarification about this too.</p>

Isaac_Kargar: <p>No. There isn’t a clear answer. I think you need a lawyer for this issue.</p>
Add Sequential model to Sequential

saluei: <p>Hi I am new to Pyttorch and may be the question be simple,<br>
I want to create Sequential model that have multiple Sequential segment or slice so that is my code</p>
<pre><code>class Net(nn.Module):
      def __init__(self):
            super(Net, self).__init__()
             #input 3 32 32
            self.slic1=nn.Sequential()
            self.slic1.add_module('conv1', nn.Conv2d(3, 6, 7,padding=3))
        
            self.layers = nn.Sequential()
            self.layers.add_module('slic1',self.slic1)
            self.layers.add_module('Act1',torch.nn.ReLU())
           self.layers.add_module('pool1',nn.MaxPool2d(2, 2))
     
           self.layers.add_module('conv2', nn.Conv2d(6, 16 ,5,padding=2))
           self.layers.add_module('Act2',torch.nn.ReLU())
           self.layers.add_module('pool2',nn.MaxPool2d(2, 2))

           self.layers.add_module('flat',nn.Flatten())
           self.layers.add_module('lin1',nn.Linear(16 * 8 * 8, 500))
           self.layers.add_module('Act4',torch.nn.ReLU())
          self.layers.add_module('lin3',nn.Linear(500, 10))
  
     def forward(self, x):
          x_out = self.layers(x)
         return x_out
</code></pre>
<p>But conv layer at self.slic1 TWO time added to main model !!! when execute<br>
net = Net()<br>
weightlist=net.state_dict()<br>
weightlist HAVE TWO time conv1 layer (layer defined in self.slic1) parameter,<br>
what Is wrong ? and how I can create  Sequential model with multiple segment</p>

albanD: <p>Hi,</p>
<p>This is because you have <code>slic1</code> both in <code>self.</code> and <code>self.layers</code>.<br>
If you only want it in <code>self.layers</code>, you should not create it as an attribute of self:</p>
<pre><code class="lang-auto">slic1=nn.Sequential()
slic1.add_module('conv1', nn.Conv2d(3, 6, 7,padding=3))
        
self.layers = nn.Sequential()
self.layers.add_module('slic1',slic1)
</code></pre>

saluei: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:">,<br>
yes my problem solved but I want to access model slice from outside and attach to them<br>
specific loss function in addition to model loss function, slice is not accessible from outside in this way</p>

albanD: <p>You can still access slice from <code>model.layers.slic1</code>.<br>
If you want to save a <code>nn.Module</code> into another one, without it being detected as a child module, you can “hide” it by putting it into a list/dict: <code>self.slic1 = [slic1,]</code> and access it as <code>self.slic1[0]</code>.</p>
Finding coefficients of a polynomial through linear regression

raphaelattias: <p>I am at the very early stages of learning Python, Pytorch and neural networks. I wanted in a simple example to find the coefficients of a polynomial that would go “as closely as possible”, in terms of least squares, of a set of “interpolating points”. So I am trying to find the coefficients of a linear models, linear in its coefficient. We know the explicit solution of this problem as the Maximum Likelihood solution but I wanted to experiment a bit with Pytorch. Here we are doing a steepest descent gradient for our optimizer and MSE for the loss.</p>
<p>When I was trying without initializing a class model, the convergence was much faster and the result a lot better. With this code the loss is, after 10000 iteration, still around 1. Whereas with no class model and a “uglier” code, I was already reaching 10 to the minus 6 and the estimated polynomial was really good.</p>
<p>Also, in this code if I put a learning rate greater than 10 to the minus 4 it diverges to infinity instantly. Whereas in my other code 0.01 is fine.</p>
<p>What could be the reason for this loss of convergence? Please excuse me if my code looks rough.</p>
<pre><code>import torch
import pdb
import numpy as np
import matplotlib
from pylab import *
import torch.nn.functional as F
from torchsummary import summary

inputs = []
labels = []
N = 3 # Number of sampling points
deg = 2 # Degree of our model polynomial

def truef(x): # True polynomial on which the data is based
   return np.sin(x)
   # return truef(x)

for x in np.linspace(2,5,N): # Create the sample inputs and their labels
   # y = np.random.randn()
    y = truef(x)
    s = []
    for i in np.arange(deg):
        s.append(x**(deg-i))
    
    inputs.append(s)
    labels.append([y+randn()/5])
    
class Net(torch.nn.Module): # Initiate the network
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = torch.nn.Linear(deg,1)
    
    def forward(self,x):
        x = self.fc1(x)
        return x
    
model = Net()
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(),lr=0.01,momentum=0.9)


y_data = torch.tensor(labels,requires_grad=False)
x_data = torch.tensor(inputs,requires_grad=False)


scatter(np.linspace(2,5,N),y_data.data)


for epoch in range(1000):
    y_pred = model(x_data)
    loss = criterion(y_pred,y_data)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(f'{loss}')


def predf(x):
    s = 0
    N = len(model.fc1.weight.tolist()[0])
    for i in np.arange(deg):
        s = s + model.fc1.weight.tolist()[0][i]*x**(deg-i)
        
    return s+model.fc1.bias.item()
    
plot(np.linspace(1.9,5.1,100),predf(np.linspace(1.9,5.1,100)))
plot(np.linspace(1.9,5.1,100),truef(np.linspace(1.9,5.1,100)))
</code></pre>
<p><a href="https://i.stack.imgur.com/vBP2v.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/vBP2v.png" alt="Estimated function in orange" width="690" height="451"></a></p>

albanD: <p>Hi,</p>
<p>I guess the main question would be how did you initialize your linear layer when you where doing it by hand?</p>
Do I have to free tensor in pytorch c++?

sh0416: <p>Hi,</p>
<p>While implementing my code using c++ API, I have a question about memory management in pyTorch.<br>
In Python, memory management is automatically handle allocation and free. What about c++?? Do I have to manually free tensor that I allocated in c++ code?</p>
<p>Thanks,</p>

tom: <p>LibTorch <a href="https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization" rel="nofollow noopener">tracks the lifetime</a> of your tensors (except when you use from_blob, in which case your tensors don’t own the memory), so you would not need to delete them except letting them go out of scope.</p>
<p>Best regards</p>
<p>Thomas</p>
Does deleting a tensor to free up GPU memory also deletes the corresponding portion of the computational graph?

coarsesand: <p>Hi,</p>
<p>I have a pytorch module, whose intermediate variables occupy a lot of memory.<br>
In order to prevent the “CUDA [from being] out of memory” I’ve deleted one of the intermediate tensors <code>dist</code> to free up GPU memory via <code>del dist</code>. I was wondering whether deleting this tensor in such a way also deletes / disfigures the corresponding portion of the computational graph that autograd is building up in the forward pass?</p>
<pre><code class="lang-auto">class E_step(nn.Module):

   def __init__(self, std):
      super(E_step, self).__init__()
      self.std = std

   def forward(self, pi, mean, features):
      # uses current parameter estimates to evaluate the responsibilities
      # 
      # INPUTS:
      # 1) pi:        (tensor: shape = [N, K])
      # 2) mean:      (tensor: shape = [N, K, F])
      # 3) features:  (tensor: shape = [N, D, F])
      # 
      # RETURNS:
      # 1) log_posteriors: (tensor: shape = [N, D, K])
      
      N, K, F = mean.shape
      N, D, F = features.shape
      
      dist = torch.empty([N, D, K], out = torch.cuda.FloatTensor())
      for k in range(K):
         dist[:,:,k] = torch.norm(mean[:, k, :] - features, p = 2, dim = 2)

      log_probs_plus_log_pi = (-0.5 * F * torch.cuda.FloatTensor([2 * math.pi * self.std**2] + dist).log() + pi.log()
      
      ############### deleting this tensor to free up GPU memory ###############
      del dist
      #################################################################
      
      log_probs_plus_log_pi = log_probs_plus_log_pi.transpose(0, 1)
      
      return log_probs_plus_log_pi - torch.logsumexp(log_probs_plus_log_pi, dim = 2, keepdim = True)
</code></pre>

ptrblck: <p>It looks like you are never using <code>dist</code>, so deleting it won’t affect the computation graph.<br>
However, you could also just remove the computation in the first place, if you don’t need it for further computation.</p>

coarsesand: <p>My bad, I missed out the <code>dist</code> when declaring the variable <code>log_probs_plus_log_pi</code>:</p>
<pre><code class="lang-auto">log_probs_plus_log_pi = (-0.5 * F * torch.cuda.FloatTensor([2 * math.pi * self.std**2] + dist).log() + pi.log()
</code></pre>
<p>Anyways, <code>dist</code> is definitely a part of the computational graph so what would happen if I deleted it from memory - would the backward pass be affected?</p>

arjds: <p>any updates on this?</p>
Very simple LSTM doesn't seem to learn anything

shamoons: <p>I’m using <code>Dataset</code>, and I have:</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        random_tensor = torch.rand(self.seq_length, self.feature_dim) * 2
        random_tensor = random_tensor - 0.5

        return random_tensor, random_tensor * 2
</code></pre>
<p>So all I’m doing is learning how to multiply a tensor by 2.</p>
<p>My model is:</p>
<pre><code class="lang-auto">class BaselineModel(nn.Module):
    def __init__(self, feature_dim=1, hidden_size=1, num_layers=2):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers)

    def forward(self, x, hidden):
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

    def init_hidden(self, batch_size):
        hidden = torch.zeros(2, 1, 1)
        cell = torch.zeros(2, 1, 1)
        return (hidden, cell)
</code></pre>
<p>So basically, a simple 1 cell, 1 dimension LSTM. My training loop is:</p>
<pre><code class="lang-auto">    model = BaselineModel(feature_dim=FEATURE_DIM, hidden_size=FEATURE_DIM)
    optimizer = optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0001)
    loss_fn = torch.nn.MSELoss(reduction='sum')

    wandb.watch(model)

    for epoch in range(250):
        model.train(True)  # Set model to training mode

        running_loss = 0.0

        for i, data in enumerate(data_loaders['train']):
            hidden = model.init_hidden(13)
            inputs = data[0]
            outputs = data[1]
            optimizer.zero_grad()

            pred, hidden = model(inputs, hidden)

            loss = loss_fn(pred, outputs)

            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            loss.backward()
            optimizer.step()

            running_loss += loss.data

        epoch_loss = running_loss / len(data_loaders['train'])
        wandb.log({"Training Loss": epoch_loss})
        print('Epoch: {}\tLoss: {:.4f}'.format(epoch, epoch_loss))
</code></pre>
<p>However this doesn’t really decrease the LOSS. Starts at 0.9490 and after 250 epochs, stays at 0.92. I realize that an LSTM is overkill for this, but I want to get a VERY basic version working before augmenting with my sequence data</p>
Gradient Computation

cetinsamet: <p>Hello,</p>
<p>I’m sorry if this question is already answered but I couldn’t find any similar question.</p>
<p>Let’s say i have two different models modelA and modelB. They work sequentially. modelA feeds modelB with its output tensors. And these two models’ weights are updated together.</p>
<p><code>optimizer = SGD(parameters=list(modelA.parameters()) + list(modelB.parameters())) #Something like that</code></p>
<p>Imagine a scenario where modelA outputs 10 tensors and modelB use not all of the tensors but 3 of them as input tensors. Well, I’m curious about if the remaining 7 tensors (not used as inputs to modelB and simply wasted) have any effect on gradient update?</p>
<p>I think that they don’t have any effect on gradient update because they are not used in loss calculation. But I just want to be sure about it because it is very important to me not to screw up in this phase.</p>
<p>Thank you very much.</p>

albanD: <p>Hi,</p>
<p>You are right: If they are not used in the loss computation, they won’t contribute to the computed gradients !</p>
Autograd.grad extremely slow

jshanna100: <p>Does anyone have an intuition why backwards applied to autograd.grad would be several hundred times slower than the forward pass? This is with Pytorch 1.4 and cuda toolkit 10.1.243.</p>

albanD: <p>Hi,</p>
<p>That would depend a lot on your forward pass. It should not do that <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Do you have a code sample that repro this? If possible on colab (<a href="https://colab.research.google.com/notebook#create=true">https://colab.research.google.com/notebook#create=true</a>) so that we can easily test it.</p>

jshanna100: <p>And here’s a graph of the full network:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e39df6f601880660f972dc01456da0a1a7677128.png" data-download-href="https://discuss.pytorch.org/uploads/default/e39df6f601880660f972dc01456da0a1a7677128" title="netGx"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39df6f601880660f972dc01456da0a1a7677128_2_393x500.png" alt="netGx" data-base62-sha1="wtAP3QFDMyUCcwCts9vLKWh0vR6" width="393" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39df6f601880660f972dc01456da0a1a7677128_2_393x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39df6f601880660f972dc01456da0a1a7677128_2_589x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39df6f601880660f972dc01456da0a1a7677128_2_786x1000.png 2x" data-dominant-color="F8F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">netGx</span><span class="informations">4100×5206 759 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jshanna100: <p>I tracked the problem to cudnn.benchmark. Only the first iteration takes very long, but if you wait for that to pass, the subsequent iterations take the correct amount of time.</p>

albanD: <p>Ho, right. This is expected then. The cudnn.benchmark takes a very long time the first time it encounters a new input size (as it tests several algorithms).<br>
If your models has many different convolutions with different input sizes, it will do that many times, which can be quite slow indeed.</p>
<p>Happy that you found the reason !</p>
How to use F.linear without defining weights and biases?

Ashima_Garg: <p>Hi,</p>
<p>I want to define a linear layer in my model (Conv layers followed by a linear layer), the input of which can be dynamic depending on the output of the last conv layer.<br>
I want to know if using F.linear is the right way for that? Because the output of the last Conv layer can become the input of the linear layer but according to this  <code>torch.nn.functional.linear(input, weight, bias=None)</code> I will need to define weights and biases for the same.</p>
<p>If there is any alternative for my problem, kindly please suggest.</p>
<p>Thanks,</p>

simaiden: <p>If you use a Linear layer you have to use fixed input size. If you want to use variable input size you may use 1x1 convolutions</p>
Change size of Tensor

eranb: <p>Hi everyone,<br>
I have a simple question,<br>
If I have a tensor with a shape of [1,3,320,320] and I want to convert him to [1,2,320,320]<br>
what is the simple way to use?</p>
<p>Best,<br>
Eran</p>

G.M: <p>Because the second tensor has less values contained, u cannot change the shape without losing some of the values in the first tensor. Let’s say the shape of A is (1, 3, 320, 320), the desired shape is (1, 2, 320, 320), a simple way to do so is  by slicing:</p>
<pre><code class="lang-auto">START, END = 0, 2
#[START, END)
b = A[ : , START:END, : , : ]
</code></pre>
Different preprocessing within same minibatch

serkansulun: <p>Hi! I am trying to implement an efficient dataloader which would yield a minibatch that includes the same sample which undergoes different preprocessing methods. e.g. a single minibatch with size 8 should include: [sample n preprocessed with filter 1, sample n preprocessed with filter 2, … , sample n preprocessed with filter 8]</p>
<p>Next, I want to do the opposite, to have a dataloader which would yield a minibatch with one preprocessing with multiple samples, and the next minibatch would include another preprocessing method.</p>
<p>To speed things up even more, I have processed all my samples with all 8 filters and saved them individually, now they just need to be loaded. But how can I control the loading of the data at the minibatch level, and still ensure parallel loading? Thanks</p>
How to train FCN 8s network using 6 channeled spectral image dataset?

dekopra: <p>Im trying to train the FCN8s model using a multispectral image dataset (Hamlin state beach park High-resolution multispectral dataset - 6 channel data and 19 classes). I’m getting the image segmented (Pixel wise classification) when I trained the network with 3 channel as input. Now trying to train the net using all the 6 channels, For that, I have changed the input channels and the weight tensor size suitable for the 6 channel training with the same hyperparameters, but after training for the same number of epochs im getting all the pixels classified into 0 class.</p>

phan_phan: <p>Do you use pre-trained weights, or do you train from scratch?</p>

dekopra: <p>In the initial layer, I have used random weights and for the rest of the layers I used pre-trained weights</p>

phan_phan: <p>The original first weight tensor is probably of size<br>
<code>[out_channels, in_channels, kernel_size, kernel_size] = [64, 3, 3, 3]</code>.</p>
<p>You could initialize your weight matrix using simply the original matrix repeated 2 times :</p>
<pre><code class="lang-auto">conv.weight = conv.weight.repeat([1, 2, 1, 1])
</code></pre>
<p>The new weight tensor will be of size <code>[64, 6, 3, 3]</code>.</p>
<p>The only possible issue is, the original weights are trained with RGB images, whereas you will use it with IR images. But I don’t think it is a problem.</p>
Multi-machine inference with PyTorch

Luan_Goncalves: <p>Hi, I’m new to distributed computation on PyTorch.<br>
I’m interested in perform a network partitioning so one piece of the network will run on the machine A and the other piece of the network will run on the machine B. The first thing I need to do is to send tensors from machine A to machine B.<br>
So I thought about use the point-to-point communication as in <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">Writing Distributed Applications with PyTorch</a>. I’m trying to adapt this code to send messages between the machines A and B but I have not been well succeed. Can anyone explain the whole pipeline for this?<br>
Any help would be appreciated!</p>

ptrblck: <p>If you want to use model sharding, <a href="https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239/2">this simple example</a> might be useful.<br>
The linked tutorial explains a distributed setup, so let me know, if I misunderstood your use case.</p>

Luan_Goncalves: <p>First of all, thanks for your attention.<br>
It is not exactly what I need but it helped me in another point. So thanks again.<br>
My issue is related to edge computing. Basically I need to run just a couple of layers in a Drone and the other layers will run in a machine equipped with a GPU.<br>
So i thought that i could send messages from Drone to my machine.<br>
It is possible to be made with PyTorch?</p>

ptrblck: <p>That’s a really interesting use case, but I’m not really sure, how well this would work.<br>
You could most likely connect the drone and your workstation to the same network and use DDP indeed.<br>
However, have you thought about the latency this would create?<br>
How long can you wait for the response?</p>

Luan_Goncalves: <p>Yes! You are right.<br>
Actually, I am interested on measuring this kind of problems because my research is concerned on 5G systems.</p>

osalpekar: <p>I think what you may be looking for is our Distributed RPC framework (<a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html?highlight=rpc" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_tutorial.html?highlight=rpc</a>), which allows you to send messages and tensors between workers. Also see the Distributed Autograd Framework (<a href="https://pytorch.org/docs/master/rpc.html#distributed-autograd-framework" rel="nofollow noopener">https://pytorch.org/docs/master/rpc.html#distributed-autograd-framework</a>) for training models that are partitioned across machines. Lastly, here is an example of training an RNN using RPC/Distributed Autograd: <a href="https://github.com/pytorch/examples/tree/master/distributed/rpc/rnn" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/distributed/rpc/rnn</a>.</p>

Luan_Goncalves: <p>Thanks <a class="mention" href="/u/osalpekar">@osalpekar</a>! It helped me a lot.<br>
Actually, I want to thank you both for your attention.</p>
How can I speed up my custom layer in PyTorch？

qi_cao: <p>Hello,</p>
<p>I have written a custom layer by using NumPy. But it’s very slow during training because there are several for loops in the code. (for batch_size, for out_channels, and for in_channels)</p>
<p>Is there any method I can try to speed up the training process？</p>
<p>I tried to use <code>Joblib.Parallel</code> but failed in Pytorch forward and backward.<br>
And I tried to change the Numpy operation to the torch operation, then use the .cuda(), but still very slow.</p>
<p>I saw the article about the C++ and CUDA Extension. But still confused and don’t know where should I start.</p>
<p>And if I want to use C++ to write the custom layer, Should I need to know C++ very well？</p>
<p>Thanks in advance.</p>

tom: <p>You probably want to look into vectorizing your function better, i.e. ideally eliminate the loops over in and out channels (the batch one might be less urgent).<br>
Unless you’ll go all the way to custom CUDA kernels (and that’s probably quite some effort) you’ll have a hard time solving this, even using C++.</p>
<p>Best regards</p>
<p>Thomas</p>

qi_cao: <p>Thanks for your reply. First, I will try to vectorizing my function. If still slow, I will take some time to learn about the CUDA.</p>
Performance degradation when GPU io and compute are parallel

miyano: <p>I am using distributeddataparallel for multi-GPU training using pytorch, and a process uses one GPU.<br>
When I follow the normal formular in pytorch, the inference time is OK, like this.</p>
<pre><code class="lang-auto">x,y = next(train_loader)
x = x.cuda(rank)
y = y.cuda(rank)
t0 = time.time()
y1 = model(x)
torch.cuda.synchronize()
inference_time = time.time()-t0
</code></pre>
<p>But when I get the data from another thread, which always read data from train_loader and input it to a queue. The code is as following.</p>
<pre><code class="lang-auto">args.data_queue=queue.Queue()
def load_data_queue(rank, dataloader, args):
    n = 0
    while True:
        try:
            x,y = next(dataloader)
            x = x.cuda(rank)
            y = y.cuda(rank)
            args.data_queue.put([feature, label])
        except StopIteration:
            print('load queue quits normally')
            return
...
t = threading.Thread(target=load_data_queue, args=(
        rank, train_loader, args), daemon=True)
t.start()
...
x,y = args.data_queue.get()
t0 = time.time()
y1 = model(x)
#torch.cuda.synchronize()
inference_time = time.time()-t0
</code></pre>
<p>The inference_time will increase a lot.<br>
To my understanding, GPU i/o should not influence GPU compute. What is causing this phenomenon？</p>

ptrblck: <p>You might run into the GIL in Python.<br>
Wouldn’t the first DDP approach work or what are the shortcomings you are facing that need the second approach?</p>

miyano: <p>Thank you for your reply. We are doing some research which acquires GPU i/o and compute to run in parallel. And when we validated our idea, we observed this problem.</p>

miyano: <p>Thanks！I change thread parallel to process parallel，and the train performance becomes normal.</p>
Computation graph optimization during training

Dale_Song: <p>Hi, is it possible or neccessary to optimize the dynamic computation graph generated during training for a higher throughput?If it is, then what is the recommended way to achieve that? Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>This is not necessary in general.<br>
If you really want to try and get the best, you should use torchscript model with cpp inference to strip away the python interpreter.</p>

Dale_Song: <p>Thank you for the reply. But my use case is to improve the training throughput, if I understand it right, torchscript can only improve performance for network inference rather than training(forward&amp;backward).And do you have any advice about how to improve pytorch forward&amp;backward efficiency?</p>

albanD: <p>You can actually perform training with torchscript.<br>
You can try to torchscript your python code during training.</p>
<p>That being said, if your network is a regular architecture, we try to make sure that the performance for these are as good as possible out of the box.</p>

111254: <p>Hi Dale,have you slove this problem, and TorchScript works?</p>

111254: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="63129">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>torchscript</p>
</blockquote>
</aside>
<p>Did torchscript can actually optimize computation graph during traing?</p>

G.M: <p>Yes, touchscript does optimize the graph at train time. See :<br>
<a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/#writing-custom-rnns" rel="nofollow noopener">https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/#writing-custom-rnns</a>.</p>
Tensor that declared in model(class) is in gpu or cpu?

seungtaek94: <pre><code class="lang-python">class model(nn.Module):
     ...

     def forward(self, x):
              a = tensor.zeros(1, 256, 64, 48)
              ...

y = model(input).cuda()
</code></pre>
<p>if i call model like this then local parameter <code>a</code> is in GPU?</p>

G.M: <p><code>.cuda()</code> only transfers the registered parameters of a model to the gpu, since <code>a</code> is a tensor created locally in <code>forward</code>, u have to transfer it to gpu mannually.</p>

seungtaek94: <p>i got it, thanks for reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Discrepancy in input and hidden tensor location

kl_divergence: <p>This works fine:</p>
<pre><code class="lang-auto">model = LSTM(num_features, output_length)
sample_input = torch.rand(128,10,7)
output = model(sample_input)
output.shape
</code></pre>
<p>But this gives an error:</p>
<pre><code class="lang-auto">model = LSTM(num_features, output_length).cuda()
sample_input = torch.rand(128,10,7).cuda()
output = model(sample_input)
output.shape
</code></pre>
<p>Returns <code>RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu</code></p>
<p>I’m using the latest version of Pytorch. Is this an intended behaviour ? I’m clearly specifying the location of both input tensor and model. But still the hidden state tensor is not on GPU. Why is this happening ? I’m using a server with multiple GPUs but I’ve specified this already:</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"]="0"
torch.cuda.set_device(0)</code></pre>

kl_divergence: <p>Solved it. Tensors declared in forward() have to be manually transferred on GPU.</p>
CUDA error: device-side assert triggered while calculating loss for LSTM

Dylan_Yung: <p>I have this code</p>
<pre><code class="lang-auto">def lstmTrain(rnn: RNN, input_line_tensor: torch.Tensor, target_line_tensor: torch.Tensor):

    rnn.train()

    criterion = nn.NLLLoss()

    hidden = rnn.initHidden()

    hidden = (hidden[0].to(DEVICE), hidden[1].to(DEVICE))

    rnn.zero_grad()

    loss = 0

    for i in range(input_line_tensor.size(0)):

        # input_line_tensor[i] is [1, input categories]

        output, hidden = rnn(input_line_tensor[i].unsqueeze(1), hidden)

        # output is [1,1,output categories]

        # target_line_tensor[i] is [1,30]

        l = criterion(output, target_line_tensor[i])

        loss += l

    loss.backward()

    for p in rnn.parameters():

        p.data.add_(-LR, p.grad.data)

    return output, loss.item() / input_line_tensor.size(0)
</code></pre>
<p>at loss += l it’s throwing the error in the title. Here is my rnn code as well</p>
<pre><code class="lang-auto">import torch

import torch.nn as nn

class LSTM(nn.Module):

    def __init__(self, input_size: int, num_layers: int, hidden_sz: int, output_size: int):

        super(LSTM, self).__init__()

        self.hidden_size = hidden_sz

        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_size= input_size, hidden_size=hidden_sz, num_layers=num_layers)

        self.fc1 = nn.Linear(hidden_sz, output_size)

        self.dropout = nn.Dropout(0.1)

        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input: torch.Tensor, hidden: torch.Tensor):

        output, hidden = self.lstm(input, hidden)

        output = self.fc1(output)

        output = self.dropout(output)

        output = self.softmax(output)

        return output, hidden

    def initHidden(self, batch_size=1):

        return (torch.zeros(self.num_layers, batch_size, self.hidden_size),

                torch.zeros(self.num_layers, batch_size, self.hidden_size))
</code></pre>

ptrblck: <p>Could you run the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace?</p>

Dylan_Yung: <p>I got</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
cuda runtime error (710) : device-side assert triggered at C:/w/1/s/windows/pytorch/aten/src\THCUNN/generic/SpatialClassNLLCriterion.cu:127

</code></pre>

ptrblck: <p>This error is most likely thrown, if you pass an invalid index.<br>
<code>nn.CrossEntropyLoss</code> (and thus also <code>nn.NLLLoss</code>, which will be called internally) expect a target tensor as a <code>LongTensor</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.<br>
This error will be thrown, if you pass indices outside of the mentioned range.</p>