akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get nan loss with CrossEntropyLoss

username: roy.mustang
cooked: <p>Hi all.<br>
I’m new to Pytorch. I’m trying to build my own classifier. I have a dataset with nearly 30 thousand images and 52 classes and each image has 60 * 80 size.<br>
This is my network (I’m not sure about the number of neurons in each layer).</p>
<pre><code class="lang-auto">class my_network(nn.Module):
    
    def __init__(self, class_num, act=F.relu):
        
        super(my_network, self).__init__()
        
        self.layer1 = nn.Linear(1 * 60 * 80, 50 * 30 * 40)
        self.act1 = act 
        
        self.layer2 = nn.Linear(50 * 30 * 40, 70 * 10 * 15)
        self.act2 = act 
        
        self.layer3 = nn.Linear(70 * 10 * 15, 90 * 5 * 8)
        self.act3 = act
        
        self.layer4 = nn.Linear(90 * 5 * 8, 80)
        self.act4 = act
        
        self.layer5 = nn.Linear(80, class_num)
        
    def forward(self, x):

        x = x.view(x.size(0), -1)

        x = self.layer1(x)
        x = self.act1(x)

        x = self.layer2(x)
        x = self.act2(x)

        x = self.layer3(x)
        x = self.act3(x)

        x = self.layer4(x)
        x = self.act4(x)

        x = self.layer5(x)
        return x
</code></pre>
<p>I’m using Cuda for my model, CrossEntropyLoss for my criterion, and SGD for my optimizer.</p>
<pre><code class="lang-auto">model = my_network(len(classes))
model = model.to(device)

learning_rate = 0.01
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
</code></pre>
<p>I use the following code for training my model.</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
      train_loss = 0.
 
      for images, labels in train_loader:
          
        images = images.to(device)
        labels = labels.to(device)
      
        optimizer.zero_grad()

        outputs = model(images)

        loss = criterion(outputs, labels)

        loss.backward()
        
        optimizer.step()

        print(loss.item())
        train_loss += loss.item()
    
      average_loss = train_loss / len(train_loader)
</code></pre>
<p>And when I run this, I get nan in output. The loss.item() returns nan in the first epoch.</p>
<pre><code class="lang-auto">nan
nan
nan
nan
...

</code></pre>
<p>also, I don’t want to use normalization for my data and I want to use them in this manner.<br>
what am I doing wrong?</p>

username: ptrblck
cooked: <p>What range are your inputs using at the moment?<br>
Is the first iteration already creating the NaN outputs or after a couple of updates?<br>
In the latter case, you could add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of the script, which would point to the operation, which created the first NaN output.</p>
Labeling Aerial images (along with probability of label)

username: Sundar_Krishna
cooked: <p>Hi ,</p>
<p>I basically want to seperate and highlight various landscapes and objects in an aerial view of an area. I came across semantic segmentation(segnet) and Unet. I initially tried jetson.utils and jetson.inference but i dont think it has a lot of options for different landscapes and objects. I also want to seperately take the labels and convert them into json format(also one of my requirement)</p>
<pre><code class="lang-auto">[
{
“label”: “water”,
“probability": 6.999387
},
{
“label”: “sand”,
“probability": 6.996159
},
{
“label”: “people",
“probability": 6.97412
},
{
“label”: “boat",
“probability": 6.002275
}
] 
</code></pre>
<p>I am looking out for more resources ,which can help me with this… links,resources etc.,</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/174bd4ca5683172b95d4e7352b3c24081cf02b1a.png" alt="Screenshot 2020-07-10 at 10.10.37 PM" data-base62-sha1="3k5rRF0H5FF82uEMGUEpOxyVJOi" width="573" height="426"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1cee39ac90db696e532656bbe9feb892b248c8fb.png" alt="Screenshot 2020-07-10 at 10.10.46 PM" data-base62-sha1="47VLoK2oJhxRoOKe6rKSN5TVFxh" width="567" height="379"> !</p>
<p>here is one example with <a href="https://github.com/dusty-nv/jetson-inference/blob/master/docs/segnet-console-2.md" rel="noopener nofollow ugc">jetson utils and inference</a>(segnet)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6f66b057e836271a923dcbe301a88b7ef1a882.png" alt="aerial2_deep" data-base62-sha1="1ujrVNMU0AntVkSR9XxjPRJbRDA" width="573" height="426"></p>
Multiple processes of the same network

username: Javier_Rodriguez_Pui
cooked: <p>Hi,<br>
I want to perform deep ensembles in my network and I have a question related to how to perform different training times in parallel.<br>
With a batch size of 16 my network takes like 3 minutes per epoch and 2GB of GPU.</p>
<p>Having enough GPU memory, I was expecting that throwing 4 processes It would take also 3 min,  but it took 12 minutes.</p>
<p>Clearly, I am doing something wrong or I don’t understand parallelism in GPU usage …</p>
<p>Could anyone help me in this matter?</p>

username: ptrblck
cooked: <p>The GPU might be utilized completely (regarding the compute), while the memory isn’t fully used (similar to your PC, where the CPU might be fully utilized, while your RAM is quite empty).<br>
Usually PyTorch kernels are implemented in such a way to try to fully use the GPU, so that running multiple models in parallel might not be possible (you would also have to use CUDA streams for it, which might be tricky).</p>
Def get_data, rootdir

username: shangguan91
cooked: <p>i want to load data by def get_data. First code is a example for getting data and transfrom if data is Gotik. however, i want to def get data and return data and labels, it wont work.</p>
<pre><code class="lang-auto">train_files = os.listdir('/content/drive/My Drive/Churches_new/Train')
data_list = []
for f in files:
    if f[-3:] == "obj":
        d = read_obj('rootdir' + f)
        if f[5:10] == "Gotik":
            d.y = torch.tensor([0])
        else:
            d.y = torch.tensor([1])
        data_list.append(transforms(d))
</code></pre>
<pre><code class="lang-auto">def get_data(rootdir):


  data=[]
  files = os.listdir('rootdir')

  for f in files:
    if f[-3:] == "obj":
        data = read_obj('rootdir ' + f)
       labels=[]
 if f[5:10] == "Gotik",
labels= "Gotik",
else labels="Romantik
         return data label

</code></pre>
<p>pls help</p>

username: ptrblck
cooked: <p>What is currently not working?<br>
Do you get any errors are unexpected results?</p>
why is inference speed of the same model with different size input tensor so unreasonable?

username: wumeng
cooked: <p><strong>my device: win10 1080ti i7-8700K</strong><br>
<strong>my model: yolov5s</strong><br>
I test inference speed with two different input size called separately bigsize:1X3X800X1376 and smallsize:1X3X416X736 in gpu and cpu mode. Test result is as follows.<br>
<strong>compute device ------------------ inputsize ------------------------ FPS</strong><br>
<strong>1080ti-------------------------- bigsize:1X3X800X1376-------------- 49</strong><br>
<strong>1080ti-------------------------- smallsize:1X3X416X736------------- 43</strong><br>
<strong>i7-8700K-----------------------bigsize:1X3X800X1376---------------1.3</strong><br>
<strong>i7-8700K-----------------------smallsize:1X3X416X736---------------5.6</strong><br>
When i record the timestamp, I use this function<br>
<code>def time_synchronized(): torch.cuda.synchronize() if torch.cuda.is_available() else None return time.time()</code></p>
<p>I can not understand why big input size get slower FPS. Thank you for your reply.</p>
How to prune weights less than a threshold in PyTorch?

username: spaul13
cooked: <p><em>How to prune weights of a CNN (convolution neural network) model which is less than a  <strong>threshold value</strong>  (let’s consider prune all weights which are &lt;= 1).</em> I want to prune less significant weights so that accuracy won’t be degraded.</p>
<p>How we can achieve that for a weight file saved in .pth format in PyTorch?</p>

username: Michela
cooked: <p>You can implement it as an extension of <code>torch.nn.utils.prune</code>. You can find instructions on how to create your custom pruning function by following the pytorch pruning tutorial: <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#extending-torch-nn-utils-prune-with-custom-pruning-functions" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#extending-torch-nn-utils-prune-with-custom-pruning-functions</a></p>
<p>In practice, in <code>compute_mask</code>, among other things, you will have to generate a mask that preserves all the entries in the tensor <code>t</code> that are above your threshold: <code>mask = (t &gt; 1.).to(t.dtype)</code></p>