akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about RoIHead Customization

username: tanabe.d.zc32s
cooked: <p>I have consulted with you several times in the Pytorch Forum.</p>
<p>I am building a custom Faster R-CNN model which outputs object’s boundary box, label, and additional attributes for a project, but I am stucking.</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
C:\Users\TANABE~1\AppData\Local\Temp/ipykernel_19556/2267599829.py in &lt;module&gt;
     17 
     18 
---&gt; 19         loss_dict = model(x1, x2)
     20 
     21         losses = sum(loss for loss in loss_dict.values())

C:\anaconda\envs\pytorch-gpu\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

C:\Users\TANABE~1\AppData\Local\Temp/ipykernel_19556/3309378347.py in forward(self, images, targets)
     84 
     85         proposals, proposal_losses = self.rpn(images, features, targets)
---&gt; 86         detections, detector_losses, leaf_age_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
     87         detections = self.transform.postprocess(detections, images.image_sizes, original_image_sizes)
     88 

C:\anaconda\envs\pytorch-gpu\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

C:\anaconda\envs\pytorch-gpu\lib\site-packages\torchvision\models\detection\roi_heads.py in forward(self, features, proposals, image_shapes, targets)
    752         box_features = self.box_roi_pool(features, proposals, image_shapes)
    753         box_features = self.box_head(box_features)
--&gt; 754         class_logits, box_regression = self.box_predictor(box_features)
    755 
    756         result: List[Dict[str, torch.Tensor]] = []

ValueError: too many values to unpack (expected 2)

</code></pre>
<p>Looking at the error statement above, I figured it was happening because the default RoIHead received more values than expected.However, it is not smart to rewrite the torchvision code itself.</p>
<p>Could you please give me some good solutions?</p>
<p>Thank you.</p>
Handling nan loss

username: thyeros
cooked: <p>Hi, pytorch gurus:</p>
<p>I have a training flow that can create a nan loss due to some inf activations, and I already know this is because of noisy dataset yet cleaning up the dataset is hard/disallowed and dataset/dataloader is fixed. Hence, I’m looking for a way to live with nan loss.</p>
<p>One thing I can do is that after backprop is over, I can reset the gradients all zero and continue for the next batch. But, this requires to iterate all the grads, which looks excessive.</p>
<p>Is there a clean way to set loss as zero and ensure all gradients become zero too automatically, rather than resetting after backprop? I tried to reset loss manually zero, but once I do backprop, still grad is abnormal. I can skip the whole backprop, but this can be a problem in DDP env, as some GPUs will still do backprop, causing deadlock (and, I don’t want additional sync up).</p>
<p>Any hint or guidance?</p>

username: JuanFMontesinos
cooked: <p>Well when you run optimizer.zero_grad() you are basically iterating over all the parameters. DUnno why you say it’s expensive.</p>
<p>Anyway. Just check if the loss is nan</p>
<pre><code class="lang-auto">if not torch.isnan(loss):
  loss.backward()
  optimizer.step()
</code></pre>
<p>And that’s all</p>

username: thyeros
cooked: <p>I should have been more clear about that. Resetting grad is mechanically not expensive, but what I meant was losing these gradients itself and not getting weight updated in DDP setup. I have N GPUs, each B samples. If there is one nan in loss, then after all-reduce all grads need to be discarded, and N*B samples  don’t contribute to the convergence.</p>
<p>Now, I’m thinking of letting nan loss backprop but hijack the Allreduces using hook?</p>

username: JuanFMontesinos
cooked: <p>You can simply remove the NaNs at some point inside the model by masking the output. If your loss is elementwise it’s pretty simple to do. If your loss depends on the structure of the tensor (i.e. a matrix multiplication) then replace the NaN by the null element. For example, tensor[torch.isnan(tensor)]=0 or tensor[~torch.isnan(tensor)]</p>
<p>In the last term. most of the losses are an average of all the samples in the batch. N*B in your case.<br>
Official pytorch losses has a flag called reduce or something similar which allows to return the value of the loss for each element of the batch instead of the average.</p>
<p>At that step you can simply remove the NaN element and do a manual average+backprop.</p>
<p>Using hooks seems too complicated to me.</p>

username: thyeros
cooked: <p>I expected that sort of, but to double confirm: So, I have to replace all Nan in the activations with something else like zero?</p>

username: JuanFMontesinos
cooked: <p>Yep but be aware of your problem.<br>
For example, if your gradient is a multiplication, then replacing by 0 does the job. You have to think of the null element (that one which is gonna make the gradient to be 0)<br>
Otherwise just do the simple trick which is removing those NaNs from the N*B loss tensor.<br>
There is a function already to replace nans in pytorch.</p>
Why random erasing should be applied as Post transfromation

username: Sanjayvarma11
cooked: <p>Hello everyone.I really appreciate this community bcz everyone replies so fast.So my doubt is i am applying data augumentation where i found about random erasing .In actual documentation</p>
<pre><code class="lang-auto">&gt; transform = transforms.Compose([
&gt;&gt;&gt; transforms.RandomHorizontalFlip(),
&gt;&gt;&gt; transforms.ToTensor(),
&gt;&gt;&gt; transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
&gt;&gt;&gt; transforms.RandomErasing(),
&gt;&gt;&gt; ])
</code></pre>
<p>Here why random erasing is applied after ToTensor() and  ToNormalize() functions.<br>
When i applied random erasing before ToTensor() and ToNormalize() it is giving me an error like image object has no attribute size.Help will be greately appreciated</p>

username: ptrblck
cooked: <p>The implementation of <code>F.erase</code> is quite simple as seen <a href="https://github.com/pytorch/vision/blob/505cd6957711af790211896d32b40291bea1bc21/torchvision/transforms/functional.py#L885">here</a> and doesn’t need <code>PIL</code> or any other image library as the backend, which is most likely why it was implemented for tensors directly.</p>

username: Haizhuolaojisite
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/sanjayvarma11">@Sanjayvarma11</a><br>
resize = transforms.Resize(size=(INPUT_HEIGHT,<br>
INPUT_WIDTH))<br>
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>
std=[0.229, 0.224, 0.225])<br>
DATA_TRANSFORMS = {<br>
‘train’:<br>
transforms.Compose([<br>
transforms.Resize(size=(256,256)),<br>
transforms.RandomCrop(INPUT_HEIGHT,INPUT_WIDTH),<br>
transforms.RandomHorizontalFlip(p=0.4),<br>
transforms.RandomVerticalFlip(p=0.6),<br>
transforms.ToTensor(),<br>
transforms.RandomErasing(p=0.5),<br>
normalize<br>
]),</p>
<p>I put random erasing after ToTensor but before normalize, and it solves the error like ‘AttributeError: ‘Image‘ object has no attribute ‘shape’.</p>
<p>Should we put this RandomErasing after or before normalize? and Why?</p>

username: ptrblck
cooked: <p>I don’t think it would matter if the <code>RandomErasing</code> transformation is applied before or after the normalization, since:</p>
<ol>
<li>
<code>RandomErasing</code> doesn’t depend on the input values and will just erase patches from it</li>
<li>
<code>Normalize</code> doesn’t depend on the input values as the <code>mean</code> and <code>stddev</code> are already provided.</li>
</ol>

username: Haizhuolaojisite
cooked: <p>I see, thanks for your reply. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Binary clasification - unbalanced data help with the last layer and the loss

username: ni_tempe
cooked: <p>I have a NLP binary classification problem with class imbalance. Class 2 is present &lt;2% of times<br>
I am using glove embedding to convert text to numbers.</p>
<p>My earlier questions advised me to not use the last layer as <code>softmax</code>, <code>logsoftmax</code> or <code>sigmoid</code>. Also I shouldn’t use <code>CrossEntropyLoss</code>.</p>
<p>I am not sure what should be my last layer and my loss function. Would appreciate suggestions:</p>
<p>My current code as below:</p>
<p>My network’s last few layers are as below:</p>
<pre><code class="lang-auto">        self.batch_norm2 = nn.BatchNorm1d(num_filters)
        
        self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)
        
        self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)
        
        self.fc3 = nn.Linear(fc2_neurons, 2)
             
        self.softmax = nn.Softmax(dim=1)
</code></pre>
<p>question 1) should I replace the last 2 lines from above with the below. Let me know if there are any other choice.<br>
I am using <code>sigmoid</code> after <code>linear</code> as I will get values between 0 and 1 and then I could use different probability cutoffs if required</p>
<pre><code class="lang-auto">self.fc3 = nn.Linear(fc2_neurons, 1)
self.sigmoid=nn.Sigmoid()
</code></pre>
<p>Loss as below</p>
<pre><code class="lang-auto">cross_entropy = nn.CrossEntropyLoss(weight=class_wts)
</code></pre>
<p>question 2) And Loss with the line shown as below? Let me know if there are any other choice</p>
<pre><code class="lang-auto">#as class_wts have class weights
BCE_loss=nn.BCELoss(pos_weight = torch.tensor (class_wts[1]/class_wts[0]))
</code></pre>
Speeding up loading data from spark

username: drj3122
cooked: <p>I’ve been working on using petastorm to train PyTorch models from spark dataframes (somewhat following <a href="https://docs.databricks.com/applications/machine-learning/load-data/petastorm.html" rel="noopener nofollow ugc">this guide</a>). I’m curious if there are any ways I can speed up data loading.</p>
<p>Here’s a basic overview of my current flow. <code>df_train</code> is a spark dataframe with three columns: x1 (float), x2 (binary 0,1), y (float). I’m using pyspark.</p>
<pre><code class="lang-auto">x_feat = ['x1', 'x2']
y_name = 'y'
spark.conf.set(SparkDatasetConverter.PARENT_CACHE_DIR_URL_CONF, "file:///dbfs/tmp/petastorm/cache")
converter_train = make_spark_converter(df_train)

with converter_train.make_torch_dataloader(batch_size=bs) as train_dataloader:
    train_dataloader_iter = iter(train_dataloader)
    steps_per_epoch = len(converter_train) // bs
    for step in range(steps_per_epoch):
      pd_batch = next(train_dataloader_iter)
      pd_batch['features'] = torch.transpose(torch.stack([pd_batch[x] for x in x_feat]), 0, 1)
      inputs = pd_batch['features'].to(device)
      labels = pd_batch[y_name].to(device)
      ... # modeling and stuff
</code></pre>
<p><code>pd_batch</code> is a dictionary with an entry for each column in the original <code>df_train</code>. My concern is that the <code>torch</code> operations might not be optimal. Something else I tried was first creating an array column in my spark dataframe for x1 and x2. I was surprised to find that each epoch was more than 2 times slower than the above strategy.</p>
<pre><code class="lang-auto">df_train = df_train.withColumn("features", array("x1", 'x2')).select('features', 'y')
# remainder same as above except torch.transpose(torch.stack([pd_batch[x] for x in x_feat]), 0, 1) was removed
</code></pre>
<p>Are there any improvements I can make here?</p>
CUDA_VISIBLE_DEVICES is no longer valid after calling torch.cuda.is_available()

username: ymwangg
cooked: <p>I found <code>CUDA_VISIBLE_DEVICES</code> is no longer valid after calling <code>torch.cuda.is_available()</code> or <code>torch.cuda.device_count()</code>. It seems like these two functions will freeze <code>CUDA_VISIBLE_DEVICES</code> after the first call. Is this an intended behavior or bug? I found this behavior caused some trouble in torch_xla with multi-processing as discussed here <a href="https://github.com/pytorch/xla/issues/3347" class="inline-onebox" rel="noopener nofollow ugc">Calling torch.cuda.is_available() with multiprocessing exhausts memory. · Issue #3347 · pytorch/xla · GitHub</a>.</p>

username: ptrblck
cooked: <p>As explained in the <a href="https://github.com/pytorch/xla/issues/3347#issuecomment-1058449067">linked issue</a>, <code>CUDA_VISIBLE_DEVICES</code> has to be set before the first CUDA call.</p>
Negative loss with UNet

username: krishmani.85
cooked: <p>Hi, when I run the UNet model implemented as listed in</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py">
  <header class="source">

      <a href="https://github.com/mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py" target="_blank" rel="noopener nofollow ugc">mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py</a></h4>


      <pre><code class="lang-py">from collections import OrderedDict

import torch
import torch.nn as nn


class UNet(nn.Module):

    def __init__(self, in_channels=3, out_channels=1, init_features=32):
        super(UNet, self).__init__()

        features = init_features
        self.encoder1 = UNet._block(in_channels, features, name="enc1")
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder2 = UNet._block(features, features * 2, name="enc2")
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder3 = UNet._block(features * 2, features * 4, name="enc3")
        self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder4 = UNet._block(features * 4, features * 8, name="enc4")
        self.pool4 = nn.MaxPool2d(kernel_size=2, stride=2)
</code></pre>



  This file has been truncated. <a href="https://github.com/mateuszbuda/brain-segmentation-pytorch/blob/master/unet.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
with the default run command python train.py</p>
<p>I see negative loss values and 0 Best validation mean DSC: 0.000000.</p>
<p><strong>loss for step: 622 = [-0.9068316221237183]</strong><br>
<strong>loss for step: 623 = [-0.9329317808151245]</strong><br>
<strong>loss for step: 624 = [-0.9376015663146973]</strong><br>
<strong>Best validation mean DSC: 0.000000</strong></p>
<p>This code is using Torch 1.10 on Nvidia GPU with CUDA 11.<br>
Is there something that needs fixing in the code (dataloader etc.)?</p>

username: ptrblck
cooked: <p>Could you check which loss function you are using and what the expected inputs are?<br>
I would guess that either the model outputs or targets do not contain the desired values.</p>

username: krishmani.85
cooked: <p>The loss is DiceLoss.</p>
Does anybody know what [-1] in optimizer.param_groups?

username: XTROLLERX
cooked: <pre><code class="lang-auto">def fit(epochs,train_set,val_dl,model,lr):

   optimizer = torch.optim.Adam(model.parameters(), lr) #defining the optimizer

   scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, lr, epochs=epochs, steps_per_epoch = len(train_set))# learning rate scheduler

   def get_lr():

    for param_group in optimizer.param_groups:  #getting the learning rates of e

      return param_group["lr"]

   history = []      

   

   for epoch in range(epochs):

      model.train()

      train_loss = []

      lrs = []

      for batch in train_set:

        loss = training_step(batch, model)

        train_loss.append(loss)

        loss.backward()

        nn.utils.clip_grad_norm_(model.parameters(), 0.1)

        optimizer.step()

        optimizer.zero_grad()

        lrs.append(get_lr())

        scheduler.step()

      #validation 

      results = validation_combine_loss(val_dl,model)

      results["lrs"] = lrs

      results["train loss"] = torch.stack(train_loss).mean().item()

      epoch_end(results,epoch)

      history.append(results)

   return history

def epoch_end(result,epoch):
  print("epoch: [{}], last_lr {:.5f}####,  Epoch_loss:{:.4f}, Epoch_accuracy {:.4f}, train_loss {:.4f}" 
.format(epoch, result["lrs"][-1]####, result["Loss"], result["Accuracy"], result["train loss"] ))

</code></pre>
<p>The areas that need help are denoted with “####” , thanks!</p>

username: ptrblck
cooked: <p>I’m not sure if you are wondering about the usage of:</p>
<pre><code class="lang-python">print("epoch: [{}], last_lr {:.5f}####,  Epoch_loss:{:.4f}, Epoch_accuracy {:.4f}, train_loss {:.4f}" 
.format(epoch, result["lrs"][-1]####, result["Loss"], result["Accuracy"], result["train loss"] ))
</code></pre>
<p>but if so:<br>
<code>[-1]</code> indexes the “last” element of the <code>list</code>.</p>

username: XTROLLERX
cooked: <p>So in this case of the [-1] index, it is going to print out the learning rates from the very last index? If so why is it necessary to implement [-1] index when using</p>
<pre><code class="lang-auto">for param_group in optimizer.param_groups:  #getting the learning rates of e

      return param_group["lr"]`

def epoch_end(result,epoch):
  print("epoch: [{}], last_lr {:.5f},  Epoch_loss:{:.4f}, Epoch_accuracy {:.4f}, train_loss {:.4f}" 
.format(epoch, result["lrs"][-1], result["Loss"], result["Accuracy"], result["train loss"] ))
</code></pre>
<p>I saw this from a tutorial and I have been trying to figure out what could be the purpose/reason for using [-1] index in this scenario?</p>

username: ptrblck
cooked: <p>I’m not familiar with the tutorial, but would guess the authors wanted to print only the last learning rate and not all learning rates created by the scheduler during the iterations.<br>
The <code>train_loss</code> is also stacked and the <code>mean</code> is taken for all iterations (one epoch).</p>
Getting gradient of two losses

username: binbbaz
cooked: <p>I am trying to get the gradients of two losses in the following code snippet but all I get is None (AttributeError: ‘NoneType’ object has no attribute ‘data’)</p>
<pre><code class="lang-auto">img = img.to(device)
#img.requires_grad = True
input = model(img)
input_prediction = input.max(1, keepdim=True)[1]

btarget = torch.tensor(2).unsqueeze(0).to(device)
x_prime.requires_grad = True
x_prime_output = model(x_prime)
x_prime_pred =  x_prime_output.max(1, keepdim=True)[1]
l_target_loss = F.nll_loss(x_prime_output, btarget)
# model.zero_grad()
l_target_loss.backward(retain_graph = True)
target_grad = l_target_loss.grad.data

l_argmax_loss = F.nll_loss(x_prime_output, input_prediction.squeeze(0))
l_argmax_loss.backward()
l_argmax_grad = l_argmax_loss.grad.data
</code></pre>
<p>What am i getting wrong, please?</p>

username: binbbaz
cooked: <p><a class="mention" href="/u/alband">@albanD</a> , could you help sir</p>

username: albanD
cooked: <p>Hey!<br>
In general, you should never user .data.</p>
<p>Also the gradient computed by the backward pass is for the model parameters (generally Tensors you create with requires_grad=True). So here you will get gradients for x_prime for example. Not the loss.</p>

username: binbbaz
cooked: <p>Thank you for your reply. I added <code>l_target_loss.retain_grad()</code> before calling <code>l_target_loss.backward(retain_graph= True)</code> and it worked. I can now get values for <code>target_grad</code> and  <code>l_argmax_grad</code> respectively. However, both values are thesame. They are both 1’s. I feel something isn’t right</p>

username: binbbaz
cooked: <p><a class="mention" href="/u/alband">@albanD</a> , I feel that I’m doing something wrong. What I actually want is the gradient of the <code>target_loss</code> with respect to the input (x) and gradient of the <code>l_argmax_loss</code> with respect to the input (x). The input <code>x</code> is <code>img</code> in the code snippet above. How do I go about it please?</p>

username: albanD
cooked: <blockquote>
<p>They are both 1’s.</p>
</blockquote>
<p>That is expected as they contain <code>d l_argmax_loss / d l_argmax_loss</code> Which is obviously 1.</p>
<blockquote>
<p>gradient of the <code>l_argmax_loss</code> with respect to the input (x)</p>
</blockquote>
<p>To get this quantity, you should:</p>
<ul>
<li>make sure x.requires_grad=True</li>
<li>compute l_argmax_loss in a differentiable manner</li>
<li>call l_argmax_loss.backward()</li>
<li>read the gradient at x.grad</li>
</ul>
Grad not updating, (solving MSE with L1 regularization, useing torch as GD)

username: Dan_ben-david
cooked: <p>im trying to find the argmin of this equation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62d23a5e953ee0b0bd8403ec5f63bfd4ce1d6075.png" alt="image" data-base62-sha1="e6dcCmEL43QJWkZOP1cKItS5OkZ" width="200" height="37"><br>
And because there is no analytical solution for this problem, im trying to use torch as my GD. and find the minimum.</p>
<p>where <strong>(W)</strong> is the toeplitz matrix of the original audio (with no noise)<br>
<strong>(x)</strong> is my attempt to cleaning the audio from an acoustic noise (x = h_original * audio_original)<br>
and what i want ot find is <strong>(z)</strong>, when <strong>(z)</strong> is the <em>room impulse response</em> im trying to find, and then comper it to the real h</p>
<p>(in this code, W = Toeplitz_x, z = h_audio, x = y_cpu)</p>
<p>so i have build a small pytorch code inorder to find this <strong>h</strong></p>
<blockquote>
<pre><code>            h_audio = torch.randn(len(audio)).reshape(-1, 1)
            lr = 0.1
            optimizer = torch.optim.Adam([h_audio], lr=lr)
            lossMSE = nn.MSELoss()
            stop = 1

            forward = torch.matmul(Toeplitz_x, h_audio)
            y_cpu = audio.to('cpu').reshape(-1,1)

            loss = lossMSE(forward, y_cpu) + LAMBDA * torch.sum(torch.abs(h_audio))
            while stop &lt; loss:
                loss.backward(retain_graph=True)
                optimizer.step()
                optimizer.zero_grad()
                h_audio.grad = None

                forward = torch.matmul(Toeplitz_x, h_audio)
                loss = lossMSE(forward, y_cpu) + LAMBDA * torch.sum(torch.abs(h_audio))
</code></pre>
</blockquote>
<p>the code it running, but it not seem like the gradient is updating… <em>h_audio.grad = None</em> even after the step.</p>
Discriminator Accuracy

username: hajar
cooked: <p>Hi,</p>
<p>I am using GAN model, and I want to calculate the discriminator’s accuracy on both real samples and fake ones.  Could someone helps me please?</p>
<p>Thank you,</p>

username: ptrblck
cooked: <p>You could calculate the accuracy as with any other classification model. Have a look at e.g. the <a href="https://github.com/pytorch/examples/blob/aef78a5b48f53956f4c7acf555a84cc9c937bc28/imagenet/main.py#L294">ImageNet example</a>.</p>

username: Najeh_Nafti
cooked: <p>what if I am using an unsupervised learning, how can I calculate the discriminator accuracy?</p>

username: ptrblck
cooked: <p>I don’t know how the accuracy calculation would work if no labels are provided.</p>

username: Najeh_Nafti
cooked: <p>what about the below function, is it correct:</p>
<pre><code class="lang-auto">d_acc = 0
d_acc += (fake_imgs == real_imgs).float().sum()
accuracy = 100*d_acc /len(train_dataloader)
</code></pre>
<p>There is another way to evaluate the discriminator in unsupervised learning?</p>

username: ptrblck
cooked: <p>I don’t think comparing image data directly might work well for a lot of use cases.<br>
Especially if you are using floating point values, you would certainly need to use a small <code>eps</code> value in the comparison. Also, you would have to think about what the “accuracy” really represents.<br>
E.g. assuming you are working on a use case where a fake image should be “close” to the input image: would it matter if the generated fake image is shifted by a single pixel (potentially low “accuracy”) or would the overall “quality” of the image matter more?</p>

username: Najeh_Nafti
cooked: <p>How can I calculate the accuracy based on the predicted scores of the discriminator?</p>
How to write a code that prints the shape after passing through each CNN layer

username: ethanlong
cooked: <p>class CNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Sequential(<br>
nn.Conv2d(<br>
in_channels=1,<br>
out_channels=32,<br>
kernel_size=5,<br>
stride=1,<br>
padding=2,<br>
),<br>
nn.ReLU(),<br>
nn.MaxPool2d(kernel_size=2),<br>
)<br>
self.conv2 = nn.Sequential(<br>
nn.Conv2d(32, 64, 5, 1, 2),<br>
nn.ReLU(),<br>
nn.MaxPool2d(2),<br>
)<br>
# fully connected layer, output 10 classes<br>
self.out = nn.Linear(64 * 7 * 7, 10)<br>
def forward(self, x):<br>
x = self.conv1(x)<br>
x = self.conv2(x)<br>
# flatten the output of conv2 to (batch_size, 32 * 7 * 7)<br>
x = x.view(x.size(0), -1)<br>
output = self.out(x)<br>
return output, x</p>
<p>The training data size is [ 60000, 28, 28 ].</p>
<p>My goal is to determine if the input volume equates to the output volume after passing through the convolutional neural layers.  I know that to do this, I need to follow strictly the given equation, padding = (kernel_size - 1)/2.  The kernel size can be any value, but I do not know whether the number of filters will affect the dimension. Do we need a softmax activation in the FC layer? What’s the effect of having it?</p>

username: suraj.pt
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=nn%20module#torch.nn.Module.register_forward_hook">Hooks</a> are a clean way to do this, but you could also simply print the shape after every conv call in your <code>forward()</code></p>
Standard built-in tools in PyTorch (not in other GitHub repos) for early stopping

username: Mona_Jalal
cooked: <p>Is there any built-in tools for early stopping in newer versions of PyTorch or should I use something like <a href="https://github.com/Bjarten/early-stopping-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Bjarten/early-stopping-pytorch: Early stopping for PyTorch</a> for Early Stopping? Imagine I want to set the patience to 5 (is that a good number for patience?).</p>
<p>Searching “early stopping” in the forum brought these results:</p>
<ol>
<li>only a question <a href="https://discuss.pytorch.org/t/pytorch-early-stopping/117650" class="inline-onebox">Pytorch Early stopping,</a>
</li>
<li>kinda a self implementation <a href="https://discuss.pytorch.org/t/early-stopping-loop-does-not-break-training-continues/57896" class="inline-onebox">Early Stopping - Loop does not 'break' training continues</a>
</li>
<li>only a question <a href="https://discuss.pytorch.org/t/how-to-apply-early-stopping-in-training-code/123309" class="inline-onebox">How to apply early stopping in training code?</a>
</li>
<li>an answer by Piotr pointing to torchsample <a href="https://discuss.pytorch.org/t/early-stopping-in-pytorch/18800/2" class="inline-onebox">Early Stopping in PyTorch - #2 by ptrblck</a>
</li>
</ol>
<p>One thing I notice about <code>torchsample</code> package is that I should change so much of my code in order to use it. Are there less invasive methods you would suggest?</p>
NN options to train for sort order?

username: pumplerod
cooked: <p>I’ve been looking into options for different sort methods and am wondering what type of NN architecture would be best suited for training to find a sort order for a list of samples.  I’ve had success, with my data set when training for an A/B comparison to predict whether A &gt; B, however each list I am attempting to sort is several thousand samples, with hundreds of thousands of samples in the training/validation data.</p>
<p>For example:<br>
if I have <code>features</code> f1…f25 for each sample and a batch size of roughly 5000, I would like to train to predict the sort location within the batch for each sample.</p>
<pre><code class="lang-auto">f1, f2, f3, ..., f25, target
.1, .3, .5, ..., .2, 0.3
.1, .1, .5, ..., .1, 0.5
.3, .4, .2, ..., .4, 0.2
.2, .2, .1, ..., .1, 0.8
.5, .2, .6, ..., .9, 0.1
.7, .9, .5, ..., .7, 0.9
.7, .7, .2, ..., .7, 0.7
.9, .6, .0, ..., .2, 0.6
.8, .1, .1, ..., .1, 0.4
.3, .1, .3, ..., .6, 1.0
...
</code></pre>
<p>Is there recommended architecture for this type of training?  I would like to compare each sample to the entire batch in order to determine where it belongs within that population.  I am able to train a standard fully connected linear layer to predict whether sample_A_f1…Af25 &gt; sample_B_f1…f25 however the process of generating a final sort order is very time consuming due to the need to compare each sample to every other sample.</p>
<p>Would there be a way to use a GRU network to speed this up?  thinking that the sorted batch could also be thought of as a kind of time-series.</p>
<p>Or perhaps there is a way to compare the single 1D vector to a 2D (image type) vector of the entire batch through convolutions?</p>
<p>I’d appreciate any thoughts you might have on where to explore.</p>
Multiple losses from two GRU models

username: iWishii
cooked: <p>I am training a model with different outputs in PyTorch, and I have four different losses for positions (in meter), rotations (in degree), and velocity, and a boolean value of 0 or 1 that the model has to predict.<br>
AFAIK, there are two ways to define a final loss function here:</p>
<p>one - the naive weighted sum of the losses</p>
<p>two - the defining coefficient for each loss to optimize the final loss.</p>
<p>More details about the model: (I have a GRU model (lets call GRU1) with two linear output layers. (both have ReLu as the activation function):</p>
<ul>
<li>First layer is a sequence of positions(in meter) and rotations(in degree),</li>
<li>Second layer gives me a boolean value of 0 or 1.</li>
</ul>
<p>Also I have another metrics called velocity that is the output from another GRU model (lets call GRU2) and the input for the GRU1.)</p>
<p>So, My question is “how to weigh these losses to obtain the final loss, correctly”?</p>

username: knoriy
cooked: <p>What is your application?</p>
<p>Think of the two methods you proposed as one, The naive approache can be seen as the sum of all losses with a coefficient of 1.</p>
<pre><code class="lang-auto">total_loss = (p_bais * p_loss) + (r_bais * r_loss) + (v_bais * v_loss) + (b_bais * b_loss)
</code></pre>
<p>The biases above are simply put hyperparameter to be tune at training time, there is no right or wrong answer to what these could be, and would simply be guided by your application (use case).</p>

username: iWishii
cooked: <p>Thank you for your comment. I`m gonna use this model later in unity engine.<br>
Since they are different values, I supposed there should be some hyper parameters to indicate their importance. I think the coefficient for both positions and rotations will be 1, since they are both play the main role in my case. On the other hand they are not relevant in the sense that one is presented in meters and the other one in degree, Then I am confused if iv to give them coefficients or not.</p>

username: knoriy
cooked: <p>I don’t fully understand. Your loss function should compute your loss between your prediction and that target. It shouldn’t matter what data your model is outputting.</p>
<p>for example:</p>
<pre><code class="lang-auto">loss_fn = nn.MSELoss()

model_out = torch.tensor([[50.0]]) # degrees
target = torch.tensor([[45.0]] ) # degrees

print(loss_fn(model_out,target))
&gt;&gt;&gt; tensor(25.)
</code></pre>
<p>if you don’t give a coefficient, it will be considered as if the coefficient is 1…</p>

username: iWishii
cooked: <p>I understand what you mean for my wrong assumption. But honestly I didn`t get my answer yet.</p>
<p>I was reading some papers similar to my case, and I see that they define the hyper-parameters for the various losses they have based on binary cross-entropy (for the boolean one), and some other coefficients from a random weighting function.<br>
I would be glad If you or anyone else have any comments about this.</p>
What does torch.bucketize do/used for?

username: knoriy
cooked: <p>Hi,</p>
<p>Could someone please explain what operation the <code>torch.bucketize</code> does, I’ve looked at the <a href="https://pytorch.org/docs/stable/generated/torch.bucketize.html" rel="noopener nofollow ugc">documentation</a>, but I can not understand what it does and how <code>torch.bucketize</code> could be used in practice.</p>
<p>Thank you in advance.</p>

username: suraj.pt
cooked: <p>Hey, <code>torch.bucketize</code> takes a continuous input and discretizes them to integer boundaries. The returned tensor contains the right boundary index for each value in the input tensor. In this example,  <code>boundaries[2] &lt; x[0][0] &lt;= boundaries[3]</code>, so <code>out[0][0] == 3</code>.</p>
<pre><code class="lang-auto">In [15]: boundaries = torch.tensor([1,3,6,7,9])

In [16]: x
Out[16]:
tensor([[6.7165, 1.9131],
        [7.0514, 8.5162]])

In [17]: out = torch.bucketize(x, boundaries)

In [18]: out
Out[18]:
tensor([[3, 1],
        [4, 4]])

</code></pre>
<p>Some applications need discrete values, this function is an easy way to bin your input according to your provided boundaries. Hope this helps!</p>

username: knoriy
cooked: <p>Thank you, that was super helpful. It makes sense now!!</p>
“Minimum” loss across several predictions

username: Andrei_C
cooked: <p>I posted this in the “vision” forum <a href="https://discuss.pytorch.org/t/minimum-loss-across-several-predictions/145405">here</a> but it occurs to me that this may be more of an autograd question. The example I use below is specific to vision however one can easily imagine equivalent non-vision problems.</p>
<p>Greetings! Looking to get some help with the following task.</p>
<p>I’m training a model that can reconstruct occluded parts of an original image. The correct reconstruction can be multimodal in pixel color space, and the different correct representations can even be very very far apart (e.g. suppose the thing to be reconstructed is a ying-yang sign in black and white, and in the data half the time the sign has white on the left and black on the right, and half the time it has black on the left and white on the right). I’d like to allow my model to make N different predictions and for the loss function to be the minimum MSE distance (minimum across the N predictions) between the prediction and the target. That way the model can say something like “I think the missing piece is a ying-yang symbol either with black on the left and white on the right, or vice-versa” and be correctly rewarded for the most accurate of the predictions.</p>
<p>Is this kind of minimum loss function possible in Pytorch? If so, has anyone used it before and can share best practices? I can imagine that there are issues related to the fact that it’s kinked around the minimum threshold and the first derivative is not easily defined for autograd.</p>
<p>Thank you!</p>
Following the receptive field through a CNN

username: chrisranderson
cooked: <p>I have an idea, and I was wondering if anyone could point me in the right direction. I’d like to:</p>
<ol>
<li>Run an image through the CNN, keeping the activations at each layer.</li>
<li>For every pixel, collect every filter activation that was influenced by that pixel.</li>
<li>The final result would be a tensor of shape [image height, image width, # of filter channels in the entire network].</li>
</ol>
<p>I was going to use network.modules() to get a list, and then find all the instances of Conv2d and MaxPool2d (etc.), and go from there… but squeezenet, for example, doesn’t appear to just apply things in the order I get from modules(). Is there some way I can traverse the computation graph, even though it is dynamic? I can figure out how to do it manually, but I’d like to implement a general-purpose function that takes in an arbitrary network.</p>

username: smth
cooked: <p>because pytorch networks are code, it’s hard to build a general purpose function to do this.</p>

username: bzcheeseman
cooked: <p>If you want the activations at every layer you could register a forward hook on every module and store them in a list with .append(). Naively I’d expect that they would then be in the order in which they executed?</p>
<pre><code class="lang-auto">handles = []
variable_storage = []
for module in net.modules():
    handles.append(module.register_forward_hook(lambda input, output: 
                                         variable_storage.push_back(output))
</code></pre>

username: rogetrullo
cooked: <p>Not sure if this will help, but I have made an implementation that allows to compute the RF numerically.<br>
Here it is:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/rogertrullo/Receptive-Field-in-Pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/8496304?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/rogertrullo/Receptive-Field-in-Pytorch" target="_blank" rel="nofollow noopener">rogertrullo/Receptive-Field-in-Pytorch</a></h3>

<p>Receptive-Field-in-Pytorch - Numerical Computation of Receptive Field in Pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: hamedB
cooked: <p>Hello <a class="mention" href="/u/rogetrullo">@rogetrullo</a></p>
<p>could you compute the receptive field region of each feature location in a feature map?and is it possible for you to share some code snippet?</p>
<p>Thank you.</p>
Create decoder equivalent to LSTM encoder

username: Spiruel
cooked: <p>Hi,</p>
<p>I’d like to create the decoder equivalent of an LSTM discriminative model. So instead of input of shape (batch_size, 45, 13) to output (batch_size, number of classes), I would like a similar model architecture to input (batch_size, number of classes) to generate an output (batch_size, 45, 13).</p>
<p>I am really stuck on how to go about this. Could someone let me know how you ho about constructing this? Similar to how one might construct an auto encoder, for example.</p>
<p>The model I am using is as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data
import os

__all__ = ['LSTM']

class LSTM(torch.nn.Module):
    def __init__(self, input_dim=13, num_classes=9, hidden_dims=128, num_layers=4, dropout=0.5713020228087161, bidirectional=True, use_layernorm=True):

        super(LSTM, self).__init__()

        self.num_classes = num_classes
        self.use_layernorm = use_layernorm

        self.d_model = num_layers * hidden_dims

        if use_layernorm:
            self.inlayernorm = nn.LayerNorm(input_dim)
            self.clayernorm = nn.LayerNorm((hidden_dims + hidden_dims * bidirectional) * num_layers)

        self.lstm = nn.LSTM(input_size=input_dim, hidden_size=hidden_dims, num_layers=num_layers,
                            bias=False, batch_first=True, dropout=dropout, bidirectional=bidirectional)

        if bidirectional:
            hidden_dims = hidden_dims * 2

        self.linear_class = nn.Linear(hidden_dims * num_layers, num_classes, bias=True)


    def logits(self, x):

        if self.use_layernorm:
            x = self.inlayernorm(x)

        outputs, last_state_list = self.lstm.forward(x)

        h, c = last_state_list

        nlayers, batchsize, n_hidden = c.shape
        h = self.clayernorm(c.transpose(0, 1).contiguous().view(batchsize, nlayers * n_hidden))
        logits = self.linear_class.forward(h)

        return logits

    def forward(self, x):
        logprobabilities = F.log_softmax(self.logits(x), dim=-1)
        return logprobabilities
</code></pre>
How to create a great training loop?

username: snowparrot
cooked: <p>Hello,</p>
<p>I am more advanced in Deep Learning, but I wanted to develop further and see what can be improved.<br>
I am currently looking for some resources to improve the training process (= the loop where the backpropagation takes place).<br>
This does not automatically have to mean better performance etc.<br>
One example I can think of is to use Tensorboard for logging. Do you have any other ideas or a good resource?</p>
<p>Thanks for the help</p>
Implement Meta-RetinaNets Weight Multiplication

username: marvib
cooked: <p>Hello,</p>
<p>I am currently attempting to implement the “Meta Coefficient Learner” from the Meta-RetinaNet <a href="https://www.bmvc2020-conference.com/assets/papers/0042.pdf" rel="noopener nofollow ugc">paper</a>. However, I am currently stuck while implementing the trainable coefficient vector (see Figure below) into this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">example</a>. So essentially I want to freeze the pretrained ResNet and apply the trainable coefficient layer such that I can then “Meta”-train the network on a new dataset.</p>
<p>So my question would be: How can I add the new coefficient vector to all convolutional weights?</p>
<p>My guess would be to add the [1x1xFilters] Vector as a convolution before each of the original convolutions. Such that I can freeze the original layer, while keeping the new layers trainable.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa0e8f43cf8215dc2c52c77075774332a1835a56.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa0e8f43cf8215dc2c52c77075774332a1835a56" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa0e8f43cf8215dc2c52c77075774332a1835a56_2_690x211.png" alt="grafik" data-base62-sha1="zG6yOiAn1o1ZEnqsEYeGESoNm3c" width="690" height="211" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa0e8f43cf8215dc2c52c77075774332a1835a56_2_690x211.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa0e8f43cf8215dc2c52c77075774332a1835a56_2_1035x316.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa0e8f43cf8215dc2c52c77075774332a1835a56_2_1380x422.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">1455×445 77.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>
Questions about torch.nn.DataParallel

username: Ford
cooked: <p>Hello! I run the codes to train a model in one machine multi cards using torch.nn.DataParallel. The codes specify gpu of input data as device 0 when loading data. The codes are written like:</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model).cuda()
device = args.gpu if args.gpu is not None else 0
for i, sample in enumerate(loader):
        video, audio, index = sample['frames'], sample['audio'], sample['index']
        video = video.cuda(device, non_blocking=True)
        audio = audio.cuda(device, non_blocking=True)
        index = index.cuda(device, non_blocking=True)

        output = model(video, audio, index)

</code></pre>
<p>For ease of explantation, I have omitted unimportant details. Now I try to add a text modality without specifying the GPU device. The codes are written as follow:</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model).cuda()
device = args.gpu if args.gpu is not None else 0
for i, sample in enumerate(loader):
        video, audio, text, index = sample['frames'], sample['audio'], sample['text'], sample['index']
        video = video.cuda(device, non_blocking=True)
        audio = audio.cuda(device, non_blocking=True)
        index = index.cuda(device, non_blocking=True)
 
        text = text.cuda()

        output = model(video, audio, text, index)
</code></pre>
<p>I wonder is this correct? Why does data need to be on device 0 when using torch.nn.DataParallel? Thanks.</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will use the default device (GPU0 if you haven’t changed it) to store the data as well as the model and will then scatter the data and the model to each device (which is why <code>DistributedDataParallel</code> is preferred as it doesn’t create an imbalanced GPU memory usage and also avoids the model replication). The detailed workflow is explained in e.g. <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.</p>

username: Ford
cooked: <p>Thank you very much! It is really helpfu to me!</p>
Fine-tuning model does'nt learn targets

username: Mukesh1729
cooked: <p>I am trying to fine-tune a model from torch. hub but after training the loss goes down but the model doesn’t learn the target at all. Any hints on what the problem could be?</p>
<pre><code class="lang-auto">#Load pre-trained model and fine-tune

model = torch.hub.load('yangsenius/TransPose:main', 

                       'tpr_a4_256x192',

                       pretrained=True)

model.final_layer =  torch.nn.Sequential(torch.nn.Conv2d(256, 18,1))                                    

model = model.to(device)

#Get model summary

summary(model,

          input_size=(3, 256, 192),

          batch_size=1

          )
</code></pre>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss(reduction="mean")

pretrain_part = [param for name, param in model.named_parameters()if 'final_layer' not in name]

# optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

optimizer = torch.optim.Adam([ {'params': pretrain_part, 'lr':1e-5 },

                                                    {'params': model.final_layer.parameters(), 'lr': 1e-4}])
</code></pre>