akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use dropout correctly in Neural network Pytorch

username: the_coder
cooked: <p>I have designed my network as follows, i am not sure whether it is right to use Dropout just after relu. I am doing a mulit class image classification task. The images are grayscale and they are 64*64 in size.</p>
<pre><code>class Neural_Network(nn.Module):
def __init__(self, input_node, hidden_node, num_classes):
    super(Neural_Network, self).__init__()
    self.input_node = input_node
    self.layer1 = nn.Linear(input_node, hidden_node) 
    self.relu = nn.ReLU()
    self.layer2 = nn.Linear(hidden_node, hidden_node2)
    self.relu = nn.ReLU()
    self.layer2_drop= nn.Dropout(p=0.5)
    self.layer3 = nn.Linear(hidden_node2, num_classes)


def forward(self, x):
    x = self.layer1(x)
    x = self.relu(x)
    out = self.layer2(x)
    out = self.relu(out)
    out = self.layer2_drop(out)
    out= self.layer3(out)



    return out</code></pre>

username: vdw
cooked: <p>Google for “dropout before or after activation” to find a whole bunch of discussion which order might or might not preferable. I don’t think there’s a hard consensus. It’s seem Dropout before the activation function is more common, but I didn’t check in detail.</p>
Slow dataloader in headless

username: jakewright
cooked: <p>Hi, I currently use slurm and I’ve noticed when I run a program from the terminal, the dataloader when initiated the first time will load everything within 10 minutes. However, when I run the program with the same configuration, code, and resources using “sbatch” mode in a headless fashion, then it will take multiple hours for the dataloader to initiate.</p>
<p>Any thoughts on what might be the problem?</p>
Is it normal that after Normalize the values aren't in the range [-1, 1]?

username: Manoel
cooked: <p>I’m trying to compute the mean and the std of my dataset. I found <a href="https://discuss.pytorch.org/t/computing-the-mean-and-std-of-dataset/34949/13">this topic</a> and implemented some ideas. However, when I applied the normalization on the dataset and checked the values. Then I found out that most values weren’t on the range[-1, 1], some greater than 1 and 2.</p>
<p>Is it normal? Or should it be best just use the scaling that the ToTensor() does?</p>
<p>I used the following transforms: Resize(250), CenterCrop(224), RandomHorizontalFlip(), ToTensor().</p>

username: ptrblck
cooked: <p>That’s expected, since <code>Normalize</code> calculates the standard scores of the input (also known as z-scores) by subtracting the mean and dividing by the standard deviation.<br>
<a href="https://en.wikipedia.org/wiki/Standard_score">This Wikipedia article</a> explains it further.</p>

username: Manoel
cooked: <p>Thank you. I was worried that I was doing something wrong.</p>
When will PyTorch support importing from ONNX?

username: Schwarz95
cooked: <p>Hi everyone,</p>
<p>I am currently working on a project for which PyTorch would be the ideal framework. Unfortunately, it does not yet support importing models from ONNX which is necessary for me. Does someone know when a version of PyTorch will be published, that supports importing .onnx models. On the ONNX github page it says that this feature will be “coming soon”.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/onnx/tutorials" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/31675368?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/onnx/tutorials" target="_blank" rel="nofollow noopener">onnx/tutorials</a></h3>

<p>Tutorials for using ONNX. Contribute to onnx/tutorials development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thanks in advance</p>

username: Oleg_Mikulchenko
cooked: <p>Yes, two ways ONNX (import-export) is available for other frameworks. Would be nice to know if/when it will be available for PyTorch, making that framework more competitive versus others.</p>

username: armin-th
cooked: <p>I’m not sure if this was a recent addition but the docs <a href="https://pytorch.org/docs/master/onnx.html#example-end-to-end-alexnet-from-pytorch-to-onnx" rel="nofollow noopener">here</a></p>
<p>mention you can do something like</p>
<pre><code class="lang-auto">import onnx

# Load the ONNX model
model = onnx.load("alexnet.onnx")

# Check that the IR is well formed
onnx.checker.check_model(model)

# Print a human readable representation of the graph
onnx.helper.printable_graph(model.graph)
</code></pre>
Differentiable version of numpy.diff()

username: Federico_Vasile
cooked: <p>Hi everyone,<br>
I’d need to count the number of times that a tensor changes its values(from the previous one to the next one), in order to include this information as a term in my loss function.<br>
Example:</p>
<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; a = [1, 1, 1, 0, 0, 1, 0]
&gt;&gt;&gt; diffs = np.diff(a)
&gt;&gt;&gt; diffs 
array([ 0,  0, -1,  0,  1, -1])
&gt;&gt;&gt; (diffs != 0).sum()
3
</code></pre>
<p>In PyTorch I did it in the following way:</p>
<pre><code>def count_diffs(x):
    diff_x = x[1:] - x[:-1]
    return (diff_x != 0).sum()

def loss_diffs(x, batch_size, num_classes):
    # x.shape == (batch_size, steps, num_classes)
    loss = 0.0
    for idx_batch in range(batch_size):
        for idx_class in range(num_classes):
            loss += count_diffs(x[idx_batch, :, idx_class])
    loss /= (batch_size * num_classes)

    return loss
</code></pre>
<p>In other words, choosing a sample and a class I have a sequence, and I would like to explicitly tell to the loss to prefer this sequence </p><pre><code>[1, 1, 1, 0, 0, 0, 0]</code></pre> over this one <pre><code>[1, 1, 1, 0, 1, 0, 0]</code></pre>
<p>Any suggestions on how to do it? My PyTorch implementation does not work, I think because it is not differentiable.<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that your <code>count_diffs()</code> method actually returns a LongTensor. And we can’t really have gradients for integer types.<br>
You will need to find a version of your loss that only uses continuous values for it to be differentiable.</p>

username: Federico_Vasile
cooked: <p>Sorry but I never faced this problem before, could you kindly show me an example where this problem occurs?</p>

username: albanD
cooked: <aside class="quote no-group" data-username="Federico_Vasile" data-post="1" data-topic="89347">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/federico_vasile/48/12933_2.png" class="avatar"> Federico_Vasile:</div>
<blockquote>
<pre><code class="lang-auto">def count_diffs(x):
    diff_x = x[1:] - x[:-1]
    return (diff_x != 0).sum()
</code></pre>
</blockquote>
</aside>
<p>This function takes a <code>x</code> that is a float Tensor (potentially with autograd info attached) and returns an integer (inside a Tensor) that represent the value you’re looking for.<br>
But integers are not continuous, so you cannot compute gradients for them.<br>
So you won’t be able to get gradients flowing back if you use this function.</p>
Fine tuning SqueezeNet for MNIST dataset

username: trancong21
cooked: <p>I want to fine tune <strong>SqueezeNet</strong> model from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/squeezenet.py" rel="nofollow noopener">torchvision/models/squeezenet.py</a> for MNIST dataset.<br>
What should i do? Thanks!</p>

username: ptrblck
cooked: <p>You could use the <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py">MNIST example</a> as your template and just change a few lines of code.</p>
<p>Since the MNIST data comes as grayscale images in the resolution <code>28x28</code>, you would need to repeat the channel to simulate an RGB image as well as resizing it to <code>224x224</code> to fit your SqueezeNet.</p>
<p>Here is a small example of the necessary changes:</p>
<pre><code class="lang-python">model = models.squeezenet1_1(pretrained=False)
    
dataset = datasets.MNIST(
    root='PATH',
    train=True,
    transform=transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Lambda(lambda x: x.expand(3, -1, -1))
    ])
)

loader = DataLoader(
    dataset,
    batch_size=10,
    shuffle=True,
    num_workers=2
)


data, target = next(iter(loader))
output = model(data)
print(output.shape)
</code></pre>

username: mehran2020
cooked: <p>Will it be Ok to discard 2 channels and convert the initial layers in_channels to 1, so it could work with MNIST as a 1-channel image dataset? Will copying the same image add any advantage?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Both approaches would be valid and I don’t know which one is better.<br>
The advantage of the posted approach to repeat the channels is that you could just use a pretrained model (which usually works on RGB data) without any manipulations or finetuning of the first layer.<br>
E.g. if you change the input channels of the first layer to 1, you could</p>
<ul>
<li>use a reduction (such as calculating the mean/median/sum of all channels),</li>
<li>initialize this channel randomly and train it,</li>
<li>or keep a single channel only (which one?).</li>
</ul>
<p>However, I haven’t run any experiments to compare these approaches.</p>
What is the most effective way to migrate an existing model to a newer version of Pytorch?

username: cytokinesis
cooked: <p>I have some models currently in production, using torch’s C++ API and traced torchscripts, ala these instructions: <a href="https://pytorch.org/docs/master/generated/torch.jit.trace.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.jit.trace.html</a><br>
I also have the model’s state dicts saved away.<br>
Overall this solution works great.</p>
<p>The models and torchscripts were generated using pytorch 1.3<br>
I want to update the next version of the system to pytorch 1.5</p>
<p>Retraining is possible, but not preferred, and certainly not a long term solution.<br>
Is there a procedure for updating my existing models, from either the saved torchscripts or the state dicts?</p>
<p>Thanks</p>

username: tom
cooked: <p>The easiest way is to instantiate the model in Python, load the state dict and trace the model again.<br>
This is something to know, even though you can run the traced model without the source, you’ll regret not keeping it around at some point.</p>
<p>Best regards</p>
<p>Thomas</p>
Why does ToTensor throw an error if the data is already a torch tensor?

username: Brando_Miranda
cooked: <p>I tried doing a ToTensor transform on data that is already a torch tensor but got this error:</p>
<pre><code class="lang-auto">TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>Why doesn’t it just do a No Op and return the original data?</p>
<p>sample code:</p>
<pre><code class="lang-auto">
# saving torch tensors

import torch
import torch.nn as nn
import torchvision

from pathlib import Path
from collections import OrderedDict

path = Path('~/data/tmp/').expanduser()
path.mkdir(parents=True, exist_ok=True)

tensor_a = torch.rand(2,3)
tensor_b = torch.rand(1,3)

db = {'a': tensor_a, 'b': tensor_b}

torch.save(db, path/'torch_db')
loaded = torch.load(path/'torch_db')
print( loaded['a'] == tensor_a )
print( loaded['b'] == tensor_b )

# testing if ToTensor() screws things up
lb, ub = -1, 1
N, Din, Dout = 3, 1, 1
x = torch.distributions.Uniform(low=lb, high=ub).sample((N, Din))
print(x)

f = nn.Sequential(OrderedDict([
    ('f1', nn.Linear(Din,Dout)),
    ('out', nn.SELU())
]))
y = f(x)

transform = torchvision.transforms.transforms.ToTensor()
y_proc = transform(y)
print(y_proc)
</code></pre>
<p>related: <a href="https://discuss.pytorch.org/t/what-is-the-recommended-format-to-save-data-in-pytorch/89047/2" class="inline-onebox">What is the recommended format to save data in pytorch?</a></p>
GLOO and infiniband

username: seliad
cooked: <p>Hi,<br>
Can GLOO work with infiniband?</p>
<p>Our RTX2080ti GPUs do not support GPUDirect/RDMA anyway, so the only thing we want is to work out of the box in reasonable BW which will not become the bottleneck, we are doing P2P communication.</p>
<p>edit: well I see that here <a href="https://github.com/facebookincubator/gloo" rel="nofollow noopener">https://github.com/facebookincubator/gloo</a> they say its supported, but I wonder if you still have anything further to say on integration with Pytorch.</p>
<p>edit2: especially since here <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html</a> its written than GLOO does not support infiniband.</p>

username: jiayisuse
cooked: <p>Hi,</p>
<p>GLOO does have an ibverbs transport <a href="https://github.com/facebookincubator/gloo/tree/master/gloo/transport/ibverbs" rel="nofollow noopener">https://github.com/facebookincubator/gloo/tree/master/gloo/transport/ibverbs</a>. However, it was never used or tested with PyTorch. That may be the reason that PyTorch doc says GLOO does not support infiniband.</p>
<p>We are about to test GLOO ibverbs transport over RDMA, and integrate with PyTorch on HPC scenarios. For now, GLOO ibverbs hasn’t been integrated to PyTorch yet.</p>

username: jiayisuse
cooked: <p>Filed an issue <a href="https://github.com/pytorch/pytorch/issues/41485" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/41485</a></p>
Loss function and batch size

username: Abhishek_Kishore
cooked: <p>If I set the  batch size as 1 then each iteration of dataloader will give me 1 row. However, if I add 4 more arbitrary rows in each iteration of dataloader then will the loss function (let’s say it is MSE) will be calculated as <code>((y-yhat)^2)/1</code> (since the batch size is 1) or will it be calculated as  <code>((y-yhat)^2)/5</code> ?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Loss function actually is not aware of batch size, It just responds to its input values which does not matter if you achieve them by using batch size or even directly reading data from storage then stacking them. So, in summary, batch size does not affect on value of loss functions directly.</p>
<p>Bests</p>

username: Abhishek_Kishore
cooked: <p>Thanks for the clarification</p>
How does one use torch.optim.lr_scheduler.OneCycleLR()

username: benihime91
cooked: <p>I wanted to use <code>torch.optim.lr_scheduler.OneCycleLR</code> while training. Can some kindly explain to me how to use.<br>
What i got from the documentation was that it should be called after each <code>train_batch</code>.</p>
<p>My confusions are as follows:</p>
<ul>
<li>
<p>Does the <code>max_lr </code> parameter has to be same with the <code>optimizer</code> <code>lr</code> parameter?</p>
</li>
<li>
<p>Can this scheduler be used with <code>Adam</code> optimizer. How is the momentum calculated then ?</p>
</li>
<li>
<p>Let’s say i trained my model for some number of epochs at a stretch now,  i wanted to train for some more <code>epochs</code>. Would i have to reset the the <code>scheduler</code> ?</p>
</li>
</ul>
<blockquote>
<p>Can anybody provide me a sort of a toy example/traininng loop that implements this scheduler ? Any help would be appreciated ?</p>
</blockquote>
<p>P.S: I am kind of new to <code>deep learning</code> &amp; <code>PyTorch</code> so my question might be somewhat silly.</p>
Torchvision.ops.nms in torchscript

username: yrath
cooked: <p>Hello, As mentioned <a href="https://pytorch.org/docs/stable/torchvision/ops.html" rel="nofollow noopener">here</a> torchvision ops is currently not supported in torchscript. I wanted to know is there a timeline on when will torchscript support these functions.</p>

username: tom
cooked: <p>The documentation might be slightly out of date if you compile both PyTorch and torchvision yourself or take the nightlies, because it seems to work for me:</p>
<pre><code class="lang-python">In [1]: import torch, torchvision                                                                                                                                                                                  

In [2]: @torch.jit.script 
   ...: def fn(a, b): 
   ...:     return torchvision.ops.nms(a, b, 0.3) 
   ...:                                                                                                                                                                                                            

In [3]: fn.graph                                                                                                                                                                                                   
Out[3]: 
graph(%a.1 : Tensor,
      %b.1 : Tensor):
  %5 : Function = prim::Constant[name="nms"]()
  %4 : float = prim::Constant[value=0.29999999999999999]() # &lt;ipython-input-2-cfe32fda8960&gt;:3:37
  %6 : Tensor = prim::CallFunction(%5, %a.1, %b.1, %4) # &lt;ipython-input-2-cfe32fda8960&gt;:3:11
  return (%6)

In [4]: fn(torch.randn(10, 4), torch.randn(10))                                                                                                                                                                    
Out[4]: tensor([2, 8, 6, 7, 0, 9, 4, 5, 3, 1])
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: yrath
cooked: <p><a class="mention" href="/u/tom">@tom</a> Thanks. It is working for me now in torchscript as well. But, the NMS now fails in the TRTserver.</p>
<pre><code class="lang-auto">Unknown builtin op: torchvision::nms.
Could not find any similar ops to torchvision::nms. This op may not exist or may not be currently supported in TorchScript.

</code></pre>

username: tom
cooked: <p>Well, you might need to load the custom ops for that. I know nothing about TRTserver.</p>
The memory address of the network structure

username: 111363
cooked: <p>Hello everyone, I found a problem when I read the memory address of CNN parameter, I found some problems!</p>
<p>import torch<br>
from torchvision import models<br>
resnet = models.resnet50(pretrained=False)<br>
print(list(map(id,resnet.parameters())))<br>
print(id(resnet.parameters()))</p>
<p>when I try to print  the id of the parameters, I found the first print have a lot of memory address, there are dozens of them. But the second print only have one address.<br>
I don’t quite understand this problem, can someone explain it? I mainly used to adjust the learning rate, but I found this strange problem.<br>
I’m not an expert on this. I hope someone can explain.</p>

username: ptrblck
cooked: <p>I don’t think it’s an issue, since the second approach would print the <code>id</code> if the created <code>Generator</code>, not the parameters.</p>

username: 111363
cooked: <p>OK，i think so.thank you very much</p>
Mini Batch training for Word Embeddings example

username: pytorch_user18
cooked: <p>Hello,</p>
<p>I am trying to modify the example shown here <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py</a> to take a mini-batch as input instead of looping through one n-gram pair at a time. While I tried doing this by modifying the layer input sizes, the network convergence slows down dramatically and essentially stalls. Is there a right way to do mini-batch training here using PyTorch?</p>
<p>Regards</p>

username: jakerhodes
cooked: <p>This is the exact issue I am having.  Did you ever get this figured out?</p>
Adversarial Training

username: shakeel608
cooked: <p>I am trying to use adversarial learning from paper</p>
<aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1903.09606.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1903.09606.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1903.09606.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1903.09606.pdf" target="_blank" rel="noopener nofollow ugc">1903.09606.pdf</a></h3>

  <p class="filesize">632.34 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Since I have implemented it, but I am confused how do I combine the two losses (my both losses here are cross entropy because I want to predict the label of emotion sample and trying to make the latent embeddings speaker invariant using the SpeakerInvariant Adversarial Training ) here one from Emotion Classifier  and other from SpeakerInvariant.</p>
<p>The second question is how do apply gradient reversal in the SpeakerInvariant Layer  in the backward step.</p>
<p><em><strong>The loss function is as:</strong></em><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a9a8f4c97fdacb65fff16799b3515261c04df46.png" alt="Screenshot from 2020-07-15 15-19-28" data-base62-sha1="aDYtu7ln2CIIFge9YAF4vHzyvCm" width="524" height="330"></p>
<pre><code class="lang-auto">class Adversarial(nn.Module):
    def __init__(self):
        super(Adversarial, self).__init__()       
        self.cnn = CNN() 
        self.emotion_classfier = EmotionClassfier()
        self.speaker_invariant = SpeakerInvariant()

    def forward(self, input):
        input         = self.cnn(input)
        input_emotion = self.emotion_classfier(input)
        input_speaker = self.speaker_invariant(input) 
       """First confusion is here, how shall handle it here and then how to implemenet this loss function  """ 
        return input_stutter, input_speaker

class EmotionClassfier(nn.Module):
    def __init__(self):
        super(EmotionClassfier, self).__init__() 
        self.fc1 = nn.Linear(128, 64)
        self.fc2 = nn.Linear(64, 5)     #5 here is number of emotion classes 
        self.relu = nn.ReLU()

    def forward(self, input):
        input = input.view(input.size(0), -1)
        input = self.fc1(input)
        input = self.relu(input)
        input = self.fc2(input)
        input = F.softmax(input,1)
        return input

class SpeakerInvariant(nn.Module):
    def __init__(self):
        super(SpeakerInvariant, self).__init__()
        self.fc1 = nn.Linear(128, 64)
        self.fc2 = nn.Linear(64, 11)   #11 here is number of speakers/labels
        self.relu = nn.ReLU()

    def forward(self, input):
        input = input.view(input.size(0), -1)
        input = self.fc1(input)
        input = self.relu(input)
        input = self.fc2(input)
        input = F.softmax(input,1)
        return input
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Backward prop per Batch of data or backward prop after one epoch

username: Henry_Chibueze
cooked: <p>Given the it won’t be normal to feed a deep learning model the training data all at once (especially if the training data is 1000+) due the available RAM and other factors, I really want to know the effects of running the backward propagation after feeding all batches of data to the model per epoch versus running the backward propagation after feeding the model per batch.</p>
<p>Thanks to anyone who answers this.</p>

username: ptrblck
cooked: <p>Generally larger batches will provide a more accurate estimate of the gradient, while smaller batches will introduce more noise (which is often seen as beneficial up to a certain degree).<br>
<a href="https://www.deeplearningbook.org/contents/optimization.html">Chapter 8.1.3 - DeepLearningBook</a> discusses these effects in more detail.</p>

username: Henry_Chibueze
cooked: <p>Thank you for your reply, but I just wanted to know if backward propagation should be carried out after each batch within one epoch or after the entire batch has been fed the model (per epoch)</p>

username: ptrblck
cooked: <p>Usually, you would update the model after each batch, but that’s not a hard rule as explained.<br>
Depending on your use case, you might want to accumulate the gradients and update the model after a couple of batches (or all of them).</p>

username: Henry_Chibueze
cooked: <p>thanks alot I think i get it now</p>
How to load a model on single GPU that was trained on dataparallel?

username: falmasri
cooked: <p>it says always that there is a mismatch in layers. I guess it is not all loaded into on GPU.</p>

username: ptrblck
cooked: <p>It’s recommended to store the <code>model.module.state_dict()</code> for data parallel models as explained <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models">here</a>.<br>
If you’ve stored the <code>model.state_dict()</code>, all keys will include the <code>.module</code> keyword, which you would have to remove as described <a href="https://discuss.pytorch.org/t/chexnet-code-error-runtimeerror-error-s-in-loading-state-dict-for-dataparallel-missing-key-s-in-state-dict-module-densenet121-features-conv0-weight/43790/3">here</a>.</p>

username: falmasri
cooked: <p>I do exactly this but when I loaded it says mismatch.</p>

username: ptrblck
cooked: <p>Could you post the error message, please?</p>

username: falmasri
cooked: <p>Error(s) in loading state_dict for Cyclic:<br>
Missing key(s) in state_dict</p>
Weight initialisation and Backpropagation modification

username: Asgard
cooked: <p>My model is used to predict values based on an minimising a loss function L. But, the loss function doesn’t have a single global minima value, but rather a large number of places where it achieves global minima.</p>
<p>So, the model is based like this:</p>
<ol>
<li>
<p>Model Input is [nXn] tensor (<strong>let’s say:</strong> <code>inp=[ [i_11, i_12, i_13, ..., i_1n],[i_21, i_22, ..., i_2n],...,[i_n1,i_n2, ..., i_nn] ]</code>) and model output is [nX1]  tensor <strong>(let’s say:</strong> <code>out1=[o_1, o_2,..., o_n ]</code>)</p>
</li>
<li>
<p>Output tensor is <code>out1</code> is passed in a function <em>f</em> to get <code>out2</code> (<strong>let’s say:</strong> <code>f(o_1, o_2, o_3,..., o_n)=[O_1,  O_2, O_3,  ..., O_n] </code>)</p>
</li>
<li>
<p>These 2 values (i.e., <code>out1</code> and <code>out2</code>) are minimised using MSELoss i.e., <code>Loss = ||out1 - out2||</code></p>
</li>
</ol>
<p>Now, there are a lot of values for <code>[o_1, o_2, ..., o_n]</code> for which the <code>Loss</code> goes to minimum.</p>
<p>But, I want the values of <code>[o_1, o_2, ..., o_n]</code> for which <code>|o_1| + |o_2| + |o_3| + ... + |o_n|</code> is maximum</p>
<p>Right now, the weights are initialised randomly:<br>
<code>self.weight = torch.nn.parameter.Parameter(torch.FloatTensor(</code>in_features<code>, out_features))</code> for some value of <code>in_features</code> and <code>out_features</code></p>
<p>But by doing this, I am getting the values of <code>[o_1, o_2, ..., o_n]</code> for which <code>|o_1| + |o_2| + |o_3| + ... + |o_n|</code> is minimum.</p>
<p>I know this problem can be solved by without using deep-learning, but I am trying to get the results like this for some task computation.</p>
<p>Is there a way to change this to get the largest values predicted at the output of the neural net?</p>
<p>Or is there any other technique (backpropagation change) to change it to get the desired largest valued output?</p>
<p>Thanks in advance.</p>
Padding values ignored by convlayers or PackedSequence for CNNs?

username: rueeggmi
cooked: <p>Hey,</p>
<p>I have a custom dataloader, that provides a different number of inputs to the network for each <code>__getitem__</code> call in form of a dict:</p>
<p><code>item.keys() = [“image”, “ground_truth”, “data_tensor_1”, “data_tensor_2”, “data_tensor_3”, …]</code></p>
<p>So for each item, the actual number of data_tensor-dict-entries can differ. (it does not necessarily need to be saved as a dict, but so far this was the easiest to perform other things in my code)</p>
<p>In the forward pass of my network (fully conv), one part is called for each of the data_tensors. So for example, if the item contains 3 data_tensors, that part is run through 3 times. This works nicely if each dataset item has the same number of elements. However, with the configuration I have now, a batch does not have a consistent number of elements so within the same batch, this part of the network would have to be run a different number of times depending on the batch entry.</p>
<p>One solution I could think of was padding all items of a batch with None-data_tensors such that the whole batch looks the same. But if I then run a tensor full of None through a convlayer, does this have an influence on the model? Basically, I would like to have a padding that’s completely ignored by my network / conv layers.</p>
<p>Another thing I was thinking of is using something like PackedSequence for RNNs. Is there a way of using them for conv layers?</p>
<p>I hope I was able to describe my problem in an understandable way.</p>
<p>Thanks a lot for any help!</p>
tf.keras.layers.TimeDistributed equivalent in PyTorch

username: VikingAlex
cooked: <p>Hi,</p>
<p>I am changing from TF/Keras to PyTorch. To create a recurrent network with a custom cell, TF provides the handy function ’ tf.keras.layers.TimeDistributed’ that handles the sequence for you and applies your arbitrary cell to each time step.<br>
Trying to implement something similar in PyTorch, I found code examples (like <a href="https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html</a>) where a simple for-loop is used. While this works, I was wondering if there is a ‘better’ way to do this, or if this is already optimal performance wise. Coming from TF, I am sure that such a simple and elegant solution with the for-loop can be optimal, but PyTorch has surprised me already many times.</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/any-pytorch-function-can-work-as-keras-timedistributed/1346">This topic</a> seems to have a few implementations. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Prediction , objective function and optimization in Python - AI

username: hpsuresh
cooked: <p>Hello, What are the different methods i can build prediction models, link objective functions and do optimization using pytorch. Pl suggest</p>

username: ptrblck
cooked: <p>I’m not sure I understand the question correctly, but it seems you are looking for the complete support of PyTorch methods?<br>
If that’s the case, the <a href="https://pytorch.org/docs/stable/index.html">docs</a> and <a href="https://pytorch.org/tutorials/">tutorials</a> might be a good starter. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Very large loss output by BCEWithLogitsLoss

username: Mona_Jalal
cooked: <p><a class="mention" href="/u/lais823">@lais823</a> and I are having very large loss output by BCEWithLogitsLoss. We are working with a vision and language problem which also has attention modules.</p>
<p>We also have followed the guidelines of <a href="http://karpathy.github.io/2019/04/25/recipe/" rel="nofollow noopener">http://karpathy.github.io/2019/04/25/recipe/</a> for training a neural network but we are unsure how to trace back and fix the problem at this point.</p>
<p>What are some suggestions you could provide us with?</p>

username: macwiatrak
cooked: <p>Hey Mona, could you please paste your model code (especially the forward call) and training loop? This should make it easier for other to help.</p>
Inverse of a complex matrix

username: K_B
cooked: <p>Hi,<br>
I need to create a complex dtype matrix and then take an inverse of it? Is there a possibility  to do so?</p>
<p>Thanks<br>
Kay</p>

username: ptrblck
cooked: <p><code>torch.inverse</code> is currently not implemented for complex types as seen in <a href="https://github.com/pytorch/pytorch/issues/33152">this feature tracker</a>.</p>
Jupyter Notebooks Python Modules Not Found

username: archy2211
cooked: <p>Hi can anyone help me with his so I have a file locate in preprocessingfolder and then i try to run this file in another folder by useing code</p>
<p>from preprocessing.preprocessing_general import preprocess<br>
this error pop up No module named ‘preprocessing.preprocessing_general’</p>
<p>I try  import import_ipynb and still not work anyone have idea how to fix this problem?</p>

username: ptrblck
cooked: <p>Most likely your current working directory doesn’t know the path to your submodule.<br>
The easiest way would be to create a <code>preprocessing.py</code> file in the same working directory, which contains the desired methods.</p>
<p>The Python import system is explained <a href="https://docs.python.org/3/reference/import.html#regular-packages">here</a> in more detail.</p>
Fix Could not identify and read probably base64 image

username: archy2211
cooked: <p>hi can anyone explain what is this error about im still student so im not reaslly sure why there is error pop up and this is my code</p>
<p>def process_base64(base64_img):<br>
if base64_img[0] == “b”:<br>
img = imread(io.BytesIO(base64.b64decode(base64_img[1:])))<br>
elif base64_img[:4] == “data”:<br>
img = imread(io.BytesIO(base64.b64decode(base64_img[base64_img.find(","):])))<br>
else:<br>
img = imread(io.BytesIO(base64.b64decode(base64_img)))<br>
return img<br>
def read_image(image):<br>
# decode base64 file<br>
try:<br>
if os.path.isfile(image):<br>
if image.endswith(“txt”):<br>
f = open(image, “r”)<br>
base64_img = f.read()<br>
img = process_base64(base64_img)<br>
f.close()<br>
elif image.endswith(“json”):<br>
f = open(image, “r”)<br>
base64_img = f.read()<br>
img = process_base64(base64_img)<br>
f.close()<br>
else:<br>
img = imread(image)<br>
else:<br>
img = process_base64(image)<br>
except:<br>
raise NotImplementedError(“Could not identify and read probably base64 image”)</p>
<pre><code>if img.shape[2] &gt; 3:
    img = img[:, :, :3]

return img</code></pre>

username: ptrblck
cooked: <p>What kind of error are you seeing?</p>
<p>PS: this issue seems to be more Python-related, so you might get a better answer on StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
LowRankMultivariateNormal throws CUDA error

username: eliaz
cooked: <p>Hi there!</p>
<p>I’m using the <code>LowRankMultivariateNormal</code> distribution in order to have a distribution of logits for every pixel of an image.<br>
I have an issue when using this distribution for squared images with a shape that is even and large (&gt;= 512x512).</p>
<p>The following code does not work on a colab notebook when the size is set to 512x512 but does for size 513x513.<br>
I also tried it on different GPUs, with the same results.<br>
It works fine when using CPUs.</p>
<pre><code class="lang-auto">import torch
from torch.distributions import LowRankMultivariateNormal

DEVICE = "cuda"

torch.manual_seed(23)
for i in range(10):
    print(i)
    distrib = LowRankMultivariateNormal(
        torch.randn(1, 512, 512, 2).to(DEVICE),
        torch.randn(1, 512, 512, 2, 10).to(DEVICE),
        torch.randn(1, 512, 512, 2).to(DEVICE).exp()
    )
</code></pre>
<p>(I added the for-loop because sometimes it works for the first iteration, although it was not the case the last few times I ran it.)</p>
<p>I don’t have any clue on how to explain this behavior.</p>
<p>[edit]<br>
The issue is for shapes that are a power of 2, not even.</p>

username: ptrblck
cooked: <p>Thanks for reporting the issue! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I could reproduce it with the latest nightly binary and have created <a href="https://github.com/pytorch/pytorch/issues/41394">this issue</a> to track it.</p>
<p>It seems, that the <code>magma_spotrf_batched</code> call creates the illegal memory access.</p>

username: eliaz
cooked: <p>Thanks for the response.<br>
Do you think that there could be a workaround in the meantime?</p>

username: ptrblck
cooked: <p>Unsure at the moment, as we are using MAGMA for this particular operation, which causes the failure.<br>
Let’s see, if the code owners might have an idea for a workaround.</p>
MultiTask training without summing losses

username: Jgrietens
cooked: <p>Hello,</p>
<p>I am trying to implement an adaption to the classical Retinanet object detector. Retinanet could be considered a multitask network with classification and localization as separate tasks. Retinanet share a common backbone with 2 heads for localization and classification.</p>
<p>The normal way of train this network is to calculate separate losses L_class and L_loc  for both tasks, summing them:<br>
L_total = L_class + L_loc.</p>
<p>The training procedure is then:</p>
<p>optimizer.zero_grad()<br>
Lclass, L_loc = loss(retinanet(x) )<br>
L_total.backward()<br>
optimizer.step()</p>
<pre><code class="lang-auto">optimizer.zero_grad()
Lclass, L_loc = loss(retinanet(x) ) 
L_total.backward()
optimizer.step()


</code></pre>
<p>I would like to have 2 separate optimizers for the classification task and the localization task. My question is how to do this correctly to make sure that both heads are trained with separate optimizers ( opt_class, opt_loc) . I would propose this to be in my training loop:</p>
<pre><code class="lang-auto"># Localization first
opt_loc.zero_grad()
_, L_loc = loss(retinanet(x) ) 
L_loc.backward()
opt_loc.step()

# Classification second
opt_class.zero_grad()
L_class,_ = loss ( retinanet(x))
L_class.backward()
opt_class.step()
</code></pre>
<p>Would this be a sound approach and, is it possible to do this without creating addtional gpu memory usage ?</p>
3D matrix of dot products

username: Zorobay
cooked: <p>Hi! I have two matrices for which I need to calculate the dot product, but only for one dimension. They are of the same shape <code>(N,M,D)</code> and I want to take the dot product of the last dimension <code>D</code> so that the result is of shape <code>(N,M,1)</code>. How would I do that? Most solutions seem to assume stuff about number of dimensions etc.</p>

username: AladdinPerzon
cooked: <p>I think you can use einsum see below for a small example. It felt a bit confusing doing dot product over multiple dimensions but it seems to make sense to me after a few examples.</p>
<pre><code class="lang-auto">N, M, D = 2, 3, 4
x = torch.ones((N,M,D))
y = torch.ones((N,M,D))
torch.einsum('ijk,ijk-&gt;ij', x, y).unsqueeze(2) # will be shape (N, M, 1)
</code></pre>

username: Zorobay
cooked: <p>That actually does seem to work! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Any source to know pytorch model size and number of parameters

username: talhaanwarch
cooked: <p>In Keras, there is a detailed comparison of number of parameters and size in MB that model takes at <a href="https://keras.io/api/applications/#:~:text=Keras%20Applications%20are%20deep%20learning,They%20are%20stored%20at%20~%2F." rel="nofollow noopener">Keras application</a> page. Is there any similar resource in pytorch, where I can get a comparison of all model pretrained on imagenet and build using PyTorch. thanks</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>There is no place for all that information but you can get them one by one. For accuracy/error please visit <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">torchvision.models</a>.<br>
For model stucture/params/size you can use <code>torchsummary</code> in the same way in Keras.</p>
<pre><code class="lang-auto">import torchvision.models as models
alexnet = models.alexnet(pretrained=True)
alexnet.cuda()
summary(alexnet, (3, 224, 224))
print(alexnet)  # strcture - optional
</code></pre>
<p>It says model size but if you want to know the size of actual pretrained model, use <code>models.&lt;model_name&gt;(pretrained=True)</code> and it will download the weights.</p>
<p>Bests</p>
Clipping input data to the valid range for imshow with RGB data

username: talhaanwarch
cooked: <p>I have calculated the mean and std for normalization using following code</p>
<pre><code class="lang-auto">mean = 0.

std = 0.

nb_samples = 0.

for data,_ in train_loader:

    batch_samples = data.size(0)

    data = data.view(batch_samples, data.size(1), -1)

    mean += data.mean(2).sum(0)

    std += data.std(2).sum(0)

    nb_samples += batch_samples

mean /= nb_samples

std /= nb_samples
</code></pre>
<p>I got these values</p>
<blockquote>
<p>[0.5820, 0.4512, 0.4023], [0.2217, 0.1858, 0.1705]</p>
</blockquote>
<p>when I put the values, it gives me this warning</p>
<blockquote>
<p>Clipping input data to the valid range for imshow with RGB data ([0…1] for floats or [0…255] for integers).</p>
</blockquote>
<p>But if I used the values calculated for imagenet dataset subset, it does not give warning.<br>
So which values should I use?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>By <code>imshow</code> I assume you are using <code>matplotlib.pyplot</code>. In this case, <code>imshow</code> expects inputs in range of <code>[0, 1]</code> or <code>[0, 255]</code> so if the input array does not have this range, it will clip input.<br>
Concerning the issue that ImageNet values do not create this warning, it is possible but does not matter. The point is that you are using mean and std in z-score (<code>(x-mean) / std</code>) so the output values can out of <code>[-1, 1]</code>. So, for visualization, you need first to unnormalize your input then clipping input data would be ok.</p>
<p>Bests</p>
How do I map Joblib's Parallel function to PyTorch's DistributedDataParallel

username: Leockl
cooked: <p>I have the following code below which uses Joblib’s Parallel and I want to implement this in PyTorch and run it with GPUs. I am reading through PyTorch’s DistributedDataParallel <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">documentation</a>, but I can’t seem to figure this out.</p>
<pre><code class="lang-auto">import numpy as np
import torch
from joblib import Parallel, delayed
from torch.nn.parallel import DistributedDataParallel as DDP

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()

def X_power_func(j):
    X_power = X**j
    return X_power

results = Parallel(n_jobs = 4)(delayed(X_power_func)(j) for j in range(8))   # how do I map this to 
                                                                             # PyTorch's
                                                                             # DistributedDataParallel
</code></pre>
<p>Any help would really be appreciated. Many thanks in advance!</p>

username: iffiX
cooked: <p>use torch.multiprocessing.pool</p>

username: Leockl
cooked: <p>Thanks <a class="mention" href="/u/iffix">@iffiX</a>. Do you know in which situations that we would use <code>torch.multiprocessing</code> and <code>DistributedDataParallel</code>?</p>

username: iffiX
cooked: <p><code>DistributedDataParallel</code> is designed for asynchronously let the model perform forward and backward process, internnaly it synchronously perform gradient reduction and parameter updating.</p>
<p><code>torch.multiprocessing</code> is a simple derivative of the vanilla <code>multiprocessing</code> module, it only replaces the default queue implementation used in the vanilla module, and implements an efficient way to pass around cuda tensors (data remains on gpu, only a pointer to data is passed to subprocess pool workers).</p>
<p>Pool is designed for carrying out general unit tasks by a group of homogeneous workers with no context, such as your:</p>
<pre><code class="lang-auto">def X_power_func(j):
    X_power = X**j
    return X_power
</code></pre>
<p>Pool is essentially the same as <code>joblib</code></p>

username: Leockl
cooked: <p>Ok many thanks <a class="mention" href="/u/iffix">@iffiX</a> for the detailed answer.</p>
<p>So essentially use <code>DistributedDataParallel</code> for neural network stuff (which involves things like forward and backward processes) that you want to parallelize, and use <code>torch.multiprocessing</code> for non-neural network stuff that you want to parallelize.</p>

username: iffiX
cooked: <p>thats correct! <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Install pytorch-nightly by conda, but got v1.3.1

username: lcxywfe
cooked: <p>After <code>conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly</code><br>
I got :<br>
The following NEW packages will be INSTALLED:</p>
<p>_pytorch_select    pkgs/main/linux-64::_pytorch_select-0.1-cpu_0<br>
cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.2.89-hfd86e86_1<br>
ninja              pkgs/main/linux-64::ninja-1.9.0-py37hfd86e86_0<br>
pytorch            pkgs/main/linux-64::pytorch-1.3.1-cpu_py37h62f834f_0<br>
torchvision        pkgs/main/linux-64::torchvision-0.4.2-cpu_py37h9ec355b_0</p>
<p>Why the nightly version is 1.3.1?</p>

username: ptrblck
cooked: <p>Could you try to update the conda environment and try to rerun the install command?<br>
Besides the old version also the cpu binary is pulled. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: lcxywfe
cooked: <p>It’s fixed now, the nightly version is updated to v1.7.0</p>