akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to calculate the mean and std of my own dataset?

username: eternity
cooked: <p>Now,I want to calculate the mean and std of my dataset(about 10000 images).<br>
I know I can get the mean by:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7ee0de14beccee20b46d1c5a5a0be879116550c9.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ee0de14beccee20b46d1c5a5a0be879116550c9" title="G(%605YC6BLH%24%EF%BF%BDE(DA)2RJK"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7ee0de14beccee20b46d1c5a5a0be879116550c9.png" alt="G(%605YC6BLH%24%EF%BF%BDE(DA)2RJK" data-base62-sha1="i6pW8rTpe4xdQznCK28IyaZdBTX" width="443" height="500" data-dominant-color="2F333B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">G(%605YC6BLH%24%EF%BF%BDE(DA)2RJK</span><span class="informations">597×673 25.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, I do not know how to get <strong>the std</strong> further effectively.<br>
Can someone tell me how to get it? Better with the corresponding code!<br>
Thanks in advance!</p>

username: ptrblck
cooked: <p>Would <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6?u=ptrblck">this answer</a> from the other thread work?</p>

username: sharkdeng
cooked: <p>Got same question here. Previously I was using ImageNet fixed Normalize technique. What is the difference between Imagenet and self dataset?<br>
When I used ImageNet Normalize, the image becomes dark(pixels are below 0). I will try to get specific mean and std of my dataset to see if the image would not be black after Normalization.</p>

username: GorkemP
cooked: <p>Just as you did for mean, you can easily adapt your code to calculate standard deviation (after you calculated the means). In addition, if you count the number of pixels (width, height) in the loop, even if your images have different sizes you can get the exact number to divide the sum:</p>
<pre><code class="lang-auto">R_channel = 0
G_channel = 0
B_channel = 0

total_pixel = 0
for idx in xrange(len(pathDir)):
    filename = pathDir[idx]
    img = imread(os.path.join(filepath, filename))

    total_pixel = total_pixel + img.shape[0] * img.shape[1]

    R_total = R_total + np.sum((img[:, :, 0] - R_mean) ** 2)
    G_total = G_total + np.sum((img[:, :, 1] - G_mean) ** 2)
    B_total = B_total + np.sum((img[:, :, 2] - B_mean) ** 2)

R_std = sqrt(R_total / total_count)
G_std = sqrt(G_total / total_count)
B_std = sqrt(B_total / total_count)
</code></pre>
Detect param update

username: iffiX
cooked: <p>I am trying to implement a mechanism which needs to be detected when parameters are updated, there are three ways to implement it:</p>
<ol>
<li>use <code>register_backward_hook</code> (problematic, it will not work sometimes, proved by experiment and statement in doc.</li>
<li>use <code>register_hook</code> on every parameter (usable, but will not be executed right after parameter update)</li>
<li>find a way to hook onto inplace operations such as <code>add_</code>, <code>copy_</code> and <code>addcdiv_</code>, but I am not sure whether inplace operations are gauranteed to be used when you want to update a parameter (using param = some_modify_func(param) is possiple). And is there any <strong>clean</strong> way to hook onto them whithout creating a wrapper and wrap parameters inside ?</li>
</ol>
<p>What’s your suggestion? Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can definitely use <code>register_hook</code> to know when gradients are computed for a Tensor (it is executed just before the gradient is accumulated and the new value is passed as an argument).<br>
But I don’t think you will be able to have a callback every time a Tensor is modified inplace <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: iffiX
cooked: <p>Thank you! Eventually I chose the second option, currently there is no better option than <code>register_hook</code> <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Implementing trainable parameters with norm=1

username: Mckinsey666
cooked: <p>I am trying to implement trainable parameters with norm always=1.<br>
Basically, I want to optimize the following variable <code>p</code> directly:</p>
<pre><code class="lang-auto">p = Variable(torch.zeros(15), requires_grad = True)
optimizer = optim.SGD([p], lr=0.1)
</code></pre>
<p>Every training step, <code>p</code> will be updated, but to prevent the values of <code>p</code> of exploding, I want to normalize <code>p</code> every step to have l2norm = 1 (but still trainable through the optimizer).  Is this possible?</p>
<p>Thank you all!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You have two options here:</p>
<ul>
<li>Projected gradient descent. Where after each step, you project p onto the space you want (here you want <code>p = p / p.norm(1)</code>)</li>
<li>Use unconstrained version by learning <code>p</code> just like you do right now, but when you use it, use <code>actual_p = p / p.norm(1)</code>. This way, you make sure that <code>actual_p</code> is always of the right norm (whatever the value of p is).</li>
</ul>
Not able to backpropagate because of tensor.data

username: saumya0303
cooked: <p>Hi team,<br>
Please find below the code snippet.I want to backpropagate till input_1 but facing an issue :-<br>
list_1=[]<br>
for i in range(10):</p>
<ul>
<li>batch = [list of frames ]</li>
<li>input_1 = torch.cat(batch,0)</li>
<li>output = model(input);</li>
<li>list_1.append(output.data)</li>
</ul>
<p>The <strong>output</strong> of a model is a tensor with <strong>required grad = True</strong> when I append output in a list(list_1) the code crashes after some iterations, while if I append output. data the code runs fine and requires_grad become false and backpropagation won’t work. What is the correct way to solve this problem?<br>
Thanks in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you use <code>.data</code>, you break the differentiable link between the element you add in the list and the net.<br>
If you don’t use it, I guess you add more and more to the list and you eventually run out of memory because of all the states you keep around.</p>
<p>If you do want to be able to compute gradients here, you should not use <code>.data</code> (in general, you should never use <code>.data</code> actually <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> ). But you will have to be careful and make sure the list does not get too big to limit the memory requirements.</p>
Adding a constant to the loss function

username: smu226
cooked: <p>Hello! I need to write a slightly modified percentage error loss function, with a threshold for the denominator. The part of code relevant to it looks like this:</p>
<pre><code class="lang-auto">threshold = 290000

def exp_rmspe(pred, target):
    loss = torch.abs(target - pred)/np.maximum(target,threshold)
    return loss.mean()
</code></pre>
<p>I have a batch size of 128. When I am using this, I get this error</p>
<blockquote>
<p>RuntimeError: bool value of Variable objects containing non-empty torch.cuda.ByteTensor is ambiguous</p>
</blockquote>
<p>How can I add that threshold constant in the loss function, without getting that error? Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Is this error coming from this function? Also to avoid any issue, I would advise against mixing numpy and torch functions, you can use <code>torch.max()</code>.</p>

username: Ignacio_Hernandez
cooked: <p>Hi, sorry for reviving this thread after a year.</p>
<p>I have a similar question about adding a constant in the loss function: let’s suppose I want to multiply the final loss by a factor, and that factor is computed based on the inputs or the outputs within a batch.</p>
<p>In my case I’m working with triplet networks and a custom loss, and now I would like to add a scale factor based on the amount of ‘easy negatives’, for example. This is part of my loss function:</p>
<pre><code class="lang-auto">def forward(self, x, labels):
        ...

        l2_norm_positive = F.pairwise_distance(x[0], x[1],2)
        l2_norm_negative = F.pairwise_distance(x[0], x[2],2)
        easy_negatives = l2_norm_negative &gt;= (l2_norm_positive + self.margin)
        easy_negatives_percent = torch.sum(easy_negatives).float()/labels[0].shape[0]
        scale_factor = 1/(1-easy_negatives_percent+self.eps)
        
        ...
</code></pre>
<p>If I remove the part where it computes the scale factor I get normal gradients after calling <code>.backward()</code>, but if I leave that section the gradients become <code>nan</code>.</p>
<p>How could I do that withouth breaking the computation graph? How should I deal with that constant (<code>scale_factor</code>)? Should I use <code>.detach()</code>?</p>
<p>Thanks in advance</p>

username: albanD
cooked: <p>Hi,<br>
If you are getting nan, it means that you’re most likely dividing by 0 somewhere in your function. Can you check the different values to confirm that?</p>
<p>You can use <code>.detach()</code> but that will have a different effect: No gradient will flow back from the scale_factor. It depends on what you want to do here.</p>
Getting the average value of a list of predictions

username: bfeeny
cooked: <p>I have a list of torch tensors, each is a tensor that holds predictions and i wish to average those. So, image 10982 predictions and we repeat that 3 times:</p>
<pre><code class="lang-auto">my_list = []
for i in range(3):
    my_list.append(torch.arange(i,10982+i, dtype=torch.float32).reshape(10982,1))
</code></pre>
<p>is there a better way (a more pytorch or even numpy way) vs just doing list aggregate functions?  It looks like the below is giving me what I want, it returns a tensor of size 10982,1 which is what I want.  I just figured there is a more proper pytorch way of doing this.</p>
<p><code>sum(my_list)/len(my_list)</code></p>
Passing tensorboard summaries when saving and loading checkpoints

username: jahan123
cooked: <p>I am using Tensorboard for visualizing the results, how can I pass the <code>summary</code> when saving a general checkpoint so I can resume visualization from same epoch?</p>
<p>code i’m using:</p>
<pre><code class="lang-auto">checkpoint = {'epoch': epoch + 1, 'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
</code></pre>
<pre><code class="lang-auto">def resume_from_checkpoint(fpath='xyz.pth', model, optimizer=None, scheduler=None):

    print('Loading checkpoint from "{}"'.format(fpath))
    checkpoint = load_checkpoint(fpath)
    model.load_state_dict(checkpoint['state_dict'])
    print('Loaded model weights')

    if optimizer is not None and 'optimizer' in checkpoint.keys():
        optimizer.load_state_dict(checkpoint['optimizer'])
        print('Loaded optimizer')
    if scheduler is not None and 'scheduler' in checkpoint.keys():
        scheduler.load_state_dict(checkpoint['scheduler'])
        print('Loaded scheduler')

    epoch = checkpoint['epoch']
    print('Last epoch = {}'.format(epoch))

    if 'rank1' in checkpoint.keys():
        print('Last rank1 = {:.1%}'.format(checkpoint['rank1']))
    return epoch
</code></pre>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    total_correct = 0
    total_loss = 0
    if epoch % 1 == 0:
        checkpoint = {'epoch': epoch + 1, 'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
        print("saving checkpoint")
        save_checkpoint(checkpoint)
    for i, traindata in enumerate(trainloader):
        images = 
        labels = 
        optimizer.zero_grad()
</code></pre>
<p>my apologies in advance if the question doesn’t belong here.</p>
How to convert a pytorch pth file to an onxx model that supports any batch size,

username: zxgm
cooked: <p>How to convert a pytorch pth file to an onxx model that supports any batch size,how to set torch.randn function batch_size variables</p>

username: zxgm
cooked: <p>I find method that use torch.onnx.export dynamic_axes variable</p>

username: CharlesHuan
cooked: <p>I search for a way that could realizations from the file endwith"pth" to the file endwith"onxx".but emmmm<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"></p>

username: leigaoxiang
cooked: <p>how did you using dynamic_axes ,i use the parammeter it raises    " ‘please determine the input size manually by --input-shape xxx’.format(key))<br>
RuntimeError: The shape of input “input” has dynamic size, please determine the input size manually by --input-shape xxx</p>
Default_collate

username: Mohamed_Nabih
cooked: <p>I tried to import default_collate from torch.utils as shown</p>
<p>from torch.utils.data.dataloader import default_collate</p>
<p>The problem is the default_collate is not found and I found _collate_fn_t are they similar functions?</p>

username: Antonio_Ossa
cooked: <p>Hi <a class="mention" href="/u/mohamed_nabih">@Mohamed_Nabih</a>,</p>
<p>I tried the import you shown and it works for me. Which version are you using?. Also, I remember I recently imported the same function using: <code>from torch.utils.data._utils.collate import default_collate</code>.</p>
<p>To check if the import you used pointed at the same funciton, I checked with the following code:</p>
<pre><code class="lang-auto">from torch.utils.data.dataloader import default_collate as dc1
from torch.utils.data._utils.collate import default_collate as dc2

print(dc1 is dc2)
# &gt;&gt; True
</code></pre>
<p>So yes, both ways import the actual <code>default_collate</code> method. I couldn’t find <code>_collate_fn_t</code>, but I think they refer to the same method (*.pyi files seem to be related with annotations, so they’re not actual declarations).</p>

username: Mohamed_Nabih
cooked: <p><a class="mention" href="/u/antonio_ossa">@Antonio_Ossa</a><br>
Thanks, Antonio for your help it works.</p>
Torch.onnx._export

username: leigaoxiang
cooked: <p>when i use torch.onnx._export to convert pth to onnx for dynamic input,but when i use dynamic_axes parameter,it raised  error as follows,</p>
<p>‘please determine the input size manually by --input-shape xxx’.format(key))<br>
RuntimeError: The shape of input “input” has dynamic size, please determine the input size manually by --input-shape xxx</p>
<p>and when i didn’t use dynamic_axes  it can convert to onnx succesfully,<br>
torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,<br>
input_names=input_names, output_names=output_names,dynamic_axes={“input”:[3]},keep_initializers_as_inputs=True, opset_version=11)</p>
<p>torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,<br>
input_names=input_names, output_names=output_names,keep_initializers_as_inputs=True, opset_version=11)</p>
I found a mistake in an official case

username: itmorn
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial</a><br>
In this link, an RNN network is specified, and I think there is something wrong with its structure. This existing structure cannot collect hidden layer information for the last character. So I think the network should be modified to the structure shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5_2_518x500.png" alt="image" data-base62-sha1="elVs620AWcrf8NZCUctohZk8Cj3" width="518" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5_2_518x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/4/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/4/6498f4aaa247c7cc0d31fdf2a9a989519a8da1d5.png 2x" data-dominant-color="EFECDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">616×594 36.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The modified network is shown below:</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = F.relu(self.i2h(combined))
        output = self.i2o(hidden)
        output = self.softmax(output)
        return output, hidden
</code></pre>
Save/Load Fails in Model with LSTM num_layers > 1

username: Oliver_Strayde
cooked: <p>I have been experimenting with a model for forecasting, which is composed like so:</p>
<pre><code class="lang-auto"># nn.Model, etc.
		self.lstm = nn.LSTM(input_size=12, hidden_size=6,
      num_layers=3, batch_first=True)
		self.linear = nn.Linear(6, 2)
		self.output_layer = nn.Linear(2, 1)
</code></pre>
<pre><code class="lang-auto">torch.save({

  'epoch': 1,

  'model_state_dict': model.state_dict(),

  'optimizer_state_dict': optimizer.state_dict(),

  'loss': loss

}, 'model.tar')
</code></pre>
<p>The model trains, evaluates and saves fine. However, it fails when I attempt to load it as per the docs:</p>
<pre><code class="lang-auto">
model = LSTModel()
optimizer = torch.optim...

checkpoint = torch.load('onelayer')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
</code></pre>
<p>I receive the following when attempting to load the model’s state dict:</p>
<blockquote>
<p>Error(s) in loading state_dict for ERNN:<br>
Missing key(s) in state_dict: “lstm.weight_ih_l1”, “lstm.weight_hh_l1”, “lstm.bias_ih_l1”, “lstm.bias_hh_l1”, “lstm.weight_ih_l2”, “lstm.weight_hh_l2”, “lstm.bias_ih_l2”, “lstm.bias_hh_l2”, “linear.weight”, “linear.bias”, “output_layer.weight”, “output_layer.bias”.<br>
Unexpected key(s) in state_dict: “mapping.weight”, “mapping.bias”, “timeLayer.weight”, “timeLayer.bias”.</p>
</blockquote>
<p>I see these keys are indeed missing, is there something I am doing wrong when saving the model?</p>

username: ptrblck
cooked: <p>I guess you might load the wrong checkpoint, since your model’s <code>state_dict</code> can be stored and loaded successfully:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.lstm = nn.LSTM(
            input_size=12, hidden_size=6, num_layers=3, batch_first=True)
        self.linear = nn.Linear(6, 2)
        self.output_layer = nn.Linear(2, 1)

model = MyModel()
sd = model.state_dict()

model = MyModel()
model.load_state_dict(sd)
&gt; &lt;All keys matched successfully&gt;
</code></pre>
<p>Are the unexpected keys familiar looking? E.g. did you use <code>mapping</code> and <code>timeLayer</code> in another script, which might have overwritten the needed checkpoint?</p>
Trainable weights

username: Prakhar_Pradhan
cooked: <p>Hello everyone!<br>
I am working with PyTorch and implementing a semantic segmentation method. The loss function used is a combination of 6 loss terms, i.e. L= L1+L2+L3+L4+L5+L6. Now my requirement is to use trainable weights with these 6 loss terms, which means L= w1<em>L1+w2</em>L2+w3<em>L3+w4</em>L4+w5<em>L5+w6</em>L6.<br>
Now in order to get <strong>trainable</strong> weights, I am using <strong>torch.autograd()</strong> class, means<br>
w1 = torch.randn(x.size(),require_grad=True), like this I am doing for all the 6 weights.<br>
Does this method takes care of backpropagation ? or Is this the correct way to use the trainable weights in PyTorch?</p>

username: ptrblck
cooked: <p>The way of creating these weights should be correct and the optimizer should be able to update them, if you pass them properly to it.</p>
<p>However, wouldn’t the optimizer try to force these weights to zeros (or rather negative values), as it would decrease the loss?</p>
Accuracy stuck for Recurrent Convolution Neural network

username: Amasu
cooked: <p>Hi, I am implementing RCNN for emotion classification in text. I am using Bi-directional LSTM then apply max-pool to get dominant features and pass it to the FC layers for classifying. Right now I am stuck at a point after achieving accuracy 83 my model gets stuck at that point. Accuracy and loss doesn’t change after that. I have attached an image of performance. Kindly suggest me anything that would help. Thank you.</p>
<p>Optimizer: SGD<br>
Loss: Focal Loss</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa86b55b8a47babf0480eb2ed6720cdfd0bcb225.png" alt="Metrics (1)" data-base62-sha1="okxMrTYAU3xvLr3NEAyoeM4m44d" width="640" height="480"></p>

username: vdw
cooked: <p>I’m not saying there isn’t anything that can be tweaked, but this looks like a pretty normal result for such a text classification task to me. Training loss going towards 100%, and test/validation loss around 80%+. Three related things to consider:</p>
<ul>
<li>
<p>Sentiment or emotion are highly subjective. I bet the if you look at the dataset you wouldn’t agree with all labels. Alternatively, give the same text to 3 different people, and in many cases you won’t get a perfect agreement, particularly for emotion where you typically have 4-7 classes.</p>
</li>
<li>
<p>Language can be extremely subtle and there are stylistic devices (sarcasm, irony, cynicism, humor) which are arbitrary difficult to learn for a network, particularly without virtually unlimited training data.</p>
</li>
<li>
<p>What a text conveys often goes beyond the words in the text, but relies on a share knowledge and understanding. Consider the simple sentence “There’s a spider on my pillow.” On it’s, this sentence is as neutral as it gets. It’s the general knowledge that most people are afraid of spiders that make this sentence associated with “fear”.</p>
</li>
</ul>
<p>Language is way too expressive to expect a network to find simple patterns.</p>

username: Amasu
cooked: <p>Makes sense. I guess</p>
Parameter defined by torch.cat does not get updated

username: PersonalFinance004
cooked: <p>I have two models, model_1 and model_2, which share a torch.nn.Parameter <code> model_1.emb</code> of shape <code>[n,dim]</code>. This parameter is defined in the init of model_1, and passed to the constructor of model_2 to define</p>
<pre><code class="lang-auto">self.shared_emb = emb
</code></pre>
<p>This shared parameter is used in the forward functions of both models, and it is properly updated by each of the two optimizers, which optimize a shared loss function</p>
<pre><code class="lang-auto">L = L_model_1 + L_model_2
</code></pre>
<p>This works fine.</p>
<p>Now, in model_2, I want to concatenate <code>self.shared_emb</code> with a second parameter. To verify that this would work, I use a dummy parameter <code>model_2.dum</code> with shape <code>[0,dim]</code>. I use the following code in the init of model_2:</p>
<pre><code class="lang-auto">self.cat_emb = torch.cat([self.shared_emb, self.dum], dim=0)
</code></pre>
<p>Now, <code>model_2.shared_emb </code> should be exactly the same thing as <code>model_2.cat_emb </code>. I replace the former with the latter in model_2’s forward function.<br>
But for some reason, this new parameter does not get updated, its weights do not change!</p>
<p>I googled, and also tried the following:</p>
<pre><code class="lang-auto">self.cat_emb = torch.cat([emb, dum], dim=0)
self.cat_emb.retain_grad()
</code></pre>
<p>but still, the parameter thus defined does not get updated. requires_grad is set to <strong>True</strong> for both emb and shared_emb. I have triple checked everything and am at my wit’s end here.</p>
<p>Any ideas why this does not work?</p>

username: ptrblck
cooked: <p><code>self.cat_emb</code> is created via an operation and is thus not a leaf variable anymore.<br>
The optimizer would thus raise an error, as it cannot optimizer non-leaf tensors, if you try to pass <code>self.cat_emb</code> to it.</p>

username: PersonalFinance004
cooked: <p>And what does that imply? Is there no way to make this work?</p>

username: ptrblck
cooked: <p>You won’t be able to update <code>cat_emb</code>, but would need to recreate it using <code>shared_emb</code> (which can be updated).</p>
Numerical Error after Addition

username: marcocarobene
cooked: <p>Hi all,<br>
I’m encountering an unexpected difference in the value of two tensors before and after addition.</p>
<p>I have received the following tensor from a Linear layer:</p>
<pre><code class="lang-auto">x = torch.tensor([ 0.2163, -0.1675, -0.0950, -0.1593, -0.0628, -0.3286,  0.0187, -0.2372,
        -0.1530,  0.2945,  0.0602,  0.1602, -0.0701,  0.0301, -0.0220, -0.1735,
        -0.1145,  0.1231,  0.0635, -0.0197,  0.0776, -0.1025, -0.0822,  0.0577,
        -0.0608,  0.0161,  0.2321, -0.1484,  0.0033,  0.1681, -0.0177, -0.1523,
        -0.0126, -0.1447, -0.0075,  0.0568, -0.2305, -0.1494, -0.3213, -0.2324,
         0.1063, -0.0171, -0.0016, -0.0474,  0.3222,  0.1174,  0.1325, -0.3385,
        -0.0653, -0.1163, -0.1045, -0.1349,  0.0684, -0.1131,  0.3090, -0.1836,
        -0.0581,  0.0181,  0.0861,  0.0921, -0.0187,  0.1681,  0.0273,  0.0134],
       device='cuda:0')
</code></pre>
<p>and I then run the following few lines:</p>
<pre><code class="lang-auto">outer = torch.ger(x, x)[0,0]
A = torch.tensor([1e6], device='cuda:0')
sigma = 1e-6

print(A)
&gt; tensor([1000000.], device='cuda:0')
print((1/sigma)*outer)
&gt; tensor([46785.6914], device='cuda:0')

A = A + (1/sigma)*outer
print(A)
&gt; tensor([1046785.6875], device='cuda:0')
</code></pre>
<p>Which shows quite a large difference in terms of the values in the decimal places. I’m not sure whether there is something I’m missing, so would be great to have this clarified. All tensors here are of dtype torch.float32.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>This small error is expected due to the limited floating point precision.<br>
As explained in <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article for single precision floats</a>, the precision limitations of decimal values are defined as:</p>
<pre><code class="lang-auto">Decimals between 2**n and 2**(n+1): fixed interval 2**(n-23)
</code></pre>
<p>For your output <code>n</code> would be <code>19</code>, since:</p>
<pre><code class="lang-python">x = torch.tensor(1000000.)
y = torch.tensor(46785.6914)
z = x + y

print(2**19 &lt; z)
&gt; tensor(True)
print(z &lt; 2**20)
&gt; tensor(True)
</code></pre>
<p>which results in an eps of <code>2**(19-23) = 0.0625</code>.<br>
The representable numbers are thus:</p>
<pre><code class="lang-python">1046785.6875 (your output) &lt; 1046785.6914 (theoretical output) &lt; 1046785.7500 (next valid number, = 1046785.6875 + 0.0625)
</code></pre>
<p>You can verify the rounding via:</p>
<pre><code class="lang-python">for i in torch.linspace(0, 2**(19-23), 10):
    print(z + i)
&gt; tensor(1046785.6875)
tensor(1046785.6875)
tensor(1046785.6875)
tensor(1046785.6875)
tensor(1046785.6875)
tensor(1046785.7500)
tensor(1046785.7500)
tensor(1046785.7500)
tensor(1046785.7500)
tensor(1046785.7500)
</code></pre>
Are dictionaries/np array safe in torch datasets?

username: Maxime_Louis
cooked: <p>Hey,</p>
<p>It may be a trivial question but I could not find a clear answer online.</p>
<p>I’m working with datasets of large images (loading each image takes a second) but the whole dataset can hold in memory, so I have decided to store the images in RAM instead of reading them on the fly.</p>
<p>I have my custom dataset class and so far I’m storing the raw images in a Manager.dict(), and in <strong>get_item</strong> I do augmentations/processing in numpy on these images before returning a tensor. So my questions are:</p>
<ol>
<li>Can I use a simple python dictionary to store these images ? (meaning is it read/write safe when using it through a dataloader with multiple workers ?)</li>
<li>Same question for a large np array (which would contain all images) ?</li>
<li>Do you think it’s faster to load all images once at instanciation of the dataset, or to load and store them in RAM during the first call to <strong>get_item</strong> for each sample ?</li>
</ol>
<p>Thank you in advance,</p>

username: ptrblck
cooked: <ol>
<li>
<p>and 2.: since <code>DataLoader</code>s use multi-processing the <code>Dataset</code>s will be copied and each worker will work in its own <code>Dataset</code> instance, so there shouldn’t be any read/write conflict. However, this would also mean that your memory usage should increase with the used number of workers.</p>
</li>
<li>
<p>You could compare both approaches, but note that the “lazy loading” will most likely not work using multiple workers, as the data would be stored on a replica and you would need to use shared arrays/dicts as given in <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">this example</a>. Also to avoid reloading the complete data, you could pre-load it outside of the <code>Dataset.__init__</code> and just pass it as an argument. Otherwise the dataset will be recreated in each epoch.</p>
</li>
</ol>
Python train.py with modelnet40.json

username: shangguan91
cooked: <p><a href="https://github.com/valeoai/LightConvPoint/blob/master/examples/README.md" rel="nofollow noopener">https://github.com/valeoai/LightConvPoint/blob/master/examples/README.md</a></p>
<p>Hey, i am reexperimenting with modelnet40, however,</p>
<p>ERROR - ModelNet40 - Failed after 0:00:00!<br>
Traceback (most recent calls WITHOUT Sacred internals):<br>
File “train.py”, line 84, in main<br>
device = torch.device(_config[“device”])<br>
TypeError: Device() received an invalid combination of arguments - got (NoneType), but expected one of:</p>
<ul>
<li>(torch.device device)<br>
didn’t match because some of the arguments have invalid types: (NoneType)</li>
</ul>

username: ptrblck
cooked: <p>It seems that <code>_config["device"]</code> contains a <code>None</code> entry, while a valid device (or device string) is needed.<br>
You could either make sure the <code>_config</code> dict contains a valid device or alternatively specify the <code>device</code> manually e.g. via:</p>
<pre><code class="lang-python">device = 'cuda' if torch.cuda.is_available else 'cpu'
</code></pre>
How to debug when Tensorboard validation loss doesn't match

username: whoab
cooked: <p>I have a validation loss of .02 during training. I see this in tensorboard.</p>
<p>After, I load up the model’s checkpoint. I set all parameters to have 0 gradient. Then I run my training script again and comment out the backward pass:</p>
<pre><code class="lang-auto">            with self.timers.record("grad"):
                self.optimizer.zero_grad()
                if self.use_fp16:
                    with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                        scaled_loss.backward()
                else:
                    loss.backward()
</code></pre>
<p>This is the plot I see:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de13d633544b4e57c83ee45c59bd2400bbf96dfe.png" data-download-href="https://discuss.pytorch.org/uploads/default/de13d633544b4e57c83ee45c59bd2400bbf96dfe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de13d633544b4e57c83ee45c59bd2400bbf96dfe_2_634x500.png" alt="image" data-base62-sha1="vGAuTUVk9lKCdbMlwnxrUbskONo" width="634" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de13d633544b4e57c83ee45c59bd2400bbf96dfe_2_634x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de13d633544b4e57c83ee45c59bd2400bbf96dfe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de13d633544b4e57c83ee45c59bd2400bbf96dfe.png 2x" data-dominant-color="D9D8D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">830×654 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The error is much higher, and also constant (I would expect variation from the minibatches).</p>
<p>Any clue what’s happening?</p>

username: ptrblck
cooked: <p>If I understand the issue correctly, you get a constant validation loss for the complete validation dataset after restoring the model, while the loss for the same setup changes before storing the model?</p>
<p>If so, could you post the model definition, so that we could have a look?</p>
Installing PyTorch on laptop without GPU

username: Wesley_Neill
cooked: <p>This is probably a silly question but…</p>
<p>I am installing PyTorch on my laptop, which does not have a GPU. Ive been using a pre-installed version of PyTorch on Kaggle and GCP.</p>
<p>I assume I should select ‘None’ for CUDA on the “getting started” page?</p>
<p>I still want access to any methods or libraries that deal with CUDA like <code>torch.cuda.is_available()</code> so that I can write my code locally and then run it on a GPU when I need it.</p>

username: Angry_potato
cooked: <p>Hey why dont u try this method in colab with hardware acceleration set as none.</p>

username: Wesley_Neill
cooked: <p>That’s actually not what I’m asking. I want to know if I lose access to any torch packages that involve CUDA interaction if I select ‘None’ for CUDA during local installation.</p>
<p>Collab, if it is anything like GCP or Kaggle’s kernels, already has PyTorch installed.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wesley_Neill" data-post="1" data-topic="87647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p>I still want access to any methods or libraries that deal with CUDA like <code>torch.cuda.is_available()</code> so that I can write my code locally and then run it on a GPU when I need it.</p>
</blockquote>
</aside>
<p>The CPU package will include these method, so that you can write device-agnostic code.<br>
If you try to execute a CUDA operation, an error will be raised. However, no errors should be raised for utilitz functions e.g. to check of a GPU is available.</p>
Custom datatype for many images to one label

username: William_Richards
cooked: <p>I have a dataset comprising of human labelled facial images of 40 participants in the following file structure:</p>
<p>root/participant1/video1/frame0001.jpg<br>
…<br>
root/participant1/video2/frame0001.jpg<br>
…<br>
root/participant2/video2/frame0001.jpg<br>
etc</p>
<p>As the video was annotated at 20-second intervals, many of these images can refer to one label within an xlsx file with the columns:</p>
<p>-participantid<br>
-videoid<br>
-segment index (which 20s interval the label belongs)<br>
-target variable.</p>
<p>I am looking for an efficient PyTorch dataset implementation to load the dataset images so that I can do leave-one-out cross-validation across participants i.e. train a CNN on all videos for all subjects with the exception of the videos of one participant and repeat until all participants have been left out once.</p>
<p>The issue I currently have is constructing a custom data structure that deals with many images to one label as well as allowing for the desired cross-validation.  What is the best way to implement this in PyTorch?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="William_Richards" data-post="1" data-topic="87629">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_richards/48/26237_2.png" class="avatar"> William_Richards:</div>
<blockquote>
<p>The issue I currently have is constructing a custom data structure that deals with many images to one label</p>
</blockquote>
</aside>
<p>I’m not sure if you really want to design the <code>Dataset</code> to return multiple images and a single target or if you rather want to repeat the target for all images in the same interval.<br>
Both approaches would be possible by writing a custom <code>Dataset</code> as described in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>.<br>
Depending how the targets are stored, you might want to create them in the <code>Dataset.__init__</code> method and lazily load the data (and index the target) in <code>__geittem__</code>.</p>
<aside class="quote no-group" data-username="William_Richards" data-post="1" data-topic="87629">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_richards/48/26237_2.png" class="avatar"> William_Richards:</div>
<blockquote>
<p>as well as allowing for the desired cross-validation.</p>
</blockquote>
</aside>
<p>I would recommend to use some methods from <code>sklearn.model_selection</code> and split the dataset indices according to your groups. E.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html"><code>GroupKFold</code></a> might be useful.</p>
<p>Once you have these split indices, you could wrap your datasets into <code>Subset</code>s with these indices and perform your cross validation.</p>
ONNX export: no bilinear interpolation support?

username: thnkim
cooked: <p>Hello,<br>
It looks the ONNX exported model use ‘nearest’ interpolation although I exported a network with ‘bilinear’ interpolation. The code I tested is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class TestNet(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        x = F.interpolate(x, (2,4), mode='bilinear', align_corners=False)
        return x

net = TestNet()

batch_size = 8
x = torch.tensor([[[[0.0, 1.0, 2.0], [1, 2, 3]]]])

print('net(x):', net(x))
torch_out = torch.onnx._export(net, x, 'net.onnx', export_params=True, verbose=False)
print('onnx export output:', torch_out.data)



#
import onnx

model = onnx.load("net.onnx")
onnx.checker.check_model(model)

import caffe2.python.onnx.backend as backend

rep = backend.prepare(model, device="CPU") # or "CPU"
outputs = rep.run(x.data.numpy())
print('caffe2:', outputs[0])
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">net(x): tensor([[[[0.0000, 0.6250, 1.3750, 2.0000],
          [1.0000, 1.6250, 2.3750, 3.0000]]]])
onnx export output: tensor([[[[0.0000, 0.6250, 1.3750, 2.0000],
          [1.0000, 1.6250, 2.3750, 3.0000]]]])
caffe2: [[[[0. 0. 1. 2.]
   [1. 1. 2. 3.]]]]
</code></pre>
<p>Is it a known issue?<br>
Thank you.</p>

username: thnkim
cooked: <p>I’ve updated the codes. The previous one I uploaded was incorrect to show the issue.</p>

username: zdaiot
cooked: <p>Hello，Have you solved the problem yet?</p>

username: ebarsoum
cooked: <p>This should be supported, did you try latest PyTorch?</p>

username: thnkim
cooked: <p>Oh, I think it is supported now. The original question was out-dated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thank you.</p>
How to compute the gradient of a vector during test process?

username: S_M
cooked: <p>Hi everyone, I want to use the gradient of a vector during test or validation epochs. Would anyone describe me by an example of how to compute the gradient of a vector in test mode?</p>
<p>thanks</p>

username: ptrblck
cooked: <p>Could you explain a bit, what “gradient of a vector” means?<br>
If you would like to calculate the gradients of all parameters (as done during training), you could just reuse the training loop and use the validation or test dataset.<br>
Note that you should not update the model, as this would of course be a data leakage.</p>

username: S_M
cooked: <p>I am interested in computing the gradient of all parameters. Do you mean, I can use “loss.backward” in test to compute the gradient of parameters but I should not use “optimizer.step()” to update the model?</p>

username: ptrblck
cooked: <p>Yes, <code>loss.backward()</code> using the test data would compute the gradients in the same way as during training.<br>
<code>optimizer.step()</code> would then update all parameters with their gradients, which would create a data leakage, since you would be training on the test dataset.</p>

username: S_M
cooked: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Training 'FasterRCNN' object without 'references\detection\engine.train_one_epoch' meets less decrease of loss

username: JoYeY
cooked: <p>Hellow everyone!<br>
I write my customized training code to train the FasterRCNN object in similar way to train the most classify-model, but it works bad with very slow cahnge of losses. I wonder whether my customized code is not suitable for training FasterRCNN, thank you.<br>
The core code as follows（faster_rcnn is FasterRCNN object with pretrained vgg16.features as backbone）:</p>
<pre><code class="lang-auto">params = [p for p in faster_rcnn.parameters() if p.requires_grad]

optimizer = optim.SGD(params, lr=0.01, momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min')

for epoch in range(num_epochs):
    start = time.time()
    
    faster_rcnn.train()
    for img, target in data_loader:
        img = list(image.to(device) for image in img)
        target = [{k: v.to(device) for k,v in t.items()} for t in target]
        loss_dict = faster_rcnn(img, target)

        losses = sum(loss for loss in loss_dict.values())

        optimizer.zero_grad()
        losses.backward()

        optimizer.step()
     
        lr_scheduler.step(losses)
</code></pre>
Intialize the multidimension weights for sequential model from txt file

username: Ganga
cooked: <p>i have Conv2d as sequential model,i am passing the multidim tensor as an input from the txt file,how do i initialize the Conv2d weight from the text file which is also an multidimension tensor<br>
here is my code:</p>
<p>import torch<br>
import torch.nn as nn<br>
import csv<br>
import torch<br>
import pandas as pd<br>
import math</p>
<p>class PrintLayer(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(PrintLayer, self).<strong>init</strong>()<br>
self.conv = nn.Conv2d(1, 1, kernel_size=5, stride=1, padding=2)<br>
a_file1 = open(“2nd layer weight.txt”,“r”)<br>
list_of_lists1 = []<br>
for line in a_file1:<br>
stripped_line1 = line.strip()<br>
line_list1 = stripped_line1.split()<br>
list_of_lists1.extend(line_list1)<br>
a_file1.close()<br>
print(len(list_of_lists1))<br>
b = torch.from_numpy(numpy.array(list_of_lists1,dtype= ‘float32’))<br>
w = torch.tensor(b)<br>
ten = w.reshape(256,96,5,5)<br>
model[0].weight.data = ten<br>
<span class="hashtag">#m</span>.weight.data = m.weight.data.fill(m.weight.data)</p>
<pre><code>def forward(self, x):
    # Do your print / debug stuff here
   # print("x is",x)
    return x
</code></pre>
<p>model = nn.Sequential(<br>
nn.Conv2d(96, 256, kernel_size=5, stride=1, padding=2),<br>
PrintLayer(),<br>
# Add Print layer for debug<br>
)<br>
a_file = open(“input_27_27_96.txt”,“r”)<br>
list_of_lists = []<br>
for line in a_file:<br>
stripped_line = line.strip()<br>
line_list = stripped_line.split()<br>
list_of_lists.extend(line_list)<br>
a_file.close()<br>
<span class="hashtag">#print</span>(len(list_of_lists))<br>
b = torch.from_numpy(numpy.array(list_of_lists,dtype= ‘float32’))<br>
x = torch.tensor(b)<br>
x = x.reshape(1,96,27,27)<br>
x =x.double()<br>
<span class="hashtag">#print</span>(x.size())<br>
output = model(x.float())<br>
print(model.state_dict())<br>
<span class="hashtag">#print</span>(list(model.named_parameters()))</p>
Adding weights to loss gives broadcast error

username: kirtanp
cooked: <p>I have a model which trains fine when I use the BCELoss without weights, but when I add weights I get a broadcast error as follows</p>
<pre><code class="lang-auto">RuntimeError: output with shape [128, 1] doesn't match the broadcast shape [128, 2]
</code></pre>
<p>The code is quite straightforward</p>
<pre><code class="lang-auto">model = LinearRegression()
model.to(device)
lr = 1e-1
optimizer = optim.SGD(model.parameters(), lr=lr)
n_epochs = 50
loss_fn = nn.BCELoss(weight=torch.tensor([3, 7]))
model = train(n_epochs=n_epochs, model=model, train_loader=train_loader, optimizer=optimizer, loss_fn=loss_fn)
</code></pre>
<p>And the code for the trainer is</p>
<pre><code class="lang-auto">def train(model, train_loader, optimizer, loss_fn, n_epochs):
    training_losses = []

    for epoch in range(n_epochs):
        batch_losses = []
        for x_train, y_train in train_loader:
            if(str(type(loss_fn)).startswith("&lt;class 'torch.nn.modules.loss")):
                y_true = y_train[:, 0]
                y_true = y_true.view(y_true.shape[0], 1)
            else:
                y_true = y_train
            model.train()   
            y_pred = model(x_train).view(y_true.shape[0], 1)
            loss = loss_fn(y_pred, y_true)
            loss.backward()    
            optimizer.step()
            optimizer.zero_grad()
            batch_losses.append(loss.item())
        training_loss = np.mean(batch_losses)
        training_losses.append(training_loss)

        print(f"[{epoch+1}] {training_loss:.3f}")
    return(model)
</code></pre>
<p>I have tried to go through the code for the loss function on git but I do not get what I can do to solve this. Sometimes I use custom losses and sometimes the pytorch loss, that is why the first few lines in the trainer look how they are. Also, I have made sure that y_train[:, 0] corresponds to the <code>target</code>. Thanks for you help!</p>

username: chetan_patil
cooked: <p>Hi, if you clearly look at the documentation, the <code>weight</code> must be same size of the <code>target</code> or <code>predictions</code>.<br>
In your case, the weight Tensor must be of size <code>[128,1]</code> and not <code>[128,2]</code>.<br>
You can use the weights the following way.</p>
<pre><code class="lang-auto">weight = torch.zeros_like(target)
weight[target==0] = 3
weight[target==1] = 7
</code></pre>
Neural Style Transfer Steps

username: jesselin
cooked: <p>Hello! This is my first post here, so forgive me if this has already been discussed or is in a tutorial, but I am building the tutorial neural style transfer network (<a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html?highlight=style%20transfer" rel="nofollow noopener">here</a>), and I am trying to simply see each 50 steps of the run, which is 300 steps by default in the tutorial. I am just looking to display the resulting image from each 50 steps once the run has been completed.</p>
<p>Again, forgive me if this is well documented, I am very new to coding, pytorch, and neural networks, and any advice or help would be much appreciated!</p>

username: tom
cooked: <p>You can take the image display code (<code>plt.</code> calls before the output image) and move it to after <code>optimizer.step()</code> (also clamp, but don’t overwrite the <code>input_img</code> (so use detach().clamp()).</p>
<p>Best regards</p>
<p>Thomas</p>