akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CPU memory not cleared after "to(cudadevice)"

username: mxs30443
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/249e6828d4e1191c8d498e989193acd0af36bf51.png" data-download-href="https://discuss.pytorch.org/uploads/default/249e6828d4e1191c8d498e989193acd0af36bf51" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/249e6828d4e1191c8d498e989193acd0af36bf51.png" alt="image" data-base62-sha1="5dWzAPfHoAptRPSoHMJg1O7CAqR" width="690" height="249" data-dominant-color="11110E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">748×270 6.83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how to free the cpu memory</p>
<p>testcode:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
import gc


@profile
def test():
    net = models.alexnet(pretrained=True)
    gpu = torch.device('cuda:0')
    net.to(gpu)
    net.cpu()
    del net
    aaa = [1,2,3,4]
    torch.cuda.empty_cache()
    bbb = [1,123,4,151]
    gc.collect()
    aa = 1
    
if __name__ is "__main__":
    print("main")
    test()
</code></pre>
<p>runcmd : python -m memory_profiler profile_try.py</p>

username: albanD
cooked: <p>Hi,</p>
<p>The only line were you could see a change here is between the <code>net.to(gpu)</code> and <code>net.cpu()</code> right?<br>
The allocator on your system might not release the memory to the system right away to improve speed. And it looks like it does this as you can see that moving your model back to the cpu use less memory than when you created it at the beginning.</p>
Problems on register_backward_hook

username: phantom90
cooked: <p>Hi. We have such sample code:</p>
<pre><code class="lang-auto">class xxx:
    def __init__(self, model):
        self.model = model
        self.gradients = None
        self.model.eval()
        self.hook_layers()

    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients = grad_in[0]
        first_layer = list(self.model.children())[0] . # Get the 1st layer.
        first_layer.register_backward_hook(hook_function)

    def generate_gradients(self, input_image, target_class):
        # Forward pass
        model_output = self.model(input_image)
        model_output = model_output.reshape(-1)
        input_image.requires_grad = True
        self.model.zero_grad()
        one_hot_output = torch.FloatTensor(model_output.size()[-1]).zero_()
        one_hot_output[target_class] = 1
        # Backward pass
        model_output.backward(gradient=one_hot_output)

        gradients_as_arr = self.gradients.data.numpy()[0]
        return gradients_as_arr
</code></pre>
<p>input_image is just pil image preprocessed by torchvision.</p>
<p>When I run the code, the error says  self.gradients.data does not exist. Actually, the format of grad_in is like (None, , None). May I know what do these dimensions mean in grad_in? How to correctly save the gradient of the first layer.</p>
<p>Many thanks~</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>input_image</code> in your example should be a Tensor. Otherwise, setting <code>requires_grad</code> on it will have no effect.</p>
<p>Also you should not use <code>.data</code> anymore. You can use <code>.detach()</code> here to replace it.</p>
<p>Are you trying to get the gradient for the input of your network? If that case, since you set require_grad=True on <code>input_image</code> which is a leaf Tensor, after the call to <code>.backward</code>, you can simply access it via <code>input_image.grad</code>.</p>
<p>Finally as you can see in the doc for register_backward_hook, it won’t work with all nn.Modules. So you might want to use <code>register_hook</code> directly on a Tensor if you want the gradients for that Tensor.<br>
Or if you have access to that Tensor during the forward, simply call <code>.retain_grad()</code> on it for its <code>.grad</code> field to be populated during the backward.</p>
Record number of samples already traversed within training loop

username: MrRobot
cooked: <h1>Question</h1>
<p>I am new to deep learning, and I am not sure if I understand the key concepts correctly.</p>
<p>I have a huge dataset of more than 1.8 million samples. The training runtime could be hours. So I decide track number of samples that have already been touched by the training loop.</p>
<p>Specifically, I have following variables related to this purpose</p>
<pre><code class="lang-auto">print_freq = 10
batch_size = 32
max_epoch = 5
</code></pre>
<p>So if I write my training loop in the following way</p>
<pre><code class="lang-auto">for epoch in range(max_epoch):
  for i, (X_train, y_train) in enumerate(dataloader):
   # do something:
   if (i + 1) % print_freq == 0:
     num_samples_traversed = ...
</code></pre>
<p>Then is it correct that <code>num_samples_traversed = print_freq * batch_size</code>.</p>

username: beaupreda
cooked: <p>Hi,</p>
<p>There is a small mistake in the way you compute <code>num_samples_traversed</code>. Right now, it will always be equal to 320. You need to accumulate <code>print_freq * batch_size</code> like this:<br>
<code>num_samples_traversed += print_freq * batch_size</code><br>
or you could use vatriable <code>i</code> to compute it:<br>
<code>num_samples_traversed = (i+1) * batch_size</code></p>
Custom loss function for Mutual Information

username: Md_Mahfujur_Rahman_0
cooked: <p>I want to create a custom loss function which will calculate the mutual information between two training datasets.</p>
<p>For an example,</p>
<p>x= dataset_1<br>
y= dataset_2</p>
<p>MI = mutual_information(x,y)</p>
<p>How can I do that in pytorch? Thank you so much in advanced.</p>

username: SimonW
cooked: <p>Define mutual information on datasets. The information theory definition of mutual information is on two RVs.</p>

username: ru6928
cooked: <p>how did you solve this?</p>
Problem with scatter_

username: xiaohan
cooked: <p>Hi all,</p>
<p>I met a problem with <code>scatter_</code>.<br>
Here is the code(it was posted here: <a href="https://discuss.pytorch.org/t/differentiable-argmax/33020/6" class="inline-onebox">Differentiable argmax</a>):</p>
<pre><code class="lang-auto">class ArgMax(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        idx = torch.argmax(input, 1)
        output = torch.zeros_like(input)
        output.scatter_(1, idx, 1)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output

if __name__ == '__main__':
    a = torch.rand((3, 2, 4, 2), requires_grad=True)
    print(a)
    argmax = ArgMax()
    b = argmax.apply(a)
    b.backward()
</code></pre>
<p>But it seems <code>scatter_</code> doesn’t work here, and I got the error:</p>
<pre><code class="lang-auto">RuntimeError: invalid argument 3: Index tensor must either be empty or have same dimensions as output tensor at /Users/distiller/project/conda/conda-bld/pytorch_1570710797334/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:133
</code></pre>
<p>Is there anything wrong in the usage of <code>scatter_</code> here?</p>
<p>Thanks in advance!</p>

username: tom
cooked: <aside class="quote no-group" data-username="xiaohan" data-post="1" data-topic="61068">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/e79b87/48.png" class="avatar"> xiaohan:</div>
<blockquote>
<p>idx = torch.argmax(input, 1)</p>
</blockquote>
</aside>
<p>I think you want this to read</p>
<pre><code class="lang-auto">idx = torch.argmax(input, 1, keepdim=True)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: xiaohan
cooked: <p>Thanks so much, Thomas! This fixed the problem.</p>
Masking binary cross entropy loss

username: ArmanMazdaee
cooked: <p>Hi,<br>
I’m trying to implement a music transcription system. Each of the labels has a shape of [88 x num_frames]. Each element represents the activation of the corresponding piano key on the corresponding frame. So the problem can be considered a binary classification problem. For making a batch each label gets pads with zeros so all of them have some number of frames.<br>
Now my question is how can I ignore these padded values in the loss function? Also as the data is heavily unbalanced how can I use class weights for computing the loss?<br>
I prefer to use binary cross entropy as the loss function.</p>

username: KFrank
cooked: <p>Hello Arman!</p>
<aside class="quote no-group quote-modified" data-username="ArmanMazdaee" data-post="1" data-topic="61065" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/armanmazdaee/48/17721_2.png" class="avatar"> ArmanMazdaee:</div>
<blockquote>
<p>Hi,<br>
I’m trying to implement a music transcription system. Each of the labels has a shape of [88 x num_frames].<br>
…<br>
For making a batch each label gets pads with zeros so all of them have some number of frames.<br>
Now my question is how can I ignore these padded values in the loss function? Also as the data is heavily unbalanced how can I use class weights for computing the loss?<br>
I prefer to use binary cross entropy as the loss function.</p>
</blockquote>
</aside>
<p>The function version of <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.binary_cross_entropy" rel="noopener nofollow ugc">binary_cross_entropy</a> (as distinct from the<br>
class (function object) version, <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a>), supports a fine-grained,<br>
per-individual-element-of-each-sample <code>weight</code> argument.</p>
<p>So, using this, you could weight the loss contribution of each frame<br>
separately, and, in particular, give the padding frames a weight of zero.</p>
<p>You can also use the <code>weight</code> argument to reweight your unbalanced<br>
data, at the granularity that is the most logical for your use case.</p>
<p>I <em>imagine</em> that you wouldn’t want to reweight individual notes (but you<br>
could).  Maybe it would make sense to reweight the combination of<br>
notes that appear together in a given frame.  Anyway, for each batch,<br>
you would  go through the batch labels, sample by sample, and create<br>
the per-sample, per-frame, per-note weights, where, again, I <em>imagine</em><br>
that the per-note weights are all equal within a given frame.</p>
<p>(Also, you will presumably prefer to use the “logits” version,<br>
<a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.binary_cross_entropy_with_logits" rel="noopener nofollow ugc">binary_cross_entropy_with_logits</a>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>
Number sequence classification task

username: Chame_call
cooked: <p>Is that possible to use rnn network to classify sequence of increasing numbers like “Increasing” and sequence of decreasing like “Decreasing”?</p>
<p>Or if I have sequences of numbers representing height of mouth between lips of speaking people and non-speaking ones then can I classify whether sequence of numbers represents speaking person or non-speaking or not?</p>
<p>Or recurrent networks cannot handle this??</p>

username: Chame_call
cooked: <p>I saw solving of text semantic classification example but in that case we have limited set of numerical values representing words while I don’t the limited one.</p>
<p>Is that possible to use recurrent net to handle my problem??</p>

username: Chame_call
cooked: <p>I could attach my notebook if it helps…</p>

username: ptrblck
cooked: <p>In NLP tasks you are usually using a mapping between word indices to dense feature tensors using <code>nn.Embedding</code>. An RNN should be able to take your (floating point) features, however I’m not sure which architecture would work best for your use case.</p>

username: Chame_call
cooked: <p>thank you for your opinion</p>
Stride tuple meaning in conv2d()

username: netqyq
cooked: <p>What is the difference between: <code>stride=2</code> and <code>stride=(2,2)</code>?</p>

username: ArmanMazdaee
cooked: <p>I believe they are the same. <code>stride=2</code> is just a shortcut for <code>stride=(2, 2)</code>.</p>
Anaconda training model

username: fedaa86
cooked: <p>i wanted to ask how do we train deep leaning simple radio logical imaging in anaconda environment ?<br>
i already collected the pictures but i need the commands used for that and the image should be .jpg ???</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">This tutorial</a> might be a good starter. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Source of error in Batch Normalization

username: seongmin
cooked: <p>Hi,</p>
<p>I am testing my understanding of torch.nn.BatchNorm2d but there is a slight error in numbers that I have failed to resolve.</p>
<p>I thought what the Batch Normalization in training mode without any update shall do</p>
<pre><code class="lang-python">(x - E[x]) / sqrt(var(X) + 1e-5)
</code></pre>
<p>which in python code</p>
<pre><code class="lang-python">(x - x.mean([0,2,3], keepdim=True)) / (x.var([0,2,3], keepdim=True) + 1e-5).sqrt()
</code></pre>
<p>But when I actually compare it to torch.nn.BatchNorm2d with the following code,</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

bn = nn.BatchNorm2d(5)
x = torch.randn(500,5,2,2)

bn_x = (x - x.mean([0,2,3], keepdim=True)) / (x.var([0,2,3], keepdim=True) + 1e-5).sqrt()
print((bn(x) - bn_x).abs().max().item())
</code></pre>
<p>I constantly get a nonzero difference in range of ~1e-3.<br>
The error decreases as the B, H, W numbers get higher.</p>
<p>I tried Bessel’s correction with different numbers but it did not work.<br>
I tried removing 1e-5 (eps parameter in nn.BatchNorm2d) but it did not work either.<br>
It might be because of the floating point precision, but the error seems quite high.</p>
<p>Could you point me to the source of this error?</p>
<p>I would like an answer to the following questions:</p>
<ul>
<li>Is my understanding of nn.BatchNorm2d correct?</li>
<li>What should I change in my formula / code to remove the difference?</li>
</ul>

username: ptrblck
cooked: <p>I guess the difference domes from the <code>var</code> calculation, which should apparently be <code>unbiased</code>.<br>
I’ve written a manual implementation some time ago <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py" rel="nofollow noopener">here</a> so feel free to compare your code to this one.</p>

username: seongmin
cooked: <p>Thank you so much for your answer. It was more than perfect.</p>
[bug] in reset method in linear.cpp

username: 7starsea
cooked: <p>If we call reset method of nn::Linear module, we will get an error arising from nn::Module::register_parameter since we have already registered the parameter in the constructor.</p>

username: yf225
cooked: <p>We recently merged <a href="https://github.com/pytorch/pytorch/pull/29832" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/29832</a> which adds <code>reset_parameters()</code> method to nn::Linear module, and it will be available in the nightly build tomorrow or in the upcoming 1.4 release.</p>
"Quick start with Cloud Partners": Google Cloud Platform is not working

username: tlaurent
cooked: <p>The Pytorch main webpage, under  “quick start with cloud partners”,  points toward the following page:<br>
<a href="https://cloud.google.com/ai-platform/deep-learning-vm/docs/pytorch_start_instance" class="onebox" target="_blank" rel="nofollow noopener">https://cloud.google.com/ai-platform/deep-learning-vm/docs/pytorch_start_instance</a></p>
<p>It used to work very well, but since last week or so, the link<br>
“Visit the AI Platform Deep Learning VM Image GCP Marketplace”:<br>
<a href="https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning" class="onebox" target="_blank" rel="nofollow noopener">https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning</a><br>
returns the error message: “You do not have sufficient permissions to view this page”</p>
<p>I am teaching a Deep Learning class with Pytorch and it used to very convenient for students to quickly set up a VM with pytorch preinstalled using this – it would be nice to contact GCP and ask them to fix it (I tried but I am still waiting for an answer)</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Thanks for reporting this. cc <a class="mention" href="/u/smth">@smth</a></p>
<p>In the meantime, you should be able to try colab  <a href="https://colab.research.google.com/notebooks/welcome.ipynb" rel="nofollow noopener">https://colab.research.google.com/notebooks/welcome.ipynb</a></p>

username: tlaurent
cooked: <p>Great, thanks for quick response!</p>

username: smth
cooked: <p>i flagged it to the GCP team, they said they are looking into it now.</p>

username: b0noi
cooked: <p>Thank you for reporting the bug we are working on the fix.</p>

username: b0noi
cooked: <p>Problem now should be fully resolved.</p>
Zero_grad and learning rate

username: elbek
cooked: <p>I am missing something here. if we zero out current gradient value with <code>optimizer.zero_grad()</code> then how does update happen to weights? For an update to happen we should have values in weight so we can do something like <code>weight-=lr*grad</code>?</p>
<p>Thanks.</p>

username: midhunharikumar
cooked: <p>You call zero grad at the start of the mini-batch. If you do not your gradients accumulate. When you zero_grad only the gradients are zero’d out not the weights. Once you do a <code>forward()</code> and <code>loss.backward()</code> the gradients will be propagated. You can accumulate the gradients by not calling <code>optimizer.zero_grad()</code></p>

username: elbek
cooked: <p>hmm, I see it is done before every <code>backward</code> call:</p>
<pre><code class="lang-auto">for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<p>here <code>data</code> is one batch then?</p>

username: midhunharikumar
cooked: <p>Yes. As you can see zero_grad() is called before beginning a loop with the data.</p>