akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Alternative way to iterate in batches on a dataset

username: antgr
cooked: <p>We iterate on batches with the following code:</p>
<pre><code class="lang-auto">for batch in iterator:
    pass
</code></pre>
<p>where the iterator has been instantiated with the following code</p>
<pre><code class="lang-auto">train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    batch_size = BATCH_SIZE,
    device = device)
</code></pre>
<p>What I want to do is something like the following:</p>
<pre><code class="lang-auto">batch = next(iterator)
</code></pre>
<p>But this obviously does not work. The reason that I want such a think is that I want to iterate in two datasets (multitask learning) and so I want to have two iterators (one for each dataset), and be able to alternate between them.</p>
<p>Can you help me on that?</p>

username: antgr
cooked: <p>OK, I did the following:</p>
<pre><code class="lang-auto">batches = iter(iterator)    # &lt;---before the loop
</code></pre>
<p>and then inside the loop,  I can iterate in batches:</p>
<pre><code class="lang-auto">batch = next(batches)
</code></pre>
How to save neuron values after activation and gradient with respect to activations using hook?

username: gebrahimi
cooked: <p>I need to save all the neuron values in each layer after the activation and gradient with respect to. I am very new to this and i know that i need to use hooks. Can someone please hint me how to do this for following network? Thanks.</p>
<p>class LeNet_5(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()

    self.conv1 = nn.Conv2d(1, 6, 5, padding=2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc3 = nn.Linear(16 * 5 * 5, 120)
    self.fc4 = nn.Linear(120, 84)
    self.fc5 = nn.Linear(84, 10)

def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.max_pool2d(x, 2)
    x = F.relu(self.conv2(x))
    x = F.max_pool2d(x, 2)

    x = F.relu(self.fc3(x.view(-1, 16 * 5 * 5)))
    x = F.relu(self.fc4(x))
    x = F.log_softmax(self.fc5(x))

    return x</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>During the forward pass. You have access to <code>x</code> directly if you need the value after each activation.<br>
If you want the gradient at this point, you can get it by doing <code>x.register_hook(your_hook_fn)</code>. Where <code>your_hook_fn</code> will be called with one argument: the gradient of x.</p>
Why the input of AvgPool3d must be variable but MaxPool3d can support tensor?

username: kmyfoer
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7_2_690x36.png" alt="image" data-base62-sha1="rZTqlArNiqWANkJ6fKoucJodoN1" width="690" height="36" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7_2_690x36.png, https://discuss.pytorch.org/uploads/default/original/3X/c/4/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/4/c439cd60a9fe4f8ab4da8225e1b7f8e2153826b7.png 2x" data-dominant-color="0C0B0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">758×40 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Which PyTorch version are you using?<br>
<code>Variables</code> are deprecated since <code>0.4.0</code>, so this error message shouldn’t be raised anymore.</p>
Problem while loading model

username: dato_nefaridze
cooked: <p>i trained model on  google-colab, then saved it with pickle module, and then i load it on my jupyter notebook.it is not working.look at that picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ecf7491bb9bab5165334674738a5fd6b58628b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ecf7491bb9bab5165334674738a5fd6b58628b2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0ecf7491bb9bab5165334674738a5fd6b58628b2_2_690x387.png" alt="image" data-base62-sha1="2719EcQmTeKuF1iJ8VUXgbq9bxg" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0ecf7491bb9bab5165334674738a5fd6b58628b2_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0ecf7491bb9bab5165334674738a5fd6b58628b2_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ecf7491bb9bab5165334674738a5fd6b58628b2.png 2x" data-dominant-color="F4F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
here i am printing net parameters and it is obvious that output is 2, if input is 10^4, but it outputs different dimentional vector.please help.</p>

username: ptrblck
cooked: <p>You are printing the method <code>model.parameters</code>.<br>
If you want to see all parameters, use <code>print(list(model.parameters()))</code>.</p>
Why grouped convolution is slower than basic convolution?

username: kiradiso
cooked: <p>I have follow module added to all blocks of res_layer4, when groups = 1(in self.tim), time cost in forward process is around 1e-3s, when groups = channel(channel wise convolution), time cost changed to 3e-2s.<br>
So,why it is much more slower than basic convolution with only 1/channel weights size?</p>
<pre><code class="lang-auto">class TEI(nn.Module):
    def __init__(self, channel, att_type='mem', reduction=8, bn=False):
        super(TEI, self).__init__()
        assert att_type in ['mem', 'se', 'none']
        self.attention = None
        if att_type is 'mem':
            self.attention = MEM3D(channel, reduction)
        elif att_type is 'se':
            self.attention = SELayer3D(channel, reduction)
        # 3 x 1 x 1 3D CW Conv
        if bn is True:
            self.tim = nn.Sequential(nn.Conv3d(channel, channel, kernel_size=(3, 1, 1), padding=(1, 0, 0), groups=1), nn.BatchNorm3d(channel))
        else:
            self.tim = nn.Conv3d(channel, channel, kernel_size=(3, 1, 1), padding=(1, 0, 0), groups=1)

    def forward(self, x):
        if self.attention is not None:
            x = self.attention(x)
        x = self.tim(x)
        return x
</code></pre>

username: ptrblck
cooked: <p>I assume you’re synchronized the code to get a proper profiling result, if you are running on the GPU.<br>
Anyway, the most likely reason is, that some convolution setups are highly optimized while others not so much.</p>

username: kiradiso
cooked: <p>Thanks a lot. maybe groups = channel will decrease the computation intensity of model. When i set groups = 8, it is faster than both.</p>

username: Eta_C
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Since <strong>groups == channels</strong> is faster than group convolution and basic convolution, I want to know something else.</p>
<p>If I set <strong>out_channels == groups == channels</strong>, it becomes a depthwise convolution. Does depthwise convolution is also highly optimized, or just because of the reduction of computation?</p>

username: ptrblck
cooked: <p>I’ve profiled the cudnn depthwise convs for FP16 in <a href="https://github.com/pytorch/pytorch/pull/22302">this PR</a> and used a heuristic to chose based on the workload.<br>
Unfortunately, the answer is again “it depends” in this case. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Score of the TransformerEncoder layer?

username: gslaller
cooked: <p>Is there a way the get the score of the this Layer?  Like the F.softmax((Q@K.t())/torch.sqrt(dim),-1) before it is multiplied with V.</p>

username: vainaijr
cooked: <p>if we do,</p>
<pre><code class="lang-auto">import torch.nn as nn, torch
x = nn.TransformerEncoderLayer(10, 2)
y = nn.TransformerEncoder(x, 1)
src = torch.randn(1, 1, 10)
x.self_attn(src, src, src)
</code></pre>
<p>then we get,</p>
<pre><code class="lang-auto">(tensor([[[-0.1861,  0.1664,  0.0857, -0.2807, -0.2680, -0.1627,  0.0585,
            0.1379, -0.0257, -0.0476]]], grad_fn=&lt;AddBackward0&gt;),
 tensor([[[1.1111]]], grad_fn=&lt;DivBackward0&gt;))
</code></pre>
<p>the second output is average attention weights over heads</p>
Set 2D tensor elements based on 2D index tensor

username: spacemeerkat
cooked: <p>I’m trying to rewrite the answer to <a href="https://stackoverflow.com/questions/58836274/set-2d-array-elements-based-on-2d-index-array/" rel="noopener nofollow ugc">this stack thread</a> in native PyTorch language (as I don’t want to be flicking back and forth between numpy and PyTorch objects.</p>
<p>However I’m struggling to make it work in tensor form, so I was wondering if anyone could give me a hand in seeing where I’m going wrong?</p>
<hr>
<p>The new form of the question in the thread goes as follows: Say you have the arrays as described in the thread as tensors…</p>
<pre><code class="lang-auto">z = torch.tensor([[[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]],

       [[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]],

       [[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]]])


i = torch.tensor([[2, 1, 2, 1],
       [1, 1, 2, 1],
       [1, 1, 1, 1],
       [1, 0, 0, 1]])


v = torch.tensor([[5., 5., 0., 4.],
       [4., 6., 8., 3.],
       [4., 0., 4., 8.],
       [7., 6., 5., 7.]])
</code></pre>
<p>And you wish to perform the answer operation:</p>
<pre><code class="lang-auto">z = np.array([*map(i.__eq__,np.unique(i))])*np.array([v]*z.shape[0])
</code></pre>
<p>…but in a PyTorch way, how does one alter this?</p>
<p>I have tried:</p>
<pre><code class="lang-auto">z = torch.tensor([*map(it.__eq__,torch.unique(it))])*torch.stack([vt]*zt.shape[0])
</code></pre>
<p>…but I still get an error relating to the use of a list inside <code>torch.tensor()</code> procedure I believe. You can see I’ve altered the second multiplication term to use <code>torch.stack()</code> instead of a list multiplication but I can’t see how to fix the list error.</p>
<p>Error:</p>
<blockquote>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>
<p>Any help on this matter would be greatly appreciated, thanks in advance!</p>

username: spacemeerkat
cooked: <p>Solved with:</p>
<pre><code class="lang-auto">z = torch.stack([*map(i.__eq__,torch.unique(i))]).type(torch.float)*(torch.stack([v]*z.shape[0])).type(torch.float)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I think you just want to use scatter for this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">i.unsqueeze_(0)
v.unsqueeze_(0)
z.scatter_(0, i, v)
</code></pre>
How to know which features are selected from an image while classifying using transfer learning?

username: sampa
cooked: <p>I am doing medical image classification using transfer learning. How to visualize which features or which part of the images are selected from training data to classify the image? Can I use CAM or Grad-CAM for this purpose? If Yes how?</p>

username: albanD
cooked: <p>Hi,</p>
<p>There are multiple methods that can be used.<br>
I think the <a href="https://github.com/pytorch/captum">captum</a> package is what you’re looking for !</p>
Get the gradient tape

username: Julie489
cooked: <p>Hi,</p>
<p>I would like to be able to retrieve the gradient tape of a given gradient computation.</p>
<p>For instance, let’s say I define the gradient of my outputs with respect to a given weights using torch.autograd.grad, is there any way to have access of its tape?</p>
<p>Thank you,<br>
Regards</p>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t really build gradient tapes per se. But graphs. You can access some limited properties of it from python. <a href="https://github.com/szagoruyko/pytorchviz">This</a> package can be used for example to plot the graph nicely.</p>
<p>Why do you want to get access to it?</p>

username: Julie489
cooked: <p>Hi alban,</p>
<p>Thank you for your answer.</p>
<p>Ok I see, I am trying to get the gradients of the output of a neural network with respect to the weights. Usually backpropagation provides the gradient of the cost function which requires to compute the gradient of the outputs. I am wondering how to get this gradient, do you have any idea ?</p>
<p>Thank you,<br>
Regards</p>

username: albanD
cooked: <p>You you have multiple outputs, it is called a Jacobian, Unfortunately, this is going to be quite expensive to compute (you can see <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">this gist</a> on how to do it).</p>

username: Julie489
cooked: <p>Thanks for the answer.</p>
<p>Let’s say I have a model N, would it not be enough to do:</p>
<p>“Jacobian = torch.autograd.grad(N.output(x), N.output.weight, grad_outputs=[v1, v2, v3])”   ?</p>
<p>Where this should give me the Jacobian-vector product with the vectors v1, v2 and v3.</p>
<p>Also, the aim of backpropagation is to get this Jacobian. Do the same process happen behind the scenes when I perform the previous line of code and when I just fit the model and the backpropagation is executed ?</p>
<p>Put simply, in terms of performance, I am wondering what is the gap between the computation of this Jacobian inside the training of tthe model and when I execute the previous line.</p>

username: albanD
cooked: <p>Unfortunately, we do not support doing the vector Jacobian product with multiple vectors at the same time. You have to use a for loop and multiple calls to backward (as is done in the gist I linked above).</p>
<blockquote>
<p>Also, the aim of backpropagation is to get this Jacobian.</p>
</blockquote>
<p>This is only true when your function has scalar output. If it has multiple outputs, then it only computes vector Jacobian product. This is a limitation to any deep learning framework and Automatic Differentiation in general.</p>
<p>The performance of using this during training is going to be quite high. It will be very dependent on your network, so you will want to try and see how slow it is.</p>
Difference between (nn.Linear + nn.CrossEntropyLoss) and (nn.LogSoftmax + nn.NLLLoss)

username: eugene
cooked: <p>In a multi-class classification, I sometimes see the following two implementations:</p>
<ul>
<li>
<code>nn.Linear</code> + <code>nn.CrossEntropyLoss</code>
</li>
<li>
<code>nn.LogSoftmax</code> + <code>nn.NLLLoss</code>
</li>
</ul>
<p>Are they both the same in terms of the following?</p>
<ul>
<li>Both are softmax classifiers</li>
<li>Mathematically</li>
<li>Model training efficiency</li>
<li>Any other differences?</li>
</ul>
<p>What are the trade-offs to consider?</p>
<p>If the intention is to do binary classification, what’s the most efficiency way to output a probability?</p>

username: ptrblck
cooked: <p>Both approaches are the same.<br>
In fact <code>nn.CrossEntropyLoss</code> just uses <code>nn.LogSoftmax()</code> + <code>nn.NLLLoss()</code> internally.<br>
<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L1550">Here</a> is the line of code.</p>
<p>For a binary classification you could use <code>nn.CrossEntropyLoss()</code> with a logit output of shape <code>[batch_size, 2]</code> or <code>nn.BCELoss()</code> with a <code>nn.Sigmoid()</code> in the last layer.</p>

username: John1231983
cooked: <p>BCEWithLogitsLoss = One Sigmoid Layer + BCELoss (solved numerically unstable problem)</p>

username: eugene
cooked: <p>Thanks for the quick response! With this, I guess I’ll need to think through it a bit harder to convince myself of it.</p>

username: NOP
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="21634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>For a binary classification you could use <code>nn.CrossEntropyLoss()</code> with a logit output of shape <code>[batch_size, 2]</code> or <code>nn.BCELoss()</code> with a <code>nn.Sigmoid()</code> in the last layer.</p>
</blockquote>
</aside>
<p>When you said binary classification you mean here just two categories, (like spam or not spam) Right?</p>

username: ptrblck
cooked: <p>Yes, by binary classification I meant a use case with two target classes (positive vs. negative).</p>

username: Neta_Zmora
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/42324cb6e83e99182daecc46cc006b690cf68847/torch/nn/functional.py#L2028" rel="nofollow noopener">Here’s</a> the permalink to the line of code <a class="mention" href="/u/ptrblck">@ptrblck</a> was pointing to in his answer (which pointed to a line in <code>master</code> - a moving target <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> )</p>

username: ptrblck
cooked: <p>Oops, thanks for the permalink <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
What causes Zombie processes?

username: wondervictor
cooked: <p>After several training steps, the script begins to evaluate the performance of the model, but it gets <strong>stuck</strong> in the evaluation step. When checking the process, some zombie processes appear.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e0b21f7917229e68c97a102aebd533fe329a70e8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e0b21f7917229e68c97a102aebd533fe329a70e8" title="51%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e0b21f7917229e68c97a102aebd533fe329a70e8_2_690x282.jpeg" alt="51%20PM" data-base62-sha1="w3KzXBCBvfMseZR3xZrPi74sexO" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e0b21f7917229e68c97a102aebd533fe329a70e8_2_690x282.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e0b21f7917229e68c97a102aebd533fe329a70e8_2_1035x423.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/e0b21f7917229e68c97a102aebd533fe329a70e8.jpeg 2x" data-dominant-color="3E3F40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">51%20PM</span><span class="informations">1166×478 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve set the <code>num_workers=0</code> and the same problem occurs.</p>

username: rpathak
cooked: <p>In linux terms, zombie process is when a process dies and its leftovers are not cleaned up by the parent-process/system. They are cleaned up eventually after sometime.</p>

username: wondervictor
cooked: <p>In other words, zombie process should be cleaned up by the system and scheduling programs. Therefore, the training process getting stuck is due to the failure of my system.</p>

username: wondervictor
cooked: <p>Thanks,  I need to check my operating system.</p>

username: rpathak
cooked: <p>It depends on how the processes which are evaluating are handled in terms of their life cycle. Its normal to have zombie process in a normal running system. Important to note is that zombie process means that process is no more active. You can check if the process which has turned into a zombie process died gracefully or because of some error. It will help to find the root cause.</p>
<p>But zombie process does not means system failure</p>

username: chunchun
cooked: <aside class="quote no-group" data-username="rpathak" data-post="5" data-topic="30320">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e9bcb4/48.png" class="avatar"> rpathak:</div>
<blockquote>
<p>You can check if the process which has turned into a zombie process died gracefully or because of some error. It will help to find the root cause.</p>
</blockquote>
</aside>
<p>Could you please tell me how to achieve this?</p>
Comparing BN output (FW and BW) on CPU and on GPU

username: Andrea_Zanetti
cooked: <p>Hey, I am writing my implementation of BN, and when I compare my implementation with the Pytorch implementation on CPU I obtain identical results with the Pytorch BN implementation, but when I set device to GPU, the FW passes are still aligned while the BW passes are not. What could it be the reason? I am also using the gradient check and I pass it, so I am not clear why I see different results when moving from CPU to GPU. Any ideas? Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>How large is the difference between the two implementations?</p>

username: Andrea_Zanetti
cooked: <p>Apologies, my bad. I did not realize that .clone() on the input tensor I used to feed the 2 lines (PT BN and mine) had effects on the backprop graph. Creating 2 indipendent inputs, I see that my BW has issues and the supposed alignment is no longer there. Thanks for the reply in any case <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Different Embeddings depending on batch size

username: Ian_L
cooked: <p>I was testing the cosine similarity of ResNet (not trained) and other models when I noticed that I would always get 1.0 for any pair of pictures.</p>
<p>I have some example code which demonstrates this:</p>
<pre><code class="lang-auto">img0 = Image.open("train/006_000.png")
img1 = Image.open("train/006_001.png")
tf_temp =transforms.Compose([transforms.Resize((256,256)),
                             transforms.ToTensor(),
                             transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225]),
                            ])
img0 = tf_temp(img0).unsqueeze(0)
img1 = tf_temp(img1).unsqueeze(0)

batch = torch.cat([img0,img1],0)
embedding_temp = net.embedding(batch)
embedding_temp_1 = net.embedding(img0)
embedding_temp_2 = net.embedding(img1)
print(F.cosine_similarity(embedding_temp[0], embedding_temp[1], dim=0).item())
print(F.cosine_similarity(embedding_temp_1, embedding_temp_2, dim=1).item())
</code></pre>
<p>Note: net.embedding is the resnet.</p>
<p>I expected the results to be the same since I only changed how many tensors I processed at a time. The first forward pass batches both tensors, but the second and third treat img0 and img1 separately. If I use the separated variant, I always get near 1.0 values. Why?</p>
<p>Edit: When I set net.embedding.eval(), both numbers are the same. However, I still get values near 1.0 all time time. I am still unsure why this is still the case.</p>
<p>Thanks.</p>

username: Oli
cooked: <p>An untrained network can output very high similarity between different images. I believe that its because one of the activations dominates the other in magnitude - leading to a very high similarity. Take a pretrained network and input two different images and it should not be close to 1 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Too small font size in documentation

username: Andrea_Garritano
cooked: <p>Hi folks!<br>
I try to learn <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">autograd from 60 minutes tutorial</a>, but i have some problem with font size of mathematical formulas.<br>
The size is so small that is impossible to read the content.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6a9b8f2ae1d905079dd2c6845b4381ed231bf34.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6a9b8f2ae1d905079dd2c6845b4381ed231bf34" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6a9b8f2ae1d905079dd2c6845b4381ed231bf34.png" alt="image" data-base62-sha1="uCZO3tdbkTgoLWGxkOYm9kJXeWU" width="690" height="76" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">809×90 7.32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I hope that it’ll be fixed!<br>
Thanks!<br>
Andrea Garritano</p>

username: Eta_C
cooked: <p>So am I.<br>
I push Ctrl and roll my the mouse wheel to zoom in to the webpage to 300%, so that I can read the content.<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: Andrea_Garritano
cooked: <p>Thanks for the reply!<br>
This is a just a workaround that I’m already using, but i think that this should fixed to improve readability.</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/andrea_garritano">@Andrea_Garritano</a> <a class="mention" href="/u/eta_c">@Eta_C</a> which browser and version are you using?<br>
It’s normally scaled on Firefox 70.0.1.</p>

username: Andrea_Garritano
cooked: <p>Google Chrome - Version 78.0.3904.108 (64 bit)</p>
How to convert gpu trained model on cpu model

username: dato_nefaridze
cooked: <p>i trained model on google_colab, then i saved it with pickle(binary file), then i downloaded it and trying to open it, but can’t, i tried many things and nothing worked, here is example:</p>
<pre><code class="lang-auto">torch.load('better_model.pt', map_location=lambda storage, loc: storage)

model=torch.load('better_model.pt', map_location={'cuda:0': 'cpu'})
</code></pre>
<p>i don’t know meaning of this code, but it doesn’t work.please help</p>

username: Juliusz_Ziomek
cooked: <p>Can you tell us exactly what problem you have? Do you get an error message? If so, what it says? Please specify exactly what kind of bug you encounter, otherwise it will be hard to help you.</p>

username: dato_nefaridze
cooked: <p>error asys:</p>
<pre><code class="lang-auto">Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
</code></pre>
<p>do you have any idea, how to load code, different from this code?</p>

username: Eta_C
cooked: <p>try</p>
<pre><code class="lang-python">torch.load('better_model.pt', map_location=torch.device("cpu"))
</code></pre>

username: dato_nefaridze
cooked: <p>i tried it :D, thanks, i will train it on cpu</p>
How to convert image dataset folder to csv format for face recognition

username: hunar
cooked: <p>how to convert image folders to csv format<br>
the paths<br>
image/john/1.jpg , 2.jpg , 3.jpg … 10.jpg<br>
image/katy/1.jpg , 2.jpg , 3.jpg … 10.jpg<br>
image/jim/1.jpg , 2.jpg , 3.jpg … 10.jpg<br>
image/andrew/1.jpg , 2.jpg , 3.jpg … 10.jpg<br>
and so on<br>
i want to make image dataset for recognizing faces , i have 400 images , each 10 images identified one person (one label)</p>
<p>thanks<br>
regards …</p>

username: alex.veuthey
cooked: <p>Use <code>os.walk()</code> to traverse your main folder, then write for each image 1 line containing the image path (you can choose absolute or relative, depending on your approach and preferences) followed by the target label.</p>
<p>Don’t forget the CSV header, so you can use Pandas for example to load the file directly in a practical database.</p>
PyTorch Intermediate Tutorials for Deep Learning Researchers in C++

username: prabhuomkar
cooked: <p>Hi PyTorch Folks!</p>
<p>My project <a href="https://github.com/prabhuomkar/pytorch-cpp" rel="nofollow noopener">C++ Implementation of PyTorch Tutorials for Deep Learning Researchers</a> is updated with Intermediate Tutorials containing:</p>
<ul>
<li>Convolutional Neural Network</li>
<li>Deep Residual Network</li>
<li>Recurrent Neural Network</li>
<li>Bidirectional Recurrent Neural Network</li>
<li>Language Model (RNN-LM)</li>
</ul>
<p>Thanks to <a href="https://github.com/mfl28" rel="nofollow noopener">Markus Fleischhacker</a> for his daily contributions which made this happen. Show some love by starring/forking/contributing to more tutorials/reporting issues.</p>
<p>Cheers!</p>
Loss value becomes NAN suddenly

username: James_Wang
cooked: <p>Hello, I was training some networks. However, I met weird problems attached below. “learning rate” means the learning rate in the current iteration. args.lr represents the initial learning rate of this program. Can you guys give me some suggestions about this problem? Thanks in advance.</p>
<p>The version of PyTorch is 1.0.0, CUDA 8.0, Ubuntu 16.04, python 3.5.</p>
<p>Iters: 351700/[07], Loss 3.5787 (3.7740), 'Prec@1 43.810 (46.888), time: 1.00 s/iter, learning rate: 4.879625646604741e-05, [args.lr]: 0.0001<br>
Iters: 351800/[07], Loss 3.4608 (3.7741), 'Prec@1 44.762 (46.922), time: 1.00 s/iter, learning rate: 4.872546293719906e-05, [args.lr]: 0.0001<br>
Iters: 351900/[07], Loss 3.7157 (3.7688), 'Prec@1 51.429 (46.960), time: 1.00 s/iter, learning rate: 4.8654671964972e-05, [args.lr]: 0.0001<br>
Iters: 352000/[07], Loss 3.1261 (3.7694), 'Prec@1 52.381 (46.949), time: 1.00 s/iter, learning rate: 4.8583883691367374e-05, [args.lr]: 0.0001<br>
Iters: 352100/[07], Loss 3.6948 (3.7712), 'Prec@1 51.429 (46.961), time: 1.00 s/iter, learning rate: 4.85130982583809e-05, [args.lr]: 0.0001<br>
Iters: 352200/[07], Loss 4.0376 (3.7710), 'Prec@1 41.905 (46.937), time: 1.00 s/iter, learning rate: 4.84423158080026e-05, [args.lr]: 0.0001<br>
Iters: 352300/[07], Loss 3.6179 (3.7683), 'Prec@1 46.667 (46.949), time: 1.00 s/iter, learning rate: 4.8371536482216515e-05, [args.lr]: 0.0001<br>
Iters: 352400/[07], Loss 4.9270 (3.7662), 'Prec@1 37.143 (46.954), time: 1.00 s/iter, learning rate: 4.830076042300044e-05, [args.lr]: 0.0001<br>
Iters: 352500/[07], Loss nan (nan), 'Prec@1 0.000 (45.216), time: 0.99 s/iter, learning rate: 4.8229987772325544e-05, [args.lr]: 0.0001<br>
Iters: 352600/[07], Loss nan (nan), 'Prec@1 0.000 (43.477), time: 0.99 s/iter, learning rate: 4.8159218672156253e-05, [args.lr]: 0.0001<br>
Iters: 352700/[07], Loss nan (nan), 'Prec@1 0.000 (41.866), time: 0.99 s/iter, learning rate: 4.8088453264449795e-05, [args.lr]: 0.0001<br>
Iters: 352800/[07], Loss nan (nan), 'Prec@1 0.000 (40.371), time: 0.99 s/iter, learning rate: 4.801769169115605e-05, [args.lr]: 0.0001<br>
Iters: 352900/[07], Loss nan (nan), 'Prec@1 0.000 (38.979), time: 1.00 s/iter, learning rate: 4.7946934094217175e-05, [args.lr]: 0.0001<br>
Iters: 353000/[07], Loss nan (nan), 'Prec@1 0.000 (37.680), time: 0.99 s/iter, learning rate: 4.787618061556734e-05, [args.lr]: 0.0001</p>

username: Joker
cooked: <p>Mate,  You’d  better   post   your  network  type  here  , and   what   task  you   are     training. (Classfication  or   object  detection)</p>
<p>Rely :<br>
I   often  encouter   this  problem  in<code> object detection</code>,  when  I   use    <code>torch.log(a)</code>,if  a  is   negative  number . It  will  be  nan  ,  because   your  loss    function  will  get  a    nan  value  as   parameter .<br>
So  you   should   debug   your  code   step   by   step  , and  see   which step   the  <code>tensor</code>   suddenly  becomes   <code>nan</code></p>
GPU memory leak using facebook maskrcnn implementation

username: gui11aume
cooked: <p>Hello,<br>
I’m working with the facebook implementation of faster rcnn <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark</a>.<br>
In this implementation, the classification/regression are defined in <a href="https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/modeling/roi_heads/box_head/roi_box_predictors.py" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/modeling/roi_heads/box_head/roi_box_predictors.py</a><br>
I would like to do multi labels classification (assume that we have N classes), so I chose to parallelize N binary classifiers (N times the cited predictor) like this.</p>
<pre><code class="lang-auto">class Roi_head(nn.Module):
    def __init__(self, cfg, args):
        super(Roi_head, self).__init__()
        self.classes_number = N
        for i in range(1, self.classes_number +1):
            setattr(self, "roi_heads%d" % i, build_roi_heads(cfg))

    def forward(self, features, proposals, targets=None):
        labels = targets[0].get_field("labels")
        for i in range(1, self.classes_number +1):
            if torch.nonzero(labels == i).squeeze(1).shape[0] != 0:
                device = labels.device
                label = torch.ones(labels.shape, dtype=torch.float32, device=device)
                targets[0].add_field("labels", label)
                x, result, detector_losse = getattr(self, "roi_heads%d" % i)(features, proposals, targets)
                try:
                    detector_losses += sum(loss for loss in detector_losse.values())
                except:
                    detector_losses = sum(loss for loss in detector_losse.values())
</code></pre>
<p>But when I try to train this, I sometimes have an uncomprehending GPU memory increase (about 100mo) during backward (not for all iterations) leading to a memory overflow…<br>
Do you have any ideas that could help me to understand what causes this leak during backward?</p>
<p>Thanks</p>

username: tengerye
cooked: <p>I think I might encounter the similar problem. May I ask have you solved it?</p>
How to swap data between cpu and gpu

username: Dale_Song
cooked: <p>hi,<br>
How to swap part of a model data including the parameters tensor and intermediates results tensor to host memory(cpu) while free the GPU memory they occupied in pytorch?</p>

username: mmisiur
cooked: <p>Why is .cpu() method not sufficient?</p>

username: Dale_Song
cooked: <p>From the docs, .cpu() returns a copy of gpu data, does it free the occupied GPU memory?</p>

username: Eta_C
cooked: <p>If you want to release the GPU memory, use <code>torch.cuda.empty_cache()</code>.<br>
I found a topic about how to release <strong>a tensor</strong> <a href="https://discuss.pytorch.org/t/freeing-gpu-memory/17713">here</a>, but I’m not sure it will work…</p>

username: Dale_Song
cooked: <aside class="quote no-group" data-username="Eta_C" data-post="4" data-topic="63442">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p>empty_cache</p>
</blockquote>
</aside>
<p>thanks, seems to be a solution.</p>

username: ptrblck
cooked: <p>Note that this approach will free the GPU memory indeed, so that other applications can use it.<br>
However, if you want to use the memory again in PyTorch, it would have to be reallocated, which might slow down your code. To avoid it, PyTorch uses a custom caching mechanism, so that the cached GPU memory can easily be reused without any allocations.</p>
FasterRCNN Fine-tuning on Custom Dataset

username: abpaudel
cooked: <p>I’m trying to fine-tune Faster RCNN model for object detection on <a href="http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d" rel="nofollow noopener">KITTI dataset</a> by following the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">PyTorch Mask RCNN tutorial</a>. But when I try to train the model, I get ‘Loss is nan’ error immediately, and the training stops. Where shall I go from here?</p>

username: MLaurenceFournier
cooked: <p>I guess there are probably many possible causes for that, but I had the same problem when I inadvertently fed this network images of range outside 0-1.</p>
Custom Distributions as Matrices

username: getzinmw
cooked: <p>Hello,</p>
<p>I am relatively new to PyTorch and distributions, so forgive any naïveté in the question.</p>
<p>I am attempting to employ PyTorch distributions in a Bayesian inferential framework. At this point I am successfully using Poisson and Gaussian distributions. However, one crucial part of my model/guide is a sort of mapping from <code>a-&gt;a'</code>. By itself these two quantities can be arranged in a square 2D matrix with <code>a</code> in rows and <code>a'</code> in columns, as an example.  Importantly, this matrix can be given as <code>P(a'|a)</code> in its forward form or <code>P(a|a')</code> in the backwards form (these two are easily computed from each other using Bayes Law).</p>
<p>I am having troubles implementing this <code>a-&gt;a'</code> conversion in a way which can facilitate the back-propagation of loss error. In actuality, both <code>a</code> and <code>a'</code> are continuous variables, however, discretization of these two quantities with a integer granularity is reasonable for the purposes of the model.</p>
<p>I have come across the idea of TransformedDistributions, but I am uncertain as to their utility. I have also considered using some combination of Categorical distributions with little success.</p>
<p>Does anyone have any thoughts on how this could be achieved?</p>
<p>Thanks for any hints/tips/tricks!</p>
Partitioning multiple networks

username: lpolari
cooked: <p>I’m interested in using/extending glow to run multiple networks concurrently. From what I found out so far, the compilation process for some network works as follows:</p>
<ol>
<li>A caffe2 or onnx network is loaded into one glow::function inside a glow:module</li>
<li>This module can be added to the HostManager where the compilation, optimizations and partitioning happen</li>
</ol>
<p>Therefor, when multiple networks are added, the DAGs of the single networks are created independantly from each other, right? If that is correct, do the overall physically available resources need to be partitioned in advance for the networks?</p>
<p>What I want to achieve is to do the partitioning, taking all the loaded networks into account. Is that something that should even be supported by glow sooner or later, or is that not desired?<br>
Maybe the most simple solution, concerning the current situation would be to just load multiple networks into one module, but that seems a bit hacky to me?</p>

username: gcatron
cooked: <p>Hey <a class="mention" href="/u/lpolari">@lpolari</a>, you are correct, when multiple networks are added the DAGs for each network are independent from each other. Physical resources are taken into account while adding a network so advanced partitioning may not be necessary.<br>
When loading partitions onto a device the Glow Runtime will try to evenly distribute used space on the devices.<br>
Example:<br>
3 Devices with 10GB of memory<br>
Network A Partitioned: A1[6GB], A2[5GB]<br>
Network B Partitioned B1[4GB], B2[8GB]<br>
Assuming you load A then B.<br>
The device loading will looks like:<br>
Dev1: A1, B1<br>
Dev2: A2<br>
Dev3: B2<br>
The limitation here is that Glow decides one network at a time so you might end up with a less than optimal assignment of resources.<br>
To get “optimal” will depend a bit on your use case. In general glow only partitions a network if it will not fit on a single device. So you usually won’t end up with many different networks loaded on a single device.<br>
If you are running many smaller networks then partitioning shouldn’t be an issue but you will have contention for devices at inference time.</p>
Gloo backend default device

username: xzhu1900
cooked: <p>Hi PyTorch experts,</p>
<p>I am trying to use torch.distributed package for my distributed training. The backend I am using is gloo.</p>
<p>Based on this doc: <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html</a>, gloo supports all_reduce on both CPU and GPU, but it seems there is no specific way to chose one over the other.</p>
<p>I am wondering, during training, does gloo perform all_reduce automatically based on the tensor’s device type? Like if the tensors are on GPU, then perform all_reduce on GPU; if the tensors are on CPU, perform it on CPU?</p>
<p>Also, when all_reduce is performed on GPU, does gloo fallback using nccl?</p>
<p>Thanks in advance!</p>

username: rvarm1
cooked: <blockquote>
<p>I am wondering, during training, does gloo perform all_reduce automatically based on the tensor’s device type?</p>
</blockquote>
<p>Yes, see <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupGloo.cpp#L720" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupGloo.cpp#L720</a>. Essentially, we check the input’s device type, and run the appropriate operation based on that.</p>
<blockquote>
<p>Also, when all_reduce is performed on GPU, does gloo fallback using nccl?</p>
</blockquote>
<p>This doesn’t happen, the GLOO backend can be built with CUDA and supports GPU operations (<a href="https://github.com/facebookincubator/gloo/blob/master/docs/cuda.md" rel="noopener nofollow ugc">https://github.com/facebookincubator/gloo/blob/master/docs/cuda.md</a>)</p>

username: xzhu1900
cooked: <p>Thanks <a class="mention" href="/u/rvarm1">@rvarm1</a>. This helps a lot!</p>
Trace ssd decode function

username: Kiki_Rizki_Arpiandi
cooked: <p>Hi everyone. I need to deploy ssd object detection model on android, but since torch ops does not available on android. I need to trace decode function (<a href="https://github.com/amdegroot/ssd.pytorch/blob/master/layers/functions/detection.py" rel="noopener nofollow ugc">https://github.com/amdegroot/ssd.pytorch/blob/master/layers/functions/detection.py</a>) to decode detection result. but I do not manage to trace it. Does any one succeed to trace the decode part ?</p>
<p>I tried to trace the function but I got the following error:</p>
<blockquote>
<p>Compiled from code export_decode_model.py(121): decode<br>
export_decode_model.py(129): forward<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py(525): _slow_forward<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py(539): <strong>call</strong><br>
/usr/local/lib/python3.6/dist-packages/torch/jit/<strong>init</strong>.py(997): trace_module<br>
/usr/local/lib/python3.6/dist-packages/torch/jit/<strong>init</strong>.py(858): trace<br>
export_decode_model.py(142): </p>
</blockquote>
<p>The above operation failed in interpreter, with the following stack trace:</p>

username: David_Reiss
cooked: <p>We don’t have a recipe for custom ops yet.  Stay turned!</p>
Is there a function that can replace torch.Tensor.max() at mobile android?

username: mawserver
cooked: <p>My model need run torch.Tensor.max() meth to get the final index of my word dictionaries.<br>
But now I don’t know what to do.</p>

username: mawserver
cooked: <p>Oh, I found I could write one myself:sweat_smile:<br>
But hopefully the mobile.Tensor will add some math util:laughing:</p>

username: David_Reiss
cooked: <p>We are not planning to implement the full tensor API in the Java bindings.  If you want some extra result calculated (like the max), your best bet is to compute it as part of your model.</p>
Why am I getting different results for batchnorm2d than expected, manual vs pytorch?

username: MLSlayer
cooked: <p>For some reason, even if I am following the formula in pytorch docs, it seems I am getting different results when trying to simulate batchnorm2d by hand vs pytorch.</p>
<p>Here is the manual version</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = t.tensor([[[[1., 2.],
...                 [3., 4.]],
...                [[3., 4.],
...                 [5., 6.]]],
...               [[[1., 2.],
...                 [30., 40.]],
...                [[15., 4.],
...                 [5., 6.]]
...                ]])
&gt;&gt;&gt; 
&gt;&gt;&gt; x.shape
torch.Size([2, 2, 2, 2])
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; x_mean = x.mean()
&gt;&gt;&gt; x_var = x.var()
&gt;&gt;&gt; eps = 1e-5
&gt;&gt;&gt; print(x_mean)
tensor(8.1875)
&gt;&gt;&gt; print(x_var)
tensor(123.3625)
&gt;&gt;&gt; print((x - x_mean) / (x_var + eps) ** .5)
tensor([[[[-0.6471, -0.5571],
          [-0.4671, -0.3770]],

         [[-0.4671, -0.3770],
          [-0.2870, -0.1970]]],


        [[[-0.6471, -0.5571],
          [ 1.9639,  2.8642]],

         [[ 0.6134, -0.3770],
          [-0.2870, -0.1970]]]])
</code></pre>
<p>Then here is the pytorch version</p>
<pre><code class="lang-auto">&gt;&gt;&gt; m = nn.BatchNorm2d(2, affine=False)
&gt;&gt;&gt; print(m(x))
tensor([[[[-0.6481, -0.5790],
          [-0.5099, -0.4407]],

         [[-0.8485, -0.5657],
          [-0.2828,  0.0000]]],


        [[[-0.6481, -0.5790],
          [ 1.3567,  2.0481]],

         [[ 2.5456, -0.5657],
          [-0.2828,  0.0000]]]])

</code></pre>
<p>I made sure to put affine=False to leave out the gamma and beta learnable params</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">this manual implementation</a>.</p>
<p>Skimming through your code it looks like the variance calculation is unbiased in your case.</p>

username: MLSlayer
cooked: <p>So yes, your implementation was helpful for finding the two bugs. First is the Bessel’s correction(i.e. diving by n-1 in the variance formula), due to unbiased=True.</p>
<p>Second is the dimensions I am getting the mean and variance from. It should have been</p>
<p><code>x.mean([0, 2, 3])</code></p>
<p>and</p>
<p><code>x.var([0, 2, 3], unbiased=False)</code></p>
Build Pytorch with Openblas and Cuda

username: Bastulli
cooked: <p>Hello, I have a AMD Threadripper CPU and want to try to avoid installing Pytorch with Intels MLK library.  I downloaded and installed OpenBLAS 0.3.7 on my Linux (Ubuntu) machine and installed Cuda drivers 10.1 for my 2080ti.  When I clone Pytorch from git and install it via “python setup.py install”, it failed to auto detect Openblas and my Cuda drivers and installs it with MLK.  Do I need to install it specifying anything?</p>

username: albanD
cooked: <p>From the <code>setup.py</code> file, you should be able to set <code>BLAS=OpenBLAS</code> to force it to use open blas.<br>
The cuda issue should be independent as the blas libraries above are cpu only. If your cuda is not detected, again from the <code>setup.py</code> file, you can set the <code>CUDA_HOME=/path/to/your/cuda</code>.</p>
Pytorch implementation of RAdam and Look Ahead

username: bing
cooked: <p>Is there a way to include RAdam and Look Ahead in pytorch models.<br>
I tried the below approach from <a href="https://discuss.pytorch.org/t/is-there-a-pytorch-implementation-of-radam-lookahead/54393/2">is-there-a-pytorch-implementation-of-radam-lookahead</a></p>
<pre><code class="lang-auto">from radam import RAdam
from optimizer import Lookahead
base_optim = RAdam(model.parameters(),lr = 0.001) 
optimizer =  Lookahead(base_optim, k=5, alpha=0.5)
</code></pre>
<p>but getting <code>ModuleNotFoundError: No module named 'optimizer'</code>.</p>

username: albanD
cooked: <p>Your error is because you import: <code>from optimizer import Lookahead</code> but <code>optimizer</code> is not an actual package I think. You need to get the file from the repo that you linked.</p>