akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Function 'SqrtBackward' returned nan values in its 0th output

zfzhang: <p>Hi,<br>
The error in title is triggered by the following lines while training:</p>
<pre><code class="lang-auto">k = (y**2).sum(dim=2,keepdim=True)
r = k.sqrt()
</code></pre>
<p>So it appears we can have a nan derivative for sqrt(0), how can we circumvent the problem in my case?</p>
<p>Thanks!</p>

zfzhang: <p>OK, I’ll avoid using sqrt() and formulate the problem in terms of r^2 instead.</p>

zl535320706: <p>can use<br>
torch.sqrt(x + <strong>1e-8</strong>)<br>
replace<br>
torch.sqrt(x)<br>
to solve this problem</p>

Fang_Wu: <p>I met the same problem. However, I don’t recommend your choice to change the sqrt into the square, since it might make the number calculated in your model bigger and bigger. I suppose it might cause a problem if you have multiple this kind of layers.</p>

zfzhang: <p>Thanks for the suggestion!</p>

xinlei_pei: <p>Thanks, that’s it! <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"> <img src="https://discuss.pytorch.org/images/emoji/apple/ghost.png?v=9" title=":ghost:" class="emoji" alt=":ghost:"></p>
What is the monte carlo gradient estimation technique used for Dirichlet (and Gamma, Beta)?

DonnyHe: <p>Dunno if autograd is the right category for this question. There’s a related <a href="https://discuss.pytorch.org/t/pytorch-reparametrization-method-for-gamma-dirichlet-von-mises-distribution/109472">thread</a> here but I didn’t see an official answer.</p>
<p>My current guess is that torch use this paper for it <a href="https://arxiv.org/abs/1806.01851" class="inline-onebox" rel="noopener nofollow ugc">[1806.01851] Pathwise Derivatives Beyond the Reparameterization Trick</a>, which is different from what tensorflow-probability uses, which is the implicit reparam gradient from DeepMind. I looked through the code base but couldn’t find a concrete reference unlike the tensorflow-probability code that explicitly mentions they use the implicit reparam gradient.</p>
<p>The reason that I am asking is because I am trying to do stochastic gradient based training of LDA, the tensorflow-probability version trains nicely, but my pytorch implementation’s gradient tend to blow up, and need special handling. 1 example of such training instability can be seen below, where there’s a significant kink near the end of training for the orange curve, and this happens consistently. This time it recovered but it can blow up at other times.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72ed86bd9b88732e1f4babf81fa1071adf215f40.png" data-download-href="https://discuss.pytorch.org/uploads/default/72ed86bd9b88732e1f4babf81fa1071adf215f40" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72ed86bd9b88732e1f4babf81fa1071adf215f40_2_690x397.png" alt="image" data-base62-sha1="goHkqoBp7sv2cGA7jUgqkMamsbS" width="690" height="397" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72ed86bd9b88732e1f4babf81fa1071adf215f40_2_690x397.png, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72ed86bd9b88732e1f4babf81fa1071adf215f40.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72ed86bd9b88732e1f4babf81fa1071adf215f40.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×414 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The tfp code and the pytorch code I wrote are identical down to the initialization scheme, and the tfp code’s optimization curve follows almost exactly like the orange curve, but without the kinks.</p>
<p>Mentioning <a class="mention" href="/u/fritzo">@fritzo</a> <a class="mention" href="/u/neerajprad">@neerajprad</a>  you guys because based on my search on github you guys contribute to the pyro/torch.distributions a lot.</p>
<p>If we are not using the implicit reparametrization, I will probably request one on github.</p>
Fast Track Course For C++

Ankit_Dixit: <p>Hello Everyone, I am new to this community. Can anyone know fast track course for C++. I am trying to look at the course in which the duration is less.</p>

Ankit_Dixit: <p>I have found a fast track <a href="https://www.interviewbit.com/courses/fast-track-cpp/" rel="noopener nofollow ugc">C++ course</a> on <a href="https://www.interviewbit.com/" rel="noopener nofollow ugc">interviewbit</a>.</p>
AttributeError: 'list' object has no attribute 'shape'

samm: <p>i’m trying to extract features and got this error from this line</p>
<blockquote>
<p>feats = model(samples).copy()</p>
<p>features = torch.zeros(len(data_loader.dataset), feats.shape[-1])</p>
</blockquote>
<p>i read that i should convert it to numpy but couldn’t write it well</p>
<p>the first error was</p>
<blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘clone’</p>
</blockquote>
<p>so i changed it to copy but got current error in the title of the post</p>

ptrblck: <p>Based on the error messages it seems that <code>feats</code> is a <code>list</code>, while you are trying to use it as a tensor.<br>
You could thus make sure that the <code>forward</code> method of your model returns a tensor or convert the list to a tensor using e.g. <code>torch.stack</code>.</p>

samm: <p>thanks but you mean that i got</p>
<pre><code>AttributeError: ‘list’ object has no attribute ‘clone’
</code></pre>
<p>as <code>feats</code> is a <code>list</code> and i’m trying to use it as a tensor ?</p>

ptrblck: <p>Yes, Python <code>list</code>s don’t have the <code>clone()</code> or <code>shape</code> operations, which are PyTorch tensor ops, so it seems you are assuming <code>feats</code> is a tensor, while it’s a <code>list</code>.</p>

samm: <p>excuse me do you mean this function</p>
<blockquote>
<p>def forward(self, x):<br>
x = self.forward_features(x)</p>
<pre><code>    if self.F4:
        x = x[3:4]

    return x
</code></pre>
</blockquote>

ptrblck: <p>Yes, the <code>forward</code> method seems to return a <code>list</code>, so you would have to check why this is the case (check the <code>type</code> of <code>x</code> and try to isolate where it’s transformed to a <code>list</code>) and/or create a tensor, if needed.</p>
transform.Normalize isn't idempotent?

chanind: <p>I’m confused by what <code>transform.Normalize</code> does. If it just normalizes a tensor to a mean and stdev, then shouldn’t it be idempotent? I find if I run the same normalization multiple times it gives different values. Also it seems that it doesn’t actually set the mean to what it says it does. An example below:</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
vec = torch.empty((3,10,10)).random_(5)

print(torch.mean(vec, axis=(1,2)))
print(torch.mean(normalize(vec), axis=(1,2)))
print(torch.mean(normalize(normalize(vec)), axis=(1,2)))
</code></pre>
<p>Running this prints:</p>
<pre><code class="lang-auto">tensor([2.0000, 1.9800, 2.1500])
tensor([6.6157, 6.8036, 7.7511])
tensor([26.7717, 28.3374, 32.6449])
</code></pre>
<p>Shouldn’t the means be <code>0.485, 0.456, 0.406</code>? Why does running <code>normalize(normalize(vec))</code> give different values than just <code>normalize(vec)</code>? Apologies if I’m missing something obvious!</p>

KFrank: <p>Hi Chanind!</p>
<aside class="quote no-group" data-username="chanind" data-post="1" data-topic="124108" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chanind/48/39331_2.png" class="avatar"> chanind:</div>
<blockquote>
<p>If it just normalizes a tensor to a mean and stdev, then shouldn’t it be idempotent?</p>
</blockquote>
</aside>
<p>No, this is not how <code>Normalize</code> works.  It does not modify your tensor<br>
to have the specified <code>mean</code> and <code>std</code>.  Rather, according to the<br>
documentation for <a href="https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.Normalize" rel="noopener nofollow ugc">torchvision.transforms.Normalize</a>, it subtracts the<br>
specified <code>mean</code> from your tensor and then divides by the specified<br>
<code>std</code> (on a per-channel basis).</p>
<p>Best.</p>
<p>K. Frank</p>
If I only have the model of PyTorch and don't know the dimension of the input, how to convert it to onnx?

Wendy-liu17: <p>I have a series of PyTorch trained models, such as “model.pth”, but I don’t know the input dimensions of the model.<br>
For instance, in the following function: torch.onnx.export(model, args, f, export_params=True, verbose=False, training=False, input_names=None, output_names=None).<br>
I don’t know the “args” of the function. How do I define it by just having the model file such as “model.pth”?</p>
Are transformers design specific to nlp

Mika_S: <p>Hi folks<br>
Is anything in transformers implementation specific to natural language modeling? Especially, wrt nn embedding.</p>

eqy: <p>Not really (see also: Vision Transformers <a href="https://arxiv.org/abs/2010.11929" class="inline-onebox" rel="noopener nofollow ugc">[2010.11929] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</a> ).</p>

Mika_S: <p>Thanks. I didn’t phrase my question correctly. I mean whether the pytorch implementation is specific to language modeling using transformers. Or nothing is limiting to use it for tabular data.</p>
Unexpected results with multi head-attention for NMT task

Aiman_Mutasem-bellh: <p>Hello all : )</p>
<p>I have applied multi-head attention (TRANSFORMER) R2L and L2R for NMT task. I have a few questions about the final results.</p>
<p>When I applied multi-pass decoding the precision is decreased by two points and recall increased two points. Why this happened? and Why precision is decreased?   Is that any way to manipulating this issue?<br>
Also, I have tried to continue training both models over new dataset and I have changed the learning rate from 1e-3 to 1e-1 but the performance is don’t improve which is unexpected and confused results.</p>
<p>Could you explain these two points?</p>
Learnable parameter as int not tensor

SungmanHong: <p>Hi, I just made learnable parameter of torch.<br>
the tensor name is <code>self.z</code><br>
Here is my neural network’s definition.</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, n_lstm=2048, n_reduced=128, n_hidden=32):
        super(LSTM, self).__init__()
        self.z = nn.Parameter(torch.tensor(1.0), requires_grad=True)
</code></pre>
<p>In my case, I want to learn the “best kernel size(1 to 12)” of max_pool1d as belows, via training of the learnable parameter, <strong>z</strong>.</p>
<pre><code class="lang-auto">def forward(self, input):
    ...
    output = F.max_pool1d(torch.cat((min_tensor, input), 2), z, stride=1)
</code></pre>
<p>z is in the range of 1 to 12.<br>
So I can train 12 times, changing z to 1 from 12 And choose the best z value looking for best validation loss.<br>
However, I want to learn and decide the z via backpropagation.<br>
Cause z should be int, not tensor, I cast-type z to int from tensor as follows.</p>
<pre><code class="lang-auto">    z = (int)(self.z.cpu().detach().numpy())
    F.max_pool1d(torch.cat((min_tensor, -QoS), 2), z, stride=1)
</code></pre>
<p>However, the z value is not change… It always return 1.0, it is static…<br>
I’m worrying that the typecast to int from tensor caused un-learnable state.<br>
but I want to find best z value via backpropagation.<br>
It is possible? What should be edit in my code?</p>
<p>Thanks.</p>

eqy: <p>The intent here looks like some kind of differentiable neural architecture search (e.g., <a href="https://research.google/pubs/pub47800/" rel="noopener nofollow ugc">DARTS: Differentiable Architecture Search – Google Research</a>)</p>
<p>However, the shape parameters to layer declarations like <code>max_pool1d</code> are not differentiable; the output of the model doesn’t directly depend on them! For example, it is similar to writing <code>y = a + b + c</code> and expecting to take a derivative to optimize how many variables we are using in our equation. One workaround (as shown in the DARTS paper) is to consider a model with several weighted branches (where the weights are differentiable and learnable), where each branch has a different <code>maxpool_1d</code> configuration. The weights of each branch are updated during the architecture search training process and the branch(es) with the highest weight(s) are chosen.</p>
DCGAN Convergence Issues

avalon1511: <p>Hi, I am implementing this DCGAN tutorial: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a> on two different datasets but my generator loss is not converging.</p>
<p>I am training the model for 75 epochs and I am calculating the average loss for each epoch by :</p>
<p><strong>average loss per epoch = sum of losses for epoch / total number of iterations in that epoch</strong></p>
<p>My final graph looks like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b1f07539e472a239424796c53a92a5a70a2a613.png" alt="D_and_G_loss_training" data-base62-sha1="m8gu9Q2wU0cjOsCOWB1ccUhGFfd" width="640" height="480"></p>
<p>How can I fix the generator loss behaviour? I tried changing the learning rate to different values but it didnt work. Thanks for the help!</p>

avalon1511: <p>If anyone is facing the same issue, I was able to solve it using the MultiStepLR method.<br>
In the training section i call the MultiStepLR at epoch 25 and 45 right after the optimizer.step() functions</p>
<p>the scheduler can be defined as</p>
<p><code>scheduler_G = MultiStepLR(optimizerG, milestones = [25, 45], gamma = gamma) scheduler_D = MultiStepLR(optimizerD, milestones = [25,45], gamma = gamma)</code></p>
<p>and it is called as</p>
<pre><code class="lang-auto">scheduler_D.step()
scheduler_G.step()
</code></pre>
Minimum Specs for PyTorch - Windows 10

Veep_Peep: <p>Hello,</p>
<p>Are there any minimum specs for using PyTorch when trying to execute code?<br>
I have been trying to play with GPT-Neo as a novice.   My computer does not have a GPU and 16 GB ram<br>
My disk grinds like crazy and it takes about 30 minutes to get output.  per change.</p>
<p>Or does this have nothing to do with PyTorch, but what I am trying?</p>
<p>Suggestions?</p>
<p>Thanks,</p>
<p>V</p>
ONNX export failed on an operator with unrecognized namespace torchvision::nms. If you are trying to export a custom operator, make sure you registered it with the right domain and version

DL_aspirant: <p>I am getting this error while exporting the model to onnx. I have tried to fiddle around with <code>operator_export_type</code> in ONNX, but its not solving the issue. I have also tried different versions such as <code>opset_version = 10</code>, <code>opset_version = 11</code>, but I am getting the same error.</p>
<p>Could someone help me with this? Anything I could try?</p>
Per Sample Gradient Calculation keeps on increasing

Pradeep_Bajracharya: <p>I was trying to calculate per sample gradient in a naive way i.e. got through one data at a time. Each data is passed through the model and the loss is calculated and backpropagated. Then I looped through each layer and added the gradient as layer.weight.grad.flatten(). The problem I am facing is that for each data the gradient is greater than the previous data. I tried to shuffle the data in a small subset and check if the gradients match but it’s the same case there too i.e. gradient keeps on increasing as I go through the data one at a time. Could anyone provide some insight to why that could be?</p>

eqy: <p>If you are using a standard optimizer are you calling <code>optimizer.zero_grad()</code> to zero out the gradients before each step? Example: <a href="https://github.com/pytorch/examples/blob/cbb760d5e50a03df667cdc32a61f75ac28e11cbf/imagenet/main.py#L300" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at cbb760d5e50a03df667cdc32a61f75ac28e11cbf · pytorch/examples · GitHub</a></p>
<p>Otherwise they will continue to accumulate with each update.</p>

Pradeep_Bajracharya: <p>I wasn’t using optimizer but I cleared the grad for each layer’s weights which solved the problem. I have a doubt if optimization based on the gradient would work or not because I cleared the gradient for the weights.</p>
How to do dot product along a given dimension

prateekagarwal3: <p>There are 2 tensors: q with dimension(64, 100, 500) and key with dimension(64, 500). I want to do dot product of key and q along the dimension of 500. Keras has a function dot() where we can give specific axes values. How can I achieve this in pytorch?</p>

dhecloud: <p>i think this is what you are looking for: <a href="https://pytorch.org/docs/stable/torch.html#torch.bmm" rel="nofollow noopener">bmm</a></p>

prateekagarwal3: <p>thanks! but I tried bmm and what It says that the 2 tensors should be of the same exact dimension</p>

zfzhang: <p>x = Tensor(B,C,D)<br>
y = Tensor(B,D)</p>
<p>If you want simplicity, you can just element-wise multiply them, then sum along that dimension:<br>
<code>(x * y.reshape(B,1,D)).sum(dim=2)</code></p>
<p>If you want to save memory, you can either use einsum():<br>
<code>torch.einsum('bcd,bd-&gt;bc', x, y)</code></p>
<p>or use bmm():<br>
<code>torch.bmm(x, y.rehsape(B,D,1)).reshape(B,C)</code></p>
Apply transforms.ToPILImage with nn.Sequential

ham1d: <p>Is there a way to apply <code>transforms.ToPILImage</code> with <code>nn.Sequential</code> e.g,</p>
<pre><code class="lang-auto">tranform_lst=[transforms.ToPILImage(),
transforms.RandomHorizontalFlip(),
transforms.ToTensor()]
transforms_func = torch.nn.Sequential(*tranform_lst)
</code></pre>
<p>I wanna use a batch-wise transform in the training loop, and my inputs are torch tensors. hence there is a need to convert to PIL and back to tensor.</p>
<p>thanks!</p>

ham1d: <p>it looks like <code>ToPILImage</code> and <code>ToTensor</code> might not be necessary after all…</p>
<p>example from colourjitter, <code>F.adjust_hue(img, hue_factor)</code>:</p>
<pre><code class="lang-auto">    if not isinstance(img, torch.Tensor):
        return F_pil.adjust_hue(img, hue_factor)

    return F_t.adjust_hue(img, hue_factor)
</code></pre>
<p>transforms functions are capable of working with both PIL and tensor types, and they do so inside <code>transforms.functional</code>.</p>
<p>I think we can safely remove <code>transforms.ToPILImage()</code> and <code>transforms.ToTensor()</code> in case batch of images are already in tensor format, and we can directly apply <code>transforms.ToTensor()</code> on them.</p>
Getting Device of model.children()

J_Johnson: <p>I am using Model Parallel. <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Single-Machine Model Parallel Best Practices — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>In the model, I assign the layers(or <code>model.children()</code>) to separate cuda devices via <code>layer.to('cuda:0')</code>.</p>
<p>How can I later call the cuda device which the layer is assigned to? In <code>torch.cuda</code>, I can get the name of the GPU via <code>torch.cuda.get_device_name(layer)</code>.<br>
<a href="https://pytorch.org/docs/stable/cuda.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/cuda.html</a></p>
<p>However, that does not tell me which cuda device it is assigned to. I can even the gpu properties of the device that layer is assigned to. But I cannot get the <code>'cuda:0'</code>. Any suggestions?</p>
<p>TIA</p>

J_Johnson: <p>Found a hackish work around:</p>
<pre><code class="lang-auto">for layer in model.children():
    try:
        print(getattr(next(layer.parameters()), 'device')
    except:
        continue
</code></pre>
<p>Any better ideas are appreciated.</p>
Accuracy drops from 100% to random when removing BatchNorm

the-dharma-bum: <p>Hello !</p>
<p>I’m doing a pretty simple 3D binary classification task using this <a href="https://github.com/shijianjian/EfficientNet-PyTorch-3D" rel="noopener nofollow ugc">EfficientNet</a>. I’ve been experimenting quite a lot around BatchNorm and I’ve found something that seems weird to me: when replacing BatchNorm3d by InstanceNorm3d or LayerNorm,  all other things being equal, my accuracy drops from 100% to random (that is, around 50%), and the loss doesn’t  improve at all.<br>
I expected a performance decrease but not something as radical as perfect to random.<br>
What should I look for ?</p>
<p>Thanks a lot !</p>
CTCLoss in PyTorch for custom data

Sanbot-kk: <p>I have <code>model_output</code> of size [T, N, C] and <code>target</code> of size [N,T]. I won’t have targets of variable length. Is this how I declare the lengths?</p>
<pre><code class="lang-auto">op_len = torch.full((N,), T, dtype=torch.long) 
target_len = torch.randint(1,T,(N,), dtype=torch.long)
train_loss = ctc_loss(model_output, target, op_len, target_len)
</code></pre>
<p>I am asking as the train loss is infinity after first iteration. There is no problem with the model. This happens only with CTCLoss( ). Any idea why?</p>

tom: <p>Having the input size as long as the target size is not a good idea.</p>
<p>If your target has repetitions, it won’t be representable and so the loss is infinite. If your target doesn’t have repetitions, there only is one valid alignment, and you might as well use per-sequence item cross entropy instead of CTC.</p>
<p>I would recommend staring a bit at the <a href="https://distill.pub/2017/ctc/" rel="noopener nofollow ugc">CTC article on Distill.pub</a> to get ideas about your modeling with special attention on the blank token aka ϵ. After reading it, take a sequence (using batch size 1 if your want) that produces inf loss and try to manually find a good alignment for CTC loss. Then you’ll have fully understood what is wrong and have achieved CTC-Zen.</p>
<p>Best regards</p>
<p>Thomas</p>
How to initialize tensors such that memory is allocated?

tjoseph: <p>Hello everyone,<br>
I am currently implementing a replay buffer and I want to preallocate the tensors for this buffer. One reason is that I want my code to crash early in case that there is not enough memory for the complete replay buffer. Everything is running strictly on <strong>CPU</strong><br>
However, I am wondering how I can allocate this tensor so that it uses the maximum possible memory size? I tried <code>torch.zeros(*my_shape, dtype=torch.float)</code> and <code>torch.rand(*my_shape, dtype=torch.float)</code> and latter seems to use a lot more memory. I am wondering whether torch is doing some smart memory saving things for former?</p>
<p>Best regards</p>

tjoseph: <p>It seems like rand needs additional memory to generate the random numbers, but then uses similar memory to zeros.</p>
If condition in the forward pass

almon: <p>Hi<br>
I am trying to implement a CNN for a binary classification in the following way:<br>
I have my input which is passed through some CNN layers. After that I have a global pooling layer. The output of the pooling is fed into a fully-connected layer and I get the output. Fairly simple.<br>
My question is about the global pooling layer. What I would like to do is: if the input instance is positive, then I want global max pooling, otherwise I want global average pooling.</p>
<p>I have implemented it this way:<br>
In the forward pass, I have an if condition that decides on the max/avg pool.<br>
When I get a batch, I split it into positive and negative batch. I will use the sign of the batches in the if statement. So i will have two outputs (one for the positive and one for the negative batch). I concatenate these and then I continue normally.</p>
<p>Everything runs, no error. But I can clearly see that something is going wrong in the predictions. The validation increases a lot.<br>
If I ‘force’ both positive and negative batch into only one branch of the pooling layer, then everything works fine.<br>
There might be something weird going on with the concatenation and maybe the backward pass.</p>
<p>What am I missing?</p>
<p>Thanks a lot for the help!</p>
RuntimeError: Function 'PowBackward1' returned nan values in its 1st output

Hawk: <p>I defined a loss function for my network, and it’s computed as:</p>
<pre><code class="lang-auto">    loss = pred.mm(pred.T) - (label.view(-1,1) == label.view(1,-1)).to(dtype=torch.float32)
    loss = loss**2
    loss = torch.pow(loss, (2-loss)**2)
    loss = loss.sum() / (label.size(0)**2 +1.0)    
    return loss
</code></pre>
<p>During training, the following error report appeared:<br>
<code>RuntimeError: Function 'PowBackward1' returned nan values in its 1st output.</code><br>
and this error is traced to the line:<br>
<code>   loss = torch.pow(loss, (2-loss)**2)</code><br>
I’m not clear about how this error is related to this line of code. Can anyone help me out?</p>

albanD: <p>Hi,</p>
<p>What is the value of the loss when it happens?<br>
This is most likely that you reach a value of the loss for which <code>loss.pow((2-loss)**2)</code> is not differentiable.</p>

Hawk: <p>Hi, I rewrote the loss and now it’s computed as:</p>
<pre><code class="lang-auto">    loss = pred.mm(pred.T) - (label.view(-1,1) == label.view(1,-1)).to(dtype=torch.float32)
    loss = (loss**2).sum()
    loss = loss / (label.size(0)**2 +1.0)
    
    return loss
</code></pre>
<p>where <code>pred</code> is computed as:</p>
<pre><code class="lang-auto">pred = 1.0 / (1.0 + (-pred).exp())
</code></pre>
<p>however, another error occured:</p>
<pre><code class="lang-auto">RuntimeError: Function 'ExpBackward' returned nan values in its 0th output.
</code></pre>
<p>which is traced to the ‘.exp’ operation.<br>
The loss value is 0.980656 when this happened. Can you figure this out?</p>

Hawk: <p>The error disappeared after I reduced the initial learning rate by half. But I’m still not clear about the mechanism behind.</p>

albanD: <p>In that case, what most likely happens is that the value of <code>pred</code> goes to infinite and thus leads to nan in gradients.</p>

xianqian: <aside class="quote no-group" data-username="Hawk" data-post="1" data-topic="82303">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/22d042/48.png" class="avatar"> Hawk:</div>
<blockquote>
<p>loss = torch.pow(loss, (2-loss)**2)</p>
</blockquote>
</aside>
<p>Hello <a class="mention" href="/u/hawk">@Hawk</a> ,<br>
You mentioned that “error is traced to this line”. Can you share how you traced it back?</p>
Model.train() with pre-trained weights makes results all 0 while model.eval() is fine

Hengyi: <p>Hello, thanks for your attention to this matter.</p>
<p>I want to continue to train a model with its pre-trained weights. When I evaluate this pre-trained model with model.eval(), everything is fine and the model will generate some reasonable results, but when I want to further train this model and set the mode with model.train(), the problem will occur. During the forward loop, all generated results will be zero after switching to model.train() statement (batchsize=1).</p>
<p>Any ideas about why this happens?</p>
<p>Thanks for your answer!</p>
Trouble using Fold/Unfold for overlap-add

sevagh: <p>I have a tensor which represents overlapping chunks (of 2D audio coefficients):</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.rand((2, 113, 12, 244)) # 12 blocks of 244 = 2928
&gt;&gt;&gt; x = x.reshape(2, 113, 12*244)
&gt;&gt;&gt; print(x.shape)
torch.Size([2, 113, 2928])
</code></pre>
<p>Each of these 12 blocks of size 244 has a 50% overlap with the next block. I would like to overlap-add on the last dimension with blocks of size 244, hop size of 122. The total coefficients should be 2928/2 = 1464.</p>
<p>I have read different forum posts, stackoverflow questions, and the documentation of Fold/Unfold, and still have trouble discovering which parameters I need to get my desired output:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; desired_output_size = (2, 113, 1464)
&gt;&gt;&gt; y = torch.nn.functional.fold(x, kernel_size=(?), stride=(?))
&gt;&gt;&gt; print(y.shape)
torch.Size([2, 113, 1464])
</code></pre>
<p>Could anybody provide some guidance? Thanks.</p>
How to backpropagate a Transformer?

morphism: <p>Hello, everyone.</p>
<p>In every source file implementing attentions and transformers,<br>
I’ve found there is no backward function inside the class of Transformer</p>
<p>I wonder why there is no backward method in the class of Transformer ?</p>
<p>Also, how to backpropagate the Transformer?<br>
In pytorch, will pytorch engine backpropagate the encoder and the decode together up thru all layers?<br>
I wonder, as I mentioned, how this works without implementing backward in Transformer class.</p>
<p>Thank you in advance</p>

ptrblck: <p>Autograd will use the backward methods of each submodule used in the <a href="https://github.com/pytorch/pytorch/blob/095cd6a0da255a5ed24d146898c53790f0e9c3f0/torch/nn/modules/transformer.py#L137-L141"><code>nn.Transformer.forward</code> method</a> to calculate the gradients so the <code>nn.Transformer</code> module doesn’t necessarily need to implement a custom <code>backward</code> method.<br>
You could check these submodules and see, how the backward methods are defined (i.e. if they are using custom ones or just other PyTorch operations with already defined backwards).</p>

morphism: <p>Dear ptrblck:</p>
<p>Thank you for your help and your kind explanation.<br>
Have a nice week and see you again.<br>
Take care</p>
Keras to PyTorch conversion

Santhosh: <p>I found this Keras implementation on Kaggle. I need to convert it to PyTorch. Can someone please help me? Also, please give an explanation if possible. I need this for my RGB dataset (3,100,100)</p>
<p>def build_base_network(input_shape):</p>
<pre><code>seq = Sequential()

nb_filter = [16, 32, 16]
kernel_size = 3


#convolutional layer 1
seq.add(Convolution2D(nb_filter[0], kernel_size, kernel_size, input_shape=input_shape,border_mode='valid', dim_ordering='th'))
seq.add(Activation('relu'))
seq.add(MaxPooling2D(pool_size=(2, 2)))  
seq.add(Dropout(.25))

#convolutional layer 2
seq.add(Convolution2D(nb_filter[1], kernel_size, kernel_size, border_mode='valid', dim_ordering='th'))
seq.add(Activation('relu'))
seq.add(MaxPooling2D(pool_size=(2, 2), dim_ordering='th')) 
seq.add(Dropout(.25))

#convolutional layer 2
seq.add(Convolution2D(nb_filter[2], kernel_size, kernel_size, border_mode='valid', dim_ordering='th'))
seq.add(Activation('relu'))
seq.add(MaxPooling2D(pool_size=(2, 2), dim_ordering='th')) 
seq.add(Dropout(.25))

#flatten 
seq.add(Flatten())
seq.add(Dense(128, activation='relu'))
seq.add(Dropout(0.1))
seq.add(Dense(50, activation='relu'))
return seq</code></pre>

ptrblck: <p>I would recommend to take a look at <a href="https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_module.html">this tutorial</a> to get familiar with writing custom <code>nn.Modules</code> in PyTorch.<br>
Based on the posted Keras model you could define these layers in the <code>__init__</code> with minor changes (change e.g. <code>Convolution2D</code> to <code>nn.Conv2d</code>) and use these layers in the <code>forward</code> method.</p>
Max number of images in training

sj_xu: <p>Hi, I am having difficulties in using dataloader. I want to train the network on ImageNet with a fixed number of training images. i.e. I hope to set max_images_train during the training. But I don’t know how to achieve this in pytorch. Thanks!</p>

eqy: <p>Are you trying to limit the number of unique images that are used? You can probably do this by editing the <code>DatasetFolder</code> dataloader:<br>
<a href="https://pytorch.org/vision/stable/_modules/torchvision/datasets/folder.html#datasetfolder" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/_modules/torchvision/datasets/folder.html#datasetfolder</a><br>
or by simply creating a modified dataset on disk with some of the images removed.</p>

sj_xu: <p>Yes, that’s what I want to do. Thank you for your reply, I will try that.</p>
Trained pytorch model in C++ showing different results

sunny1: <p>I have trained my model and saved it to use in C++ environment. When i am trying to predict the results in C++ it is showing different results in comparison to my pytorch model. I have used the exact as shown in pytorch tutorial. This is my code for C++.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;memory&gt;
#include &lt;iostream&gt;
int main(int argc, const char* argv[])
{
 if (argc != 2) {
    std::cerr &lt;&lt; "usage: main &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }

  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    torch::jit::script::Module module;
    module = torch::jit::load("/home/sunny/neural/diffusion2.pt");
    // Create a vector of inputs.
    double x;
    std::cout&lt;&lt;"Please enter the value of x ";
    std::cin&gt;&gt;x;
    double t;
    std::cout&lt;&lt;"Please enter the value of t ";
    std::cin&gt;&gt;t;
    
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(torch::tensor({{x,t}}));
    // Execute the model and turn its output into a tensor.
    torch::Tensor output = module.forward((inputs)).toTensor();

    std::cout &lt;&lt; "u(x,t) = " &lt;&lt; output &lt;&lt; std::endl;
    
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
  }

}
</code></pre>
<p>One more thing. I tried many times to save my model to use in C++ and it show different results every time . Do anyone know what is going wrong?</p>

eqy: <p>Does your model have any nondeterministic layers like dropout that require <code>model.eval()</code> to fix their behavior at inference time?</p>
How to handle different input image sizes?

billconan: <p>Say I have an image model, how can I handle different image sizes?</p>
<p>Should I generate multiple onnx models for different sizes?</p>
<p>Do I do resizing or padding?</p>
<p>Thanks</p>

eqy: <p>The typical approach is to simply crop and resize to the trained model’s input size (so that the scale of the objects in terms of the number of pixels is close to what was seen at training time). You can generate different models for different size, but you still need to account for the scale issue; a model trained at 224x224 will start losing accuracy without finetuning around <code>400x400</code> resolution because of this scale mismatch problem.</p>
<p><a href="https://arxiv.org/abs/1906.06423" class="inline-onebox" rel="noopener nofollow ugc">[1906.06423] Fixing the train-test resolution discrepancy</a> is a good paper for more details on how to handle resizing</p>
How can i get features map for images?

samm: <p>As i got from the concept of captioning that i need the features for the images not to classify it. So if i have a classification model, the classification head can be dropped. does that code doing the concept ?  the code <a href="https://github.com/facebookresearch/dino/blob/main/vision_transformer.py" rel="noopener nofollow ugc">here</a></p>
<p>the modifications are</p>
<blockquote>
<p>def gradient(model, freeze: bool):<br>
transformer = VisionTransformer(num_classes=0)<br>
for parameter in transformer.parameters():<br>
parameter.requires_grad_(not freeze)</p>
</blockquote>
<blockquote>
<p>def vit_small(patch_size=16, **kwargs):<br>
model = VisionTransformer(         patch_size=patch_size, embed_dim=384, depth=12, num_heads=6, mlp_ratio=4,         qkv_bias=True, norm_layer=partial(nn.LayerNorm, eps=1e-6), **kwargs)         gradient(model, freeze=True)<br>
return model</p>
</blockquote>
<p>Does that right ?</p>
<p>Does after that i need re -train the model for my datasets after the modifications ?</p>
<p>What should result expected ?</p>

eqy: <p>Usually if you want to get feature maps from a model the typical approach is to edit the <code>forward</code> function in the model definition to return the intermediate feature maps in addition to the final output: e.g., <a href="https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model/119/6" class="inline-onebox">How to extract features of an image from a trained model - #6 by fmassa</a></p>
<p>Usually some kind of finetuning would be needed (at least), as classification is kind of a different domain from captioning.</p>

samm: <p>excuse me and thanks for replying … if i have a detection model and need to use it for captioning . Does i need freezing any layer of the model or the freezing is just for classification model ?</p>

eqy: <p>I don’t think there is a strict answer for this because it depends on what the entire model architecture is and not just the backbone. So you can try training with or without freezing though without freezing probably gives you better results.</p>
Tensor index operation

wxystudio: <p>I have 2 tensor a, b. a.shape = [2, n], b.shape = [n]. The element in b[i] is 0/1, which means we choose a[0][i] or a[1][i].<br>
I want to get tensor c, c.shape = [n], c[i] = a[0][i] or a[1][i], depending on b[i] = 0 or 1.</p>
<p>So how can I get c without for loop?</p>

eqy: <p>I think you can use <code>torch.where</code> here:</p>
<pre><code class="lang-auto">import time
import torch

n = 2048
a = torch.randn(2, n)
b = torch.randn(n) &gt; 0
t1 = time.time()
out1 = torch.empty(n)
for i in range(n):
    if b[i]:
        out1[i] = a[1][i]
    else:
        out1[i] = a[0][i]
t2 = time.time()
out2 = torch.where(b, a[1], a[0])
t3 = time.time()
print(torch.allclose(out1, out2))
print(t2-t1, t3-t2)
</code></pre>
<pre><code class="lang-auto">True
0.0062215328216552734 3.409385681152344e-05
</code></pre>
Gradient of output of network with respect to parameters

Rahul_Vashisht: <p>Hi,<br>
I need to calculate the gradient of output of network with respect to the parameters of network ( say with respect to first layer weights). Any suggestion how I can do this. I am doing 3-class classification so output of network is a tensor of length 3.</p>

eqy: <p>Usually this is done with some <code>criterion</code> like mean-squared error or cross-entropy loss and an <code>optimizer</code> like stochastic gradient descent.</p>
<p>The MNIST example is a good starting point for a typical training loop: <a href="https://github.com/pytorch/examples/blob/2639cf050493df9d3cbf065d45e6025733add0f4/mnist/main.py#L41" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at 2639cf050493df9d3cbf065d45e6025733add0f4 · pytorch/examples · GitHub</a></p>

Rahul_Vashisht: <p>Yeah, I am working on the case where I need to calculate gradient of output ( not some loss function) wrt parameters. Is there any way I can do this.</p>

eqy: <p>What happens if you simply define a loss function that is the identity with respect to the output? (assuming that the output is a scalar)</p>

Rahul_Vashisht: <p>Hi,<br>
So my output is not scalar. Is that case is there any function which I can use.</p>

eqy: <p>In case you want to compute something like a Jacobian, you can take a look at <a href="https://pytorch.org/docs/0.3.1/autograd.html#torch.autograd.grad" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch master documentation</a></p>

Rahul_Vashisht: <p>Hi,<br>
torch.autograd.functional.jacobian works.<br>
Thanks for the reply.</p>
Knowledge Graph Embedding model ranking results

tetedefer12: <p>I am running a model that computes the KGE embeddings of triples for a link prediction task. I am faced with a code, that I understand line by line, but I don’t understand how it serves the purpose of ranking predictions.</p>
<p>Usually, in link prediction, the model is tested on a triple such as (Obama, spouse, ?), and the model has to predict the tail at (?). What happens is that the model calculates the likelihood score of each entity in the dataset, then ranks all entities based on that score. Afterwards, the rank of the correct answer (in this case, Michelle Obama), is saved, and should be the output of the code I am struggling with.</p>
<p>What I don’t understand is the following: Why are they comparing scores in this code, and why are they summing them up??</p>
<p>In the below code, <code>scores</code> is a 2D array where each entry is a list with the likelihood score of each entity in the dataset (size: n_testing_triples x n_dataset_entities). <code>targets</code> is the score of the correct tail for each testing triple (size: n_testing_triples)</p>
<p>This seems to sum up the number of entries that have a score &gt;= targets. Why is that useful for the ranking?</p>
<pre><code class="lang-auto">ranks[0: batch_size] += torch.sum(
                  (scores &gt;= targets).float(), dim=1
.cpu()
</code></pre>
<p>Full code : <a href="https://github.com/HazyResearch/KGEmb/blob/master/models/base.py" class="inline-onebox" rel="noopener nofollow ugc">KGEmb/base.py at master · HazyResearch/KGEmb · GitHub</a></p>
How to allocate large intermediate variables in advance?

Haozhi: <p>I have a large intermediate variable and I allocated the memory before doing optimization. But RuntimeError happens in the second epoch. When I moved the variable allocation into the iteration, the error disappeared, but it will allocate a large memory every epoch (I thought this process will take a lot of time). Could anyone explain why the RuntimeError happens and how can I avoid it? Can I save time by allocating large intermediate variable in advance?</p>
<pre><code class="lang-auto">var1 = torch.rand(128, 128, requires_grad=True)
m1 = var1.new_empty((4, *var1.shape)) # intermediate variable
a = torch.linspace(1, 10, 4).reshape(4, 1, 1)

for i in range(10):
    #m1 = var1.new_empty((4, *var1.shape)) # No error if this line is uncommented
    m1[:] = torch.exp(var1[None, :] ** 2)
    l = loss(m1, data)
    l.backward()
    with torch.no_grad():
        var1 -= var1.grad * 0.1
        var1.grad.zero_()

#RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results #have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the #first time.
</code></pre>
Comparing two DL models across various datasets

anilkunchalaece: <p>I’m testing two DL models M1, M2 across datasets D1,D2,D3 and D4.</p>
<p>When using D1 and D4, M2 is showing a better performance but for both D2 and D3 there is no performance improvement.</p>
<ul>
<li>Is this normal ?</li>
<li>Or Do I have to change my architecture So M2 should always produce better performance compared to M1 for all datasets ?</li>
</ul>
How to do multi-task training with Soft parameter sharing mechanism?

YH-WEI: <p>Thanks for the help.<br>
I want to build a series of LSTM or self-attention-based autoencoder models. They are used to process time-series data output by multiple sensors. Each time-series data obtained by a sensor corresponds to an LSTM or self-attention channel, and channels are softly related to each other like the figure.<br>
what should I do to link the channels?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274.png" data-download-href="https://discuss.pytorch.org/uploads/default/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274_2_690x271.png" alt="image" data-base62-sha1="2ZnPU3KmsfgGfsi8mu5hyutAuoY" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274_2_690x271.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274_2_1035x406.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14f4b5a3ed3aa8a8827c144b8d16c6dd5ad77274.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1180×464 66.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How can i enuser of freezing layers?

prog_asker: <p>i have a classifier model and tried to freeze some layers and need to ensure that this layers are already freeze … is there any way to know that ?</p>

ptrblck: <p>You can check the <code>.requires_grad</code> attribute of all parameters, which should be frozen, and make sure it’s returning <code>False</code> and could additionally check that their <code>.grad</code> attribute is <code>None</code> after a <code> backward</code> operation.</p>
Concatenate Tensors with different dimension

4bach: <p>Hi everyone,</p>
<pre><code class="lang-auto">torch.rand(1,4,2,2)
tensor([[[[0.2244, 0.5357],
          [0.6145, 0.6231]],

         [[0.0133, 0.0747],
          [0.2936, 0.9928]],

         [[0.8611, 0.5119],
          [0.7976, 0.1552]],

         [[0.7479, 0.0291],
          [0.3519, 0.8065]]]])
</code></pre>
<p>I want to add an artificial class to my prediction with zero as prob, so i want to add a zero tensor of dim(2,2).</p>
<p>I want a result like this :</p>
<pre><code class="lang-auto">tensor([[[[0.2244, 0.5357],
          [0.6145, 0.6231]],

         [[0.0133, 0.0747],
          [0.2936, 0.9928]],

         [[0.8611, 0.5119],
          [0.7976, 0.1552]],

         [[0.7479, 0.0291],
          [0.3519, 0.8065]],
         
         [[0.0,0.0],
           [0.0,0.0]]
]])
</code></pre>
<p>with now a tensor of size (1,5,2,2)</p>
<p>How can i do, when I use stack or cat it says this error :</p>
<p>RuntimeError: stack expects each tensor to be equal size, but got [1, 4, 2, 2] at entry 0 and [2, 2] at entry 1</p>
<p>How can I do to overcome the problem of the dimension, is there a solution ?</p>
<p>Thank you for your help !</p>

4bach: <p>torch.cat((t,torch.zeros(2,2).unsqueeze(dim=0).unsqueeze(dim=0)),dim=1)</p>
<p>Works !</p>