akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch Tutorial Error--CUDA

username: badger
cooked: <p>I was running the CIFAR10 tutorial and when I got to the CUDA portion, both locally and on colab (with a GPU runtime) I get the following error:</p>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
<p>This was after I ran the commands in the tutorial:</p>
<pre><code class="lang-auto">net.to(device)
inputs, labels = data[0].to(device), data[1].to(device)
</code></pre>
<p>And my device was confired as cuda:0</p>

username: badger
cooked: <p>I hate it when I figure out the answer after I post.</p>
<p>If it isn’t clear you need to set to  .to(device) inside the training loop, not just once.</p>
How to freeze part of the weight matrix during training?

username: Asgard
cooked: <p>I have:</p>
<pre><code class="lang-auto">model = Sequential(Linear(8,5), Linear(5,3), Linear(3,1))
</code></pre>
<p>Now, for <code>model[1].weight</code>, I want to freeze part of this weight matrix during the training procedure i.e.,</p>
<pre><code class="lang-auto">model[1].weight[0:3,0:3]
</code></pre>
<p>values shouldn’t vary during the backpropagation. Apparently, <code>model[1].weight[0:3,0:3].detach()</code> isn’t working as the weights vary during the training. And <code>model[1].weight[0:3,0:3].requires_grad=False</code> throws error.</p>
<p>So, how to achieve this? Thank you in advance.</p>

username: InnovArul
cooked: <p>I am not sure if any predefined way exists in pytorch.<br>
You may need to write a custom linear layer to achieve it, something like below:</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/InnovArul/ee2986feff215fed94401ec83367408b">
  <header class="source">

      <a href="https://gist.github.com/InnovArul/ee2986feff215fed94401ec83367408b" target="_blank" rel="noopener">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/ee2986feff215fed94401ec83367408b" target="_blank" rel="noopener">https://gist.github.com/InnovArul/ee2986feff215fed94401ec83367408b</a></h4>

  <h5>linear_partial_freeze.py</h5>
  <pre><code class="Python">import torch, torch.nn as nn                                                         
import torch.optim as optim, torch.nn.functional as F                                
                                                                                     
class CustomLinear(nn.Module):                                                       
    def __init__(self, mask):                                                        
        super().__init__()                                                           
        self.register_buffer("mask", mask)                                           
        out_channels, in_channels = mask.shape                                       
        self.weight = nn.Parameter(torch.randn(out_channels, in_channels))       
        </code></pre>
    This file has been truncated. <a href="https://gist.github.com/InnovArul/ee2986feff215fed94401ec83367408b" target="_blank" rel="noopener">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
KL divergence for (Multivariate)Normal with diagonal covariance matrix

username: JakobHavtorn
cooked: <p>I use <code>torch.distributions</code> for variational inference and thus use <code>torch.distributions.kl_divergence</code> to compute analytical KL divergences. However, there is no registered KL divergence for normal distributions with diagonal covariance when they are defined by wrapping <code>Normal</code> with <code>Independent</code>. The code example below shows this in detail.</p>
<p>The KL divergence can be correctly computed if we use the <code>MultivariateNormal</code> class to instantiate the covariance matrix, but this is rather inefficient both in terms of memory and compute time when the covariance matrix is diagonal.</p>
<p>Is there any other way to go here or do I simply have to implement the analytical formula myself?</p>
<pre><code class="lang-auto">import torch
# Define diagonal normal with MultivariateNormal (to compute KL wrt.)
mu = torch.Tensor([0, 0])
scale1 = torch.Tensor([1, 1])
cov1 = torch.diag_embed(scale1)
diag_normal_mv1 = torch.distributions.MultivariateNormal(loc=mu, covariance_matrix=cov1)

# Define diagonal normal with Independent
scale2 = torch.Tensor([0.5, 1.5])
diag_normal = torch.distributions.Independent(torch.distributions.Normal(mu, scale2), 1)

# Define diagonal normal with MultivariateNormal
cov2 = torch.diag_embed(scale2**2)
diag_normal_mv2 = torch.distributions.MultivariateNormal(loc=mu, covariance_matrix=cov2)

torch.distributions.kl_divergence(diag_normal_mv1, diag_normal_mv2)  # &gt; 0.9345
torch.distributions.kl_divergence(diag_normal_mv1, diag_normal)  # &gt; NotImplementedError
</code></pre>

username: hf_Morii
cooked: <p>Hi,  I have a problem that has been bothering me for a long time.<br>
I wanna implement the JS divergence for (Multivariate)Normal with diagonal covariance matrix.<br>
could you give me some advice? Thanks</p>

username: zplizzi
cooked: <p>Did anyone figure out a solution to the original question here? Or are willing to share an implementation of the analytical formula?</p>

username: InfT
cooked: <p>I’ve the same problem.</p>
Updating model parameters with a new loss function

username: sahar.mi
cooked: <p>hi everyone, I’m trying to update the parameters of a pretrained masked language model with a new custom loss function, but it seems that the gradient of parameters are none. I used a pretrained bert for masked language model and a pretrained classifier. I have concatenated a vector to the batch and made a new batch, when I want to optimize parameters of language model, I see that the gradients of the parameters are none so it can not update the weights. but without concatenation everything seems fine. Can you please advice what should I do with this issue?</p>
<pre><code class="lang-auto">loss = loss_function(scores, true_labels)
loss.backward()
for param in maskedlm_.parameters():
    print(param.grad)
update_weights(maskedlm)
optimizer.step()
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sahar.mi" data-post="1" data-topic="137022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sahar.mi/48/30432_2.png" class="avatar"> sahar.mi:</div>
<blockquote>
<p>but without concatenation everything seems fine</p>
</blockquote>
</aside>
<p>My guess would be that you are creating a non-leaf tensor with the concatenation.<br>
Could you check the <code>is_leaf</code> attribute of the parameters before and after the concatenation?<br>
If it’s showing <code>False</code> afterwards (and <code>True</code> before), re-wrap the new tensor in an <code>nn.Parameter</code>.</p>

username: sahar.mi
cooked: <p>Thanks for your suggestion, I checked is_leaf attribute and all of them were True, but the gradients of parameters are None, do you have any other suggestion? maybe there is a problem with my pretrained classifier or loss function?<br>
by the way I used this code snippet for checking the leaf</p>
<pre><code class="lang-auto">for parameter in maskedlm_.parameters():
     print(parameter.data.is_leaf)
</code></pre>

username: ptrblck
cooked: <p>Your code will explicitly return the parameters only, which all should be leaves. You should check the <code>is_leaf</code> attribute of the parameters before and after concatenation manually.</p>
<pre><code class="lang-python">a = nn.Parameter(torch.randn(1))
b = nn.Parameter(torch.randn(1))

print(a.is_leaf)
&gt; True
print(b.is_leaf)
&gt; True

c = torch.cat((a, b), dim=0)
print(c.is_leaf)
&gt; False
</code></pre>

username: sahar.mi
cooked: <p>Thank you, your suggestion was so helpful and I found the problem at last <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch on AMD Cpu and Nvidia Rtx 3080

username: Hussein
cooked: <p>Hello</p>
<p>I have to train deep learning models on videos that require time consuming augmentations.</p>
<p>I found that most of the training and testing  time is related to the preprocessing on the CPU.</p>
<p>Currently, I have Intel 10700k processor. I use Nvidia RTX 3080 GPU for training and testing with pytorch.</p>
<p>My question is:</p>
<p>If I switch to using AMD processor does that affect pytorch in any way?</p>
<p>Can I train on Nvidia GPU while using AMD processor for the videos augmentations?</p>
<p>Many Thanks<br>
Hussein</p>

username: my3bikaht
cooked: <p>Main bottleneck here is actually Python itself and how it manages CPU (GIL for example). If you can parallelize your calculations I would advise you to implement multiprocessing for the CPU part. 10700k is rather fast 8-core CPU, switching to AMD won’t give you significant speed improvements (except switching to something fast and with a lot of cores, like 5650x or threadripper)</p>

username: Hussein
cooked: <p>Thanks <a class="mention" href="/u/my3bikaht">@my3bikaht</a> for your help. Yes I agree with you. After doing some experiments I found that the Cpu is fast and can handle the operations efficiently.<br>
The bottleneck is using one process to do the work.<br>
Using multiple processes will decrease the runing time and use the resources effectively.</p>
<p>Thank you very much man</p>
NLLLoss not implemented for

username: ivang90
cooked: <p>Hi all, I have problem with NLLLoss, I am getting error message: RuntimeError: “nll_loss_out_frame” not implemented for ‘Long’</p>
<p>This is my code:<br>
for input_tensor, target_tensor in train_dataloader:<br>
encoder_decoder.zero_grad()<br>
log_probs = encoder_decoder((input_tensor,target_tensor))<br>
predicted = log_probs.argmax(dim=1)<br>
loss = loss_function(predicted, target_tensor)</p>
<p>Shape of tensors are predicted.shape = (1,4) and targer_tensor.shape = (1,4).</p>
<p>I tried to change to other tensor type like int, but it doesn’t help. What i am doing wrong here?</p>

username: KFrank
cooked: <p>Hi Ivan!</p>
<aside class="quote no-group quote-modified" data-username="ivang90" data-post="1" data-topic="137337" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/f475e1/48.png" class="avatar"> ivang90:</div>
<blockquote>
<p>I am getting error message: RuntimeError: “nll_loss_out_frame” not implemented for ‘Long’<br>
…<br>
Shape of tensors are predicted.shape = (1,4) and targer_tensor.shape = (1,4).</p>
</blockquote>
</aside>
<p>May I respectfully suggest that you read the documentation for<br>
<a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.NLLLoss.html" rel="noopener nofollow ugc">NLLLoss</a>?</p>
<p>What types does <code>NLLLoss</code> expect / require for the <code>input</code> (your<br>
<code>predicted</code>) and <code>target</code> (your <code>target_tensor</code>) tensors passed<br>
into it?  What shapes?</p>
<p>Best.</p>
<p>K. Frank</p>

username: ivang90
cooked: <p>Ohhh, I think I was very tired yesterday how I didn’t saw obvious thing here what I did even after reading documentation. I just had to call loss function using log_probs and target_tensor like:<br>
loss = loss_function(log_probs, target_tensor)<br>
Now everything works, sorry for confusion.</p>
<p>Thank you Frank,<br>
Ivan</p>
How to load from two datasets with different batch size

username: Mountchicken
cooked: <pre><code>  I have two datasets, suppose they are A and B.  And I want to create one dataloader that can read from both of them with 512 samples from A and only 1 sample from B. And dataset A is the main one, the loop should end when A finishes its iteration. 
 Currently, my solution is below but it's time-consuming. Could any help me about this😣
</code></pre>
<pre><code class="lang-auto">dataset_A = lmdbDataset(*args)
dataset_B = lmdbDataset(*args
dataloader_A = torch.utils.data.Dataloader(dataset_A, batch_size=512,shuffle=True)
dataloader_B = torch.utils.data.Dataloader(dataset_B, batch_size=1,shuffle=True)
...
for batch_A in dataloader(dataset_A):
      batch_B = next(iter(dataloader_B)
      total_batch = torch.cat([batch_A, batch_B], dim=0)
      out = model(total_batch)
</code></pre>

username: OrielBanne
cooked: <p>Hi-</p>
<p>first - your dataset is a part of your dataloader -<br>
you do not need to mention ‘for batch_A in dataloader(dataset_A)’, but you can: for batchA in dataloader_A</p>
<p>fix this, and show what is your error flag and we can go from there</p>
Loss.backward() gives RuntimeError: CUDA error: device-side assert triggered

username: laro
cooked: <p>I’m trying to use BCELoss without any success.</p>
<pre><code class="lang-auto">loss = BCELoss()
opt   = optim.AdamW  (model.parameters(), lr=0.01, betas=(0.9, 0.99), weight_decay=0.001)
</code></pre>
<pre><code class="lang-auto">loss = Loss(z, y)
opt.zero_grad() 
loss.backward()
</code></pre>
<p>z, y have the shape: (128, 2)</p>
<p>I’m getting error (from <code>loss.backward()</code>):</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<ol>
<li>What is wrong ?</li>
<li>What do I need to change ?</li>
</ol>
Where to set the manual seed

username: alex_rachkov
cooked: <p>I have three files in my current pipeline:</p>
<ol>
<li>net.py that defines a network I want to use</li>
<li>data.py that reads the data and creates a dataloader object</li>
<li>train.py that runs the experiment</li>
</ol>
<p>I can think of randomness coming in play with setting the initial weights of the network, its dropout regularization, as well as the dataloader object fetching batches.</p>
<p>My question is, where do I need to set the random seed? Do I need to set it within all three files? Or is it enough for it to be set within train.py and then the code will be able to use it when I load in the network and dataloader objects?</p>

username: ptrblck
cooked: <p>Setting the seeds at the beginning of your main script should work if you are concerned about the PRNG usage in other files/modules. The <code>DataLoader</code> object might be a bit more tricky so take a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html#reproducibility"><code>DataLoader</code> section in the Reproducibility Docs</a>.</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 768]], which is output 0 of TanhBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detectio

username: guanghui_ma
cooked: <p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 768]], which is output 0 of TanhBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<pre><code>def forward(self, input, labels):
    result = self.bert(**input).pooler_output

    label_one = [idx for idx, dat in enumerate(labels) if dat.item() == 0]
    label_two = [idx for idx, dat in enumerate(labels) if dat.item() == 1]
    alpha = 0.5
    lam = np.random.beta(alpha, alpha)
    temp_result = result.detach()  # 拷贝一个共享内存，合并操作不能进行梯度修改
    # 把原始节点的特征， 替换为新节点的特征
    if label_one:
        label_one_tensor = torch.tensor(label_one).to(device)
        deep_label_one = copy.deepcopy(label_one)
        random.shuffle(deep_label_one)
        label_one_tensor_shuffle = torch.tensor(deep_label_one).to(device)
        # with torch.no_grad():
        result[label_one_tensor] = lam * temp_result[label_one_tensor, :] + (1 - lam) * temp_result[label_one_tensor_shuffle, :]

    print(result)
    if label_two:
        label_two_tensor = torch.tensor(label_two).to(device)
        deep_label_two = copy.deepcopy(label_two)
        random.shuffle(deep_label_two)
        label_two_tensor_shuffle = torch.tensor(deep_label_two).to(device)
        # with torch.no_grad():
        result[label_two_tensor] = lam * temp_result[label_two_tensor, :] + (1 - lam) * temp_result[label_two_tensor_shuffle, :]

    data = self.head(result)
    return data, self.fc(data), labels</code></pre>

username: ptrblck
cooked: <p>I guess these inplace operations might be causing the issue:</p>
<pre><code class="lang-python">result[label_one_tensor] = lam * temp_result[label_one_tensor, :] + (1 - lam) * temp_result[label_one_tensor_shuffle, :]

result[label_two_tensor] = lam * temp_result[label_two_tensor, :] + (1 - lam) * temp_result[label_two_tensor_shuffle, :]
</code></pre>
<p>and you might need to store the output in a temporal tensor and <code>torch.cat</code> or <code>torch.stack</code> them to the final result tensor.</p>
Can we use model.eval() and model.train() alternatively in Pytorch?

username: Lingfeng_Shen
cooked: <p>Hi! I’m trying to get two representation of the same input in model.eval() and model.train(), respectively. I write the following codes:</p>
<pre><code class="lang-auto">for id, batch in enumerate(loader):
    model.train()
    rep1 = model(batch.text)
    model.eval()
    rep2 = model(bacth.text)
</code></pre>
<p>However, it seems it didn’t work, so is there any ways to do that?</p>

username: ptrblck
cooked: <p>Could you describe what exactly is not working? Do you see any errors using it?</p>

username: Lingfeng_Shen
cooked: <p>Thanks for your attention. I just found I put model.eval() in the forward process, and it works now, I’ll be more careful next time.</p>
Extract core model from Siamese Network

username: Malaker
cooked: <p>I have trained a model with the following general architecture:</p>
<ol>
<li>Model takes two inputs</li>
<li>Runs both of them through an embedding model, which returns some embedding corresponding to both inputs, say, shape a → [batch_size, 748] and b -&gt;[batch_size, 748].</li>
<li>Take the absolute difference, ie torch.abs(a - b), and pass to fully connected layer.</li>
<li>Train model with cross entropy loss.</li>
</ol>
<p>Now, I want to <strong>extract the embedding model</strong> such that I can pass <strong>one input</strong> and get an embedding. How can this be achieved?<br>
The “forward hook” seems to not be applicable here since the number of inputs in my Siamese Network are two and thus I won’t be able to pass a single input and get the embedding.</p>
<p>Thank you.</p>

username: Malaker
cooked: <p>It’s actually pretty simple if you know how the embedding model is defined.<br>
Initialize a new model with the same configuration of layers as the embedding model.<br>
And say, siameseNetwork is the network with the embedding model within it as a layer, called “embedding_model”. You can find this by <code>print(siameseNetwork)</code>. Then do something like:</p>
<pre><code class="lang-auto">embedding_model = EmbeddingModel()
embedding_model.load_state_dict(siameseNetwork.embedding_model.state_dict())
</code></pre>
<p>Now <code>embedding_model</code> will have the correct weights.</p>
My weight initialization method is shown in 2. However, the weight value remains unchanged after initialization. Is there a proper weight initialization method that can be trained?

username: xiaojin
cooked: <p>2.My weight initialization method<br>
w=torch.tensor([[ 1.3562],[ 1.3095], [-0.6110],[ 1.6676], [-0.3462],[-1.8372],[-1.0733],[ 1.5269],[-0.9958],<br>
[-0.7507]], requires_grad=True)<br>
net.hidden1.weight=torch.nn.Parameter(w)</p>

username: xiaojin
cooked: <p>1.my network<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93f1f1846c46669abbb2f056fffe1bbad8ab125d.png" data-download-href="https://discuss.pytorch.org/uploads/default/93f1f1846c46669abbb2f056fffe1bbad8ab125d" title="network"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93f1f1846c46669abbb2f056fffe1bbad8ab125d_2_690x185.png" alt="network" data-base62-sha1="l6My4uUVR99HNvQywRwp1VeyBGJ" width="690" height="185" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93f1f1846c46669abbb2f056fffe1bbad8ab125d_2_690x185.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93f1f1846c46669abbb2f056fffe1bbad8ab125d_2_1035x277.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93f1f1846c46669abbb2f056fffe1bbad8ab125d_2_1380x370.png 2x" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network</span><span class="informations">1399×376 57.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Vulkan workflow

username: dennism
cooked: <p>I was able to build pytorch with USE_VULKAN=1 and go through the workflow for generating a Vulkan backend model.</p>
<p>I copied my aar files to my android project:</p>
<blockquote>
<pre><code>implementation files('pytorch_android-release.aar')
implementation files('pytorch_android_torchvision-release.aar')
</code></pre>
</blockquote>
<p>2nd step, run on Android, I’m getting the error</p>
<blockquote>
<p>java.lang.UnsatisfiedLinkError: dlopen failed: library “libpytorch_jni_lite.so” not found</p>
</blockquote>
<p>Some of the other posts with this error suggest using LiteModuleLoader but that doesn’t seem quite the right solution to me for the Vulkan backend… or is it?</p>
<p>For desktop python support w/ Vulkan I gathered I needed to build PyTorch.  Is this the same for the Android libraries?  Or can I use the prebuilt implementation from Gradle dependencies instead of building the aar files?</p>
Does the current version libtorch support TensorRT as inference backend?

username: ItalianSC
cooked: <p>Hi:<br>
Does the current version libtorch support TensorRT as  inference backend?</p>

username: narendasan
cooked: <p>You can use TensorRT as an inference backend for LibTorch by installing Torch-TensorRT: <a href="https://github.com/nvidia/Torch-TensorRT/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/Torch-TensorRT: PyTorch/TorchScript compiler for NVIDIA GPUs using TensorRT</a></p>
<p>Instructions on how to use the backend integration can be found here: <a href="https://nvidia.github.io/Torch-TensorRT/tutorials/use_from_pytorch.html" class="inline-onebox" rel="noopener nofollow ugc">Using Torch-TensorRT Directly From PyTorch — Torch-TensorRT master documentation</a></p>
Backward through gradient update

username: robert-lieck
cooked: <p>I am trying to figure out why this minimal example is not working as expected:</p>
<pre><code class="lang-python"># two variables
x = torch.Tensor([1])
y = torch.Tensor([1])
x.requires_grad = True
y.requires_grad = True

# loss (think: training loss)
loss = x * y
loss.backward()
print("loss", loss)
print("x", x, x.grad)
print("y", y, y.grad)

# update x using gradient (which depends on the value of y)
new_x = x - 1e-1 * x.grad
print("new_x", new_x)

# different loss using updated value (think: validation loss)
new_loss = new_x ** 2
print("new_loss", new_loss)
y.grad.data.zero_()  # clear grads from first pass
x.grad.data.zero_()  # clear grads from first pass
new_loss.backward()  # compute gradient for new loss

# Why is grad of y zero? 
print("x", x, x.grad)
print("y", y, y.grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-nohighlight">loss tensor([1.], grad_fn=&lt;MulBackward0&gt;)
x tensor([1.], requires_grad=True) tensor([1.])
y tensor([1.], requires_grad=True) tensor([1.])
new_x tensor([0.9000], grad_fn=&lt;SubBackward0&gt;)
new_loss tensor([0.8100], grad_fn=&lt;PowBackward0&gt;)
x tensor([1.], requires_grad=True) tensor([1.8000])
y tensor([1.], requires_grad=True) tensor([0.])
</code></pre>
<p>Changing the value of y <strong>does</strong> change the value of new_loss, so shouldn’t it’s gradient be non-zero? What am I missing?</p>

username: robert-lieck
cooked: <p>OK, adding <code>create_graph=True</code> to the first <code>backward</code> call fixes the problem. From the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html" rel="noopener nofollow ugc">docs</a>:</p>
<blockquote>
<p><strong>create_graph</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a><em>,</em> <em>optional</em>) – If <code>True</code>, graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults to <code>False</code>.</p>
</blockquote>
<p>I kind of expected a warning or error, similar to when calling <code>backward</code> twice (without explicitly retaining the graph), but I guess it might be hard to detect if cutting the computational graph at that point is intended or not…</p>
Debugging why network parameters don't change

username: yohad
cooked: <p>Hi,</p>
<p>I have a somewhat weird network architecture, where I use two networks (one output is the second weights). I can run it and do <code>.backward()</code> and <code>optimizer.step()</code> without any errors. But when I look at the first network parameters between batches they don’t change.</p>
<p>I have absolutely no idea what I’m doing wrong and would like to just look at the backward’s computation graph and see where it stops. I’ve tried <code>pytorchviz</code> but from what I understood it is no longer maintained.</p>
<p>Does anyone know how I can debug this problem?</p>

username: googlebot
cooked: <p>between backward() and step(), you can check .grad of all named_parameters() to locate the problematic segment where gradients disappear</p>
<p>or, add hooks (either to tensors or modules) to do a similar analysis on non-parameter tensors (i.e. intermediate outputs) - this requires more effort, but you’ll be able to narrow it down to individual operations, if necessary</p>
<p>though, it is likely that your “two network” link is the problem point.</p>
Torch.istft onesided output error

username: jBloodless
cooked: <p>I try to do some DSP thing and I need to convert frame into single bin and then restore the frame from that bin. Here’s minimal example:</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

frame = torch.rand(1536)
bin = torch.stft(frame, n_fft=frame.shape[0], hop_length=frame.shape[0], center=False,
                     return_complex=True, window=torch.hann_window(frame.shape[0]))
rest_frame = torch.istft(bin, n_fft=frame.shape[0], hop_length=frame.shape[0], center=False,
                        return_complex=True, window=torch.hann_window(frame.shape[0]))

plt.plot(rest_frame-frame)
plt.show()
</code></pre>
<p>I expected to get the same frame, but instead I got</p>
<p><code>RuntimeError: Cannot have onesided output if window or input is complex</code></p>
<p>in istft. torch.stft produces the bin that I need, but I still can’t make istft work. What do I do wrong?</p>

username: nateanl
cooked: <p>Hi <a class="mention" href="/u/jbloodless">@jBloodless</a>, thanks for sharing. In this case, I suggest changing the <code>return_complex</code> to False in <code>torch.istft</code>.</p>
<p>Also the <code>n_fft</code> and <code>hop_length</code> and window length need to be carefully selected, as <code>torch.istft</code> requires the settings to pass NOLA check to make the signal invertible. Check this doc for more information: <a href="https://gauss256.github.io/blog/cola.html" class="inline-onebox" rel="noopener nofollow ugc">Invertibility of overlap-add processing</a></p>
How to fix the GPU memory allocation with changeable training size?

username: c_cj
cooked: <p>I have a task, for one epoch the image size is raised from 256 to 512. However, the GPU memory allocation is very unstable during the training with a changeable size. For example, if the image size change from 256 to 264, the costed GPU memory will dramatically change from 9GB to 20GB and back to 10GB. And the training speed is very slow. Although my model can be trained in 512x512 size, it also causes OOM by the unstable memory allocation. How to fix the GPU memory allocation with the largest image size in pytorch?</p>

username: ptrblck
cooked: <p>For variable input shapes I would recommend to start with the largest one and reduce the size if this would be possible. This would allow you to reuse the freed memory without running into memory fragmentation issues.</p>

username: c_cj
cooked: <p>Thanks. It is useful!</p>
ReduceLROnPlateau verbose not correct?

username: Kalle
cooked: <p>I have implemented <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html" rel="noopener nofollow ugc">ReduceLROnPlateau</a> in my code and I have a question about verbose.</p>
<p>When I set verbose=True and patience=1 it gives me the following result:</p>
<pre><code class="lang-auto">...
...
Epoch: 9/100
validation_loss: 0.9176
Epoch: 10/100
validation_loss: 0.8771
Epoch     9: reducing learning rate of group 0 to 2.5000e-04.
validation_loss: 0.8286
Epoch: 12/100
validation_loss: 0.8203
Epoch: 13/100
validation_loss: 0.8343
Epoch: 14/100
validation_loss: 0.8316
Epoch    13: reducing learning rate of group 0 to 2.5000e-05.
...
...
</code></pre>
<p>What I find strange is that I would expect it to print:</p>
<pre><code class="lang-auto">Epoch     10: reducing learning rate of group 0 to 2.5000e-04.
</code></pre>
<p>Instead of:</p>
<pre><code class="lang-auto">Epoch     9: reducing learning rate of group 0 to 2.5000e-04.
</code></pre>
<p>As I see it the learning rate changes at epoch 10 and not at epoch 9? Again at the other verbose message I would expect it to be epoch 14 instead of epoch 13.</p>
<p>So what am I missing here?</p>

username: ptrblck
cooked: <p>You are either manually passing the <code>epoch</code> argument to <code>step</code> (and might be passing the wrong value) or otherwise the internal <code>last_epoch</code> counter will be increased and used instead as seen <a href="https://github.com/pytorch/pytorch/blob/94b6fa6f8b0cd730ad7401dc52880a9eb8a6e405/torch/optim/lr_scheduler.py#L902-L933">here</a>. I don’t know when exactly you are calling the learning rate scheduler, so could you post a minimal, executable code snippet showing the epoch mismatch?</p>

username: Kalle
cooked: <p>So the following is a minimal piece of my code:</p>
<pre><code class="lang-auto">scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimize, verbose=True)

for epoch in range(n_epochs):
    for i, (local_batch, local_labels) in enumerate(training_generator):
        local_batch, local_labels = local_batch.to(device), local_labels.to(device)
        ...
    model.eval()  # prep model for evaluation
    for local_batch, local_labels in validation_generator:
        local_batch, local_labels = local_batch.to(device), local_labels.to(device)
        output = model(local_batch)
        ...
        # calculate the loss
        val_loss = criterion(output, local_labels)
        # record validation loss
        running_val_loss += val_loss.item()

    valid_loss = running_val_loss / len(validation_generator)
    
    scheduler.step(valid_loss)
</code></pre>
<p>I tried to update my <em>scheduler.step(valid_loss)</em> with <em>scheduler.step(valid_loss, epoch=epoch+1)</em> and now it prints correctly. But why was the parameter epoch needed?</p>
<pre><code class="lang-auto">Epoch    19: reducing learning rate of group 0 to 2.0000e-04.
Epoch: 20/100
...
</code></pre>
How to use pretrained FastText-Embeddings for LSTM Tagger

username: Lady_Hangaku
cooked: <p>I want to use german pretrained fasttext embeddings for my LSTM tagger model.<br>
There are a few options to get the full fasttext embedding collection. Which would you recommend using?<br>
And how do I load the embeddings for each text of the training data so that the embedding layer of the model already gets the fasttext representation?<br>
Can you maybe give me an example code or a tutorial which I can recreate? Didn’t find anything useful yet. For training data I use CoNLL2003 format.<br>
Thank you!</p>

username: AbdulsalamBande
cooked: <p>Here is something better.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
data_dictionary = {"a":1,"apple":2,"milk":3}
n_tokens = 3
embedding_size = 8
embedding = nn.Embedding(n_tokens, embedding_size)
pretrained_fasttext_embeddings  = torch.rand((n_tokens,embedding_size))
embedding.weight.data.copy_(pretrained_fasttext_embeddings)
print(embedding.weight)
tensor([[0.3748, 0.4387, 0.9356, 0.7790, 0.7401, 0.3412, 0.1741, 0.4702],
        [0.6634, 0.8787, 0.9448, 0.2775, 0.5960, 0.6934, 0.6094, 0.0103],
        [0.8099, 0.9782, 0.4780, 0.0253, 0.5966, 0.0216, 0.5862, 0.6692]],
       requires_grad=True)
[0.3748, 0.4387, 0.9356, 0.7790, 0.7401, 0.3412, 0.1741, 0.4702] is the embedding for "a"
0.6634, 0.8787, 0.9448, 0.2775, 0.5960, 0.6934, 0.6094, 0.0103] is the embedding for "apple"
[0.8099, 0.9782, 0.4780, 0.0253, 0.5966, 0.0216, 0.5862, 0.6692] is the embedding for "milk"

</code></pre>

username: Lady_Hangaku
cooked: <p>Thank you! What kind of punctuations are missing between “pretrained”, “fasttext” and “embeddings”?<br>
And do I then give the nn.Embedding() layer a list of strings, therefore a list of the tokens?</p>
Training multiple small models in parallel

username: Yitao_Chen
cooked: <p>I need to train multiple small models in parallel to speedup the training process using a node with four GPUs. The training process of each model uses about 50% of my GPUs.<br>
In my use case, all the models are independent, so there is no synchronization between any of them.<br>
I tried multiple methods, but nothing is working. Appreciate if anyone can give me some advice on this.</p>
<p>The following are the methods I tried and the problems with them:</p>
<ol>
<li>
<p>DDP<br>
Training time of a single model using DDP is longer than training a without it, even I turn off the synchronization between models. I am not sure where the additional overhead comes from.</p>
</li>
<li>
<p>Multiprocessing<br>
The behavior of using multiprocessing is strange with cuda.<br>
I created multiple threads and each of the thread runs a training process.<br>
But the program only works for some times; when it does not work, it hangs without sending any error message.</p>
</li>
</ol>
<p>Any advice is appreciated!</p>

username: Bihy
cooked: <p>Hi, I’m facing the same challenge here. did you find any solution? Thank you.</p>
Out option for operators

username: jay_thakur
cooked: <p>Hello,<br>
I wanted to know how to find out which operator supports “out” option?</p>
<p>for e.g. clamp does <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" class="inline-onebox" rel="noopener nofollow ugc">torch.clamp — PyTorch 1.10.0 documentation</a><br>
while silu don’t <a href="https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html" class="inline-onebox" rel="noopener nofollow ugc">SiLU — PyTorch 1.10.0 documentation</a></p>
<p>is there any rule on which it depends or something?</p>
Large RAM usage at the start of training

username: spadel
cooked: <p>Hey guys, I have a training loop, which looks as follows:</p>
<pre><code class="lang-auto"> for epoch in range(arg.epochs):
        for step, (seq, target, reference_val) in enumerate(TrainLoader):
            model.train()
            seq, target, reference_val = seq.to(device), target.to(device), reference_val.to(device)
            model.zero_grad()
            preds = model(seq)
            loss = criterion(preds, target)
            loss.backward()
            if step % 100 == 0:
                if arg.BCELoss:
                    print('reference: ', reference_val[:5].unsqueeze(-1), '\n target: ', target[:5], '\n pred: ',
                          torch.sigmoid(preds[:5]))
                else:
                    print('reference: ', reference_val[:5].unsqueeze(-1), '\n target: ', target[:5], '\n pred: ',
                          preds[:5])

            optimizer.step()
            if optimizer.param_groups[0]["lr"] &gt; arg.lr_end:
                scheduler.step()
          
</code></pre>
<p>At the time of the first print statement, about 4GB RAM are consumed. However, during the very first optimizer step, almost 10 GB are necessary. Later during the training the network will stay constantly at 4GB. Does anyone know what causes this behavior and how to prevent it?</p>

username: my3bikaht
cooked: <p>Don’t see anything out of ordinary in your training pass. Maybe something related to precision during optimizer run?</p>
<p>UPD. right, in this case memory consumption will increase every run</p>
Quantization aware training support cuda?

username: Xiaoyuan_WANG
cooked: <p>Hello,everyone.</p>
<p>I‘m now trying use pytorch for quantization. According to the documentation,there are three types, dynamic quantization,static quantization and static quantization aware training.</p>
<p>And i have some questions related to the GPU and CPU, we know that pytorch doesn’t provide quantized operator implementation on CUDA, and quantization aware training occurs in full floating point can support CUDA, it is still not clear enough for me, i need to use the model to do prediction on C++ by using libtorch, once I do quantization aware training, can the model run on GPU?</p>
<p>May someone answer my question?<br>
thanks a lot and best wishes!</p>

username: Atopis
cooked: <p>I want to know too. if u have the answer, please tell me. thank u <img src="https://discuss.pytorch.org/images/emoji/apple/purple_heart.png?v=10" title=":purple_heart:" class="emoji" alt=":purple_heart:"></p>
Predicts image and retrieve id image from dataset

username: sara_adam
cooked: <p>Hi<br>
I’m a beginner using PyTorch models<br>
I need to know the name or id image prediction and store the results inside CSV file (id or name image and predictions)</p>
<p>def visualize_model(model, num_images=10):</p>
<pre><code>was_training = model.training

model.eval()



images_so_far = 0

fig = plt.figure()

with torch.no_grad():

    for i, (inputs, labels) in enumerate(dataloaders['val']):

        inputs = inputs.to(device)

        labels = labels.to(device)

        outputs = model(inputs)

        _, preds = torch.max(outputs, 1)

    

        for j in range(inputs.size()[0]):

       

            images_so_far += 1

            ax = plt.subplot(num_images//2, 2, images_so_far)

            ax.axis('off')

            ax.set_title('predicted: {}'.format(class_names[preds[j]]))

            imshow(inputs.cpu().data[j])

            if images_so_far == num_images:

                model.train(mode=was_training)

                return

    model.train(mode=was_training)
</code></pre>
<p>the code prints the image and predicts, is there a way to retrieve the name or id image?</p>
<p>sorry I am a beginner in vision</p>

username: ptrblck
cooked: <p>I assume you are referring to the image name (or path), not the class name.<br>
If so, you could create a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and return the image name in the <code>__getitem__</code> additionally to the data and target tensors. Inside the <code>DataLoader</code> loop you would then have 3 values and could print the image name as well:</p>
<pre><code class="lang-python">for i, (inputs, labels, name) in enumerate(dataloaders['val']):
    ...
</code></pre>
Taking advantage of JIT optimization from C++

username: erick
cooked: <p>Hello! I’m relatively new to pytorch, so this question may not make complete sense.</p>
<p>We are using a primarily C++ code base, so we are considering defining our models in C++. We are finding that for a toy example (GELUs), the python code which is converted to torchscript and loaded in C++ is significantly faster (2-6x) than implementing the same operation directly in C++. I have since learned that this is due to the JIT interpreter fusing certain operations into a single CUDA kernel.</p>
<p>Is this surprising? Is there anyway to take advantage of these optimizations from a model defined in C++?</p>

username: ptrblck
cooked: <p>You could check what exactly is fused in the operation by printing the graph via <code>.graph_for</code> for this module. This should show the fused operations and would give you more information about the difference in your timing.</p>
<p>I also assume you’ve written a custom CUDA implementation or are you comparing the GPU kernel vs. your C++ (host) code?</p>
How to tell if pypi version of torchaudio is gpu or cuda?

username: mmmmoth
cooked: <p>How can i tell if the torchaudio on pypi is cuda or cpu based?</p>

username: ptrblck
cooked: <p><code>pip list | grep torch</code> will return the packages with <code>torch</code> in their name and the pip wheels using a CUDA runtime would have the <code>+cu113</code> (or another CUDA version) suffix.<br>
E.g.</p>
<pre><code class="lang-python">pip list | grep torch
torch                         1.11.0.dev20211101+cu113
</code></pre>

username: mmmmoth
cooked: <p>But on the site <a href="https://pypi.org/project/torchaudio/" class="inline-onebox" rel="noopener nofollow ugc">torchaudio · PyPI</a><br>
It doesn’t have any additional information. Also, how are you getting a 1.11.0dev version? Is that a self compiled version?</p>

username: ptrblck
cooked: <p>Yes, you are right. The provided link seems to depend on the default CUDA runtime, i.e. CUDA 10.2.<br>
Based on <a href="https://github.com/pytorch/audio/pull/1737">this PR</a> it seems that <code>torchaudio</code> isn’t directly linking to this runtime, but depends on <code>torch</code> to provide all symbols.<br>
In that case, you could check the corresponding <code>torch.version.cuda</code> output, as I cannot find an internal <code>torchaudio</code> flag which would indicate the needed CUDA runtime version.</p>
<p>The <code>1.11.0.dev...</code> version is used in the nightly binaries, which you can also select via the install instructions.</p>
F.CROSS_ENTROPY weight parameter does not seem to have an effect

username: Makuta11
cooked: <p>Hey Guys</p>
<p>I am currently building a multi-task ML model using both F.binary_cross_entropy and F.cross_entropy. Since my datasets are quite unbalanced i am implementing class weights and giving them as argument to both functions to combat this. However, i am experiencing something quite peculiar when adding it to F.cross_entropy. It seems to have absolutely no effect on the loss, see below (code ran in VSC debugger):</p>
<pre><code class="lang-auto">F.cross_entropy(out_AU_intensities[i][AU_idx], lab[AU_idx] - 1, weight = self.cw_int[i])
&gt;tensor(1.3951, grad_fn=&lt;NllLossBackward0&gt;)

F.cross_entropy(out_AU_intensities[i][AU_idx], lab[AU_idx] - 1)
&gt;tensor(1.3951, grad_fn=&lt;NllLossBackward0&gt;)
</code></pre>
<p>And the actualy rescaling is as follows:</p>
<pre><code class="lang-auto">self.cw_int[i]
&gt;tensor([5.5556e+00, 2.8571e+01, 1.0000e+05, 1.0000e+05])
</code></pre>
<p>To me it looks like it simply is not scaling the weights. However, it does work when i call the F.binary_cross_entropy:</p>
<pre><code class="lang-auto">F.binary_cross_entropy(out_AU, AUs, weight = self.cw_AU)
&gt;tensor(20765.5312, grad_fn=&lt;BinaryCrossEntropyBackward0&gt;)
F.binary_cross_entropy(out_AU, AUs)
&gt;tensor(0.6949, grad_fn=&lt;BinaryCrossEntropyBackward0&gt;)
self.cw_AU
&gt;tensor([1.0000e+00, 1.0000e+05, 1.2500e-01, 1.0000e+05, 1.0000e+05, 1.0000e+05,
        4.0000e-02, 1.6667e-01, 1.6667e-01, 3.3333e-01, 1.0101e-02, 1.9231e-02])
</code></pre>
<p>Has anyone else experienced this problem, or am i just miss-understanding the implementation</p>

username: ptrblck
cooked: <p>I guess your target might contain the same target labels for all samples in the batch, which would then remove the weighting during the normalization step.<br>
<a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">This post</a> shows you how the weighting is implemented internally and as you can see, the weighted loss will be normalized with the sum of the weights in the last step. If all labels are equal, weighting won’t have any effect.</p>
ValueError: Expected more than 1 spatial element when training, got input size torch.Size([16, 2048, 1])

username: fllci
cooked: <p>Hi all. I am using CycleGAN on 1-D tensors and keep getting this error. I searched for this error all over the blogs and everyone is saying it is because of batchnorm. Yet I only use instancenorm in my model.<br>
Some say, increase your batchsize, I started with batch size=1 and increased to different numbers, did not work. Not sure at this point what to do.</p>
<blockquote>
<p>fake_horse = gen_H(zebra)<br>
Traceback (most recent call last):</p>
</blockquote>
<blockquote>
<p>File “”, line 1, in <br>
fake_horse = gen_H(zebra)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)</p>
<p>File “C:\Users\furka\Downloads\GANs\CycleGAN\generator_model.py”, line 51, in forward<br>
x = layer(x)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)</p>
<p>File “C:\Users\furka\Downloads\GANs\CycleGAN\generator_model.py”, line 16, in forward<br>
return self.conv(x)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\container.py”, line 139, in forward<br>
input = module(input)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\modules\instancenorm.py”, line 57, in forward<br>
return F.instance_norm(</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\functional.py”, line 2325, in instance_norm<br>
_verify_spatial_size(input.size())</p>
<p>File “C:\Users\furka\anaconda3\lib\site-packages\torch\nn\functional.py”, line 2292, in _verify_spatial_size<br>
raise ValueError(“Expected more than 1 spatial element when training, got input size {}”.format(size))</p>
<p>ValueError: Expected more than 1 spatial element when training, got input size torch.Size([16, 2048, 1])</p>
</blockquote>

username: ptrblck
cooked: <p>As described in the docs:</p>
<blockquote>
<p>The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch.</p>
</blockquote>
<p>so I think this is expected as your the stats would be calculated from each sample in the temporal dimension, which won’t be possible for a single value (the <code>stddev</code> would be NaN).</p>