akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Attention layer visualize

username: Rohan_Kumar
cooked: <p>How to visualize attention layer, specifically in this code?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">bentrevett/pytorch-seq2seq/blob/master/3 - Neural Machine Translation by Jointly Learning to Align and Translate.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Neural Machine Translation by Jointly Learning to Align and Translate\n",
    "\n",
    "In this third notebook on sequence-to-sequence models using PyTorch and TorchText, we'll be implementing the model from [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473). This model achives our best perplexity yet, ~27 compared to ~34 for the previous model.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "As a reminder, here is the general encoder-decoder model:\n",
    "\n",
    "![](assets/seq2seq1.png)\n",
    "\n",
    "In the previous model, our architecture was set-up in a way to reduce \"information compression\" by explicitly passing the context vector, $z$, to the decoder at every time-step and by passing both the context vector and input word, $y_t$, along with the hidden state, $s_t$, to the linear layer, $f$, to make a prediction.\n",
    "\n",
    "![](assets/seq2seq7.png)\n",
    "\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
SMOTE for Image data

username: Griffintaur
cooked: <p>I have seen in many papers using SMOTE for image data to cater to unbalance in image data?<br>
How to use SMOTE in Pytorch? any links or gist to implement would be quite helpful<br>
Is it even make sense to use SMOTE for image data as it uses K -NN for synthetic data creation?</p>
AttributeError: 'OrderedDict' object has no attribute 'cuda'

username: pvskand
cooked: <p>I have trained my model and now want to test it but I was getting this error while testing<br>
<code>AttributeError: 'OrderedDict' object has no attribute 'cuda'</code></p>
<p>The error comes after this:<br>
<code>model = torch.load(snapshot_path) model.cuda()</code></p>
<p>Is there any mistake in my training that it is not able to find the cuda parameter, since I had done the training on GPU.</p>

username: ruotianluo
cooked: <p>How do you save your model?</p>

username: pvskand
cooked: <p><code>torch.save(model.state_dict(),''model.pth')</code></p>

username: smth
cooked: <p>if you saved the state dict, you have to load the model via <code>load_state_dict</code>. Either read the documentation or look at our examples.</p>

username: haqkiemdaim
cooked: <p>hi there!</p>
<p>Have you found the solution for this issue? What would it be ya ?<br>
Really need to know this <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

username: pvskand
cooked: <p><a class="mention" href="/u/smth">@smth</a>’s answer is the solution to this.<br>
Use - <code>model.load_state_dict(torch.load(PATH))</code> and for more details refer to the documentation (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>)</p>
pretrained=True and torch.no_grad()

username: John1231983
cooked: <p>In VGG network, I often saw the <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" rel="nofollow noopener">neural_style_tutorial</a></p>
<pre><code class="lang-auto">cnn = models.vgg19(pretrained=True).features.to(device).eval()
</code></pre>
<p>It will extract VGG features from pre-trained weight. If I train my network on the custom dataset, then to extract my network feature from the trained model. Instead of using <code>pretrained=True</code>, can I use <code>torch.no_grad()</code>?</p>
<pre><code class="lang-auto">cnn = my_model.eval()
with torch.no_grad():
     feature = cnn('fc')
</code></pre>

username: ptrblck
cooked: <p>The <code>pretrained</code> argument is passed to the initialization of the <code>vgg19</code> model, which will load the pretrained <code>state_dict</code>. If you have a custom trained model, you should load this <code>state_dict</code> manually via <code>model.load_state_dict</code>.</p>
<p><code>torch.no_grad</code> makes sure that all operations in the block won’t be tracked by Autograd.<br>
E.g. intermediate variables, which are created in your forward pass, won’t be stored and you can save some memory. It’s independent of the model or if you are using a pretrained model.</p>
<p>Could you explain your issue a bit more and where you are stuck?</p>

username: John1231983
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Thanks for valuable comments. In my case, I want to use pre-trained model to compute the  Style Loss. It is done by extract the feature from the pre-trained model such as VGG. However, I will use my custom network that also trained on ImageNet. Instead of using VGG network, I will use my network.</p>
<p>As you comment, I understand that we just change the <code>pretrained=True</code> by <code>load_state_dict</code>. Because the styleloss will be added on total loss with backward function. So, I think I should not use torch.no_grad(). Am I right? If I use torch.no_grad(), then styleloss.backward() will be useless.</p>

username: John1231983
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Please confirm my understanding</p>

username: ptrblck
cooked: <p>Sorry for the late follow-up.<br>
Yes, you are right. If you want to call <code>backward</code> on the loss, you shouldn’t wrap the forward pass in a <code>torch.no_grad</code> block.<br>
This is useful for validation, test, inference, where no <code>backward</code> will be called and thus the intermediate variables can be freed to save memory.</p>
Hook on a tensor in forward pass of a module?

username: gslaller
cooked: <p>Given the following code:</p>
<pre><code class="lang-auto">class Example(nn.Module):
  def __init__(self):
    super().__init__()
    self.l1 = nn.Linear(1,3)
    self.l2 = nn.Linear(3,2)
  
  def forward(self, x):
    mid = self.l1(x)
    res = self.l2(mid)
    return res
</code></pre>
<p>is there a way to have a hook the the <code>mid</code> variable in the forward call?<br>
something like:</p>
<pre><code class="lang-auto">example.forward.mid.register_hook(lambda x: print(x))
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Why not add <code>mid.register_hook(lambda x: print(x))</code> to your <code>forward</code> function?<br>
You can also return <code>mid</code> from the function so that the caller can add the hook.</p>

username: gslaller
cooked: <p>I was going through a code of someone else, this is what i ended up doing.</p>

username: SunQpark
cooked: <p>How about <code>example.l1.register_hook(lambda x: print(x))</code>, since <code>example.l1</code> is also an <code>nn.Module</code></p>

username: albanD
cooked: <p><code>register_hook</code> do not exist for <code>nn.Module</code>. And the closest thing that exists, <code>register_backward_hook()</code> is not working properly at the moment and should not be used.</p>

username: SunQpark
cooked: <p><code>register_forward_hook</code> takes more complex functions than <code>lambda x: print(x)</code> as input. Try this example.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class Example(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(1,3)
        self.l2 = nn.Linear(3,2)
    
    def forward(self, x):
        mid = self.l1(x)
        res = self.l2(mid)
        return res


if __name__ == "__main__":
    model = Example()
    dummy_input = torch.randn([1])
    print(dummy_input)

    def hook(module, input, output):
        print(output)
    
    model.l1.register_forward_hook(hook)
    out = model(dummy_input)
</code></pre>
1 conv layer with 2 channels output and 2 conv layers with 1 channel output. Are they same with each other?

username: Oshrin
cooked: <pre><code class="lang-auto">self.conv1 = nn.Conv2d(32, 1, 1)
self.conv2 = nn.Conv2d(32, 1, 1)
# VS
self.conv = nn.Conv2d(32, 2, 1)
</code></pre>

username: JuanFMontesinos
cooked: <p>I don’t understand the question at all. Conv is equivalent  to stack conv1 and conv2 if it’s what you are asking.</p>

username: albanD
cooked: <p>Could you be clearer about what you try to accomplish here?</p>
<p>Yes these two will have the same number of parameters.<br>
And depending on how you use them, you can get an output of the same size by using <code>torch.cat</code>.<br>
Was that the question?</p>
Save model as string

username: Philipp_Singer
cooked: <p>Hi all,</p>
<p>is there a simple way that allows me to save a trained model as some kind of string so that I can copy the string manually to a new script loading it without saving/loading files?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>In python, when you save the model, you don’t actually save the code.<br>
The state_dict only contains the weights of your network.</p>
<p>So you can keep your code and copy/paste the values of the Tensors. But that might be too many to be possible to do.</p>
Is there 'same' padding in pytorch?

username: wonchulSon
cooked: <p>I mean using 2x2 kernel, how can I match input and output shape same??</p>
<p>Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you use a kernel of even size, I don’t think there is a way to have the output size being the same with just a convolution.</p>
HTTPError while ImageNet dataset

username: vihas_makwana
cooked: <p>HTTPError Traceback (most recent call last)</p>
<p><a href="https://nptszgc59b-b5278c2458b95a70-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20191017-080046-RC00_275259420#" rel="nofollow noopener"></a> in () ----&gt; 1 DA=torchvision.datasets.ImageNet("/",download=True,transform=transforms.ToTensor())</p>
<p><a href="https://nptszgc59b-b5278c2458b95a70-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20191017-080046-RC00_275259420#" rel="nofollow noopener">/usr/lib/python3.6/urllib/request.py</a> in http_error_default(self, req, fp, code, msg, hdrs) 648 class HTTPDefaultErrorHandler(BaseHandler): 649 def http_error_default(self, req, fp, code, msg, hdrs): --&gt; 650 raise HTTPError(req.full_url, code, msg, hdrs, fp) 651 652 class HTTPRedirectHandler(BaseHandler):</p>
<p>HTTPError: HTTP Error 404: Not Found</p>

username: albanD
cooked: <p>Hi,</p>
<p>You might want to upgrade your torchvision.<br>
But as you can see in the new code <a href="https://github.com/pytorch/vision/blob/d409c1173511becdc4607cf4628d73bd0fa9ba5b/torchvision/datasets/imagenet.py#L41-L45" rel="nofollow noopener">here</a>, the dataset is not available publicly unfortunately. You will need to to download the dataset by yourself.</p>
Is there any good way to implement CP decomposition?

username: moskomule
cooked: <p>Hi all, I’d like to implement CP decomposition to downsize the tensors. As far as I know, PyTorch does not support these factorization for more than 2D. (I know for 2D there are some factorization functions)</p>
<p>CP decomposition factorize <code>I*J*K</code> tensor (we call it X) into <code>I*R, J*R, K*R</code> tensors (U, V, W respectively). Naively, I use stochastic gradient descent to get U, V and W. When composing, you need to add R tensors <code>Y_1, Y_2,...Y_R</code>, where</p>
<pre><code class="lang-auto">Y_r = outer(outer(U[:,r], V[:,r]), W[:,r]) 
</code></pre>
<hr>
<p>I implemented this naively as below,</p>
<pre><code class="lang-auto">def outer(t1, t2):
    h, w = t1.size()
    c = t2.size()[0]
    return t2.repeat(h, w, 1) * t1.unsqueeze(2).repeat(1, 1, c)

out = variable(torch.Tensor(I, J, K))
for r in range(R):
     out += outer(torch.ger(U[:, r], V[:, r]), W[:, r])
</code></pre>
<p>When X is small then it works, but when X is large OutofMemoryError is thrown because <code>Y_r</code> is <code>I*J*K</code> tensor.</p>
<p>I think there might be two way to avoid this error, first is get <code>ΣY_r</code> directly from <code>U, V, W</code> instead of composing tensors from vectors. The other is creating and adding <code>outer(torch.ger(U[:, r], V[:, r]), W[:, r])</code> effectively. But still I don’t have any idea to improve.</p>
<p>Does anyone has good solutions to this? Thank you for advance.</p>

username: Kerem_Can_Turgutlu
cooked: <p>You can use tensorly package and choose pytorch as backend. Gives same results if you were to implement it yourself and say optimize the norm with Adam.</p>

username: moskomule
cooked: <p>Thank you. Yes, recently I found tensorly. It’s super great!</p>

username: Xin_Luo
cooked: <p>You can use einsum for this situation if you need more specially operation on tensor.<br>
like this “torch.einsum(‘ir,jr,kr-&gt;ijk’, self.A, self.B, self.C)”</p>
RNN Pack Padded Sequence Hidden State

username: Rafael_R
cooked: <p>Hi,</p>
<p>When using the pack padded sequence, what should be the final lstm state that is passed to the decoder?</p>
<p>Should it be hidden or output?</p>
<pre><code class="lang-auto">packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths, batch_first=True)
output, hidden = self.text_LSTM(packed, None)
output, _ = torch.nn.utils.rnn.pad_packed_sequence(output, batch_first=True)
</code></pre>
<p>Cheers!</p>

username: InnovArul
cooked: <p>The <code>output</code> contains the output of LSTM for all timesteps for all sequences.</p>
<p>The <code>hidden</code> has two components.</p>
<ol>
<li>The output of last timestep (of the individual sequences).</li>
<li>The cell state after last timestep</li>
</ol>
<p>In my understanding, you may want to pass the first component of the <code>hidden</code> variable (<code>hidden[0]</code>) to the decoder, as it represents the encoding of the whole sentence after having seen all the words in the sentence. Again, its a design choice about your network.</p>

username: Rafael_R
cooked: <p>thanks <a class="mention" href="/u/innovarul">@InnovArul</a>  at some places like <a href="https://stackoverflow.com/questions/48831585/lstm-in-pytorch" rel="nofollow noopener">here</a> i see output[:,-1,:] passed, are hidden[0] and output[:, -1,:] the same?</p>

username: InnovArul
cooked: <p>Yes, they are same. In addition, <code>output</code> contains the output from all the timesteps, but <code>hidden[0]</code> has only the output of the last timestep.</p>
Compute the accuracy by one-hot prediction and one-hot label

username: shirui-japina
cooked: <p>Now there is:</p>
<ul>
<li>
<strong>prediction_one_hot</strong>:</li>
</ul>
<pre><code class="lang-auto">tensor([[0, 1],
        [0, 1],
        [1, 0],
        [0, 1],
        [0, 1],
        [0, 1],
        [0, 1],
        [1, 0]], dtype=torch.uint8)
</code></pre>
<ul>
<li><strong>label_one_hot</strong></li>
</ul>
<pre><code class="lang-auto">tensor([[0, 1],
        [1, 0],
        [1, 0],
        [0, 1],
        [0, 1],
        [0, 1],
        [0, 1],
        [1, 0]])
</code></pre>
<p>Then how can I compute the accuracy?<br>
This problem should not be difficult, but I have been thinking for a long time and can’t think of it. I thought there maybe a function for tensor in PyTorch can work, but I did not find that <a href="https://pytorch.org/docs/master/torch.html" rel="nofollow noopener">here</a>.</p>

username: JuanFMontesinos
cooked: <p>Scikitlearn has a good implementation but it is for numpy.<br>
Anyway you can code it yourself. The starting point <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">def classification_metric(pred_labels, true_labels):
    pred_labels = torch.ByteTensor(pred_labels)
    true_labels = torch.ByteTensor(true_labels)

    assert 1 &gt;= pred_labels.all() &gt;= 0
    assert 1 &gt;= true_labels.all() &gt;= 0

    # True Positive (TP): we predict a label of 1 (positive), and the true label is 1.
    TP = torch.sum((pred_labels == 1) &amp; ((true_labels == 1)))

    # True Negative (TN): we predict a label of 0 (negative), and the true label is 0.
    TN = torch.sum((pred_labels == 0) &amp; (true_labels == 0))

    # False Positive (FP): we predict a label of 1 (positive), but the true label is 0.
    FP = torch.sum((pred_labels == 1) &amp; (true_labels == 0))

    # False Negative (FN): we predict a label of 0 (negative), but the true label is 1.
    FN = torch.sum((pred_labels == 0) &amp; (true_labels == 1))
    return (TP, TN, FP, FN)
</code></pre>

username: shirui-japina
cooked: <p>Thanks for your help.<img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=9" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
How to use BERT for ENTITY extraction from a Sequence without classification in the NER task?

username: Ajj
cooked: <p>My requirement here is given a sentence(sequence), I would like to just extract the entities present in the sequence without classifying them to a type in the NER task. I see that BERT has BertForTokenClassification for NER which does the classification.</p>
<p>Can somebody give me an idea of how to do <strong>entity extraction/identification using BERT?</strong></p>
Loading model on GPU

username: Rutu_Pandya
cooked: <p>It currently takes approximately 8-9 seconds to execute the following code:</p>
<pre><code>        device = torch.device("cuda:0")
        self.model = self.model.to(device)
        self.model.load_state_dict(torch.load(self.validate_model))
        for parameter in self.model.parameters():
            parameter.requires_grad = False
</code></pre>
<p>The actual inference time is only 3 seconds, where the model segments a structure on 74 2d images of size 512x512.</p>
<p>Is there any way to speed up the model loading on the GPU?</p>

username: ptrblck
cooked: <p>If your <code>state_dict</code> contains CPU tensors, you’ll trigger the host to device copy twice: first for the model parameters, then again for the <code>state_dict</code>.<br>
If that’s the case, you could first restore the model and push it only once to the GPU.</p>
<p>How often are you resetting the model that you worry about the performance?<br>
Usually you’ll setup the model once and the training will take much longer than the initialization.</p>

username: Rutu_Pandya
cooked: <p>Thanks for your response ptrblck!<br>
This is actually part of the inference, and not training.<br>
Originally the entire model was in a .tar format and was being loaded using torch.nn.DataParallel. But since we only had 1 GPU and are trying to cut down as much time as possible, I re saved the checkpoint only using the following:</p>
<p>checkpoint = torch.load(self.validate_model)<br>
self.model.module.load_state_dict(checkpoint[‘state_dict’])<br>
torch.save(self.model.module.state_dict(), path)</p>
<p>And then loading this newly saved state dict for inference without using torch.nn.DataParallel.</p>
<p>Is this not the right way to approach this? I am new to PyTorch.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Your approach sounds right. However, you would only need to load the model once during the startup of your inference application or are you recreating the model for each prediction?</p>
The speed of tensor to device

username: auroua
cooked: <p>I am running two different segmentation models. The input have the same size and data type. But the speed of the following code is quite different.<br>
What caused the difference.<br>
<code>Model 1 code</code></p>
<pre><code class="lang-auto">            # images  torch.float32 cpu False torch.Size([2, 3, 480, 480]) 
           # targets torch.int64 cpu False torch.Size([2, 480, 480])
            end2 = time.time()
            images = images.to(self.device)
            targets = targets.to(self.device)
            data_to_device = time.time() - end2  
</code></pre>
<p>The <code>data_to_device</code> of this model is 0.0568695068359375.</p>
<p><code>Model 2 code</code></p>
<pre><code class="lang-auto">        print(images.dtype, images.device, images.requires_grad, images.size())
        print(target_mask.dtype, target_mask.device,
              target_mask.requires_grad, target_mask.size())
        end2 = time.time()
        images = images.to(device)
        target_mask = target_mask.to(device)
        data_to_device = time.time() - end2
        print('=============', data_to_device)
</code></pre>
<p>The output of model 2 from the first iteration is</p>
<pre><code class="lang-auto">2019-07-14 18:23:21,835 agfcoo.trainer INFO: Start training
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.0012090206146240234
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.5653738975524902
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.28435635566711426
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.26169490814208984
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.2824215888977051
torch.float32 cpu False torch.Size([2, 3, 480, 480])
torch.int64 cpu False torch.Size([2, 480, 480])
============= 0.2888615131378174
</code></pre>
<p>This computer have one RTX 2060.<br>
The pytorch version is 1.1.<br>
The cuda is 10.0.</p>
<p>What caused the difference?</p>

username: Luoplus7
cooked: <p>Hi, I have the same question with you.<br>
Same data but very different speed with pytorch1.0.<br>
Have you solved it?</p>

username: ptrblck
cooked: <p>Since CUDA operations are executed asynchronously, you should synchronize before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.</p>
<p>Could you add it and profile the code again?</p>
Accessing tensor elements using another "index" tensor

username: stevethesteve
cooked: <p>Let’s say I have a tensor A of shape (BatchxDim1xDim2x…) and a tensor B of shape (Batch)<br>
B = Tensor([1,3,7,2,…])<br>
that encodes indices that I am interested in.<br>
What is the best way to obtain a tensor C such that<br>
C[i,j,k,…] = A[i,B[i],j,k,…]  ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <a href="https://pytorch.org/docs/stable/torch.html#torch.gather" rel="nofollow noopener">torch.gather</a> function is built just for that.<br>
You might have to expand your B to get the behavior you want here.</p>

username: stevethesteve
cooked: <p>Yes, this works. Thanks a lot!</p>
Create a lmdb DataLoader

username: Mendel123
cooked: <p>I’m trying to load a LMDB dataset with <code>torch.utils.data.Dataloader</code> like below.</p>
<pre><code class="lang-python">class LmdbDataset(Dataset):
    def __init__(self,lmdb_root):
        super().__init__()
        self.env = lmdb.open(lmdb_root, max_readers=32, readonly=True)
# create dataloder
 data = LmdbDataset('train')
loader = DataLoader(data,sampler=DistributedSampler(data,shuffle=True),num_workers=16)
#run 
</code></pre>
<p>However,  some erros occur.</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:
    for batch_idx, batch in enumerate(dataloader):
  File "/miniconda/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 278, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/miniconda/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 682, in __init__
    w.start()
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/miniconda/envs/py36/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle Environment objects

</code></pre>
<p>So could anyone give me some advice to solve this problem?<br>
Thanks very much.</p>

username: alex.veuthey
cooked: <p>See <a href="https://github.com/pytorch/examples/issues/526" rel="nofollow noopener">this issue</a>.</p>
requires_grad=False get Error

username: Bing_Ponng
cooked: <p>I try to set some layer freeze in my network, when I freeze it and run it on a single GPU it will work fine. However, when I try to run it on multiple GPU I will get " get RuntimeError: you can only change requires_grad flags of leaf variables.".</p>

username: JuanFMontesinos
cooked: <p>For me it sounds (I have never seen that error), that you are freezing layer after calling dataparallel. I would say that replicated models are pointing to original model and that’s why they are no longer leaf variables. Thus, when you try to freeze the model it just throws that error.<br>
Could you try to freeze weights before calling dataparallel?</p>
Colab RuntimeError: "CUDA error: invalid device function"

username: BoltzmannBrain
cooked: <p>Google colab notebook comes packaged with PyTorch and GPU support – see <a href="https://stackoverflow.com/a/57782206/3765905" rel="nofollow noopener">this related SO answer</a>, and then you should go change notebook runtime to GPU from CPU (from the top Colab menubar).</p>
<p>Yet trying to train a model from GPyTorch always fails on</p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/gpytorch/utils/cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     19     """
     20     try:
---&gt; 21         L = torch.cholesky(A, upper=upper, out=out)
     22         # TODO: Remove once fixed in pytorch (#16780)
     23         if A.dim() &gt; 2 and A.is_cuda:

RuntimeError: CUDA error: invalid device function
</code></pre>
<p>Why?</p>
<p>Relevant info:</p>
<pre><code class="lang-auto">import torch
torch.cuda.is_available()

True
</code></pre>
<pre><code class="lang-auto">print(torch.__version__)
print(gpytorch.__version__)
print(torch.backends.cudnn.version())

1.3.0+cu100
0.3.6
7603
</code></pre>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue?</p>

username: BoltzmannBrain
cooked: <p>Running <a href="https://github.com/cornellius-gp/gpytorch/blob/master/examples/08_Deep_Kernel_Learning/Deep_Kernel_Learning_DenseNet_CIFAR_Tutorial.ipynb" rel="nofollow noopener">this GPyTorch notebook</a> will do the trick.</p>
DistributedDataParallel consumes much more gpu memory

username: turboxin
cooked: <p>Would DistributedDataParallel wrapper cost much GPU memory? In my case, the model cost around 7300MB when loaded into a GPU. However, when wrapped in DistributedDataParallel and run in the distributed mode, it costs 22000MB GPU momery.<br>
Is it caused by the DistributedDataParallel wrapper? Are there any methods to save memory usage? Thanks!</p>

username: ptrblck
cooked: <p>That’s a huge increase in memory. Which model and batch sizes are you using?</p>

username: turboxin
cooked: <p>I am using pointpillars from this repo <a href="https://github.com/traveller59/second.pytorch" rel="nofollow noopener">https://github.com/traveller59/second.pytorch</a>, with DataParallel changed to DistributedDataParallel, the batch size is 2 per gpu.</p>

username: pritamdamania87
cooked: <p>How many nodes are you using and how many GPUs per node? Also, which communication backend are you using?</p>
<p>Also, it might be helpful to debug if you could share the code you’re using to initialize and train using DistributedDataParallel?</p>
Is there a cross_entropy in c++?

username: markl
cooked: <p>I found <code>nll_loss</code> but I didn’t find <code>cross_entropy</code> in this list <a href="https://pytorch.org/cppdocs/api/namespace_at.html#functions" rel="nofollow noopener">https://pytorch.org/cppdocs/api/namespace_at.html#functions</a></p>
<p>Is it missing? Also - what is <code>binary_cross_entropy</code>?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> is calling <code>F.log_softmax</code> and <code>F.nll_loss</code> internally as described <a href="https://discuss.pytorch.org/t/code-for-loss-function-without-c-aten-implementation/58725/2">here</a>.<br>
As mentioned in the linked topic, <a class="mention" href="/u/yf225">@yf225</a> is actively coordinating the development of the C++ API.</p>
<p><code>binary_cross_entropy</code> is used for binary or multi-label classification use cases.</p>
Cross entropy loss on text generation

username: Rohan_Kumar
cooked: <p>I was trying to read up on some seq to seq models for translation, and i saw that in a very common model, the loss was used as cross entropy loss and the way it was used was</p>
<p>dimension sizes -&gt;</p>
<pre><code class="lang-auto">trg = [(trg sent len - 1) * batch size]
output = [(trg sent len - 1) * batch size, output dim]
</code></pre>
<p>where the output dim was the target vocab size. now my question is how is this loss working given the dimensions are not equal, what is it computing and how it is computing?</p>
<p>my one theory is that they perform output.argmax(1) and then compute the loss.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects the output tensor to contain the class logits in dim1, while this particular dimension is missing in the target tensor, which contains the class indices.</p>
<p>A vanilla multi-class classification output would therefore be defined as <code>[batch_size, nb_classes]</code>, while the target would only be <code>[batch_size]</code>.</p>
<p>For a segmentation use case, you could use <code>[batch_size, nb_classes, height, width]</code> for the output and <code>[batch_size, height, width]</code> for the target.</p>
<p>In your example, it seems the temporal dimension was collapsed with the batch size into dim0.</p>
Using the model in real life

username: Rohan_Kumar
cooked: <p>So, i created this model for translation that works nice enough, now i want to use as a “product”, like i want to be able to input a sentence and compute it’s translation, how do i go about doing that, do i make a custom data loader for it, wouldn’t i run into trouble with drop_last = True for data loader.</p>

username: ptrblck
cooked: <p>How would you like to deploy your model?<br>
If you would like to feed the inputs “live” , you wouldn’t necessarily need a <code>DataLoader</code>.<br>
<a class="mention" href="/u/jspisak">@jspisak</a> mentioned Flask as a valid approach <a href="https://discuss.pytorch.org/t/deploy-pytorch-1-0-into-production-flask-python-vs-nodejs-c-addons/56402/2">here</a>, if you don’t need low latency and also highlights a proposal of Amazon for a high performance PyTorch C++ serving platform.</p>
Output of torch.autograd.grad

username: Harkirat_Behl
cooked: <p>What is the return value of torch.autograd.grad function? The common way is to just use it like this:<br>
gradients = torch.autograd.grad(loss, model.parameters())</p>
<p>which is fine. But I have also seen people using this:<br>
(gradients,) = torch.autograd.grad(loss, model.parameters())</p>
<p>I just want to know the exact return values of the grad function.</p>

username: albanD
cooked: <p>The autograd.grad function returns an object that match the <code>inputs</code> argument.<br>
When you do <code>(foo,)</code> you unpack a tuple of size 1 into the variable <code>foo</code>: <code>(foo,) = (2,)</code> will make <code>foo == 2</code>.</p>

username: Harkirat_Behl
cooked: <p>Thanks Alban. hope you are doing good.</p>
Using dataparallel and cuda with non trainable vars?

username: tueboesen
cooked: <p>I have the following very simple resnet that I currently use:</p>
<pre><code class="lang-auto">class ResNetSimple(nn.Module):
    def __init__(self, block, layers, channels):
        super().__init__()
        assert len(layers) == 3
        layers_ = []
        chan = channels
        for sublayers in layers:
            layers_.append(block(chan,chan+1))
            chan += 1
            for layer in range(sublayers):
                layers_.append(block(chan, chan))
        self.layers = nn.Sequential(*layers_)
        self.channels_out = chan

    def forward(self, x):
        x = self.layers(x)
        return x



class ResnetBlock(nn.Module):
    def __init__(self, channels_in,channels_out,dt=0.1):
        super().__init__()
        self.dt = dt
        self.nl = nn.Tanh()
        self.linear = nn.Linear(channels_in,channels_out)
        self.channels_in = channels_in
        self.channels_out = channels_out
        if channels_in != channels_out:
            self.scaling = torch.zeros((channels_in,channels_out))
            for i in range(min(channels_out,channels_in)):
                self.scaling[i,i] = 1

    def forward(self, x):
        a = self.linear(x)
        a = self.nl(a)
        if self.channels_in == self.channels_out:
            res = x + self.dt * a
        else:
            res = x @ self.scaling + self.dt * a
        return res
</code></pre>
<p>Afterwards I call:</p>
<pre><code class="lang-auto">descriptor = nn.DataParallel(descriptor).cuda()
</code></pre>
<p>where descriptor is the network defined above.</p>
<p>Now the problem comes when I actually run data through this network, since self.scaling does not get transferred to the gpu with the above. How do I ensure that this parameter gets correctly transferred as well in the above?</p>

username: ptrblck
cooked: <p>Since <code>scaling</code> doesn’t require gradients, you should register this tensor as a buffer via</p>
<pre><code class="lang-python">self.register_buffer('scaling', torch.zeros(...))
</code></pre>
<p>This will make sure to push this tensor to the specified device(s).</p>

username: tueboesen
cooked: <p>That worked perfectly, thank you!</p>
Num_workers>0 creates memory error in SLURM?

username: Vedant_Sanil
cooked: <p>Hi,</p>
<p>I’m trying to run my code on a SLURM cluster with the following configuration - 1 node, 2 Nvidia 1080Ti GPUs, 8 CPUs and 8GBs of RAM per CPU.</p>
<p>I’m implementing ResNeXt, with a dataset that contains about 1million 32x32 images. When I try running this code with torchvision.datasets.ImageFolder and num_workers = 4-8, it throws an “exceeded virtual memory” error by requesting for 341GB of data! That seems a little absurd. This error is thrown at the first trainLoader loop as it is preparing the first batch for the code.</p>
<p>Initially, I assumed this was an error with my program, but my program works just fine with num_workers = 8 on Google Colab. My program only works when I set num_workers=0. At num_workers=2, it works for 2 epochs before throwing the same error. Any solution for this would really be appreciated.</p>

username: pritamdamania87
cooked: <p>Are you using DistributedDataParallel or DataParallel for this? It seems like your question is more related to torchvision or the PyTorch dataset/dataloader (the ‘distributed’ tag is not appropriate for this). Maybe tag this with ‘vision’?</p>
Speech to text in pytorch

username: SHASHANK_KUMAR_MISHR
cooked: <p>I have to create a reusable library that can convert a paragraph of spoken english to written english. For example, “two dollars” should be converted to $2. Abbreviations spoken as “C M” or “Triple A” should be written as “CM” and “AAA” respectively. How to proceed with this problem in pytorch.</p>

username: PCerles
cooked: <p>Not really a question that calls for neural nets, as verbalization is a very rule-based problem. I suggest you take a look at <a href="https://github.com/google/sparrowhawk" rel="nofollow noopener">https://github.com/google/sparrowhawk</a>, which tackles the inverse problem ($2 --&gt; two dollars). Not too hard to flip stuff around after spending time learning about FSTs. There is some work on doing this with neural nets but they tend to be trained on corpuses generated by rule-based grammar methods.</p>
Dataloader is slow when iter's number is a multiple of workers

username: nero
cooked: <p>In gereral, the speed of dataloader is very fast (0.001 s/batch), but when the number of iterations is a multiple of workers, It’s getting very slow (80 s/batch).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6421af6c642b440add808d4f66912fc979600663.png" data-download-href="https://discuss.pytorch.org/uploads/default/6421af6c642b440add808d4f66912fc979600663" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6421af6c642b440add808d4f66912fc979600663.png" alt="image" data-base62-sha1="ehNUSjqxQM0r0zsv4eFhMxLULJN" width="690" height="455" data-dominant-color="100F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">702×463 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: wen_lei
cooked: <p>I have the same question.</p>

username: Mughees
cooked: <p>facing the same issue</p>

username: Mughees
cooked: <p>Resolved the issue by moving to SSD. Working fine now.</p>
Are there any plans for ragged tensors in pytorch?

username: Santosh-Gupta
cooked: <p>Ragged tensors are useful in cases where labels have a varying length of inputs. For example, graph embeddings may vary in how many nodes they are connected to. Another example is Bert: the inputs may be a series of texts, which usually are not uniform in length, so inputs of a particular batch need to be padded to match the maximum sample length.</p>
<p>Both keras and original tensorflow have ragged tensors, I am wondering if there are any plans to introduce these to Pytorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>We are currently implementing NestedTensors that are similar. You can find the RFC and progress in that issue: <a href="https://github.com/pytorch/pytorch/issues/25032" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25032</a></p>
Regression with large target differences

username: mshuaibi
cooked: <p>I’m dealing with a dataset that consists of 100 images with target values around (0-10), when I train my ML model on this I get proper convergence to an RMSE of 0.001. However, when I need to retrain this model and include a few data points orders of magnitude larger(1k-10k), I get, unsurprisingly, significant large losses that start around an RMSE of 300+ and decay to at most 5. Does anyone have any thoughts/approaches to dealing with regression models that contain target data that vary on orders of magnitude? Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure, if your new dataset only contains values in the larger range or just a few samples.<br>
In the former case, you could try to normalize the target to a specific range (e.g. z-score) and revert the normalization for the actual prediction for your validation and test set.</p>
<p>In the latter case, this approach would also be possible, but I’m afraid the smaller values might be treated as noise and you might overfit on the larger values.</p>

username: mshuaibi
cooked: <p>The scenario is the latter of the cases. I have already tried scaling the targets, but as you mentioned - the small large values dominate and the smaller values despite being the majority are treated as noise. Not too sure if batch normalization would help here? Or possibly working in logspace?</p>
Memory/computation on cuda / cpu

username: Faolian
cooked: <p>I’m having a problem of running out of cuda memory during training, and some parts of it seem a bit odd, I’ll explain the current setup and my thoughts of how to fix it, but I have a feeling it will not work.</p>
<p>Running on cuda i have<br>
I have 3 networks, policy, q1, q2. The q networks in particular are quite large and  fairly complex<br>
A tensor for an entropy param (it’s a size(1) tensor)</p>
<p>When entering the train loop, i sample a batch of size batch_size from a replay buffer. This is used with target nets on cpu with no grad to calculate q_target values.</p>
<p>I then move the relevant parts of the replay buffer to cuda and run them into my q1 network. I move the q_target results (and yes these are fully detached, just tensors of values) to cuda also.</p>
<p>I do mse_loss between the q1 outputs and the q_targets.</p>
<p>I then call backward() on the loss, then optim.step(), and i get this error<br>
RuntimeError: CUDA out of memory. Tried to allocate 530.00 MiB (GPU 0; 4.00 GiB total capacity; 2.32 GiB already allocated; 518.99 MiB free; 9.51 MiB cached)</p>
<p>What I’m finding curious is that this is roughly the same values even if i bring the batch size down from 100 to 5, and similarly still happens if I remove the q2 network.</p>
<p>This suggests to me that largely the data tensors and networks sitting on gpu dont affect the memory usage much, but the size of the computation graph / optimisation is what’s causing the problem?</p>
<p>If this is the case and i simply cannot run a network of this size on cuda i do have vastly more space on cpu, but since i use the q network to evaluate the policy network (with the reparametization trick) i assume gradient/computation graphs do not work across devices? And if I moved or created a copy of the q network on cuda to run this it would presumably now being in even larger computation graph fail also?</p>
<p>Any thoughts on how to resolve this would be greatly appreciated.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can actually move stuff between cpu and cuda and the gradients will still be computed properly.<br>
Note that the backward ops are performed on the same device where the forward op was done.<br>
So moving the result of your gpu computation to cpu won’t reduce the cpu usage.</p>

username: Faolian
cooked: <p>So, to clarify, this should mean that if my loss function for the policy network were just loss = q1(state, policy_output), where policy_output was calculated on gpu but q1 was on cpu, i would move the policy_output to cpu (this would be required for it to function I think?), and then a call of loss.backward() would correctly backprop through the policy network?</p>
<p>My thanks for your help</p>

username: albanD
cooked: <p>Yes, you will have to move policy_output to the cpu yourself (pytorch never move stuff between the cpu and gpu automatically). But then yes the backprop will work just fine. Note that the q1 backprop will be computed on the cpu and the policy network one will be computed on the gpu in your particular case.</p>
Is multiplication by 1 a no-op

username: vadimkantorov
cooked: <p>If I do sth like <code>x * (y if condition else 1)</code>, will multiplication by 1 skip copying / allocating new memory?</p>

username: tom
cooked: <p>No, you would need to do <code>x * y if condition else x</code> (if short circuits, so x * y will not be evaluated if the condition is false).</p>
<p>Experiments:<br>
Multiplication is not a no-op</p>
<pre><code class="lang-python">x = torch.randn(12)
y = x * 1
print(x.data_ptr(), y.data_ptr())
</code></pre>
<p>if short-circuits:</p>
<pre><code class="lang-python">x = []
print(x[1] if len(x) &gt; 0 else None)
</code></pre>
<p>Note that skipping allocation is not a good option here:</p>
<ul>
<li>
<code>== 1</code> is fishy for floating point (see the thread with a mask the other day),</li>
<li>people will use + 0 or * 1 for getting copies…</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: vadimkantorov
cooked: <p>Yeah, maybe fixing this would require the user also casing for 1, if an inplace op is used after multiplication.</p>

username: tom
cooked: <p>Let’s just say there is nothing to fix. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Train,test validation

username: SHASHANK_KUMAR_MISHR
cooked: <p>I have a  data frame in which two columns are there text_data and distractor. How to divide in train,test and validation in pytorch.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d9b53f010f976da5f254e34ff675f5f241af750.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8d9b53f010f976da5f254e34ff675f5f241af750" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d9b53f010f976da5f254e34ff675f5f241af750_2_690x218.jpeg" alt="Capture" data-base62-sha1="kcI6VHAWL9JUAkQYPNxgPWqVMKk" width="690" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d9b53f010f976da5f254e34ff675f5f241af750_2_690x218.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d9b53f010f976da5f254e34ff675f5f241af750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d9b53f010f976da5f254e34ff675f5f241af750.jpeg 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">985×312 43.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>If you’ve already created a <code>Dataset</code> instance for your data frame, you could use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="nofollow noopener">torch.utils.data.random_split</a> to create non-overlapping <code>Datasets</code>.<br>
Alternatively, you could also split the data before creating the <code>Dataset</code>s e.g. via <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" rel="nofollow noopener">sklearn.model_selection.train_test_split</a> or just by manually indexing the data frame.</p>