akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compute gradient for conditionals?

username: avani_shukla
cooked: <p>I want to create mask from score threshold and multiply it with weight matrix.<br>
To create mask I’m using condition shown in code. but for all score parameters grad are zero.<br>
How to compute gradient for score parameters ?</p>
<pre><code class="lang-auto">class SuperMaskMLP(torch.nn.Module):
  
    def __init__(self,num_features,hidden_nodes_list,num_classes):
        super(SuperMaskMLP, self).__init__()
        
        num_hidden_layes = len(hidden_nodes_list)
        self.hidden = torch.nn.ModuleList()
        self.score = torch.nn.ModuleList()

        if num_hidden_layes==0:
          self.hidden.append(torch.nn.Linear(num_features, num_classes))
          self.score.append(torch.nn.Linear(num_features, num_classes))
        else:
          self.hidden.append(torch.nn.Linear(num_features, hidden_nodes_list[0]))
          self.score.append(torch.nn.Linear(num_features, hidden_nodes_list[0]))
          for k in range(num_hidden_layes-1):
              self.hidden.append(torch.nn.Linear(hidden_nodes_list[k], hidden_nodes_list[k+1]))
              self.score.append(torch.nn.Linear(hidden_nodes_list[k], hidden_nodes_list[k+1]))      
          self.hidden.append(torch.nn.Linear(hidden_nodes_list[num_hidden_layes-1], num_classes))
          self.score.append(torch.nn.Linear(hidden_nodes_list[num_hidden_layes-1], num_classes))

        
    # input : features
    # output: logits , probabilities
    def forward(self, x):
        out = x
        i=0
        for layer in self.hidden[:-1]:
          wmask = self.score[i].weight.clone()
          wmask[self.score[i].weight &gt;= (self.score[i].weight.mean())] = 1
          wmask[self.score[i].weight &lt; (self.score[i].weight.mean())] = 0
          bmask = self.score[i].bias.clone()
          bmask[self.score[i].bias &gt;= (self.score[i].bias.mean())] = 1
          bmask[self.score[i].bias &lt; (self.score[i].bias.mean())] = 0 
          w = layer.weight * wmask
          b = layer.bias * bmask
          out = torch.matmul(out,torch.t(w)) + (b)
          out = F.relu(out)
          i+=1

        wmask = self.score[i].weight.clone()
        wmask[self.score[i].weight &gt;= (self.score[i].weight.mean())] = 1
        wmask[self.score[i].weight &lt; (self.score[i].weight.mean())] = 0
        bmask = self.score[i].bias.clone()
        bmask[self.score[i].bias &gt;= (self.score[i].bias.mean())] = 1
        bmask[self.score[i].bias &lt; (self.score[i].bias.mean())] = 0 
        w = self.hidden[-1].weight *wmask
        b = self.hidden[-1].bias * bmask
        logits = torch.matmul(out,torch.t(w)) + (b)
        probas = F.log_softmax(logits, dim=1)
        return 1,logits, probas
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The derivative tells you how much the output will change for a very small change in the input.<br>
If you use your weights in a condition like that. Changing them a little won’t change the output at all. So all the gradient will be 0.<br>
So all zeros are the right gradients here I’m afraid.</p>
Subblocks of Node

username: almeetb
cooked: <p><code>torch/csrc/jit/ir/ir.h</code> explains the concept of sub-blocks of a node and state that inputs to a block that represent control flow act as equivalent phi-nodes by defining a new Value to represent any term that has multiple definitions depending on how control flowed.</p>
<p>Is there an example where I can see the creation of these new Values?</p>
<p>Thanks!</p>

username: Michael_Suo
cooked: <p>The easiest way to see example IR is to compile a small torchscript snippet using <code>torch.jit.script()</code> and calling <code>.graph</code> on the resulting object.</p>
Numpy operations in pytorch autograd

username: ku294714
cooked: <p>Hello All,</p>
<p>I have come across a situation where i have a function written in matlab and i call it in my pytorch model to perform final computation on my model output. But during back propagation, my model parameter’s gradients are not calculated.</p>
<p>I reproduced the issue with a toy example for better understanding:</p>
<p>import torch<br>
import numpy as np<br>
def b_func(x):<br>
abc = torch.tensor([[1]],dtype = torch.float)<br>
x = abc + x<br>
return x<br>
X = torch.tensor([[2]],dtype = torch.float, requires_grad = True)<br>
W = X ** 2<br>
Y = W ** 3<br>
print(Y)<br>
Z = b_func(Y)<br>
print(Z)<br>
Z.backward(retain_graph=True)<br>
print(X.grad)</p>
<p>This is the expected behavior of pytorch and there is no issue here. However, when i perform the below code:</p>
<p>def a_func(x):<br>
abc = np.array([[1]])<br>
x = x.detach().numpy()<br>
x = abc + x<br>
x = torch.tensor(x, requires_grad = True)<br>
return x<br>
XX = torch.tensor([[2]],dtype = torch.float, requires_grad = True)<br>
WW = X ** 2<br>
YY = W ** 3<br>
print(Y)<br>
ZZ = a_func(Y)<br>
print(ZZ)<br>
ZZ.backward(retain_graph=True)<br>
print(XX.grad)</p>
<p>What changes shall i make for gradients to flow in this scenario. Apart from using pytorch tensors.</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you use .detach(), you break the graph and so gradients cannot be computed anymore.</p>
<p>But the root issue is that for the autograd to work, the autograd engine needs to be able to know how to compute the gradient for each operation that is done.<br>
Unfortunately, if you don’t use pytorch ops, it cannot know how to compute the gradients and so you won’t be able to get gradients.</p>
<p>You will either have to re-implement it using pytorch operations.<br>
Or you will have to write a custom function where you tell the autograd how to compute the backward for that part where it does not know how to do it. See doc <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a> on how to do this.</p>

username: ku294714
cooked: <p>Thank <a class="mention" href="/u/alband">@albanD</a> for quick response. I had that those solutions in my mind, I wish there was a workaround.<br>
Thank you for the response, it makes things more clear now.</p>
How to divide the dataset when it is distributed

username: sunshk1227
cooked: <p>Hi everyone!</p>
<p>I am a beginner in PyTorch. Now I want to divide a dataset into two parts: the train set and validation set when using <code>torch.distributed</code>. I know that on a single GPU  I can do this using a sampler:</p>
<pre><code class="lang-auto">indices = list(range(len(train_data)))
train_loader = torch.utils.data.DataLoader(
      train_data, batch_size=args.batch_size,
      sampler=torch.utils.data.sampler.SubsetRandomSampler(indices[:split]),
      pin_memory=True, num_workers=2)
</code></pre>
<p>But when I want to train it in a parallel way using <code>torch.distributed</code>, I have to use another sampler, namely, <code>sampler =  torch.utils.data.distributed.DistributedSampler(train_data)</code></p>
<p>So how should I do to use the two samplers, so that I can divide the dataset and distribute it at the same time?</p>
<p>Thank you very much for any help!</p>

username: sunshk1227
cooked: <p>Yeah, I find a solution with the help of Szymon Maszke. Use <code>torch.utils.data.random_split</code> instead. Namely,</p>
<pre><code class="lang-auto">train_data, val_data = torch.utils.data.random_split(
      train_data, (num_train, num_val))
</code></pre>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> just in case if there are different suggestions.</p>
Multiply sparse tensor with dense tensor on GPU

username: dkoutsou
cooked: <p>I have the following implementation of page rank using sparse tensors:</p>
<pre><code class="lang-auto">i = torch.LongTensor(idx)
values = torch.FloatTensor([1] * len(idx))
M = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571, 4847571]))
N = M.shape[1]
v = torch.rand(N, 1).float()
values = torch.FloatTensor([(1 - self.d)/N] * len(indices))
temp = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571,
                                                                   4847571]))
if torch.cuda.is_available():
     v = v.cuda()
     M = M.cuda()
     temp = temp.cuda()

v = v / torch.norm(v, 1)
M_hat = self.d * M + temp
for i in range(self.num_iter):
     v = torch.mm(M_hat, v)
</code></pre>
<p>On CPU everything runs fine. On GPU I am getting the following error:</p>
<pre><code class="lang-auto">    v = torch.mm(M_hat, v)
RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch.
</code></pre>
<p>Is this a known issue? Should I do something differently?</p>
<p>This is my configuration:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2
OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2
Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 440.82
cuDNN version: Could not collect
Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.5.0
</code></pre>
<p>Thanks very much for the help!</p>

username: ptrblck
cooked: <p>Could you try to use <a href="https://pytorch.org/docs/stable/sparse.html#torch.sparse.mm"><code>torch.sparse.mm</code></a>, which is the equivalent of <code>torch.mm</code> for a sparse and dense matrix?</p>
<p>Also, could you post all shapes and values of the tensors so that we can reproduce this error?<br>
The error message isn’t really helpful in this case.</p>

username: dkoutsou
cooked: <blockquote>
<p>Also, could you post all shapes and values of the tensors so that we can reproduce this error?<br>
The error message isn’t really helpful in this case.</p>
</blockquote>
<p>Yes, sorry about that but it’s really hard to give you the weights since they were parsed using <a href="https://snap.stanford.edu/data/soc-LiveJournal1.html" rel="noopener nofollow ugc">this</a> graph. The shapes can be deduced from the code though. M and M_hat are (4847571, 4847571) and v is (4847571, 1).</p>
<p>Actually the error is the same as <a href="https://github.com/pytorch/pytorch/issues/37583" rel="noopener nofollow ugc">this</a> one and it has been solved in the current master.</p>
<p>Thanks for taking the time though!</p>
Slicing with assignment in Libtorch - Pytorch C++

username: bjornamr
cooked: <p>I have some code I am trying to convert to C++. What I am trying to understand is to convert this bit of code:</p>
<pre><code class="lang-auto">import torch
# first slice

X = torch.randn(32, 4468)
cond_mask = X &lt;= 0
keep_val = torch.randn(32, 4468)

Y = torch.randn(32,4468)
Y[cond_mask] = keep_val[cond_mask]

# second slice
Z = torch.randn(32, 4468)
batch = 32
idx_l = torch.arange(0, 32, dtype=torch.long)
label = torch.tensor([3962, 3962, 1768, 1768, 3532, 3532,  245,  245,  201,  201,  900,  900,
        4176, 4176, 1539, 1539, 1932, 1932, 4017, 4017, 1807, 1807, 4211, 4211,
        4439, 4439,  701,  701,  554,  554, 1829, 1829])

Z[idx_l, label] = Y[idx_l, label]
</code></pre>
<p>I did not know what to do with the first slice, but I tried with the second.<br>
I tried this on the second slice in c++, but this did not succeed.</p>
<pre><code class="lang-auto">auto tensoroptions = torch::TensorOptions().dtype(torch::kInt32);
auto idx_l = torch::arange(0, 32).to(tensoroptions);
Z.index_put({idx_l, label}, Y.slice(idx_l, label));
</code></pre>
<p>The reason i tried to change the datatype was to avoid thist error, but I still got the same error.</p>
<blockquote>
<p>error: no viable conversion from ‘at::Tensor’ to<br>
‘int64_t’ (aka ‘long long’)<br>
output.index_put({idx_, label}, cos_theta_m.slice(idx_, label));</p>
</blockquote>
<p>Hopefully someone can explain to me the way to go with slicing with assignment in C++.</p>
<p>Thank you in advance.</p>
Unresonable GPU memory consumption when truncating padding tokens

username: dkajtoch
cooked: <p>I am training pytorch model with Contrastive Loss (pytorch-metric-learning) and there are two ways I can batch text examples. In the first method, I am padding only up to maximal sequence length in a batch. In the second case, I am padding always up to 510 tokens no matter what the sequence size is. In general, tensor size in the first method is always smaller or equal to the second method. Yet, what I am observing is that smaller batch (the whole training) is actually consuming more memory than the constant, larger batch. Additionally, it produces CUDA out of memory error at some point in training (during a backward call). Green line in the Figure corresponds to memmory consumption for a smaller, dynamic batch. Grey curve for larger, constant size batch. I would expect an inverse situation. What is going on? How to even debug this kind errors?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/816698138e52de78e1a25a043b406e6c02887dcc.png" alt="image" data-base62-sha1="isJoPdZ7MGQnfllFsndDblZqyYs" width="359" height="245"></p>
<p>I am training with pytorch-lightning, pytorch==1.5.0 and amp O2 (cuda memory also shows up without amp)</p>

username: ptrblck
cooked: <p>Is the padding the only difference between these approaches?<br>
If so, could you show a code snippet to demonstrate this behavior?<br>
My first guess would be memory fragmentation, but that’s way too large and the memory should also be reused, so you shouldn’t run out of memory.</p>

username: dkajtoch
cooked: <p>Thanks for answer <a class="mention" href="/u/ptrblck">@ptrblck</a>! Yes, padding is the only difference in this configuration. I am using huggingface transformers library and noticed that the amount of padding actually changes slightly loss value due to dropout. Anyway, I will try to prepare full example using some public dataset (I can’t share company’s data) and reach out to you.</p>
How to use Visdom to visualize the training process?

username: ky_Pa
cooked: <p>I ran the program on the cloud server and opened Visdom before training the model. Why can’t I open my website? How can I see the training process through Visdom?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c8ff0f761793ca19d5bb27cc6accf655d5a097f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c8ff0f761793ca19d5bb27cc6accf655d5a097f" title="微信图片_20200514194230"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c8ff0f761793ca19d5bb27cc6accf655d5a097f.png" alt="微信图片_20200514194230" data-base62-sha1="6mdoTcOOXHRAcSGPMBftv5bKmhF" width="690" height="478" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200514194230</span><span class="informations">717×497 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52e69f9fbf5845ea8e9cd48bf88a2642cbc34215.png" data-download-href="https://discuss.pytorch.org/uploads/default/52e69f9fbf5845ea8e9cd48bf88a2642cbc34215" title="微信图片_202005141942301"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52e69f9fbf5845ea8e9cd48bf88a2642cbc34215.png" alt="微信图片_202005141942301" data-base62-sha1="bPngc6zmUSNKmrt2XXdHHSf6Elv" width="690" height="403" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_202005141942301</span><span class="informations">976×571 27.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/284975c68133d4916b3384f769b0abe6cc0bd708.png" alt="微信图片_202005141942302" data-base62-sha1="5Kou9e0yfq3fUxoArw7Wr211W2s" width="460" height="457"></p>

username: ptrblck
cooked: <p>The error message points to an issue with your DNS configuration.<br>
I’m not sure, if it’s related to Visdom or your current server / website setup.</p>

username: mathematics
cooked: <p>This title reminds me to ask question.<br>
Is there any way to visualize training process of GAN’s ,while training.<br>
Also it would be great if we visualize and save in video format.</p>
<p>Thank you for your idea.</p>

username: ptrblck
cooked: <p>I’ve used <code>visdom</code> to plot pictures of different epochs, but <code>visdom</code> has also video support (haven’t used it yet).</p>

username: ky_Pa
cooked: <p>I opened Visdom in the local terminal, and gave me the URL http: // localhost: 8097 and it can be opened. However, it is the URL http: // 2be6dfaf3048: 8097 that can not be opened on the cloud server. Is this because I am using someone else ’s server? How can I troubleshoot this error?</p>
Tensor indexing with conditions

username: xuesongwang
cooked: <p>Hi everyone,<br>
suppose I have a tensor:  x: [0, 1, 4, 5, 6]<br>
and I want to index elements in any of this list: [4, 5, 6],<br>
similar to: np.where(x in [4, 5, 6], True, False) (this also doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">),<br>
Expected outcome: [False, False, True, True, True]<br>
Are there any solutions? Thanks.</p>

username: ptrblck
cooked: <p>Could you explain the use case a bit more or give an example output for the provided tensors?<br>
Would you like to select a random value from <code>[4, 5, 6]</code>?</p>

username: xuesongwang
cooked: <p>Hi, I think I didn’t explain the example enough.<br>
So the input x can be: [1, 2, 3, 4, 5, 6]<br>
I want to locate the index of elements <em>i</em> in x where  <em>i</em> also belongs to y:[4, 5, 6]<br>
The expected outcome  is: [False, False, False, True, True, True].</p>
<p>I managed to achieve this by numpy:<br>
index = np.isin(x.numpy(), [4, 5, 6])</p>
<p>I am wondering if pytorch has similar functions. Thank you.</p>

username: ptrblck
cooked: <p>PyTorch doesn’t have the same <code>isin</code> implementation, but you might take a look at the <a href="https://github.com/numpy/numpy/blob/v1.17.0/numpy/lib/arraysetops.py#L572-L593">numpy implementation</a> and maybe use their method?</p>
<p>Alternatively, you could of course use a list comprehension as:</p>
<pre><code class="lang-python">x = torch.arange(7)
y = torch.tensor([4, 5, 6])

ret = torch.stack([y_ == x for y_ in y]).sum(0).bool()
</code></pre>

username: xuesongwang
cooked: <p>Yes, .sum(0) is a smart trick. Thank you</p>
High gpu memory overhead

username: IsCoelacanth
cooked: <p>I was doing a GPU memory usage study for deployment and I noticed by default PyTorch is blocking out ~800mb of VRAM. I thought it might be model specific so I tried moving a Linear(1,1) layer to gpu and it took the same amount of memory, but moving another Linear(1,1) didn’t take any memory.<br>
happens in 1.4 and 1.5</p>

username: ptrblck
cooked: <p>This memory is most likely allocated by the CUDA context and not parameters or tensors.<br>
Depending on the CUDA version, used device etc. you might see an allocation of ~700+MB.</p>
Slicing a 3D tensor using two index tensor

username: srikara
cooked: <p>Hi,<br>
I have one 3D tensor <code>X</code> of shape <code>[5, 16, 128]</code> and I also have two 2-D tensors <code>P</code> and <code>N</code> containing indices of <code>X</code> in some order. <code>P</code> is of shape <code>[5,6]</code> and N is of shape <code>[5,10]</code>.</p>
<p>How do I slice <code>X</code> using <code>P</code> and <code>N</code> to get two tensors of shape <code>X1 = [5,6,128]</code> and <code>X2 = [5,10,128]</code>?</p>
<pre><code class="lang-auto">X = torch.randn(5, 16, 128)
P = torch.tensor([[0, 1, 2, 3, 4, 5],
        [0, 1, 2, 3, 4, 5],
        [0, 1, 2, 3, 4, 5],
        [0, 1, 2, 3, 4, 5],
        [0, 1, 2, 3, 4, 5]])
N = torch.tensor([[ 6,  7,  8,  9, 10, 11, 12, 13, 14, 15],
        [ 6,  7,  8,  9, 10, 11, 12, 13, 14, 15],
        [ 6,  7,  8,  9, 10, 11, 12, 13, 14, 15],
        [ 6,  7,  8,  9, 10, 11, 12, 13, 14, 15],
        [ 6,  7,  8,  9, 10, 11, 12, 13, 14, 15]])

X[P, :] = ? 
X[N, :] = ?
*** IndexError: index 5 is out of bounds for dimension 0 with size 5
</code></pre>
<p>I tried <code>X[P, :]</code> since I thought the 3rd dim (of shape 128) would get carried forward with <code>:</code> but I get an error. How do I do this in pytorch? (in Tensorflow i am aware of <code>tf.gather</code>)</p>
<p>thanks,<br>
srk</p>
<p>Note: first first dim of index tensors <code>P</code> and <code>N</code> is same as <code>X</code> and second dim of <code>P</code> and <code>N</code> (6+10) add to the second dim of <code>X</code> (16).</p>

username: kunalpratap
cooked: <p>If you want to simply split X into two tensors of shape [5,6,128] and [5,10,128] then you can use</p>
<p>X1,X2 = torch.split(X,[6,10])</p>
Memory error when loading glove embedding

username: abhigenie92
cooked: <pre><code class="lang-auto">vectors = Vectors(name='glove.6B.200d.txt')  
TEXT.build_vocab(train_data, vectors=vectors, max_size=25000, unk_init=torch.Tensor.normal_)
</code></pre>
<p>On this machine, I can’t access the internet so I have manually copy them to the machine.</p>
<p>I get this error while loading the word vectors:</p>
<pre><code class="lang-auto">RuntimeError: Vector for token b'zsombor' has 223 dimensions, but previously read vectors have 300 dimensions. All vectors must have the same number of dimensions.
</code></pre>
<p>I have already specified a max_size, I am not sure what is happening.</p>

username: ptrblck
cooked: <p>I’m not deeply familiar with <code>torchtext</code>, but the error message seems to indicate that you’ve created these vectors previously and are now changing their dimension?<br>
The code snippet looks as if you are initializing <code>vectors</code> new. Could you explain the use case a bit?</p>
Load model weights into a new created model

username: verance
cooked: <p>Hello, I trained a Unet Segmentation Model without <code>Dropout</code> in the <code>SegmentationHead</code> module.<br>
I added the module to my network but I dont want to retrain it from scratch. I was wondering if I could initialize my model with dropout module on and then just load the weights from the model without dropout.</p>
<p>I did this:</p>
<pre><code class="lang-auto">model = smp.Unet(
    encoder_name=ENCODER, 
    encoder_depth=ENCODER_DEPTH,
    encoder_weights=ENCODER_WEIGHTS, 
    decoder_channels=DECODER_CHANNELS,
    classes=len(CLASSES), 
    activation=ACTIVATION,
    decoder_attention_type=attention,
    #aux_params=dict(dropout=0,classes=len(CLASSES))
)
</code></pre>
<p>Then I loaded the weights:</p>
<p><code>model.load_state_dict(torch.load(path+"/model_loss.pth"))</code></p>
<p>However, I am getting the following error:</p>
<blockquote>
<p>‘Unet’ object has no attribute ‘copy’</p>
</blockquote>
<p>Any idea if this is possible and if so, what am I doing wrong?</p>
<p>Kind regards</p>

username: ptrblck
cooked: <p>Are you able to load the <code>state_dict</code> without any modifications to the model?<br>
The current error message seems to indicate that <code>torch.load</code> might load the complete model, not a <code>state_dict</code>.<br>
Could this be the case, i.e. did you save the complete model instead of the <code>state_dict</code>?</p>

username: verance
cooked: <p>Yes, I used <code>torch.save()</code> method to save the model at its best epoch.</p>

username: verance
cooked: <p>I managed to make this work, I loaded the model and saved the dict and then I created a Unet with dropout in the architecture and loaded the dict.</p>
<p>However, I am getting this error:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for Unet:
	Missing key(s) in state_dict: "segmentation_head.1.weight", "segmentation_head.1.bias". 
	Unexpected key(s) in state_dict: "segmentation_head.0.weight", "segmentation_head.0.bias".
</code></pre>
<p>I believe this is because of the modification I did in the <code>SegmentatioHead</code> by adding dropout. I want to load the unexpected keys as they are the missing keys. Is there any way to do this?</p>

username: ptrblck
cooked: <p>You could iterate the <code>state_dict</code> and change the name of these keys as shown <a href="https://discuss.pytorch.org/t/how-do-i-update-old-key-name-in-state-dict-to-new-version/76026/3">here</a>.</p>
How to train the paralleled layers?

username: malcolmyanguci
cooked: <p>My network is not sequential. It is a list of a nn.sequential of a same structure. When I code the forward function, I used a loop, which I feel is not efficient. How could I paralleled train the list of nn.sequential rather than train each nn.sequential one by one? Appreciate any help.</p>
<pre><code class="lang-auto">class my_net(nn.Module):
    def __init__(self,num_ring):
        super(my_net, self).__init__()
        self.num_ring = num_ring        
        self.my_list = nn.ModuleList()
        for i in range(num_ring):
            self.my_list.append(nn.Sequential(
                nn.Conv1d(64, 128, 1),
                nn.BatchNorm1d(128),
                ))
    def forward(self,x,ring):
        # ring [B,N]
        #x.shape is [B,N,D]
        x_new = []
        for i in range(self.num_ring):
            #first found the corresponding xi to i
            idx = (ring==i)
            xi = x[idx].unsqueeze(0)
            xi = xi.transpose(2,1)
            # Then train on corresponding nn.Sequential
            #xi[1,D,M_i]-&gt;xi[1,128,M_i]
            xi = self.pooler_list[i](xi)
            x_new.append(xi)
        x_new = torch.cat((x_new),dim=2)
        return x_new
</code></pre>
<p>Note each xi has different dimension. M_i maybe different</p>
Problem with multiprocessing with GPU

username: Chalkhous
cooked: <p>Whenever I try and use multiprocessing with my device as a gpu, I get this error.</p>
<p>THCudaCheck FAIL file=C:\w\b\windows\pytorch\torch/csrc/generic/StorageSharing.cpp line=245 error=801 : operation not supported<br>
Traceback (most recent call last):<br>
File “c:/Users/chalk/OneDrive/Documents/Monash Uni/FYP/Rubik’s Cube/test2.py”, line 103, in <br>
p.start()<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\process.py”, line 112, in start<br>
self._popen = self._Popen(self)<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\context.py”, line 223, in _Popen<br>
return _default_context.get_context().Process._Popen(process_obj)<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\context.py”, line 322, in _Popen<br>
return Popen(process_obj)<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\popen_spawn_win32.py”, line 89, in <strong>init</strong><br>
reduction.dump(process_obj, to_child)<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
File “C:\Users\chalk\Anaconda3\lib\site-packages\torch\multiprocessing\reductions.py”, line 242, in reduce_tensor<br>
event_sync_required) = storage.<em>share_cuda</em>()<br>
RuntimeError: cuda runtime error (801) : operation not supported at C:\w\b\windows\pytorch\torch/csrc/generic/StorageSharing.cpp:245<br>
(base) PS C:\Users\chalk\OneDrive\Documents\Monash Uni\FYP\Rubik’s Cube&gt; Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\spawn.py”, line 105, in spawn_main<br>
exitcode = _main(fd)<br>
File “C:\Users\chalk\Anaconda3\lib\multiprocessing\spawn.py”, line 115, in _main<br>
self = reduction.pickle.load(from_parent)<br>
EOFError: Ran out of input</p>
<p>For example if I run the example code found here with cuda enabled, <a href="https://github.com/pytorch/examples/blob/master/mnist_hogwild/train.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/mnist_hogwild/train.py</a> I get the error.</p>
<p>Also, whenever I set num_workers &gt; 0, I also get an error.</p>
<p>What’s going on here?</p>

username: ptrblck
cooked: <p>Could you try to add the if-clause protection as explained in the <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">Windows FAQ</a>?</p>

username: Chalkhous
cooked: <p>Do you mean to add the freeze_support() line? I just tried that and it didn’t change anything. If you mean putting the if <strong>name</strong> == “<strong>main</strong>” line I already had that, as could be seen in the example code.</p>

username: Chalkhous
cooked: <p><a href="https://pytorch.org/docs/stable/notes/windows.html#cuda-ipc-operations" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/windows.html#cuda-ipc-operations</a></p>
<p>Looking at this link, does this literally mean its impossible to set num_workers on Windows? Also with the sharing CPU tensors, I have set that, but still can’t share the model thats on the GPU.</p>
Crop images loaded on GPU using coordinates provided to eliminate host-device transfer

username: evgeniititov
cooked: <p>Hi guys,</p>
<p>Could you please give me a hand with the following. Thanks in advance.</p>
<p>Let’s say I create a batch of images (N video frames) and move them to GPU. Then, I have a list of coordinates that represent objects on the images that we earlier moved to GPU. I want to <em>crop</em> images using those coordinates (select only those regions) and then run only those areas through my network.</p>
<ol>
<li>Does anyone know a method that I could use to extract only those areas? I’ve tried using indexing [x1: x2, y1: y2, :] but it didn’t quite work. Probably I am doing something wrong.</li>
<li>Could I combine them in a batch using .cat() or since they are already on GPU I could process them together?</li>
</ol>
<p>Just a bit of background. I am building a system where multiple nets work one after another. Moving data back and forth (device-host) is expensive, so I thought I would move images to GPU only ones. I first detect objects of class A, then within this object A I want to find objects B and then classify them. So, the idea is to move images only ones and then just move coordinates back and forth.</p>
<p>Thanks a lot!</p>
<p>Regards,<br>
Eugene</p>
Load state dict only from old model.pth file

username: zwacke
cooked: <p>I have an old model that I saved (complete model) in a .pth file. When I try to load it now, I get an exception from the unpickler, that the python module in which I defined the model cannot be found.<br>
I was not aware that when saving full .pth models, the pickler only saves references to the documents that contain the definitions of the underlying python objects, and not the objects themselves. I refactored my codebase massively since then and can not reinstate the old version of the code, much less the former file directoy structure. I would have, however, the python object for the exact model.</p>
<p>Is there any way I can salvage my model? In particular I am only interested in the state_dict, i.e. the tensors that should still be in the .pth file. I tried looking at the source-code of torch.serialized, but I did not really grasp what is going on when models are saved/loaded and how I could get hold of the parameters.</p>
<p>As a side note, the whole saving/loading functionality with pytorch is in an unsatisfying situation, especially when working in teams on models and codebase. There are two options. One requires to save the model definitions including all used setting parameters separately from the state dict, the other requires to hold alot of legacy code that cannot be refactored. Is there no way to make it work in a way that pytorch models can be saved as they are and reused as they are?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="zwacke" data-post="1" data-topic="81167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/bc79bd/48.png" class="avatar"> zwacke:</div>
<blockquote>
<p>Is there no way to make it work in a way that pytorch models can be saved as they are and reused as they are?</p>
</blockquote>
</aside>
<p>You could script the model and export it so that it’ll be usable in standalone applications (e.g. in C++).<br>
However, that wouldn’t store the source code of the model, which would be needed for further experiments.</p>
<p>You could try to use the patch tool via:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.Module.dump_patches = True
model=torch.load("mymodel.model")
</code></pre>
<p>which should create patch files with the diffs. I’ve never used it before, so I’m not sure, how easy it would be to restore your code.</p>
Runtime error shape is invalid for input

username: lee64456546
cooked: <p>i am trying to get the input and output information of a network. When debugging, i got this error, Runtime, shape ‘[-1, 400]’ is invalid for input of size 384. I tried different values, but can’t find the correct value. Is there a way to solve this issue? Thanks.</p>
<pre><code class="lang-auto">
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<pre><code class="lang-auto">input_shape = (3, 21,21)
        dummy_input = torch.randn(6,*input_shape)

        graph = torch.jit._get_trace_graph(model,  args=dummy_input, _force_outplace=False, _return_inputs_states=False)
</code></pre>
<p>Error meaasge:</p>
<pre><code class="lang-auto">RuntimeError: shape '[-1, 400]' is invalid for input of size 384
</code></pre>

username: ptrblck
cooked: <p>Since your input shape is <code>[batch_size, 3, 21, 21]</code>, the flattened activation before the linear layer will have the shape <code>[batch_size, 16, 2, 2]</code>, so you should change the input features to <code>16*2*2</code> as well as the flattening operation (or use <code>x = x.view(x.size(0), -1)</code> instead).</p>
CUDA - Crash with no error message with multiple GPU

username: fisch_std
cooked: <p>I am actually facing a problem on my project.<br>
The learning process (using CUDA) is randomly crashing <strong>without any error message</strong> (most of the time). It can happen after 1min or after 35min.<br>
It only happens if I use both my GPU’s (mismatched 1080Ti and 2080Ti).<br>
If I use only one everything is fine.</p>
<p>We had a power problem (insufficient power supply) but now it is solved and the problem is still here.<br>
Please note that rarely a message about memory access appears (during the crash), not sure it is related. It says an instruction referenced memory that could not be written, always memory with very small address (0x00000…00020)or very big (0xFFFFFFFF…FF).<br>
We also observed that if the model is more heavy the crash happens sooner.</p>
<p>What could cause such crashes  ?<br>
Thanks for your help</p>

username: ptrblck
cooked: <p>Are you running the script in a terminal of notebook?<br>
The latter might hide the actual error message?<br>
Also, how did you verify that your power supply issue is solved?<br>
Are the GPUs running fine on their own?</p>

username: fisch_std
cooked: <p>Running in visual code terminal, or separate terminal. (same problem)<br>
Concerning power supply, we had also drops in CPU frequency, they are now gone.<br>
Running the same code on a single GPU is fine (just slower)</p>

username: ptrblck
cooked: <p>What are the specs of your current PSU?</p>

username: fisch_std
cooked: <p>New setup is dual 750W.</p>
<p><strong>Update:</strong><br>
Something worth knowing is that we do Model sharing when using MultiGPU. We run 2 architecture (same nature and complexity) on 2 sets of data. We do no use DataParrallel.</p>

username: ptrblck
cooked: <p>Do you have another PSU with more power by chance?<br>
The 1080Ti built into a workstation might need 600-650 Watts, while the 2080Ti might need additional ~270+ Watts. I don’t know how your system setup looks like, so I just used public information from tomshardware etc.</p>

username: fisch_std
cooked: <p>Hello,</p>
<p>Sadly we don’t, but we are convinced that it is not a problem.<br>
According to the server company the power supplies are not simple redundancy but share their power.<br>
And that is reflected but the measures we made on our side.<br>
We measured a power draw of roughly 850w at peak (which indeed follows your estimation).<br>
That shows that in fact power delivery is capable of more than one unit (750w) , therefore we should have a capacity of 1500w (more than enough) with the 2 units<br>
Thank you for your help !</p>

username: fisch_std
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could that be an incompatibility between the 2 cards at the CUDA level ?</p>

username: ptrblck
cooked: <p>I’m not aware of compatibility issues as long as the CUDA version is not too old for either device.</p>
Hidden paramaters in LSTM

username: dachosen1
cooked: <p>I’ve seen examples where users are initializing a hidden parameter for a LSTM network. I was under the impression that the hidden states were managed under the hood. What’s the advantage of manually initializing the hidden parameters.</p>
<blockquote>
<p>class CharLSTM(nn.Module):</p>
<pre><code>def __init__(self, tokens, n_hidden=256, n_layers=2,
                           drop_prob=0.5, lr=0.001):
    super().__init__()
    self.drop_prob = drop_prob
    self.n_layers = n_layers
    self.n_hidden = n_hidden
    self.lr = lr
    
    self.chars = tokens
    self.int2char = dict(enumerate(self.chars))
    self.char2int = {ch: ii for ii, ch in self.int2char.items()}
    
    self.lstm = nn.LSTM(len(self.chars), n_hidden, n_layers, 
                        dropout=drop_prob, batch_first=True)
    
    self.dropout = nn.Dropout(drop_prob)
    self.fc = nn.Linear(n_hidden, len(self.chars))
  

def forward(self, x, hidden):
    ''' Forward pass through the network. 
        These inputs are x, and the hidden/cell state `hidden`. '''
    r_output, hidden = self.lstm(x, hidden)
    out = self.dropout(r_output)
    out = out.contiguous().view(-1, self.n_hidden)
    out = self.fc(out)
    return out, hidden


def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    weight = next(self.parameters()).data
    
    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_())
    
    return hidden
</code></pre>
</blockquote>

username: sharvil
cooked: <p>LSTM states are taken care of under the hood but you have to provide an initial state from which to start the recurrent network. You’d usually want to use zeros for the initial state but there are cases where you might want to start with a different value (e.g. when performing truncated backprop through time). If you call the LSTM layer with no initial state, it assumes you want a zero state and takes care of generating the zeros tensor for you.</p>
Hard time understanding bcewithlogitloss

username: Sanjayvarma11
cooked: <p>Hello everyone recently i am working on a assignment where i have to predict mask of image using foreground_background image,background image.I used bcewithlogitloss bcz i changed my target value as combination of 1 and 0 where 1 is for foreground and 0 is for background.So my results are pretty good but i am not still convinced why i used bcewithlogitloss.It is a reconstruction problem where i am using probabilty distribution(bcewithlogitloss).SO can anyone plz hepl me why i can use bcewithlogitloss in this example.Thank you</p>
<p><a href="https://stackoverflow.com/questions/tagged/pytorch" rel="nofollow noopener">pytorch</a></p>
Early stopping at loss increassing or accuracy decreasing?

username: WaterKnight
cooked: <p>What do you think that is the better approach for EarlyStopping?</p>
<p>Is a good idea to earlystop at higher valid_loss at validation? And early_stop at accuracy decreasing in test?</p>

username: Sanjayvarma11
cooked: <p>it depends.Initially the early stopping was used bcz if we train our model too deep then it may not generalize well on real life situations as it is learning from training samples.So we are stopping the train once loss is below the particular threshold.It may be helpful for real life machine learning examples and if you want to get higher accuracy at test samples it is better to train for longer time.</p>
Unavailable instructions PyTorch CPU

username: Bauke_Brenninkmeijer
cooked: <p>Hi Guys, I’m trying to run MobileNetv3 on pytorch (1.4.0+cpu and 1.5.0+cpu), but I’m getting the error<br>
<code>AttributeError: 'SelectAdaptivePool2d' object has no attribute 'flatten'</code>.</p>
<p>Is this just an unavailable instruction for PyTorch CPU or am I doing something wrong? If so, is there any comprehensive list that shows which models do and don’t run on the cpu?</p>

username: ptrblck
cooked: <p>I assume you are using <a class="mention" href="/u/rwightman">@rwightman</a>’s implementation of this model?<br>
If so, note that the <code>flatten</code> argument seems to have been added 13 days ago <a href="https://github.com/rwightman/pytorch-image-models/blob/dab9935b363ddcaa99bac43cf7f3f91aee79aa33/timm/models/layers/adaptive_avgmax_pool.py#L70-L101">here</a>.<br>
Are you using different versions of the repository?</p>

username: rwightman
cooked: <p>That woud appear to be my impl. Unexpected error, especially given that the default remained to not flatten when that was added. A lot of GPU usage with that change in and no problems noticed.</p>
<p><a class="mention" href="/u/bauke_brenninkmeijer">@Bauke_Brenninkmeijer</a> I just ran on a clean environement, PyTorch 1.5.0 cpu (conda) and no such issue. Any chance you’ve got multiple PyTorch or timm versions in the environment/path? It wasn’t long ago that flatten didn’t exist in PyTorch.</p>

username: Bauke_Brenninkmeijer
cooked: <p><a class="mention" href="/u/rwightman">@rwightman</a> Thanks for the fast response. I’ll create a new environment and see if it makes a difference. I’ll report back.</p>

username: Bauke_Brenninkmeijer
cooked: <p>Aaah I got the issue. I was running an old version of timm in the environment where I exported and a new version of timm in the environment where it was imported. Hence the exported model did not have the <code>.flatten</code> attribute.</p>
<p>Again, thanks for the help! Sorry for not updating first :).</p>
Transformer tgt argument

username: Adamits
cooked: <p>Hi,</p>
<p>I am trying to use the <code>Transformer</code> for the first time as an encoder-decoder for a tagging task. I am finding the input arg <code>tgt</code> a bit underdescribed. The docs give us: <code>tgt – the sequence to the decoder (required).</code></p>
<p>Is this intended to be the embedding for the previously decoded timestep? A tensor of all previously decoded timesteps? Also, given that there is a <code>tgt_mask</code> input, is this simply for batching, if one sample stops decoding, but the others continue? Or does this imply that at train time it is preferable to get the entire gold target sequence, with outputs that have not been decoded yet masked? I cannot think of another use case for masking the attn between tgt timesteps. Even then, at test time, what would you be providing the model since we do not have a gold sequence?</p>
<p>Could someone point me to any examples that clarify the usage of <code>tgt</code> and <code>tgt_mask</code>?</p>

username: Adamits
cooked: <p>In the basic case, my assumption is we want (if using student forcing) something like:</p>
<pre><code class="lang-auto">for i in range(0, sequence_length):
    out = transformer(input_representation,
                      torch.stack(decoded))

    # self.out encodes into out space
    # Softmax to compute cross-entropy or take max as prediction
    preds = self.softmax(self.out(preds))
    _, indices = torch.max(preds)
    decoded.append(embed(indices))</code></pre>
Kernel Restarting while using CUDA as device

username: sgaur
cooked: <p>I switch back too windows 10 from ubuntu 18.04 LTS and all of my codes that previously run on Ubuntu are not working on windows.</p>
<p><strong>Getting below Error</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b230c1db3b53bbe8cf54d1aad2da016bc1fe5624.png" data-download-href="https://discuss.pytorch.org/uploads/default/b230c1db3b53bbe8cf54d1aad2da016bc1fe5624" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b230c1db3b53bbe8cf54d1aad2da016bc1fe5624.png" alt="image" data-base62-sha1="pqls0dt0JrRhEosZhAV7IJFrhCk" width="690" height="321" data-dominant-color="979798"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×442 24.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Pytorch Version = 1.5.0</strong><br>
<strong>NVIDIA-SMI =431.90</strong><br>
<strong>Driver Version: 431.90</strong><br>
<strong>CUDA Version: 10.1</strong></p>
<p>If I do basic Code run it works Fine using cuda but my Deep-learning models are not working.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72da6fff2892e974cbf3ab572f4f96c89227130b.png" data-download-href="https://discuss.pytorch.org/uploads/default/72da6fff2892e974cbf3ab572f4f96c89227130b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72da6fff2892e974cbf3ab572f4f96c89227130b.png" alt="image" data-base62-sha1="go2qLNpU7Qpwt1De4XOukJKLzrJ" width="690" height="185" data-dominant-color="F2F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">824×222 6.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Anshumaan_Dash
cooked: <p>Try to get the pytorch installation instruction from <a href="https://pytorch.org/" rel="nofollow noopener">https://pytorch.org/</a></p>
<p>The pytorch libraries you have might not be compatible with your system.</p>

username: sgaur
cooked: <p>This one is the new one “conda install pytorch torchvision cudatoolkit=10.2 -c pytorch”<br>
Do you want me to install it ?</p>

username: Anshumaan_Dash
cooked: <p>You did install it once in your system. Right?</p>

username: sgaur
cooked: <p>Yes I don’t remember which version but I am uninstalling now and install it again</p>