akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get the gradient of a function

username: zyl1024
cooked: <p>This should be a basic question, but maybe I am missing something. Is there a way to automatically derive an analytical function of the gradient, and use this function further?</p>
<p>For example, we have a vector $x$ and the sum of square function: (sorry don’t know how to insert latex math…)</p>
<p>$f(x) = \sum_i x_i^2$</p>
<p>We can define the gradient of $f(x)$ as</p>
<p>$\nabla f(x) = [2x_1, 2x_2, …, 2x_d]^T$</p>
<p>Then we can define a “sum of gradient” function as</p>
<p>$g(x) = \sum_i \nabla f(x)_i = \sum_i 2x_i$</p>
<p>which is, itself, a scalar function of $x$. But I don’t know how to implement $\nabla g(x)$ using pytorch’s auto-diff.</p>
<p>Currently I have,</p>
<pre><code class="lang-auto">import torch
def f(x):
    return torch.sum(x**2)
def grad(func, x):
    y = func(x)
    y.backward()
    gr = x.grad
    return gr
def sum_of_grad(func, x):
    gr = grad(func, x)
    return torch.sum(gr)
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
s = sum_of_grad(f, x)
s.backward()
print(x.grad)
</code></pre>
<p>But <code>s.backward()</code> will cause an error because <code>gr = x.grad</code> does not have gradient information. So is what I am trying to do compatible with pytorch’s autodiff infrastructure or is it not supported? I know autodiff is technically not the same as symbolic differentiation so I won’t be surprised if this is not possible, but just want to make sure.</p>

username: albanD
cooked: <p>Hi</p>
<p>If you want to be able to backprop through another backprop, you have to run the first one with <code>create_graph=True</code> (see the doc for more details).<br>
Also I would recommend using autograd.grad when doing higher order derivatives as it avoid any issue with multiple backward accumulating gradients at the same place.</p>
<pre><code class="lang-python">def grad(func, x):
    y = func(x)
    gr = autograd.grad(y, x, create_graph=True)[0]
    return gr
</code></pre>
Expand a tensor, varying the expansion dimension by row

username: jkuck
cooked: <p>I would like to expand a 2d tensor to a 3d tensor, varying which dimension is expanded in each row of the tensor.  For example, I’d like to take this 2d tensor:</p>
<pre><code class="lang-auto">values = torch.tensor([[1, 2],
					   [3, 4],
					   [5, 6],
					   [7, 8],
					   [9, 10]])
</code></pre>
<p>And transform it to this 3d tensor:</p>
<pre><code class="lang-auto">tensor([[[ 1,  1],
         [ 2,  2]],

        [[ 3,  4],
         [ 3,  4]],

        [[ 5,  5],
         [ 6,  6]],

        [[ 7,  7],
         [ 8,  8]],

        [[ 9, 10],
         [ 9, 10]]])
</code></pre>
<p>where the dimension to expand for each row is given by these indices:</p>
<pre><code class="lang-auto">torch.tensor([0, 1, 0, 0, 1])
</code></pre>
<p>The following code produces the desired result, but is inefficient:</p>
<pre><code class="lang-auto">def expand_slow(expand_indices, values):
    expanded_values = torch.stack([
      row_val.expand([2,-1]).transpose(expand_indices[row_idx], 1) 
      for row_idx, row_val in enumerate(torch.unbind(values, dim=0))
                                   ], dim=0)
    return expanded_values

expand_indices = torch.tensor([0, 1, 0, 0, 1])

values = torch.tensor([[1, 2],
                       [3, 4],
                       [5, 6],
                       [7, 8],
                       [9, 10]])

expand_slow(expand_indices, values)
</code></pre>
<p>What’s the best way to perform this operation using efficient pytorch functions?  Thank you!</p>
What is the official way to download Tensoboard with conda for Pytorch?

username: pinocchio
cooked: <p>I was going through the tutorial <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">https://pytorch.org/docs/stable/tensorboard.html</a> and it seemed it had the command using pip but I use conda. I would like to avoid issue in the future. Is:</p>
<pre><code class="lang-auto">conda install tensorboard
</code></pre>
<p>really all I need to do? I couldn’t find docs from pytorch confirming this.</p>

username: simaiden
cooked: <p>Try <a href="https://anaconda.org/conda-forge/tensorboard" rel="nofollow noopener">with</a></p>
<pre><code class="lang-auto">conda install -c conda-forge tensorboard 
</code></pre>

username: pinocchio
cooked: <p>how is that different?</p>
Reduce model's final classification accuracy variance

username: amirhf
cooked: <p>Hello,</p>
<p>I am training a ResNet18 classification model on a dataset of 27K images with 7 classes. I initialize the layers with ImageNet weights and I’m <strong>not</strong> using any fixed seed. However the final classification results have large variance: such as 80.70%, 80.37%, 80.31%, 81.81%, and 81.58%. I use a batch size of 256 for both train and test, with a learning rate of 0.01 initially decayed by a factor of 10 every 20 epochs. I run for 60 epochs in total and I use SGD with momentum 0.9 and weight decay 0.0005. I use the standard cross entropy function for scores. My code uses the template from the standard imagenet.py from pytorch repo. I intend to publish the results in a research paper, but the numbers vary a lot. Mentioned numbers have a mean 80.954 and std ($\sigma^2$) of 0.38.</p>
<p>Are there any other configurations/measures that I can apply to decrease this variance?</p>
Any body has datasets for cnn model please

username: Muhammad_Izaz
cooked: <p>i want to check my cnn model. or show me a site for datasets</p>

username: Dwight_Foster
cooked: <p><a href="http://Kaggle.com" rel="nofollow noopener">Kaggle.com</a> has many great datasets that can be used to train a cnn.</p>

username: Muhammad_Izaz
cooked: <p>thank you brother @#</p>
Concatenate torch tensor along given dimension

username: Bixqu
cooked: <p>In tensorflow you can do something like this</p>
<pre><code>third_tensor= tf.concat(0, [first_tensor, second_tensor])
</code></pre>
<p>so if first_tensor and second_tensor would be of size [5, 32,32], first dimension would be batch size, the tensor third_tensor would be of size [10, 32, 32], containing the above two, stacked on top of each other.</p>
<p>How can I do this with torch variables?  Or ar least with torch tensors?</p>

username: aditya
cooked: <p>Check out <a href="http://pytorch.org/docs/torch.html?#torch.cat" rel="nofollow noopener"><code>torch.cat</code></a>. It works both on torch tensors and variables</p>
<p>for example:<br>
<code>third_tensor = torch.cat((first_tensor, second_tensor), 0)</code></p>

username: Bixqu
cooked: <p>Thanks, I wrote my own function that transforms it to numpy.array and then uses numpy.concatenate, then transforms back… not sure how slow this is though.</p>

username: aditya
cooked: <p>Cool! but you won’t be able to backpropagate by converting to numpy and back (if you so desire). But <code>torch.cat</code> would allow torch autograd to do its “magic” <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . Not sure about the performance issue between the two, but I guess the native way of concatenating would be faster.</p>

username: Bixqu
cooked: <p>Thanks, switched to torch.cat.</p>

username: albertotono
cooked: <p><a class="mention" href="/u/bixqu">@Bixqu</a> be also mindful of this property  x = torch.cat([y, z])<strong>.view(-1)</strong><br>
.view is a special parameter that can change the shape of the tensor without changing its contents.</p>
Loss backward from sum of all values in array

username: stuck
cooked: <p>Hi all,</p>
<p>I have an array of loss values called loss_arr:<br>
<code>[tensor([0.6828], grad_fn=&lt;SubBackward0&gt;), tensor([0.152], grad_fn=&lt;SubBackward0&gt;)]</code></p>
<p>When I tried to backward the total loss, I tried to do:</p>
<pre><code class="lang-auto">loss = sum(loss_arr)
loss.backward()
</code></pre>
<p>However it gave me:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>If I backward like:</p>
<pre><code class="lang-auto">loss_arr[0].backward()
</code></pre>
<p>for example, there is no problem.</p>
<p>So, how can I backward the total loss?<br>
And, why am I failed to backward?</p>
<p>Thank you all, have a nicely quarantined days…</p>

username: tux
cooked: <p>Hi,</p>
<p>Can you try torch.sum instead of sum ?</p>
ValueError about batchsize while doing semantic segmentation

username: LaserLumiere
cooked: <p>Hi,</p>
<p>I’m using UNet implemented by pytorch for multimodal semantic segmentation. The size of original image is <strong>512×512×3(RGB)</strong>. After processing by DataLoader, the input size of the network is <strong>[1,3,512,512]</strong>, and I know the first dimension is batchsize (I set the batchsize to 1). And the output size of the network is <strong>[1,2,512,512]</strong> since num_class = 2.</p>
<p>But I met a problem when computing CrossEntropyLoss. The error is <strong>ValueError: Expected input batch_size (1) to match target batch_size (3)</strong>. I think this error has something to do with the mask. The size of mask is the same as the original image (512×512×3). I have tried this size in another UNet network implemented by keras for single-modal segmentation, and it works well. I try to process the mask using Convert(‘L’), and the error changes into <strong>ValueError: Expected input batch_size (1) to match target batch_size (512)</strong>. I’m a beginner, and I’m so confused about it. How to fix this error?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I assume you are using <code>nn.CrossEntropyLoss</code> for your segmentation use case.<br>
If that’s the case, then you would have to pass the target mask in the shape <code>[batch_size, height, width]</code> as a <code>LongTensor</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.</p>
<p>Based on the shape of your mask, it seems it might be an RGB image?<br>
If so, you would need to map the colors to class indices first.</p>
Learning to Rank in PyTorch

username: SushantC
cooked: <p>Hi,</p>
<p>Is there any future plan to roll out a Learning to Rank library in PyTorch similar to TensorFlow Ranking (<a href="https://github.com/tensorflow/ranking" rel="nofollow noopener">https://github.com/tensorflow/ranking</a>)?</p>

username: Rowing0914
cooked: <p>It’s a bit old tho, i found this resource from our community!</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="https://ptl2r.github.io/" target="_blank" rel="nofollow noopener">[“ptl2r”]</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://ptl2r.github.io/" target="_blank" rel="nofollow noopener">[“ptl2r”]</a></h3>

<p>[“Learning to Rank in PyTorch”]</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Error while loading a dataset from torchvision.datasets

username: cijerezg
cooked: <p>I am trying to load a MNIST from torchvision.datasets. As shown here <a href="https://pytorch.org/docs/stable/torchvision/datasets.html" rel="nofollow noopener">pytorch website</a> .</p>
<p>dataset = torchvision.datasets.MNIST(‘data_nn/’)<br>
data_loader = torch.utils.data.DataLoader(dataset,batch_size=1,shuffle=True,num_workers=4)</p>
<p>But when I am going to iterate over the data_loader, that is<br>
“”"<br>
for image,label in data_loader():<br>
print(image)<br>
“”"</p>
<p>I get the following error:</p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;</p>
<p>So it is clear to me what is happening but I don’t really know how to fix it.</p>
<p>Something I could is dataset.data and then pass that to data_loader but that only gives me images without labels.</p>
<p>Thanks!</p>

username: charan_Vjy
cooked: <p>You would need to convert the PIL images to a Tensor. There is an inbuilt function in pytorch that does this, which is shown below.</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.MNIST("data_nn/",transform=torchvision.transforms.ToTensor()) 
</code></pre>
<p>Also change the data loader part as shown below:</p>
<pre><code class="lang-auto">for step,(img,label) in enumerate(data_loader):
    print(img.shape)
</code></pre>
I am unable to load a simple RNN model

username: GlazeDonuts
cooked: <p>This is what my model look like</p>
<pre><code class="lang-auto">class myRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, corp_size):
        super(myRNN, self).__init__()
        self.hidden_size = hidden_size
        self.corp_size = corp_size
        self.op_size = output_size
        self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size)
        self.fc = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, h_0):
        out, h_n = self.rnn(input, h_0)
        out = out.reshape(self.corp_size, self.hidden_size)
        out = self.fc(out)
        out = self.softmax(out)
        return out, h_n
    
    def initHidden(self):
        return torch.zeros(1, self.hidden_size)
</code></pre>
<p>This is how I have saved it after training</p>
<pre><code class="lang-auto">MODEL_STORE_PATH = "/media/khurshed2504/Data/NLP/Models/Dino_Names_RNN"
torch.save(model.state_dict(),MODEL_STORE_PATH)
</code></pre>
<p>But, when I try to load the state directory  in another notebook,</p>
<pre><code class="lang-auto">MODEL_STORE_PATH = "/media/khurshed2504/Data/NLP/Models/Dino_Names_RNN"
testmodel = myRNN(vocab_size, 64, vocab_size, corp_size)
testmodel.load_state_dict(torch.load(MODEL_STORE_PATH, map_location = torch.device(device)))
testmodel.to(device)
</code></pre>
<p>It gives me an error saying **AttributeError: ‘Tensor’ object has no attribute ‘copy’<br>
**</p>
<p>I have saved and loaded many models and this is the first time that I am coming across this error</p>
<p>The error log looks like:</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-25-97d0abbc9b0a&gt; in &lt;module&gt;
      3 # testmodel = torch.load(MODEL_STORE_PATH)
      4 # testmodel = testmodel.to(device)
----&gt; 5 testmodel.load_state_dict(torch.load(MODEL_STORE_PATH, map_location = torch.device(device)))
      6 testmodel.to(device)
      7 # HIDDEN_STORE_PATH = "/media/khurshed2504/Data/NLP/Models/English_Names_Hidden"

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    801         # copy state_dict so _load_from_state_dict can modify it
    802         metadata = getattr(state_dict, '_metadata', None)
--&gt; 803         state_dict = state_dict.copy()
    804         if metadata is not None:
    805             state_dict._metadata = metadata

AttributeError: 'Tensor' object has no attribute 'copy'
</code></pre>

username: ptrblck
cooked: <p>Could you recheck the path please and make sure that <code>torch.load</code> returns a valid <code>state_dict</code>?</p>
<p>Your code seems to work for me:</p>
<pre><code class="lang-python">model = myRNN(1, 1, 1, 1)
sd = model.state_dict()

model = myRNN(1, 1, 1, 1)
model.load_state_dict(sd)
&gt; &lt;All keys matched successfully&gt;
</code></pre>
Lstm bidirectional - size mismatch

username: funmath
cooked: <p>I am a newbie with lstm and confused with dimensions when going from lstm to the linear layer. I get the following error: # size mismatch, m1: [4096 x 128], m2: [64 x 3]. I know I have to flatten in some way but do not know how to.</p>
<p>For reference batch_size=64, hidden_dim=64, tagset_size=3, embedding_dim=64. Below is my class LSTM (bidirectional lstm). The error is in the forward() when it runs tag_space = self.fc(lstm_out). I think its because dimensions going from lstm to linear are not correct, but I cannot figure out why (I added print statements so you can see the dimensions).</p>
<p>Here is my class:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

 def __init__(self, embedding_dim, hidden_dim, batch_size, vocab_size, tagset_size, layer, direct, dropout):
     super(LSTM, self).__init__()
     self.hidden_dim = hidden_dim
     self.word_embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
     self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=layer, bidirectional=True,          dropout=0.5, batch_first=True)
     self.direct = 2 if direct else 1
     self.layer = layer
     self.fc = nn.Linear(2*hidden_dim, tagset_size)
     self.hidden = self.init_hidden()



 def init_hidden(self):
     hid = torch.autograd.Variable(torch.zeros(self.layer * self.direct, batch_size,  self.hidden_dim))
     cel = torch.autograd.Variable(torch.zeros(self.layer * self.direct, batch_size,     self.hidden_dim))
     hid = hid.to(device)
     cel = cel.to(device)
     return hid, cel

 def forward(self, sentence):
     print(sentence.size()) -&gt; torch.Size([64, 64])
     embeds = self.word_embeddings(sentence)
     print(embeds.size()) -&gt; torch.Size([64, 64, 64])
     lstm_out, self.hidden = self.lstm(
embeds, self.hidden) 
     print(lstm_out.size()) -&gt; torch.Size([64, 32, 128])
     ###### Below line is the error
     tag_space = self.fc(lstm_out)
     tag_scores = F.log_softmax(tag_space, dim=2)
     return tag_scores
</code></pre>

username: ptrblck
cooked: <p>Your output should be returned with the shape <code>[batch, seq_len, num_directions * hidden_size]</code> and it seems you are directly feeding this input to the linear layer.<br>
Note that linear layers accept inputs as <code>[batch_size, *, nb_features]</code>, where the asterisk can be any dimensions, such that the linear layer will be applied on each of these dimensions separately.</p>
<p>I’m not sure what your exact use case is, but you could use:</p>
<pre><code class="lang-python">output.view(batch, seq_len, num_directions, hidden_size)
</code></pre>
<p>to separate the direction dimension, if you need to slice the output tensor.</p>
Does losing index mess up backpropagation?

username: blade
cooked: <p><strong>In short:</strong></p>
<p>if while computing the loss from the reconstructed output with dimensions <code>(B,N,N,T)</code>, I ignore the first index on the last element, meaning</p>
<p><code>recon = rec[:, :, :, 1:]</code></p>
<p>Would this screw with the back-propagation process? Also, is there a quick test to see if my implementation (any implementation in general) is breaking the back-propagation magic of PyTorch?</p>
<p><strong>In details:</strong></p>
<p>I’m trying to implement a Graph VAE loss function from the paper: <a href="https://doi.org/10.1186/s13321-019-0396-x" rel="nofollow noopener">https://doi.org/10.1186/s13321-019-0396-x</a></p>
<p>Graphs are discrete objects that are constructed of nodes and edges. The loss function shows if the reconstructed edges and nodes have the correct type. Basically a classification problem. According to the proposed method, I need to ignore one type and compute the loss based off of other types.</p>
<p>Now my question is if the reconstructed edges are presented using <code>recon_adj</code> with dimensions <code>(B,N,N,T)</code> where <code>T</code> is the type of the edge. Would eliminating one type using</p>
<p><code>recon_adj = rec_adj[:, :, :, 1:]</code></p>
<p>mess with the backprop?</p>

username: charan_Vjy
cooked: <p>When you index out an element, you are simply making sure that a particular activation at the end does not contribute to the loss function and subsequently does not contribute to changing the weights and biases. This would not break backprop. To illustrate this point with code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
batch_size = 6 ### Any arbitarary number
fc1 = nn.Linear(12,3)
input = torch.randn(batch_size,12) ### Making sure the second dimension is qual to 12
target = torch.randn(batch_size,2)
output = fc1(input)[:,:-1]
loss = F.mse_loss(output,target)
loss.backward()
</code></pre>
Convolution without summation over kernels

username: 9616
cooked: <p>Hi, is there a smart way to get the output of a convolution without summing over the kernels spatially?</p>
<p>i.e from input [B, 4, 6, 16, 16] to output [B, 6, 14, 14, K, K] where K=3 is the kernel size in a convolution operation.</p>
<p>This can be achieved trivially with unfold, but is it possible using a Conv2d/3d layer with a fancy filter?</p>
Question about torch.distributed p2p communication

username: yijing
cooked: <p>Hi,<br>
I have a question about the p2p communication in torch.distributed. Suppose we set up a group with 3 processes using command <em>init_process_group(backend=‘gloo’, init_method=“tcp://10.0.0.1:8888”, rank=args.rank, world_size=3)</em> on three different nodes with IP 10.0.0.1 to 10.0.0.3. When we are sending tensors from 10.0.0.2 to 10.0.0.3, how is the underlying network traffic routed? Is it directly from 10.0.0.2 to 10.0.0.3 or from 10.0.0.2 to 10.0.0.1 and then to 10.0.0.3? Probably the answer is obvious but I couldn’t find it based on the doc’s description. Thanks in advance!</p>
<p>Yijing</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/yijing">@yijing</a></p>
<p>The message will directly send from  10.0.0.2 to 10.0.0.3.</p>
<p>In <code>init_process_group</code>, the <code>init_method=“tcp://10.0.0.1:8888”</code> is only for rendezvous, i.e., all process will use the same ip:port to find each other. After that communications don’t need to go through master.</p>
<p>BTW, if you are using p2p comm, <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">torchrpc</a> might be useful too. <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" rel="nofollow noopener">Here</a> is a tutoral.</p>
IDE Setup: linking forward and __call__

username: Sam_Shleifer
cooked: <p>Do any pytorch developers/users have a python IDE configuration (or other tooling) that allows them to easily jump between invocations of a model and the definition of <code>model.forward</code>? I am in pycharm, where “show usages”, “jump to caller” etc. work for most class methods but not this,  but happy to switch to VSCODE or another IDE.</p>
<p>Thanks!</p>
Appropriate ways to share weights between modules

username: Peyash
cooked: <p>What are the appropriate ways to share/tie parameters between modules?<br>
For example, I have two <code>Sequential</code> containers containing multiple modules and I want to share/tie some of the module weights. I can’t reuse module in my case (e.g., sharing weights of <code>Embedding</code> with <code>Linear</code>)</p>
<p>Does setting one module weights to another do the job like this example below?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/word_language_model/model.py#L34" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/word_language_model/model.py#L34" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/word_language_model/model.py#L34</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="24" style="counter-reset: li-counter 23 ;">
<li>
</li>
<li>    # Optionally tie weights as in:</li>
<li>    # "Using the Output Embedding to Improve Language Models" (Press &amp; Wolf 2016)</li>
<li>    # https://arxiv.org/abs/1608.05859</li>
<li>    # and</li>
<li>    # "Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling" (Inan et al. 2016)</li>
<li>    # https://arxiv.org/abs/1611.01462</li>
<li>    if tie_weights:</li>
<li>        if nhid != ninp:</li>
<li>            raise ValueError('When using the tied flag, nhid must be equal to emsize')</li>
<li class="selected">        self.decoder.weight = self.encoder.weight</li>
<li>
</li>
<li>    self.init_weights()</li>
<li>
</li>
<li>    self.rnn_type = rnn_type</li>
<li>    self.nhid = nhid</li>
<li>    self.nlayers = nlayers</li>
<li>
</li>
<li>def init_weights(self):</li>
<li>    initrange = 0.1</li>
<li>    self.encoder.weight.data.uniform_(-initrange, initrange)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
DataParallel index error during forward

username: Shiro
cooked: <p>Hello,</p>
<p>I am currently trying to train a NLP model on multi GPU (4) with two inputs : the text encoded in long tensor and the lengths of each item in the batch.<br>
Unfortunately I got an error I do not really understand. I have followed the tutorial on pytorch so I was wondering what I was doing wrong. My code works well without the dataparallel function. For the information I am using also apex.</p>
<p>So the piece of code :</p>
<pre><code class="lang-auto">if cfg.multigpu:
    model = nn.DataParallel(model)
    device = "cuda:0"

....
scores = model(data["Text"].to(device), data["Lengths"].to(device))
</code></pre>
<p>The error :</p>
<blockquote>
<p>File “/home/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 151, in forward<br>
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])<br>
File “/home/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 156, in replicate<br>
return replicate(module, device_ids, not torch.is_grad_enabled())<br>
File “/home/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/replicate.py”, line 166, in replicate<br>
setattr(replica, key, buffer_copies[j][buffer_idx])<br>
IndexError: list index out of range</p>
</blockquote>
<p>I tried to fix the “device_ids” args in DataParallel, but I got the same issue.<br>
Do I forget to add some line of codes to make dataparallel work?</p>
UNet -- Test Loss not Decreasing

username: CCL
cooked: <p>I’m using <code>BCEWithLogitsLoss</code> to optimise my model, and Dice Coefficient loss for evaluating train dice loss &amp; test dice loss. However, although both my train BCE loss &amp; train dice loss decrease after each epoch, my test dice loss doesn’t and plateaus early on. I have already tried batch norm and dropout, as well as experimented with both Adam and SGD using from 1 to 0.00001 as the learning rate. The only difference between experiments is that the value at which the loss plateaus is different, but all experiments plateau at approx the 1st epoch.</p>
<p>Notably, as my dataset size is extremely small (100 images in total), I opted for a 80-20 train-test split, and augmented the trainset (via random cropping, rotations, etc) to 640 images. Could my problem be related to the limited dataset size?</p>
<p>Here are my training graphs (first 30 epochs) for reference:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b064ccfde96908ec7ea554b8be80a0db8dcc007.png" alt="one_kind" data-base62-sha1="1zwKuRbNZjBu3jZKjuNHRvouaXB" width="320" height="240">  <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f82f31ca3bc5e54590289b71279a68a85b451bb.png" alt="two_kinds" data-base62-sha1="dCVUUVJhcVee79w9T3p20tN7KAj" width="320" height="240"></p>

username: charan_Vjy
cooked: <p>Since the train loss is consistently decreasing, it implies that your model has low bias. The plateued valid loss could indicate high variance. Furthermore, you have tried standard methods to avoid overfitting. Therefore, this seems to be the best you can do with the current model.<br>
Below are three possible conclusions.</p>
<ol>
<li>The train dataset size is small. No matter how well you learn on train data, you are unable to generalize to new distributions - &gt; obtain more data.</li>
<li>The valid dataset could be of a vastly different distribution as compared to the train dataset - - &gt; Try cross validation and verify whether it is indeed the size of the dataset rather than its distribution.</li>
<li>There is a possibility of a mixup in the testloader. Just before you give the input to the model, check and see whether you are giving in the image correctly. Make sure you apply the same scaling that was done during training. - - &gt; fix if present.</li>
</ol>
<p>Try these out and see how it goes.</p>
No module named 'torch.hub' and 'torch.utils.model_zoo'

username: Yimi
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a60581108c8b1f3313c561779ee6fdf616105c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a60581108c8b1f3313c561779ee6fdf616105c6" title="H(Q2{5I@CNA_%7(SRMI(WF"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a60581108c8b1f3313c561779ee6fdf616105c6.png" alt="H(Q2{5I@CNA_%7(SRMI(WF" data-base62-sha1="hsAFnO8NxzGd56ZttSc9WRSbgWi" width="690" height="318" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">H(Q2{5I@CNA_%7(SRMI(WF</span><span class="informations">861×398 65.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The Github code I ran needs Pytorch &gt;= 1.4.0, while Pytorch in my server is 1.3.1. Does the version of Pytorch cause this problem? How can I fix it?</p>

username: ptrblck
cooked: <p><code>torch.hub</code> and <code>torch.utils.model_zoo</code> were already in <code>1.3.1</code>, so could you please double check your version?</p>
<p>Generally, we recommend to use the latest version, as it ships with bug fixes as well as new features.</p>

username: Yimi
cooked: <p>Thanks. I updated the version to 1.4.0 and then this problem was solved.</p>
Linear layer with custom connectivity

username: wpron
cooked: <p>Hi there,</p>
<p>I am new to PyTorch, but I want to create a verysimple linear layer with custom connections.<br>
As an example, I want to create the following connectivity:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/78a4dc8cf6d1407feb9f4d3b77879c01bd050725.png" alt="my_net" data-base62-sha1="hdgvEUOqLS75vswwciIMGzMCQi9" width="77" height="97"></p>
<p>My code does not work:</p>
<pre><code>class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.l1 = nn.Linear(n_feats, 3, bias=False)
    self.t = Variable(torch.randn(N_BATCH, 2))
    self.neurons = [nn.Linear(2, 1), nn.Linear(2, 1)]

  def forward(self, x):
    x = F.relu(self.l1(x))
    self.t[:, 0] = self.neurons[0](x[:, :2])
    self.t[:, 1] = self.neurons[1](x[:, 1:])
    return t
</code></pre>
<p>Besides being ugly, the I expect to have (n_feats x 3 + 2 * 2 + 2) parameters. My model displays only (n_feats x 3) parameters.</p>
<p>What is the correct way to do this?</p>

username: alexis-jacq
cooked: <p>You can either force the weights of the non-existent connexions to equal zero, or do (I think it’s what you are trying) two different fully connected layers. the second solution seems more elegant.</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(2,1)
        self.fc2 = nn.Linear(2,1)

    def forward(self, x):
        splitted = torch.split(x,1,1)
        x1 = torch.cat(splitted[:2],1)
        x2 = torch.cat(splitted[1:],1)
        x1 = F.relu(self.fc1(x1))
        x2 = F.relu(self.fc1(x2))
        return torch.cat([x1,x2],1) 
</code></pre>

username: Lee_Jim
cooked: <p>no such thing as ugly or elegant, cpux, write any is ok</p>

username: uhmbg
cooked: <p>How would the backward work? In the same way? Or is there a change to be made?</p>
Equivalent for tf.squared_difference

username: blade
cooked: <p>Is there a corresponding function for TensorFlow’s <code>squared_difference</code> in PyTorch? or should I just create a function</p>
<pre><code class="lang-auto">def squared_difference(input, target):
    return (input - target) ** 2
</code></pre>
<p>to define it? For the sake of completeness:</p>
<blockquote>
<p><strong><code>tf.math.squared_difference</code>:</strong> Returns (x - y)(x - y) element-wise.</p>
</blockquote>

username: charan_Vjy
cooked: <p>The squared_difference function that you have used should be adequate. Just looked at the forums. There seems to be nothing that explicitly computes what you are looking for.</p>
loss.backward(retain_graph=True) vs summing losses and called .backward() once

username: oneil512
cooked: <p>Hi,</p>
<p>I am training an RNN, and computing a loss for each step of the output sequence. Is calling loss.backward(retain_graph=True) for each step of the output more memory efficient/faster than summing loss = loss_step1 + … + loss_stepn and then calling loss.backward()? I have tried both, and they seem to give the same accuracy. But I’m not sure what is different behind the scenes.</p>

username: albanD
cooked: <p>Hi,</p>
<p>summing the losses and calling a single backward is going to be more efficient unless each loss work with independent networks and it will be the same.<br>
But basically you can’t go wrong with doing the sum and a single backward !.</p>
PoissonNLLLoss stirling's approximation term not getting added

username: mrityunjay
cooked: <p>I’m encountering some unexpected behavior in PyTorch’s  <code>PoissonNLLLoss</code>. The  <code>full</code>  parameter whether set to either  <code>True</code>  or  <code>False</code>, it returns the same answer.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

criterion = nn.PoissonNLLLoss(log_input = True, full = False, reduction = 'sum')

input = torch.tensor([1.0, 1.0, 1.9, 1.6, -1.9, 3.7, -1.0, 0.5], requires_grad = True)
target = torch.tensor([1.0, 3.0, 1.0, 2.0, 1.0, 4.0, 2.0, 1.0])

loss = criterion(input, target)
print('loss:', loss)
loss.backward()
print('grad:', input.grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">loss: tensor(39.1890, grad_fn=&lt;SumBackward0&gt;)
grad: tensor([ 1.9331, -0.3169,  6.3966,  3.3222, -0.9567, 41.0032, -1.8361,  0.7298])

</code></pre>
<p>When the value of <code>full</code> is changed to <code>True</code>, there is no change in result. It seems that Stirling’s Approximation term is not getting added. Can someone tell what’s wrong?</p>

username: mrityunjay
cooked: <p>Ohh. Got it! It is already solved here <a href="https://github.com/pytorch/pytorch/pull/28637/files" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/28637/files</a>. Seems like I need to upgrade PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
When to reset LSTM states for binary classifcation of time series

username: ppmt
cooked: <p>I am working on a binary classifier for time series data with one feature. However, I am unsure when exactly to reset the hidden and cell states and why.</p>
<p>Currently I am using the following classifier:</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):
    def __init__(self,
                 input_features_size=1,
                 hidden_layer_size=30,
                 num_layers=1,
                 output_size=2):
        super().__init__()
        self.lstm = nn.LSTM(input_size=input_features_size,
                            hidden_size=hidden_layer_size,
                            num_layers=num_layers,
                            batch_first=True)
        self.linear = nn.Linear(hidden_layer_size, output_size)
        # number of features in hidden layer
        self.hidden_layer_size = hidden_layer_size
        # number of input features (i.e., features per sequence point)
        self.input_features_size = input_features_size
        # number of layers
        self.num_layers = num_layers
        # size of batches used in gradient descent
        self.batch_size = None
        # tuple with hidden state (a.k.a. short term memory) and cell state (a.k.a. long term memory)
        self.hidden_cell = (None, None)

    def init_hidden(self, batch_size):
        self.batch_size = batch_size
        self.hidden_cell = (torch.zeros(self.num_layers, batch_size, self.hidden_layer_size),
                            torch.zeros(self.num_layers, batch_size, self.hidden_layer_size))

    def forward(self, x_batch):
        batch_size = x_batch.shape[0]
        self.init_hidden(batch_size)
        lstm_out, self.hidden_cell = self.lstm(x_batch, self.hidden_cell)
        predictions = self.linear(lstm_out[:, -1, :])
        return predictions
</code></pre>
<p>The hidden and cell state get reset before every forward pass. The reasoning behind this is that the individual time series instances are seen as independent of each other. Thus, the ‘kept information’  from the previous instance is not relevant for the next one etc. I suppose the actual learning of what information to keep in hidden and cell state when parsing an instance is done by updating the weights of the individual gates.</p>
<p>Is this reasoning correct?</p>
<p>I have found examples resetting the hidden state only once before training, every epoch, and before every forward pass on different sites leaving me quite confused.</p>
<p>Cheers,<br>
ppmt</p>
Input with different length

username: JY_Zhang
cooked: <p>My input data is point clouds, number of points in every file is different, around 5k~2w, when I set batch_size&gt;1, seems like I need to pad them to the same length,   I am puzzled about computation, does padding part cost extra computation?</p>

username: charan_Vjy
cooked: <p>Yes, you would have to pad all the point clouds to the maximum point cloud size. Apart from the very process of padding, even operations on the padded sequence will be computationally more expensive as compared to doing it without padding. The cost would be proportional to the amount of padding required for each point cloud which indirectly depends on the size of the maximum point cloud.</p>

username: JY_Zhang
cooked: <p>Thanks for your reply！</p>
Mixing stochastic gradients with autograd

username: darleybarreto
cooked: <p>Hi there,</p>
<p>I am trying to implement Algorithm 1 of <a href="https://arxiv.org/pdf/1808.02078.pdf" rel="nofollow noopener">this</a> paper, where I need to evaluate a gradient composed by two terms, one I can compute with the usual <code>.backward</code> which is the g_\theta^{mod}(.) (the log likelihood of the model) and the other, g_\theta^{ent}, is the gradient of the entropy which is given by the Hamiltonian Monte Carlo estimate of the stochastic gradient, thus I cannot perform the backward pass on this. How can I apply these gradients on my model?</p>
<p>Thanks in advance.</p>
Why there's not a single metric in pytorch?

username: braindotai
cooked: <p>I’ve been curious for over a year, why not a single metric is available in Pytorch? I’ve got no problem with defining one from scratch and keep copying and pasting it to every project, but wouldn’t it be a bit nicer if we have some.<br>
At least some basic ones like accuracy, topk accuracy, categorical accuracy… etc. Like we have in Tensorflow, for its api consisteny I can give full marks to tensorflow.<br>
I’m just wondering is there any good reason for not to have any?</p>

username: charan_Vjy
cooked: <p>You could convert your tensors into numpy arrays and use scikit learn’s metric functions. That should be pretty straight forward. However, we could actually try writing native metric functions in pytorch.</p>

username: braindotai
cooked: <p>Thanks for pointing it out, I never thought of that.</p>

username: damicoedoardo
cooked: <p>You should check also the <code>Ignite</code> library from Pytorch, you will love it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
<a href="https://github.com/pytorch/ignite" rel="nofollow noopener">Ignite library</a></p>

username: braindotai
cooked: <p>I’ve seen it, loved it for sure, then moved to pytorch lightning loved it for sure as well, found some pitfalls, especially when training a multi model system like gan, so now I’m developing my own wrapper, and by using it I’ve never been this much productive!! I’m loving what I’ve built. May be I’ll share more about it later.</p>
[Meta] What can I do if I can't get support in this forum?

username: pmeier
cooked: <p>The <a href="https://github.com/pytorch/pytorch/blob/master/.github/ISSUE_TEMPLATE/questions-help-support.md" rel="noopener nofollow ugc">question template</a> of for an <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">issue on PyTorch’s GitHub</a> states that one should use this forum as a</p>
<blockquote>
<p>primary mean[s] of support</p>
</blockquote>
<p>Does this mean that the issue tracker the <em>secondary</em> mean of support?</p>
<ul>
<li>If yes, is there a rule of thumb which time has to pass before I “escalate” my question?</li>
<li>If no, what can I do if my question gets no attention here?</li>
</ul>

username: justusschock
cooked: <p>This just means, that the issue tracker is primarily for reporting bugs and stuff that is related to the development. There is such a huge amount of issues, that your question may not be noticed by the developers. Additionally questions can usually be answered by other users as well.</p>
<p>These users usually hang around in this forum, which makes it more likely to get an answer here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: pmeier
cooked: <p><a class="mention" href="/u/justusschock">@justusschock</a></p>
<blockquote>
<p>the issue tracker is primarily for […] <strong>stuff that is related to the development</strong>. (emphasis mine)</p>
</blockquote>
<p>Do you mean <em>stuff that is related to development of PyTorch</em> or does that include <em>stuff that developers might ask while developing with PyTorch</em>?</p>
<p>In my concrete case: I’m working on a framework that builds upon and thus deeply integrates <code>torch</code>. I need support with something that other <em>users</em> of PyTorch most likely cannot answer.</p>
<hr>
<p>P.S. If you want to give it a shot, here is my <a href="https://discuss.pytorch.org/t/how-to-correctly-annotate-subclasses-of-nn-module/74317">original question</a> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: justusschock
cooked: <p>Most likely it’s a mixture of both <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> However, the devs might focus more on the stuff that is related to development of PyTorch.</p>
<p>PS: Regarding your question: I don’t have any idea <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Theory question on autograd

username: James_e
cooked: <p>I have some questions regarding the theory of autograd:</p>
<p>1.) Are y and x the input and output respectively?<br>
2.) Where does v come from? and how is it computed?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8429ff18a289d060a86c64f2e4988dd6b803149.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8429ff18a289d060a86c64f2e4988dd6b803149" title="Screenshot 2020-03-24 at 23.31.42"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8429ff18a289d060a86c64f2e4988dd6b803149_2_690x315.png" alt="Screenshot 2020-03-24 at 23.31.42" data-base62-sha1="szAeH6NE4TU9LcwpAN9flGR9UbT" width="690" height="315" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8429ff18a289d060a86c64f2e4988dd6b803149_2_690x315.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8429ff18a289d060a86c64f2e4988dd6b803149_2_1035x472.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8429ff18a289d060a86c64f2e4988dd6b803149_2_1380x630.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-03-24 at 23.31.42</span><span class="informations">1662×760 92.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>When you write <code>y = f(x)</code>, that means that x is the input. And y is the output.</li>
<li>v can be any Tensor you want. A special example is if you function has a single output (like a loss function in NN) then J will be a simple row and by setting v = 1, you get the gradient of your function.</li>
</ol>
IDSGAN RunTime error

username: headsun21
cooked: <p>Hi !<br>
I’m new to pytorch and try to run IDSGAN. (<a href="https://github.com/imoken1122/IDSGAN" rel="nofollow noopener">https://github.com/imoken1122/IDSGAN</a>)<br>
I got RunTime error(below) during running IDS-WGAN.py</p>
<hr>
<h2>Traceback (most recent call last):<br>
File “C:/논문/IDSGAN-master/IDSGAN-master/IDS_WGAN.py”, line 47, in <br>
ids_model.load_state_dict(param)<br>
File “C:\anaconda3\envs\tutorial\lib\site-packages\torch\nn\modules\module.py”, line 830, in load_state_dict<br>
self.<strong>class</strong>.<strong>name</strong>, “\n\t”.join(error_msgs)))<br>
RuntimeError: Error(s) in loading state_dict for Blackbox_IDS:<br>
size mismatch for layer.6.weight: copying a param with shape torch.Size([33, 66]) from checkpoint, the shape in current model is torch.Size([16, 66]).<br>
size mismatch for layer.6.bias: copying a param with shape torch.Size([33]) from checkpoint, the shape in current model is torch.Size([16]).<br>
size mismatch for layer.9.weight: copying a param with shape torch.Size([16, 33]) from checkpoint, the shape in current model is torch.Size([16, 16]).</h2>
<p>Waiting for your help. Thanks.</p>

username: charan_Vjy
cooked: <p>The model from which you are loading the checkpoint from and the model that you are currently using differ in certain layers as mentioned in the traceback. I just had a look at the model. Did you tweak the input to the model instantiation? I have pasted some code from the tutorial below to clarify the question.</p>
<blockquote>
<p>D_G_INPUT_DIM = len(train_data.columns) → Are you using different train data?<br>
ids_model = Blackbox_IDS(D_G_INPUT_DIM,2) → What is your D_G_INPUT_DIM?<br>
param = th.load(‘save_model/IDS.pth’)<br>
ids_model.load_state_dict(param)</p>
</blockquote>
Why is it so hard to train GANs?

username: lycaenidae
cooked: <p>I will be starting a new project related to my master studies but my supervisor warned me about GAN training. He told me it would be tough if I don’t have any prior experience in training GANs. I read several blog posts, but I am really wondering why is it so hard to train them? What do you suggest to tackle that problem?</p>

username: ptrblck
cooked: <p>From my experience implementing some GANs from scratch (a couple of years ago now), I would recommend to have a lot of patience. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
I had to run a lot of experiments to try to figure out what causes e.g. mode collapse etc.</p>
<p>I think there are a lot of good repositories now, which will work as a starter, also there was a lot of progress in this field.</p>
How to get variance inside each cluster?

username: Mah
cooked: <p>Hello! I have this code and I want to get the variance inside each cluster. Thanks for helping.</p>
<p>import torchvision.datasets as dsets<br>
train_dataset = dsets.MNIST(root=’./data’, train=True, transform=transforms.ToTensor(), download=True)<br>
from sklearn.cluster import KMeans<br>
num_class=10<br>
data = train_dataset.data.numpy()<br>
data=data.reshape(-1,28*28)<br>
kmeans = KMeans(n_clusters=num_class, random_state=0).fit(data)<br>
center=kmeans.cluster_centers_<br>
centers=torch.from_numpy(center)</p>

username: ptrblck
cooked: <p>This doesn’t seem to be PyTorch-specific, as you are using <code>sklearn</code>.<br>
You could probably use the <code>labels_</code> to get all samples for the clusters and then calculate the variance between these samples.</p>