akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Modify predicted class label to category label before computing the loss

username: saywwhaaaat
cooked: <p>I am doing semantic segmentation task and I already train a model with a dataset that provides class label.</p>
<p>Now, I have a new dataset that only provide category label rather than the class labels I used to train. But, to utilize the pretrained model, I have to either adapt the prediction layer channels from number of classes to number of category; or, I map the predicted class labels to category labels, then use this to compute cross entropy loss as before.</p>
<p>For me, the second option sounds optimal, because all layers in the pretrained model can be applied.</p>
<p>But, if I modify the predicted label, what would be the consequence to autograd?</p>
<p>Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="saywwhaaaat" data-post="1" data-topic="80087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/48db29/48.png" class="avatar"> saywwhaaaat:</div>
<blockquote>
<p>But, if I modify the predicted label, what would be the consequence to autograd?</p>
</blockquote>
</aside>
<p>I’m citing <a class="mention" href="/u/kfrank">@KFrank</a> here:</p>
<blockquote>
<p>The number-one rule is that your output means whatever you train it to mean…</p>
</blockquote>
<p>I’m not sure what kind of classes and categories you are using and how similar they are.<br>
E.g. if you remap “husky” and “bulldog” to a “dog” class, your model might still work fine.<br>
Depending how the remapping is done, you might need to retrain the model.</p>

username: saywwhaaaat
cooked: <p>Basically 1 category  is like a collection of class labels, who belongs to this category. For example flat as a category includes road, road line, sidewalk.  The pretrained model is trained on class level, but what I want to fine tune is to predict category label.</p>
<p>I just want to know modify the prediction before computing the loss, whether this is a legal operation, because I feel like manually modification is not differentiable. Maybe I can add a additional mappling layer right after the pretrained model, that learn how to map class label to category?</p>

username: ptrblck
cooked: <p>Since the mapping is “hard-coded” you could keep the model with the class outputs and later use the prediction to map it to the corresponding category.</p>
<p>However, another way would be to let the model learn to combine the classes to categories.<br>
To do so you could add another layer, get the class predictions as its input, and output the categories.<br>
This might allow the model to learn that e.g. 5 “medium predictions” of flat classes are a stronger signal than a single higher prediction of a non-flat class, assuming that the target is the flat category.</p>
Is loss function reduction="something" have significant effect on training performance?

username: ThaiThien
cooked: <p>I want to consider 2 case:</p>
<p>First, with reduction = sum</p>
<p>crit = nn.MSELoss(reduction=‘sum’).to(device)<br>
…<br>
for data, label in batch:<br>
output = model(data)<br>
loss = crit(output, data)<br>
loss.backward()<br>
print(loss.data[0])</p>
<p>and second, with no reduction, we sum when we want output (print something, write log)</p>
<p>crit = nn.MSELoss().to(device)</p>
<p>…<br>
for data, label in batch:<br>
output = model(data)<br>
loss = crit(output, data)<br>
loss.backward()<br>
print(loss.data[0].sum)</p>
<p>The first example is no sum, we sum when we need to print out. The second example is we define reduction=‘sum’. I want to know if it have impact on training process.</p>
<p>That is for my case. But I don’t think it is importance information here: The code was suppose for computer vision problem. Output is a 2D matrix, with batch also, so output is (batch, a, b), input is (batch, channel, a, b).</p>

username: russellizadi
cooked: <p>Depending on the definition of your specific loss function, the reduction may affect the training performance. One of the advantages of <code>reduction=mean</code> is that it makes the update term independent of the batch size. So if you want to use a constant learning rate, you better take the average of loss values. Again, it totally depends on the task and the derivation of the loss function.</p>

username: nmtp
cooked: <p>Just curious, is there ever a time when the reduction=“sum” would be preferable? It seems like a lot more effort to keep track of batch size but I’ve seen “sum” used as well</p>
What's faster? 1 tall convolution or many short convolution stacked?

username: makslevental
cooked: <p>this is strictly about inference rather than training</p>
<p>let’s say i want to produce a stack of 50 filtered images from one 1 channel input image. is it faster to convolve with a stack of 50 filters or to convolve with (e.g.) 10 stacks of 5 filters and then <code>torch.stack</code>?</p>
<p>right now i’m performing the former - building a convolutional unit with 50 filters and then convolving with the image. i profiled and i’m surprised by how slow it is: ~3s for one inference pass. does that seem right?</p>
<p>just in case anyone XYs me: i have a very specific reason for doing this and I am in fact needing 50 filtered images.</p>
<p>thanks!</p>
<p>Edit: I should probably mention that the kernels are pretty wide as well ~50x50 - ~100x100</p>

username: Stacie_Davis
cooked: <p>Maybe just %time both approaches if you are using jupyter? As it is taking just  3s, you can certainly experiment.<br>
Interested in your findings as well!</p>

username: makslevental
cooked: <p><code>%time</code> actually led me astray - it was nvprof that revealed to me the real time.</p>

username: Stacie_Davis
cooked: <p>And which one of them turned out to be faster?</p>

username: makslevental
cooked: <p>i haven’t compared them head to head. what i’m saying is when i timed my implementation initially using just <code>%timeit</code> i was misled to believe that much of my time cost was host to device copy and back whereas nvprof reveals that it’s actually the convolution.</p>
Understanding TensorboardX graph

username: paul_shuvo
cooked: <p>I’m inspecting my model in the tensorboardX. Following is the <code>forward</code> method:</p>
<pre><code class="lang-python">    def forward(self, x):
        x = self.net(x)
        x = x.view(x.shape[0], -1)  # flat
        x = self.ext_net(x)
        x = F.log_softmax(x, dim=1)

        return x
</code></pre>
<p>In the graph, I don’t understand what these values and nodes in blue and green boxes represent. Why is there a separate dataflow from <code>input</code> to <code>ext_net</code> (<em>FCNCustom</em>)? Shouldn’t there be only one coming through <code>net</code>?</p>
<p><a href="https://i.stack.imgur.com/YeMno.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/YeMno.png" alt="enter image description here" width="430" height="499"></a></p>
Doubt regarding usage of model.eval() function in renet18 architecture

username: MJN
cooked: <p>here is the code snippet that I used to calculate the error and accuracy of the data. Not sure if the accuracy logic is correct or not.</p>
<ol>
<li>I read that model.eval() helps to change the functionality of dropout and batch normalization but not sure what it really means.</li>
<li>Can I calculate error and accuracy of renet18 model without using model.eval()?</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c23cc4a3766ddb8a9d8e125ba085623b940b3133.png" data-download-href="https://discuss.pytorch.org/uploads/default/c23cc4a3766ddb8a9d8e125ba085623b940b3133" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c23cc4a3766ddb8a9d8e125ba085623b940b3133_2_619x500.png" alt="image" data-base62-sha1="rIiP6QzW0OblHYi8B0PrSnbhURR" width="619" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c23cc4a3766ddb8a9d8e125ba085623b940b3133_2_619x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c23cc4a3766ddb8a9d8e125ba085623b940b3133.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c23cc4a3766ddb8a9d8e125ba085623b940b3133.png 2x" data-dominant-color="31322D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">809×653 65.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: iamgroot42
cooked: <p>The behaviour of certain layers changes when the model is in inference mode i.e. <code>model.eval()</code>. For dropout, this means the dropout layer won’t drop out any neurons. For batch normalization, this means the layer’s summary statistics won’t be updated with the given data. You <strong>can</strong> calculate your metrics without putting the model in evaluation mode, but there is no good reason to do that.</p>
<p>To try it yourself: if you’re not using <code>model.eval()</code>, you’ll notice that your evaluation metrics (and loss) will change across multiple runs of your evaluation script.</p>
Multiple indices of specific value

username: tani
cooked: <p>Hello,<br>
I would like to get in a tensor of shape (Nb_of_values, Nb_max_indices) all the indices of specific values from a list without using “for loop”.</p>
<p>Bellow you can find my script. In order to optimize it on GPU I wish to avoid the use of a loop.<br>
<strong>Is there a pytorch function for that?<br>
Does somebody have an idea?</strong><br>
Thank you in advance.</p>
<pre><code class="lang-auto">import torch

max_idx = 6
data_idx = torch.tensor([2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0]).cuda()

max_number_data_idx = data_idx.shape[0]

filled_matrix = torch.zeros([max_idx, max_number_data_idx], dtype=torch.int8, device='cuda')
filled_matrix.fill_(-1)
for i in range(max_idx):
   same_idx = (data_idx == i).nonzero().flatten()
   filled_matrix[i][:same_idx.shape[0]] = same_idx

</code></pre>
"Learnable" parameter does not want to learn

username: Aeglen
cooked: <p>I am trying to use “mult” as learnable weights in this model, yet despite being in model.parameters() and requires_grad=True the weights aren’t being updated (and indeed mult.grad = None). What could have gone wrong?</p>
<pre><code class="lang-auto">    class NeuralNetB(nn.Module):
        def __init__(self):
            super(NeuralNetB, self).__init__()
            self.mult = nn.Parameter(torch.rand(3), requires_grad=True).to(device) #todo: why not learnable?
            self.fc1 = nn.Linear(6, 64)
            self.relu = nn.LeakyReLU()
            self.fc2 = nn.Linear(64, 16)
            self.fc3 = nn.Linear(16, 3)
            self.drop = nn.Dropout(0.4)
            self.fc4 = nn.Linear(30, 3)

        def forward(self, x, tc):
            print(self.mult)
            x2 = x/tc.view(tc.shape[0],1,1) #get partial averages
            x2 = x2*(self.mult*10).view(1,3,1) #multiply by weights
            out = torch.sum(x2, dim=1) #sum them up
            #print(self.mult)
            out = self.fc1(out)
            #out = self.relu(out)
            out = self.fc2(out)
            out = self.relu(out)
            out = self.fc3(out)
            #out = self.drop(out)
            #out = self.fc4(out)
            return out
</code></pre>
<p>Training:</p>
<pre><code class="lang-auto">            y_pred = model(train_xB[v], train_count[v])
            model.zero_grad()
            loss = criterion(y_pred, train_yB[v])
            loss.backward()
            optimizer.step()
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>nn.Parameter()</code> always require gradients, no need to specify it.<br>
Also when you do <code>.to()</code> you get a new Tensor (which is not a leaf Tensor !) Didn’t you get a warning when you did the optimizer step?</p>
<p>You should change the definition to <code>self.mult = nn.Parameter(torch.rand(3, device=device))</code></p>

username: Aeglen
cooked: <p>Ah that worked, thanks so much! There was no warning.</p>
Question on training LSTM to play Snake

username: waw_waw
cooked: <p>Hi,</p>
<p>I follow the DQN tutorial and trained a CNN to play a game, now I want to switch to LSTM but got a problem.</p>
<p>I found the tutorial for LSTM, and it recommends using (time sequence, Batch size, features). In my code, my input to LSTM is (4,32,100), where 4 means 4 consecutive frames, 32 is the batch size, 100 is a vector representing the current state.</p>
<p>Then I add a nn.Linear layer after LSTM, and the input size of linear layer is 4<em>32</em>lstm_output_size, and here comes the problem. In trainig, the batch size is 32-&gt;(4,32,100), but in testing, the batch size is1 -&gt; (4,1,100), which will cause an error.</p>
<p>I tried to train the LSTM with batch size 1, but it will take significantly longer time, is there any way that can let me train the LSTM with batch=32 and do inference with batch=1?</p>

username: ptrblck
cooked: <p>The number seems to be formatted a bit wrong, but I understand that the input features to the linear layer are defined as <code>seq_len * batch_size * lstm_output_size</code>?<br>
You shouldn’t use the batch size as the number of input features, as this will limit your use case to only this particular size, and assumes that the samples in the batch are somehow related to each other.</p>
<p>Instead, <code>permute</code> the LSTM output so that the batch size is at dim0 and reset the <code>in_features</code> to e.g. <code>seq_len * lstm_output_size</code>.<br>
This will allow you to use arbitrary batch sizes.</p>
<p>Alternatively, you could also use <code>batch_first=True</code> when you are creating an instance of <code>LSTM</code>, which will accept and return the batch size in dim0.</p>
Custom loss is not updating the weights

username: Rajan_Lagah
cooked: <p>I do have list of history of states.<br>
Then by numpy i calculate <strong>discounted_rewards</strong><br>
Then i multiply output of model with discounted_rewards using <strong>torch.mm</strong><br>
then</p>
<pre><code class="lang-auto">  print(self.global_model.state_dict())
    print("total_loss",total_loss)
    total_loss.backward()
    self.opt.step()
    print(self.global_model.state_dict())
</code></pre>
<p>it’s output is</p>
<blockquote>
<p>(‘dense1.weight’, tensor([[ 0.3997, -0.1907,  0.1120,  0.3016],<br>
[ 0.1156,  0.0646,  0.1802,  0.3558],<br>
[ 0.0321,  0.2537,  0.0879,  0.2441],<br>
[-0.2952, -0.0886, -0.3235,  0.3006]])), (‘dense1.bias’, tensor([ 0.1927,  0.3048, -0.3551, -0.0302])), ('dense2.weig</p>
<p>total_loss.backward() tensor(2.5806, dtype=torch.float64, grad_fn=)</p>
<p>(‘dense1.weight’, tensor([[ 0.3997, -0.1907,  0.1120,  0.3016],<br>
[ 0.1156,  0.0646,  0.1802,  0.3558],<br>
[ 0.0321,  0.2537,  0.0879,  0.2441],<br>
[-0.2952, -0.0886, -0.3235,  0.3006]])), (‘dense1.bias’, tensor([ 0.192</p>
</blockquote>
<p>and</p>
<pre><code class="lang-auto">self.opt = torch.optim.SGD(self.global_model.parameters(),lr = 0.01)
</code></pre>
<p>So it is not updating the weights. what i am missing ?</p>
<p>Model is</p>
<pre><code class="lang-auto">
        self.dense1 = torch.nn.Linear(4,4) 
        self.dense2 = torch.nn.Linear(4,4) 
        self.dense3 = torch.nn.Linear(4,4) 
        self.dense4 = torch.nn.Linear(4,4) 
        self.probs = torch.nn.Linear(4,2) 
        self.values = torch.nn.Linear(4, 1)
</code></pre>

username: cskarthik7
cooked: <p>Can you share the loss function code and the part of the program where you have called the criterion?</p>

username: Rajan_Lagah
cooked: <p>Hey i have notice that<br>
If i do</p>
<pre><code class="lang-auto">            print(self.local_model.state_dict())
            print("total_loss.backward()",total_loss)
            total_loss.backward()
            opt_2 = torch.optim.SGD(self.local_model.parameters(),lr = 0.01)
            opt_2.step()
            self.opt.step()
            print(self.local_model.state_dict())
</code></pre>
<p>Then it show weight change.<br>
But i want to update weights using loss of global_model.<br>
I am using local_model to predict the result<br>
then i want to apply grad to global model<br>
A3C stuff<br>
Do you know how to ?</p>

username: cskarthik7
cooked: <p>The predictions and weights will be same all the time if you don’t backpropogate local_model and the loss function will always return a constant value because since the gradients of local_model are not being updated,it will give same prediction every time and labels are also constant so if you are able to interpret this then you will notice that loss will come out constant all the time so while you backpropogate through global_model, the gradients will be same.</p>
Run a torch model inside a parallelized loop

username: Nicolas_Pozin
cooked: <p>Hi everyone,</p>
<p>I run a python loop which is parallelized with multiprocessing lib. Each iteration of the loop calls a trained torch model so as to make a prediction. It all works on CPU.</p>
<pre><code>    for _ind in inds:
        # process each sample
        f = pool.apply_async(function_launching_torch_model,
                             [some_parameters])
</code></pre>
<p>It appears that the torch model is very slow, much more than when the loop is sequential. I guess there’s a “conflict” in the use of the procs?<br>
Anyone would have a suggestion to keep a parallel loop while getting good performance?</p>
<p>Thnaks in advance !</p>
Get loss as a map of loss per pixel

username: Yakirs
cooked: <p>I’m working on a semantic segmentation project.</p>
<p>I can get the loss using this (simplified):</p>
<p>loss = nn.BCELoss(predicted_weights, gt_weights)</p>
<p>But I want to get the loss per pixel. I want to know how much loss each pixel contributed to the loss, and change the weights between them before I’m doing the backward propagation.</p>
<p>Thanks</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/yakirs">@Yakirs</a> Maybe you can set <code>reduction='none'</code>. For example,</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

y = torch.rand(4, 10, 10)
y_true = torch.randint(0, 1, size=(4, 10, 10))
loss = nn.BCELoss(reduction='none')
loss(y, y_true).shape

&gt; torch.Size([4, 10, 10])
</code></pre>

username: Sanjayvarma11
cooked: <p>hi but for a multi-dimension output how would plan on doing backpropagation</p>
Does autograd save copies of constant buffers?

username: apsod
cooked: <p>If i have a pytorch module <strong>M</strong> with a constant matrix buffer T, where M.forward(x) = x @ T, If i apply this module <strong>M</strong> at several points in a deep model, will autograd save one copy of the (constant) matrix T per invocation, or will it “realize” that the matrix T is constant and just have one copy?</p>

username: ptrblck
cooked: <p>If <code>T</code> is registered as a buffer, it will be stored in memory once.<br>
The output created by e.g. <code>x @ self.T</code> will of course be a new tensor.<br>
Could you explain your question a bit more, as I think I might have misunderstood the problem?</p>

username: apsod
cooked: <p>So given a deep model <code>f=(f_3 . M . f_2 . M. f_1)</code>, when we evaluate <code>f(x)</code>, <code>M</code> is applied twice. Once to <code>f_1(x)</code> and once to <code>f_2(M(f_1(x)))</code>, the output at each point will of course be a new tensor.<br>
If <code>T</code> is not constant, and if we needed to compute the gradient w.r.t. T and the input to M, we would need to save the both T and the inputs to M in the forward pass to be able to compute the backward pass. But since T is constant, and we only need to compute the gradient w.r.t. x, the backward pass of M is itself constant (it’s T).<br>
The question is whether I can safely assume that autograds computation graph is constructed in such a way that it does not store unnecessary copies of T. (This might very well be a naive question, with an obvious answer, but I’m not super knowledgeable about the inner workings of autograd <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>

username: ptrblck
cooked: <p>Yes, Autograd should figure this out.<br>
Here is a small example, which shows that leaf variables are treated in a special way. I.e. inplace modifications are now allowed, if the value of the variable is needed to calculate the gradient:</p>
<pre><code class="lang-python"># Both require gradients
x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=True)

out = x * y
out.backward()

# Modify inplace
x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=True)

y[0] = 2 # error
out = x * y
out.backward()
&gt; RuntimeError: leaf variable has been moved into the graph interior

# Modify inplace of constant
x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=False)

y[0] = 2 # works
out = x * y
out.backward()
</code></pre>
<p>While the inplace modification works for “constant” tensors.</p>

username: apsod
cooked: <p>I see. And the following also fails:</p>
<pre><code class="lang-auto">x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=False)

y[0] = 2 # Works
l1 = x * y
y[0] = 3 # Does not work
l2 = l1 * y
out.backward()
</code></pre>
<p>Which makes sense.<br>
Thanks for the reply!</p>
Save model each epoch

username: Chaoying_Wu
cooked: <p>I want to save model for each epoch</p>
<p>but my training process is using model.fit();<br>
not using for loop<br>
the following is my code:<br>
model.fit(inputs, targets, optimizer, ctc_loss, batch_size, epoch=epochs)<br>
torch.save(model.state_dict(), os.path.join(model_dir, ‘savedmodel.pt’))</p>
<p>any suggestion to save model for each epoch</p>
<p>thanks in advance</p>

username: ptrblck
cooked: <p>Did you define the <code>fit</code> method manually or are you using a higher-level API?<br>
In the former case, you could just copy-paste the saving code into the <code>fit</code> function. In the latter case, I would assume that the library might provide some “on epoch end” - callbacks, which could be used to save the model.</p>
Sequence multilabel classification accuracy problem

username: omph
cooked: <p>I’m trying to implement a multi-label classification task, and currently my model has Embedding, GRU, 2x Linear layers.</p>
<p>I have padded the data, and its shape is (seq_len x batch) where seq_len is the longest sequence in that batch. Targets are multi-hot encoded as I’m using BCEWithLogitsLoss.</p>
<p>I have a weird issue that when using batch size &gt; 1, I get much lower accuracy (0.3) when using batch size = 1 (0.8). I suspected it might be padding thing, but I was also able to reproduce this with same length sequences. I’m trying my luck here if anyone has encountered something similar and what the problem was?</p>

username: ptrblck
cooked: <p>How large is the batch size for the low accuracy?<br>
Huge batch sizes tend to give a worse accuracy, but the gap seems to be too large for it.</p>
<p>Could you post the code of your model and training routine, so that we could have a look, if maybe a slicing operation is wrong?</p>

username: ptrblck
cooked: <p>I assume your <code>input</code> has the size <code>[batch_size, seq_len]</code>?<br>
If so, then <code>self.gru</code> would get an input of <code>[batch_size, seq_len, embedded_dim]</code>, while it expects an input of <code>[seq_len, batch_size, input_size]</code> in the default setup.</p>
<p>If my assumptions are correct, you could either <code>permute</code> the input or use <code>batch_first=True</code> while creating the <code>nn.GRU</code>, which would then expect an input of <code>[batch_size, seq_len, features]</code>.</p>

username: omph
cooked: <p>Actually it is seq_len x batch_size. I printed shapes for all layers:</p>
<pre><code class="lang-auto">inputs shape:  torch.Size([377, 2])
embeds shape:  torch.Size([377, 2, 100])
hidden[0] shape:  torch.Size([2, 64])
output shape:  torch.Size([2, 126])
</code></pre>

username: ptrblck
cooked: <p>The shapes look correct.<br>
I’m currently unsure, what might be wrong <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Did you verify that your <code>collate_fn</code> is creating valid batches?</p>

username: omph
cooked: <p>Afaik yes. But anyway this helps as at least it’s not some trivial bug in the model, but might be related to something else. Thanks for your help!</p>
Giving more than one path to Dataloader

username: arian
cooked: <p>I have multiple path with folders containing images, is it possible to use pytorch Dataloader reading from all of them?</p>

username: ecdrid
cooked: <p>Yes it’s definitely possible! Try searching Google or the docs next time;</p>
<p>You are looking for  <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder" rel="nofollow noopener"><em>torchvision.datasets.ImageFolder()</em>!</a></p>

username: arian
cooked: <p>Tnx for your response, I have multiple sources for training. My data folders are like below:</p>
<pre><code class="lang-auto">Train/
         class1/
                   pos/
                   neg/

          class2/
                   pos/
                   neg/
</code></pre>
<p>When I’m using this with datasets.ImageFolder() it gives wrong labels to images, for example all negative should be 0 and all positives should be 1. how can I do this?</p>
Fix backbone and task1 parameters, fine-tune task2 's parameters

username: nieyan
cooked: <p>I test task1’s acc after resume model. And accuracy value match to resume file at first. But after some iterations of training , task1’s accuracy has changed. Don’t know what’s wrong about training. I think fix backbone and task1 parameters should get always the same task1’s accuracy .</p>
<p>code of fix backbone and task1 parameters:</p>
<pre><code class="lang-auto">for p in self.model.parameters():
    p.requires_grad = False
for q in self.model.task2.parameters():
    q.requires_grad = True
</code></pre>

username: ptrblck
cooked: <p>Besides setting the <code>requires_grad</code> attribute to <code>False</code>, you might need to call <code>.eval()</code> on the layers involved for “task1”.<br>
This will make sure to disable dropout and use the running stats of batchnorm layers.<br>
If you keep the model in <code>train()</code> mode, the batchnorm stats will be updated and might thus change the accuracy during evaluation.</p>

username: nieyan
cooked: <p>Thank you very much !!! <img src="https://discuss.pytorch.org/images/emoji/apple/100.png?v=9" title=":100:" class="emoji" alt=":100:"> <img src="https://discuss.pytorch.org/images/emoji/apple/100.png?v=9" title=":100:" class="emoji" alt=":100:"> <img src="https://discuss.pytorch.org/images/emoji/apple/100.png?v=9" title=":100:" class="emoji" alt=":100:"></p>
Improving training time for LSTM model

username: dachosen1
cooked: <p>I created an LSTM model and rented a GPU and CPU in the cloud for training. The problem is that the model isn’t using all the available resources. My CPU utilization is less than 5% and my GPU is at ~20%. The GPU utilization does follow a sin wave pattern. Which I suspect is due to turning my GPU for validation.</p>
<p>Is there a way to speed up the training time by using 100% of the CPU and GPU?</p>

username: ptrblck
cooked: <p>If you GPU is not fully utilized, your training might suffer from bottnecks such as data loading or small workloads on the GPU.<br>
In the former case, you could try to use synthetic data and check the GPU utilization again.<br>
If you are dealing with a small model and small batches, the actual computation on the GPU might be too small so that the overhead from the data transfer to the device as well as the kernel launches might be visible.</p>

username: dachosen1
cooked: <p>I believe the bottle neck is in data loading. I’m using Dataset folder to load data in batches. What do you mean by using synthetic data?</p>
<p>would increasing the number of workers in the Dataloader method increase the number workload on the GPU?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="dachosen1" data-post="3" data-topic="79716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/47e85d/48.png" class="avatar"> dachosen1:</div>
<blockquote>
<p>What do you mean by using synthetic data?</p>
</blockquote>
</aside>
<p>Instead of loading real data you could initialize random data, push it to the GPU and profile the training loop without the data loading:</p>
<pre data-code-wrap="python"><code class="lang-python"># for a multi-class classification
data = torch.randn(batch_size, channels, height, width, device='cuda')
target = torch.randint(0, nb_classes, (batch_size,), device='cuda')

for epoch in range(10):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
</code></pre>
<aside class="quote no-group" data-username="dachosen1" data-post="3" data-topic="79716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/47e85d/48.png" class="avatar"> dachosen1:</div>
<blockquote>
<p>would increasing the number of workers in the Dataloader method increase the number workload on the GPU?</p>
</blockquote>
</aside>
<p>Generally yes, but too many workers might again slow down your system. The sweet spot depends on your current workstation and you would have to profile specific values.</p>
DataLoader creating small batches with leftovers that cause errors

username: jpc
cooked: <p>I have a validation set that is 1302 examples long. I create a DataLoader object from the data set and pass my desired batch size, which is 50.</p>
<p>During training, I iterate over the DataLoader, which provides a batch of size 50 and the associated labels. However, the last batch causes an error to be throw at the nn.LSTM layer because it expects a hidden[0] size of (a, 50, b) but instead received size (a, 2, b).</p>
<p>It appears DataLoaders provides a final batch with the leftover 2 examples.</p>
<p>Are there any solutions, possibly specific to the LSTM class, that do not involve either throwing out those last two examples or finding a different batch size the data can be equally divided into?</p>
<p>My training set size is a prime number, so I cannot find a number they are both divisible by, and I want to avoid throwing out data as much as possible.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You could initialize the LSTM states using the currently incoming input shape instead of the global batch size.<br>
If that’s not possible for some reason, you would have to remove these samples using <code>drop_last=True</code> in the <code>DataLoader</code>.</p>
How to input a tensor with shape (1899, 14, 30491) into LSTM

username: SamYuen101234
cooked: <p>I have a problem about the how to fit my data into a LSTM model.</p>
<p>The shape of the training features is: torch.Size([1899, 14, 30491]) and the shape of training labels is: torch.Size([1899, 30490]). 14 is the window size and 30491 is the 30490 products in a shop plus 1extra 0/1 feature. 1899 is the time series that 1899 days.<br>
I know keras this can be simply written as</p>
<pre><code class="lang-auto">regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
layer_1_units=40
regressor.add(LSTM(units = layer_1_units, return_sequences = True, input_shape = (X_train.shape[1], X_train.shape[2])))
</code></pre>
<p>The X_train.shape[1] is 14, the X_train.shape[2] is 30491.</p>
<p>My model in pytorch is:</p>
<pre><code class="lang-auto">class M5_predictor(nn.Module):
    def __init__(self, in_features, out_features, n_hidden, n_layers, dropout):
        super(M5_predictor, self).__init__()
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.linear = nn.Linear(n_hidden, out_features)
        #self.drop = dropout
        #self.sigmoid = nn.Sigmoid()
        self.rnn1 = nn.LSTM(input_size=in_features,
                            hidden_size=n_hidden
                            #num_layers=n_layers,
                            )
    def forward(self, in_data):
        #print(in_data.view(in_data.shape[1], in_data.shape[0], in_data.shape[2]).shape)
        batch_size, seq_len, in_features = in_data.size()
        rnn_out, hidden = self.rnn1(in_data)
        output = self.linear(rnn_out)
        return output
</code></pre>
<pre><code class="lang-auto">n_hidden = 40
n_layers = 1
dropout = 0.2
lr = 0.001
in_features = 30491
out_features = 1

# build model
criterion = nn.MSELoss()
model = M5_predictor(in_features, out_features, n_hidden, n_layers, dropout).to(device)
optimizer = Adam(model.parameters(), lr=lr)
</code></pre>
<p>How can I modify my code on pytorch that let the data fit the model and give outputs with shape (batch_size, 30490)</p>
Type_as also changes tensor's device

username: bluehood
cooked: <p>As per the title: <code>type_as</code>, surprisingly, also changes the tensor’s device, and not just its type.<br>
That’s not what I expected from <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.type_as" rel="nofollow noopener">the documentation</a>.</p>
<p>What is the correct way to change a tensor’s type to the type of another tensor, without changing its device?</p>
<p>Cheers,<br>
Enrico</p>

username: ptrblck
cooked: <p>Hi Enrico!</p>
<p>That seems indeed a bit strange.<br>
Does the alternative work fine?</p>
<pre><code class="lang-python">a = a.type(b.type())
</code></pre>
<p>If so, the documentation (and method naming) seems to be inconsistent.</p>

username: MariosOreo
cooked: <p>I tried the <code>tpye_as</code> method, it does not change the tensor’s device.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
<pre><code class="lang-auto">a = torch.randint(0,5,(2,2),device='cuda:0')
b = torch.randn(2,2,device='cuda:1')
a = a.type_as(b)
a.device 
-&gt; device(type='cuda', index=0)
b.device
-&gt; device(type='cuda', index=1)
</code></pre>

username: drewhayward
cooked: <p>It seems like the tensor device will be changed to cuda:0 if is originally on the CPU. Is that intended behavior?</p>
<pre><code class="lang-auto"># a gets moved to cuda:0
a = torch.randint(0,5,(2,2),device='cpu')
b = torch.randn(2,2,device='cuda:0')
a = a.type_as(b)
a.device
-&gt; device(type='cuda', index=0)
b.device
-&gt; device(type='cuda', index=0)

# a is not moved
a = torch.randint(0,5,(2,2),device='cuda:0')
b = torch.randn(2,2,device='cuda:1')
a = a.type_as(b)
a.device
-&gt; device(type='cuda', index=0)
b.device
-&gt; device(type='cuda', index=1)
</code></pre>

username: ptrblck
cooked: <p>This behavior is tracked in <a href="https://github.com/pytorch/pytorch/issues/33662">this issue</a>.<br>
Note that <a href="https://github.com/pytorch/pytorch/issues/37863">this issue</a> suggests to deprecated <code>type_as</code> and <code>type</code> as also discussed in the first link.</p>
Questions about Learnable parameters

username: oasjd7
cooked: <pre><code class="lang-auto">optimizer = [includes params of Encoder + Classifier1 + Classifier2]
# Network Flow
Encoder  -&gt; Classifier1 -&gt; Backprop loss1 from Classifier1.
         -&gt; Classifier2 -&gt; Backprop loss2 from Classifier2.

I want to add learnable parameter 'w' to loss1 like 'loss1 * w'
Is it possible that w is adjusted by Classifier2 not Classifier1 ?


</code></pre>
Label smoothing for CrossEntropyLoss

username: kaiyuyue
cooked: <p>Hi:<br>
Are there some methods to hack the code to implement the <a href="https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/ops/losses/losses_impl.py#L706" rel="nofollow noopener">label smoothing</a> for <code>CrossEntropyLoss</code>? Because the <code>target</code> must be <code>torch.LongTensor</code> to hinder the soft target (<code>torch.FloatTensor</code>).</p>

username: kaiyuyue
cooked: <p>Mirror issue: <a href="https://github.com/pytorch/pytorch/issues/7455" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/7455</a></p>
RuntimeError: Error attempting to use dtype torch.float64 with layout torch.sparse_coo and device type cuda

username: xiaoqingli
cooked: <p>I’m trying to load a sparse matrix on GPU,but I got this RuntimeError. There is no error when I run the same code on Windows.</p>
<pre><code class="lang-auto">import torch
import torch.sparse
import scipy.io as scio

imdb_path='./data/proMatrix_512.mat'
device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")

mat = scio.loadmat(imdb_path)['systemMatrix']
mat = mat.tocoo()
i = torch.LongTensor([mat.row, mat.col])
v = torch.DoubleTensor(mat.data)
ASRmtx = torch.sparse.DoubleTensor(i, v, mat.shape).to(device)
</code></pre>

username: ptrblck
cooked: <p>Could you post an executable code snippet with randomly initialized tensors to reproduce this issue?<br>
The adapted code from the docs seems to work on Linux and Windows for <code>float64</code>:</p>
<pre><code class="lang-python">nnz = 5
dims = [5, 5, 2, 2]
I = torch.cat([torch.randint(0, dims[0], size=(nnz,)),
               torch.randint(0, dims[1], size=(nnz,))], 0).reshape(2, nnz).cuda()
V = torch.randn(nnz, dims[2], dims[3]).cuda()
size = torch.Size(dims)
S = torch.sparse.DoubleTensor(I, V.double(), size)
</code></pre>
How to get hidden state output of all RNN layers for intermediate time steps

username: amitdedhia
cooked: <p>I am a beginner to PyTorch. I am trying to implement Many-to-Many RNN where the entire input sequence is processed first, and after the last step processing, the output is being generated.  Something like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd875a9c510d6d7568f20d4fe152bed99058c8cb.png" alt="image" data-base62-sha1="tkbSj6rOoG80jFxSCiquxGYzRub" width="198" height="207"></p>
<p>For efficiency, I am using <code>pack_padded_sequence</code>. At the end of last time-step (of input sequence) process, I <em>unpack</em> using <code>pad_packed_sequence</code>. Each input sequence length is different. I need to pass on intermediate step’s hidden state outcome to the next timestep to generate output sequence.</p>
<p>This is no problem if I have just one rnn layer. However, I am not able to do this if there are multiple RNN layers. I get intermediate hidden state output of last layer of multi layered RNN.  The <code>nn.RNN</code> return the hidden state of outermost RNN layer for intermediate steps. Is there a way to get hidden state output of all RNN layers for intermediate steps?</p>
Does Custom C++ Class Release GIL?

username: Jingwang_Ling
cooked: <p>Hi everyone, I’m trying the pytorch 1.5 new custom c++ class and want to release GIL during C++ execution to allow multithreading. However I found there is nowhere for me to insert <code>py::call_guard&lt;py::gil_scoped_release&gt;()</code> as we often do with c++ extensions.</p>
<p>So I wonder if pytorch releases GIL for custon c++ class by default? Thanks a lot!</p>
How to interpolate batch of tensors over channel's dimension

username: Sac_dev
cooked: <p>Hi,</p>
<p>I’m implementing a CNN-VAE with skip-connection layers in encoder and decoder, to implicitly optimize the information flow from input data and latent representation.</p>
<p>I am aware that ResBlock use identity short-cut mapping if the resolution (HxW) and the channel depth is kept unchanged, and otherwise use a convolution in the shortcut to make a appropriate upsampling/downsampling (example : 1x1 convolution if only channel depth is changing).</p>
<p>My question is the following, according to that paper :  <a href="https://arxiv.org/search/stat?searchtype=author&amp;query=Zheng%2C+H" rel="nofollow noopener">Huangjie Zheng</a>, they stipulated that a skip-connection short-cut is add in each layer, using non-trainable upsampling / downsampling shortcut functions, that therefore doesn’t increase network complexity.</p>
<p>That means that an interpolation throughout channel is necessary, which is not a feature enable by nn.functional.Interpolate.<br>
What would be a pytorch implementation to do so ? My turn-around is to move to 3D volumetric data with torch.unsqueeze, perform a trilinear interpolation and go back to 2D data. Is there a proper way ?</p>
<p>Thanks for your help</p>

username: ptrblck
cooked: <p>You could <code>permute</code> the activation, treat the channels as a spatial dimension, apply <code>F.interpolate</code> on it, and <code>permute</code> the activation back to its original layout.</p>
Apply a skimage (or any) function to output before loss

username: EdgarCardenasDeLaHoz
cooked: <p>Lets say the output to my model is a stack of images of size [B, C, M, N].</p>
<p>I wish to binarize each image channel and apply a skimage function such as remove_small_objects to the images, and then apply a loss function to those images.</p>
<p>What would be the best way for me to use one of these function to the tensors images and still be able to keep the gradients and use loss functions?</p>
<p>As an extension to this, how could one use a function skimage.measure library to that produces an array of centroids and then apply a loss function to the prediction and target centroids that propagates back into the model parameters?</p>

username: ptrblck
cooked: <p>Since <code>skimage</code> uses numpy under the hood, if I’m not mistaken, you would have to use a custom <code>autograd.Function</code> and implement the <code>backward</code> method manually.<br>
Autograd won’t be able to create the backward pass automatically.</p>
<p>I’m not sure which operations are applied for the mentioned operations, but you might be able to implement them using PyTorch operations. If that’s the case, you wouldn’t need to write the <code>backward</code> manually.</p>
Compiling c from torchvision

username: yuhong
cooked: <p>Hi folks,</p>
<p>I’ve been trying to use the built in NMS function in torchvision.ops but I keep getting this error:<br>
undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_29E<br>
I have already looked at other posts, but have not found any answers.</p>
<p>torch 1.4<br>
torchvision 0.4<br>
gcc 7.4.0-ubuntu~18.04.1</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/761329ef70ce774312ae88d8bcfe19606252e084.png" data-download-href="https://discuss.pytorch.org/uploads/default/761329ef70ce774312ae88d8bcfe19606252e084" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/761329ef70ce774312ae88d8bcfe19606252e084.png" alt="image" data-base62-sha1="gQxoMMeTFqzgR9gbBYh1mLBu8kc" width="690" height="133" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1529×295 17.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Does anyone know how to solve this?<br>
Best,<br>
Yuhong</p>

username: ptrblck
cooked: <p>How did you built this op and could you post the build log, please?<br>
Did you also build torchvision manually with GPU support?</p>

username: yuhong
cooked: <p>Yes I am using GPU and building it using python 3.7<br>
This issue has been solved using torchvision 0.5.0</p>
How to modify the tensor class or use custom data type?

username: 111313
cooked: <p>Hi All</p>
<p>I want to use custom data type or modify the tensor class to accommodate these custom datatype.<br>
Actually, I need to test the effect of the variable bit width.<br>
For example, I want to replace float with unsigned int a : 3; or something like that.</p>
<p>I think I should modify the tensor class to accommodate these custom data types.<br>
Am I on the right direction? Or Is there other ways to do it more easily?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/111313">@111313</a><br>
I don’t think you can test bit wise datatype. pytorch currently support these types.<br>
‘<a href="https://github.com/pytorch/pytorch/blob/master/c10/core/ScalarType.h#L26" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/c10/core/ScalarType.h#L26</a>’</p>
How to write a custom point-wise cuda kernel?

username: Separius
cooked: <p>I want to write a custom point-wise cuda kernel and I can do that easily with <code>cpp_extension</code>, and a custom cuda kernel luncher. The problem is that I don’t know how to set a good block_size.<br>
As far as I can tell <code>CUDA_tensor_apply2</code> handles those parameters and simplifies my work. But I need <code>CUDA_tensor_apply3</code> which has been removed from the aten.<br>
It seems like I should use <code>TensorIterator</code> with <code>gpu_kernel</code> but I can not include it into my code. Should I include <code>&lt;ATen/native/CUDALoops.cuh&gt;</code>? If so, I can’t and there is no such file in my conda installation (I’m using torch==1.5 with cudatoolkit==10.2)</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/separius">@Separius</a><br>
CUDALoops.cuh is not exposed.<br>
TensorIterator is exposed. To use it, you can do<br>
<span class="hashtag">#include</span> &lt;ATen/native/TensorIterator.h&gt;</p>
<p>For elementwise cuda kernel block setup, you can take a look at here, we have basic thread and block setup here.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6fcabf619d2aeacc9867f41af8b36ddac06c3d25/aten/src/ATen/native/cuda/Loops.cuh#L10" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6fcabf619d2aeacc9867f41af8b36ddac06c3d25/aten/src/ATen/native/cuda/Loops.cuh#L10" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/6fcabf619d2aeacc9867f41af8b36ddac06c3d25/aten/src/ATen/native/cuda/Loops.cuh#L10</a></h4>
<pre class="onebox"><code class="lang-cuh"><ol class="start lines" start="1" style="counter-reset: li-counter 0 ;">
<li>
</li>
<li>#pragma once</li>
<li>
</li>
<li>#include &lt;ATen/detail/FunctionTraits.h&gt;</li>
<li>#include &lt;ATen/native/TensorIterator.h&gt;</li>
<li>#include &lt;ATen/cuda/detail/OffsetCalculator.cuh&gt;</li>
<li>
</li>
<li>namespace at { namespace native {</li>
<li>
</li>
<li class="selected">#define NUM_THREADS (C10_WARP_SIZE * 2)</li>
<li>#define THREAD_WORK_SIZE 4</li>
<li>#define BLOCK_WORK_SIZE (THREAD_WORK_SIZE * num_threads)</li>
<li>
</li>
<li>constexpr int num_threads = NUM_THREADS;</li>
<li>constexpr int thread_work_size = THREAD_WORK_SIZE;</li>
<li>constexpr int block_work_size = BLOCK_WORK_SIZE;</li>
<li>
</li>
<li>// `needs_dynamic_casting` compares the types expected by iterator</li>
<li>// (i.e. dtypes of the operands) with the actual type of the arguments</li>
<li>// of func_t</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>And within CUDALoops.cuh, you can check gpu_kernel_impl() to see how we assign thread and block size for different cases. Basically, for 1d tesnor, we set thread to 512 and each thread handle 1 item, for multi-dimension tensor, we set thread number to 128, but each thread handle 4 items.</p>
Reading csv.gz file in torch

username: abhigenie92
cooked: <p>Pandas’s read_csv works for csv.gz as well. Is there a way to achieve similar with PyTorch.?<a href="https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Dataset" rel="nofollow noopener">https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Dataset</a> doesn’t seem to have such an option.</p>