akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Obtaining a tensor from a list of tensors with requires_grad=True

melody_marks: <p>In the forward method my my model, I come across a list of tensors. All of those tensors have requires_grad=True. However, when I try to obtain a tensor from this list using torch.FloatTensor(myTensorList), the requires_grad of the resulting tensor is False, which is breaking the graph for computing gradients. Will setting the requires_grad=True when casting my list to a FloatTensor will keep my graph concrete? Is it safe to do this? If not, what else can I do?</p>

albanD: <p>Hi,</p>
<p>It is because <code>FloatTensor</code> creates a brand new Tensor from raw values.<br>
You can use <code>torch.cat(myTensorList)</code> or <code>torch.stack(myTensorList)</code> to do this.</p>
Loss.backward runtime error

hawkeye101: <p>I’m getting the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "NN.py", line 57, in &lt;module&gt;
    loss.backward()
  File "/home/arvind/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/arvind/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>This is the relevant code.</p>
<pre><code class="lang-auto">x_train=torch.empty(train_size,5,requires_grad=True)
x_train=x_total[1:train_size,:]
y_train=torch.empty(train_size,1,requires_grad=True)
y_train=y[1:train_size]

model = nn.Sequential(nn.Linear(n_in, n_h),
   #nn.ReLU(),
   nn.Linear(n_h, n_out),
   nn.Sigmoid())

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

for epoch in range(50):
   y_pred = model(x_train)

   y_pred = torch.Tensor(list(y_pred))
   y_train = torch.Tensor(list(y_train))
   loss = criterion(y_pred, y_train)
   print('epoch: ', epoch,' loss: ', loss.item())

   optimizer.zero_grad()

   #loss.requires_grad = True
   loss.backward()

   optimizer.step()
</code></pre>

albanD: <p>Hi,</p>
<p>When you do <code>y_pred = torch.Tensor(list(y_pred))</code> you convert the Tensor to a python list and then back to a Tensor. But we cannot track gradients in the python list and so this breaks the gradient computation.<br>
You can just use <code>y_pred</code> as is. Or if you want to make it 1D use <code>y_pred = y_pred.view(-1)</code>.</p>
How to get the confidence score of all class for a bounding box while using pretrained object detector in pytorch

spaul13: <p>for a pretrained object detection model in pytorch and for each bounding box predicted by the model how to get the confidence score for each of the 80 COCO classes for that bounding box?</p>
<p>I have put the code I am using for object detection using pretrained fasterRCNN Resnet-50 FPN model</p>
<pre><code>    img = Image.open(img_path) # Load the image
    transform = transforms.Compose([transforms.ToTensor()]) # Defing PyTorch Transform
    img = transform(img) # Apply the transform to the image
    pred = model([img.cuda()]) # Pass the image to the model
    pred_class = [COCO_INSTANCE_CATEGORY_NAMES[i] for i in list(pred[0]['labels'].cpu().numpy())] # Get the Prediction Score
    pred_boxes = [[(i[0], i[1]), (i[2], i[3])] for i in list(pred[0]['boxes'].cpu().detach().numpy())] # Bounding boxes
    pred_score = list(pred[0]['scores'].cpu().detach().numpy())
</code></pre>
<p><strong>pred only provides all possible bounding boxes and the best possible class for the bounding box but how to get the confidence score for all possible classes for one bounding box?</strong></p>
<p>Any help will be highly appreciated.</p>
Unable to import torch now

Paul_Gureghian1: <p>I installed ‘pip install torch torchvision’ into my virtual venv env on Linux, and when I run a file to import torch I get this. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f83fa42ffdb0152222eb309f2fa492bb554721fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/f83fa42ffdb0152222eb309f2fa492bb554721fd" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f83fa42ffdb0152222eb309f2fa492bb554721fd_2_690x431.png" alt="pytorch" data-base62-sha1="zq6LltipgVnuUeg6Bfss7JrDeWx" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f83fa42ffdb0152222eb309f2fa492bb554721fd_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f83fa42ffdb0152222eb309f2fa492bb554721fd_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f83fa42ffdb0152222eb309f2fa492bb554721fd.png 2x" data-dominant-color="BBC2C0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">1280×800 199 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Which PyTorch and <code>cudatoolkit</code> versions were installed using your command?</p>

Paul_Gureghian1: <p>Actually, I needed the cpu version. all good now. thanks</p>
Learning to run a power network in a sustainable world - NeurIPS 2020 competition

marota: <p>“Learning to run a power network in a sustainable world” (<a href="https://l2rpn.chalearn.org/" rel="nofollow noopener">https://l2rpn.chalearn.org/</a><a href="http://lnkd.in/duXYUCz" rel="nofollow noopener"> <strong>)</strong> </a>) is listed among 2020 official NeurIPS competitions!</p>
<p>We will be thrilled if you come join us now (<a href="https://competitions.codalab.org/competitions/24902" rel="nofollow noopener">https://competitions.codalab.org/competitions/24902</a><a href="http://lnkd.in/dmbKVkD" rel="nofollow noopener"> <strong>)</strong> </a>) to unleash the power of AI for these critical real-world power grid infrastructures, especially in times of crisis. What would it be either today and tomorrow without reliable electricity ?<br>
We are pleased to answer any related questions here or on Discord Channel: <a href="https://www.google.com/url?q=https%3A%2F%2Fdiscord.gg%2FcYsYrPT&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNGqXjpx8cJuSUnOC1TlJDr5KrPUsw" rel="nofollow noopener">https://discord.gg/cYsYrPT</a></p>
CUDA out of memory is a frequent error

Aryaman_Sriram: <p>I’ve been training VGG16 from scratch on the flower recognition dataset and after the first epoch itself CUDA runs out of memory. Is there a way out of this? I recently studies tensorflow.js and it had a command called tf.tidy() which would delete tensors as soon as their usage was over for proper memory usage. Does torch  have a provision like that?</p>

Avi_Cooper: <p>What batch size are you using? If the batch size is too large, it will cause a out of memory error.</p>

blgnksy: <p>Can you post your training code?</p>
Questions about data normalization

Manoel: <p>I’m training my model on a dataset that I’ve normalized. But I thought that I just had to normalize the train set and not the validation or test sets. However, my validation loss was really inconsistent and it never converged, until I changed the transforms and add the normalized transforms on it. Then the validation loss started to work as expected.</p>
<p>My question is: When I normalize my train set I must normalize my validation and test set as well? If so, why? When I have a single image to predict, will I have to normalized it?</p>

Nikronic: <p>Hi,</p>
<p>Normalization is some kind of transformation, let’s say your data contains weights of humen then if you train your model with normalized data, model learns to predict for instance using input in range of <code>[-1, 1]</code> and it also predict in a range similar to that.  Then if you test your model using original values for instance [1, 200]`, your model has no idea about these huge variance and bias so it cannot  predict in the structure it was trained for.<br>
Test/validation is a step to ensure your model had a reliable train stage, so everything need be consistent between train and test/validation, if not, how can one make sure model is working fine?</p>
<p>Bests</p>

Manoel: <p>Perfect. Thank you very much.</p>
Evaluate log_prob with different distribution in one-line

11169: <p>Suppose I have data, x = [batch, width, height], y = label<br>
I would like to calculate log_prob of x for each label.<br>
For example, log_prob of x with label = 1 is calculated by Normal distribution with mean = label, variance = 1.<br>
So, I calculated log_prob by initializing distributions for each label as below</p>
<pre><code class="lang-auto">dist = [Gaussian(mean = i, std = 1) for i in range(10)]
for x, label in data_loader:
      log_prob += dist[label].log_prob(x)
</code></pre>
<p>But I want to calculate it with one line <strong>dist.log_prob(x)</strong>, but I don’t know how to calculate it with one line.<br>
Would you anyone help me? or is it impossible?</p>
Cannot iterate through a DataLoader object

costa98: <p>Hello, I have the ucf 101 dataset loaded as follows:</p>
<pre><code>dataComplete = torchvision.datasets.UCF101('directory', frames_per_clip=16, step_between_clips=1, frame_rate=None, fold=1, train=True, transform=transformF, _precomputed_metadata=None, num_workers=1, _video_width=0, _video_height=0, _video_min_dimension=0, _audio_samples=0)
data = torch.utils.data.DataLoader(dataComplete, batch_size=3, shuffle=True)
#where transformF = transforms.ToTensor()
</code></pre>
<p>and I am trying to access the tensors stored in the dataLoader object, but everytime I try to execute:</p>
<pre><code>for n_batch in data:
  #training...
</code></pre>
<p>I get the error</p>
<pre><code>TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>I tried using dataComplete instead and using enumerate(), range() ecc. but it shows the same error</p>

Nikronic: <p>Hi,<br>
Error says it needs PIL but you passed tensor.</p>
<aside class="quote no-group" data-username="costa98" data-post="1" data-topic="89862">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/90ced4/48.png" class="avatar"> costa98:</div>
<blockquote>
<p><code>TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="costa98" data-post="1" data-topic="89862">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/90ced4/48.png" class="avatar"> costa98:</div>
<blockquote>
<p><code>#where transformF = transforms.ToTensor()</code></p>
</blockquote>
</aside>
<p>You are converting a tensor to tensor using <code>ToTensor</code> which only accepts PIL images.<br>
Remove that line.</p>
<p>In the <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.UCF101" rel="noopener nofollow ugc">docs of UCF101</a> it says it returns a <em><strong>tensor</strong></em>.</p>
<p>Bests</p>

costa98: <p>Thanks for your reply, understood, but now I keep getting the error</p>
<pre><code class="lang-auto">RuntimeError: stack expects each tensor to be equal size, but got [1, 0] at entry 0 and [2, 18432] at entry 1
</code></pre>
Pytorch regularising against saved model

Imran_Rashid1: <p>I have a neural network in PyTorch that has been trained on a dataset and the model is saved as <strong><code>saved_model.pth.tar</code></strong>.</p>
<p>The current loss of the model is a simple MSE loss <code>loss = torch.nn.MSELoss(target,predictions)</code>.</p>
<p>Now I am training the network on a new dataset and I want to <em>regularise</em> the model against the saved model (<code>saved_model.pth.tar</code>) so that the model is penalised if the weights of the neural network are far away from the saved model.</p>
<p>In essence what I want to do is:</p>
<pre><code class="lang-auto">regularisation_loss = (existing_weights - saved_weights)**2
loss =  torch.nn.MSELoss(target,predictions) + regularisation loss
</code></pre>
<p>The neural network is quite complex and so I wanted to ask if there is a general way of enforcing regularisation against a saved model.</p>
Prepare Yolo v3 Custom Dataset

chandra_sutrisno: <p>Hi all,</p>
<p>I am planning to explore Yolo model and use my custom dataset. I have all the images but quite confuse to prepare the dataset. I notice I must feed my model with data with specific format like below:</p>
<p>object-id center_x center_y width height</p>
<p>I do not understand with this setting. I understand that object-id is the class id and both width and height is images width and height but how can I calculate center_x and center_y? I understand there is a tool called LabelIMG but I want to do it manually.</p>
<p>Please advise</p>

l00p: <p>The Yolo format required params normalised wrt. the dimensions of the image.</p>
<p>So an object of class 3 with its center 20% of the way down from the top of the image and 30% of the way across from the left with a width equal to 10% of the image width and a height equal to 15% of the  image height would be labelled as:</p>
<p>3 0.3 0.2 0.1 0.15</p>
<p>See here for more description: <a href="https://github.com/AlexeyAB/Yolo_mark/issues/60" rel="nofollow noopener">https://github.com/AlexeyAB/Yolo_mark/issues/60</a></p>
Problem with torch.pow()

mohit117: <p>Hello,</p>
<p>In my CNN model there is a MLP which has 4 fully connected layers with last one having just one node. This output I call <code>alpha</code> and has 2 dimensions as in this sample output of <code>alpha</code> <code>[[0.1023]]</code> . I make sure that minimum values never goes below 0.1 using <code>alpha=torch.clamp(alpha,min=0.1,max=2)</code></p>
<p>I need now need to manipulate some other Pytorch tensor <code>x</code> with <code>alpha</code> . Both are in <code>torch.float</code> type. The tensor <code>x</code> has the usual 4 dimensions of batch x channel x height x width.</p>
<p>So I have tried the following:-</p>
<ul>
<li>
<code>x**alpha[0][0]</code>: This however causes Nan just after the first iteration for both <code>alpha</code> and <code>x</code>. I tried several other variants <code>x**alpha</code>, <code>x.pow(alpha[0][0])</code> and <code>x.pow(alpha)</code>. But all had the same problem. <em><strong>How to solve this?</strong></em>
</li>
<li>
<code>x*alpha[0][0]</code>: This however runs nicely I did not encounter any problem even after 100,000 iterations. So why <em><strong>raising to power is problematic while simple multiplication is alright?</strong></em>
</li>
<li>Interestingly for <code>x.pow(alpha.item())</code> Nan problem goes away but is this the correct way to do? Does this cause problem in backpropagation?</li>
</ul>
<p>I use a very small learning rate of <code>1e-4</code> with Adam Optimiser. I am using Pytorch 1.3.1.</p>
<p>Thankyou verymuch</p>

mohit117: <p>I do not know if this helps, but i found a way to circumvent the issue though I still dont understand why the original issue came up.</p>
<p><code>alpha=torch.clamp(alpha,min=0.1,max=2)</code><br>
with</p>
<pre><code class="lang-auto">m = torch.nn.Threshold(0.1,0.1)
alpha=m(alpha)
</code></pre>
<p>Now allows me to use <code>x**alpha[0][0]</code>. Think there is a bug with <code>clamp</code>.</p>

tom: <p>Hi Mohit,</p>
<p>I am relatively confident, that there is no problem with clamp,.</p>
<p>What’s the value range of x?<br>
If you have values of x &lt;= 0, you might expect non-integral powers and derivatives to be problematic.</p>
<p>Best regards</p>
<p>Thomas</p>

mohit117: <p>Thankyou for your reply.</p>
<p>The tensor x is between 0 to 1. With torch.min i checked it is 0. But its low light data. So average value of x is 1e-4 or 1e-3 and no greater 1e-2. rarely it would reach 1e-1.</p>
<p>But what ever it might why using clamp gives nan and nn.Threshold does where th job of both is to clamp any value less than 0.1 to 0.1 for alpha.</p>
A bug in the code may be the reason

agcala: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/multi-class-classification/47565/5">Multi-class classification</a>:</p>
<aside class="quote no-group" data-username="Rojin" data-post="5" data-topic="47565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rojin/48/11062_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/multi-class-classification/47565/5">Multi-class classification</a></div>
<blockquote>
<pre><code class="lang-auto">test_loader_X = torch.utils.data.DataLoader(dataset=tr_latent_X,
                                             batch_size=bs,
                                             shuffle=False)
</code></pre>
</blockquote>
</aside>
<p>It seems to me there is a bug in the code this should have said</p>
<pre><code class="lang-auto">test_loader_X = torch.utils.data.DataLoader(dataset=te_latent_X,
                                             batch_size=bs,
                                             shuffle=False)
</code></pre>

ptrblck: <p>That might be the case. Would you mind continuing the discussion in the linked topic and ping the author, in case she is not aware of this typo?</p>
Pytorch 1.4 works, but Pytorch 1.5.X gives ‘RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation’ error

TJKlein: <p>Hi there,</p>
<p>I got a weird issue. When using Pytorch 1.4 my code runs just fine.<br>
But when using 1.5.1 I am getting the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation’.</p>
<p>I am using CrossEntropyLoss.</p>
<p>Cheers,<br>
Tassilo</p>

ptrblck: <p>This error might be raised due to the fixed inplace checked in optimizers.<br>
From the <a href="https://github.com/pytorch/pytorch/releases/tag/v1.5.0">1.5.0 release notes</a>:</p>
<blockquote>
<p><code>torch.optim</code>  optimizers changed to fix in-place checks for the changes made by the optimizer (<a href="https://github.com/pytorch/pytorch/pull/33640">#33640</a>, <a href="https://github.com/pytorch/pytorch/pull/34211">#34211</a>)</p>
</blockquote>
<blockquote>
<p>If this causes your code to fail, there are two possible reasons:<br>
Reason 1: The value of that parameter was actually saved and used and we were computing incorrect gradients in previous versions of PyTorch. This would result in an error message mentioning incorrect version numbers. You should replace code that uses  <code>self.my_param</code>  by  <code>self.my_param.clone()</code>  to make sure the saved version is different from the one that is modified by the optimizer.<br>
Reason 2: You know what you’re doing and change the values back to the right thing before the next backward. However, you’re running into an error because the version counter cannot be decremented. Open an issue with your particular use case and we will help you to work around the version counter issue.</p>
</blockquote>
<p>Could you take a look at the examples and check, if it would fit your use case?</p>
About the confusion matrix

Foolish: <p>The number of pictures in the confusion matrix is far less than the actual number of pictures in the test set. What is going on?</p>

ptrblck: <p>Your code might miss some test predictions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Without seeing any code, I can just speculate, so please feel free to post the code used to calculate the confusion matrix.</p>
RuntimeError: cudnn RNN backward can only be called in training mode during backpropagation

AnupKumarGupta: <p>Hi. I am trying to perform an adversarial attack on a model, where I have to calculate gradients of loss over inputs, something like this:</p>
<pre><code class="lang-auto">model.eval()
for prediction != target:
     prediction = model(input)
     loss = criterion(prediction, target)
     loss.backward()
     input = input - k * (input.grad())
</code></pre>
<p>But, following error is encountered <code>Pytorch cudnn RNN backward can only be called in training mode</code>. I tried the solutions posted  <a href="https://discuss.pytorch.org/t/pytorch-cudnn-rnn-backward-can-only-be-called-in-training-mode/80080/6">here</a>, which are:</p>
<ol>
<li>Call <code>.train()</code> on the rnn module after using <code>model.eval()</code>. Used this as follows:</li>
</ol>
<pre><code class="lang-auto">model.eval()
for prediction != target:
     prediction = model(input)
     loss = criterion(prediction, target)
     model.train()
     loss.backward()
     input = input - k * (input.grad())
     model.eval()
</code></pre>
<p>It doesn’t help. The error is still there.</p>
<ol start="2">
<li>
<p>call <code>.eval()</code> only on the necessary modules: I have 2 doubts regarding this solution.<br>
a. My model under consideration is quite big, it is advisable to go for this?<br>
b. Does it affect my prediction accuracy in any way?</p>
</li>
<li>
<p>disable <code>cudnn</code> via <code>torch.backends.cudnn.enabled = False</code>. Going for this solution, I think I won’t be able to GPU. Without the GPU the running times is 4-5 times to that of time taken for execution with the GPU.</p>
</li>
</ol>
<p>Can you please suggest me workaround/solution? Thanks in advance.</p>

ptrblck: <p>Could you post a (minimal) model definition including all necessary shapes and arguments to reproduce this issue?</p>
<ol>
<li>
<p>The predictions shouldn’t be affected in any way.</p>
</li>
<li>
<p>The GPU will still be used, if you disable cudnn. As the call suggests cudnn won’t be used and the native CUDA implementations will be called instead.</p>
</li>
</ol>
Freezing a part of the tensor

Guitaricet: <p>Hi!</p>
<p>I am trying to freeze all my network weights except <em>some</em> of the output embeddings (nn.Linear columns) using register_backward_hook.</p>
<p><a href="https://discuss.pytorch.org/t/freezing-part-of-the-layer-weights/9457/3">This discussion</a> was very helpful, but I still do not understand what is the right way to do it.</p>
<p>Backward hook for the layer has <code>grad_input</code> and <code>grad_output</code>. As far as I saw, <code>grad_input</code> was always equal to <code>grad_output</code> (this is the final layer of the model) and they both have shape <code>(bach_size, seq_len, vocab_size)</code>.</p>
<p>I have three questions:</p>
<ol>
<li>
<p>Ideally, I would like to modify the value of the weight gradient <code>(hidden_size, vocab_size)</code> and I tried to access it inside the hook via <code>self.weight.grad</code>. However it is always None even though this is happening in the function registered via <code>.register_backward_hook</code> so it is called after <code>.backward()</code>. Why does this happen and how to access the gradient?</p>
</li>
<li>
<p>Ok, let’s forget about the weight gradient tensor. I can modify all values corresponding to the particular embedding in grad_input right? (According to the documentation though, I should not modify, but to copy them and return new values).<br>
So it would look like this:</p>
</li>
</ol>
<pre><code class="lang-python">def hook(self, grad_input, grad_output):
    new_grads = grad_input[0].detach().clone()
    new_grads[:, :, freeze_ids] = 0.
    return new_grads, grad_input[1]
</code></pre>
<p>Should I modify grad_input or grad_output in this case?</p>
<ol start="3">
<li>Is ADAM going to update the values even after we zeroed the gradients? ADAM has momentum values which depend on the previous gradients and not on the current gradient and my setup implies that ADAM will have nonzero betas for these parameters.</li>
</ol>
<p><strong>UPD:</strong></p>
<ol>
<li>Still have no idea about question 1.</li>
<li>This hook seem to work with SGD (as far as my tests go)</li>
<li>ADAM or SGD with momentum do update these values if you initialize the hook in the middle of the training process. Help me plz =(</li>
</ol>

ptrblck: <p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook"><code>register_backward_hook</code></a> might nor properly work for “complex” modules as described in the <code>Warning</code> box in the docs.<br>
I would thus recommend to use <code>param.register_hook</code> directly, which will only get the gradient as its input.</p>
Convert Sklearn MLPRegressor to Pytorch

marya: <p>Hello, I am trying converting my MLP regression model to pytorch. This is my first time using Pytorch. below is the simple MLP model:<br>
reg=MLPRegressor()<br>
reg.fit(x_train, y_train)<br>
pred=reg.predict(x_test)<br>
Can someone help me to figure out what changes I need to make in order to convert this sklearn model to Pytorch model? Tahnks in advance for your help.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/sklearn-model-to-pytorch-model/33532">This post</a> might be useful, as it’s dealing with the same problem. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Related to optimizer and loss function in transfer learning

Santanu_Bhattacharje: <p>If I change the <code>requires_grad</code> to <code>True</code> or <code>False</code> midway after a few training epochs, do I need to reinitialize the optimizer or loss function?</p>

ptrblck: <p>Setting the <code>.requires_grad=False</code> attributes of (some) parameters will avoid calculating the gradient for these parameters.<br>
Since the optimizer would still have the references to these parameters, it would perform the <code>step()</code> operation on zero gradients and would in the simplest case basically “skip” these parameters.</p>
<p>However, if the optimizer uses some internal estimates for each parameter or you’ve been using weight decay before, these parameters might still be updated even if they get a zero gradient.<br>
If that’s not desired, you would have to reinitialize the optimizer (and try to restore the running estimates of the remaining parameters).</p>
Calculating the model overall accuracy

alameer: <p>Sorry, if this is a very simple question. I would like to calculate the over all accuracy of a model during training. I’m writing to a tensor-board test loss, and test count error. What do i need to do to get the overall accuracy across the whole training please? Many Thanks</p>
<pre><code class="lang-auto">loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
        
        global_step += 1
    
    mean_test_error = mean_test_error / len(loader_test)
    print("Test count error: %f" % mean_test_error)
   
    if mean_test_error &lt; best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)</code></pre>

ptrblck: <p>You could store the number of correctly predicted samples during training and divide it by the total number of samples after each epoch is done.<br>
Also, the <a href="https://github.com/pytorch/examples/blob/8df8e747857261ea481e0b2492413d52bf7cc3a8/imagenet/main.py#L293-L297">ImageNet example</a> uses some utility classes to track the losses and accuracy, which might be useful for your use case.</p>
CNN Image Classification

shivkhaira: <p>I am trying to build image classification with CNN. It did first with Restnet18 which works. But when i try to make my own CNN model the outputs are all same.</p>
<p>This is my modal:</p>
<p>class Network(nn.Module):</p>
<pre><code>def __init__(self):

    super(Network,self).__init__()

    self.cn1=nn.Conv2d(3,16,3,1,1)

    self.cn2=nn.Conv2d(16,32,3,1,1)

    self.cn3=nn.Conv2d(32,64,3,1,1)

    self.cn4=nn.Conv2d(64,128,3,1,1)

    self.pool=nn.AvgPool2d(2,2)

    self.fc1=nn.Linear(128*14*14,1024)

    self.fc2=nn.Linear(1024,512)

    self.fc3=nn.Linear(512,256)

    self.fc4=nn.Linear(256,2)

def forward(self,x):

    x= F.relu(self.pool(self.cn1(x)))

    x = F.relu(self.pool(self.cn2(x)))

    x = F.relu(self.pool(self.cn3(x)))

    x = F.relu(self.pool(self.cn4(x)))

    

    x=x.view(-1,128*14*14)

    

    x=self.fc1(x)

    x=self.fc2(x)

    x=self.fc3(x)

    x=self.fc4(x)

    x=F.log_softmax(x,dim=1)

    return x
</code></pre>
<p>Batch size is 20. So then i checked the it is giving same output for all the images in the batch.</p>

ptrblck: <p>I would recommend to add non-linearities between the linear layers and try to overfit a small data sample (e.g. just 10 samples) by playing around with the hyper parameters, such as the learning rate.<br>
Once your model can successfully overfit the small data sample, you could try to scale up the problem again.</p>
'collections.OrderedDict' object has no attribute 'to'

xud: <p>I have trained my model  on GPU but I was getting this error while training<br>
<code>AttributeError:</code>‘collections.OrderedDict’ object has no attribute ‘to’ ,<br>
the code comes after this<br>
my_resnet18 =torch.load(’…/resnet18model/best_model.pth’)<br>
my_resnet18.fc =torch.nn.Sequential(nn.Linear(512, 16),nn.Tanh())<br>
my_resnet18 = my_resnet18.to(device)<br>
Is there anything wrong with this?<br>
my pytorch version is 1.0.0</p>

xud: <p>The problem has been solved thanku</p>

dong-mvp: <p>How did it work out?</p>

zhujunjie-JLU: <p>can you tell me how to solve it?</p>

ptrblck: <p>Most likely <code>torch.load()</code> returned a <code>state_dict</code>, which would create the issue, if you are trying to call <code>to()</code> on the <code>OrderedDict</code>.<br>
The proper way of restoring the model is to initialize the model and load the <code>state_dict</code> afterwards:</p>
<pre><code class="lang-python">model = models.resnet18()
state_dict = torch.load(...)
model.load_state_dict(state_dict)
model.to(device)
</code></pre>
<p>CC <a class="mention" href="/u/dong-mvp">@dong-mvp</a></p>
Help with imagenet

gamebo5000: <p>I am trying to use this imagenet script: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>  on a pretrained mobilenetV2 model using my own custom dataset. Unfortunately, I am not having any luck. I am still new to Pytorch, having just learned about it last month (June 2020). Is there a link with a tutorial or example to help better understand the code and how to modify it to fit my needs?</p>

ptrblck: <p>I would recommend to check out the <a href="https://pytorch.org/tutorials/">tutorials</a> first and in particular <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html">this transfer learning tutorial</a>.<br>
This should explain a lot about the underlying <code>Dataset</code> and <code>DataLoader</code> usage as well as about the general training structure. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to convert the at::Byte tensor to boolean tensor in c++?

lqx: <p>In recent, a pooling extension has been used in my code. But I use a higher version pytorch than introduced in the original repo. The pooling function has used two functions named <code>masked_scatter_</code> and <code>masked_fill_</code>. In previous version of pytorch, it could be fine. However, when I use pytorch==1.4.0 the two functions output tens of <code>warning</code>. It refers to the variable <code>mask</code> is defined with <code>at::kByte</code> scalar type while the two function require the <code>mask</code> to become <code>torch.bool</code> type. Otherwise the program will output warning when the function is called. So my question is how to convert the type of <code>mask</code> from <code>at::kByte</code> to <code>torch.bool</code> (It is easy in python just using one line <code>var.bool()</code>). But how to implement that in C++ version. I can not find any  reference in doc.</p>

ptrblck: <p>Would <code>tensor.to(at::kBool)</code> work?</p>

21fa417e3fb06e56040c: <pre><code class="lang-auto">single tensor
torch::Tensor t = torch::randn({1});		
float f = t.item().toBFloat16();

multi tensor
std::memcpy(dataptr, tensor.data_ptr(), sizeof(torch::kU8/*tensortype*/) * tensor.numel());
</code></pre>
Where to use batch normalization?

MUnique: <p>Hi,<br>
I have a deep neural network for classifying videos. I was wondering what layers should have batch normalization. Is it empirical to find out that? When and where should I use the batch normalization? What happens if I use more than enough batch normalizations?</p>

Sudarshan_Babu: <p>BatchNorm helps in the training of most Deep Networks, that said if your batch size is small (less than 8), it is known training with  BatchNorm yields sub optimal results, and using other normalization tricks like GroupNorm is helpful.</p>
<p>In ResNets BatchNorm is placed usually before a skip connection or just before the activation layer (ReLU).</p>
<p>Also, if you are using a shallow network (less than 6 layers), you can train your model without BatchNorm, and might experience marginally slow convergence, but forward pass for networks without BatchNorm is certainly less time consuming, and hence it might be a better trade off to train your model without BatchNorm, but experiments are certainly needed to evaluate this trade off, as things often depend on your dataset and other problem specific parameters.</p>
Def update_target(target, original, update_rate):

MERAH_Samia: <p>I’d seen this in the implementation of Baysian CNN with MC Dropout here<a href="https://github.com/sungyubkim/MCDO/blob/master/Bayesian_CNN_with_MCDO.ipynb?fbclid=IwAR18IMLcdUUp90TRoYodsJS7GW1smk-KGYovNpojn8LtRhDQckFI_gnpOYc" rel="nofollow noopener">here</a><br>
Could anyone tel me what is for please?<br>
def update_target(target, original, update_rate):<br>
for target_param, param in zip(target.parameters(), original.parameters()):<br>
target_param.data.copy_((1.0 - update_rate) * target_param.data +<br>
update_rate*param.data)</p>
Nn.init.xavier_uniform_

MERAH_Samia: <p>Hello!<br>
Does anyone know what these two lines of code do Please?<br>
nn.init.xavier_uniform_(self.conv1.weight)<br>
nn.init.constant_(self.conv1.bias, 0.0)</p>

Mahmoud_Abdelkhalek: <p>The first line fills <code>self.conv1.weight</code> with samples from a uniform distribution, while the second line fills <code>self.conv1.bias</code>  with zeros.</p>
<p>See <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.normal_" rel="nofollow noopener">here</a> for more information.</p>

MERAH_Samia: <p>Am gonna see, Thank you so much</p>

MERAH_Samia: <p>Do you know when i could use the uniform distribution please</p>

Mahmoud_Abdelkhalek: <p>That depends on your application. See <a href="https://www.deeplearning.ai/ai-notes/initialization/" rel="nofollow noopener">here</a> for a good tutorial on this.</p>

MERAH_Samia: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"> <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
What is a fast method of calculating the difference in intensity between a local image patch and its four neighbors?

Amr_Mustafa: <p>Hello! I am trying to implement a computer vision paper using PyTorch. The thing is it is using a custom loss function that involves iterating over all local image patches of specific size and calculating the sum of differences in intensity between that patch and its four neighbors.</p>
<p>I am trying to find an efficient and fast solution since this would be included in the training loop. I searched online for answers and found that PyTorch provides a method called  <code>unfold</code>  and I used it to get the patches I need to work with but I do not know how to find the neighbors of a certain patch since the spatial structure of the input image is lost when using that method.</p>
<p>To be more concrete, here is the loss function in question.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77e72adf7a22851d063271e55c1998c5f0fb15b2.png" alt="image" data-base62-sha1="h6I5LwFef9vONiGpqD0IBmNNldM" width="427" height="67"></p>
<p>It is from the Zero-Deep Curve Estimation paper for low-light image enhancement.</p>
Question: Does torchaudio.load spawn a new process for loading mp3?

antimora: <p>Lots of python libraries for reading mp3 files require spawning new processes that execute cli utilities. For example, AudioSegment runs ffpeg underneath. This is suboptimal for my particular use case because I convert files and I write to a database which works better with in memory files.</p>
<p>From reading torchaudio source code I could not get a straightforward answer since it uses different backends and C++ wrapper code.</p>
<p>Could you please confirm  torchaudio.load does/does not spawn a new process for loading mp3?</p>
Pytorch tensor division hangs in multiprocess linux

Lewis_Liu: <p>Hi,</p>
<p>I’m experiencing a hanging in process. it happens at the following codes</p>
<p>samping_priorities = (self.priority_memory[0:upper_bound] / self.priority_memory[0:upper_bound].sum()).cpu().detach().numpy()</p>
<p>batch_idx = T.LongTensor(np.random.choice(upper_bound, batch_size,p=samping_priorities[0:upper_bound].cpu().detach().numpy()))</p>
<p>samping_priorities is a 2000000*1 tensor.<br>
upper_bound is the range I’m interested in and upper_bound+=1 through iterations</p>
<p>at the beginning everything is okay. then I noticed when upper_bound exceeds 32768, the process hangs between the first line and second line</p>
<p>It works fine on my windows workstation but hangs in linux cluster. What could be the cause and how can i fix it</p>

mrshenli: <aside class="quote no-group" data-username="Lewis_Liu" data-post="1" data-topic="89786">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lewis_liu/48/12953_2.png" class="avatar"> Lewis_Liu:</div>
<blockquote>
<p>at the beginning everything is okay. then I noticed when upper_bound exceeds 32768, the process hangs between the first line and second line</p>
</blockquote>
</aside>
<p>This sounds like a <code>int16_t</code> overflow bug (or that might hit some different branch). Could you please create an issue in <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">pytorch repo</a>? Thanks!</p>
Inconsistent Results for reloading the model

gabrielsantosrv: <p>Hey Guys,</p>
<p>I’m facing an issue with trained models, during the training in my own dataset I get a CIDEr Score around 0.03 in the validation set, but when I save and then reload the best model and try to evaluate it the score is about zero… But it is even stranger because it works perfectly with MS COCO dataset.</p>
<p>Here is my training script:<br>
<a href="https://github.com/gabrielsantosrv/AoANet/blob/master/train.py" rel="nofollow noopener">train.py</a></p>
<p>I save the model as follow:</p>
<pre><code class="lang-auto">    def save_checkpoint(model, infos, optimizer, histories=None, append=''):
        if len(append) &gt; 0:
            append = '-' + append
        # if checkpoint_path doesn't exist
        if not os.path.isdir(opt.checkpoint_path):
            os.makedirs(opt.checkpoint_path)

        checkpoint_path = os.path.join(opt.checkpoint_path, 'model%s.pth' %(append))
        torch.save(model.state_dict(), checkpoint_path)
        print("model saved to {}".format(checkpoint_path))

        optimizer_path = os.path.join(opt.checkpoint_path, 'optimizer%s.pth' %(append))
        torch.save(optimizer.state_dict(), optimizer_path)
        with open(os.path.join(opt.checkpoint_path, 'infos_'+opt.id+'%s.pkl' %(append)), 'wb') as f:
            utils.pickle_dump(infos, f)

        if histories:
            with open(os.path.join(opt.checkpoint_path, 'histories_'+opt.id+'%s.pkl' %(append)), 'wb') as f:
                utils.pickle_dump(histories, f)
</code></pre>
<p>And I reload it:<br>
<a href="https://github.com/gabrielsantosrv/AoANet/blob/master/eval.py" rel="nofollow noopener">evaluation script</a></p>
<pre><code class="lang-auto">model.load_state_dict(torch.load(opt.model, map_location='cpu'))
model.cuda()
model.eval()
</code></pre>
<p>Do you have any idea about what may be happening?</p>

Valerio_Biscione: <p>Sorry if it sounds like a stupid question, but do you set it to eval mode during validation (when you get a score of around 0.03)?</p>

gabrielsantosrv: <p>Yes, I set model.eval(), compute the score, and then I set model.train() again to keep on the training</p>
How to do Cosine similarity between 2 matrices with same concept as of scikit-learn?

SOUMYA_SUVRA_GHOSAL: <p>I have one tensor x of shape [64,324]. Think of 64 as batch size and 324 as number of features. I want to perform cosine similarity with another tensor w such that that the output tensor is of size [64,10] , where 64 is same as batch size and 10 is output features. I could not find any tensor w which serves this purpose. <strong>w</strong> should be independent of batch size.</p>

tom: <p>For x of shape 64x324, if w is 324x10 the product x @ w is 64x10 and is 64 x 10 scalar products (so higher = closer alignment).<br>
You effectively compare the batch features against 10 example features and measure alignment to each of them.</p>

SOUMYA_SUVRA_GHOSAL: <p>Thanks very much for your reply</p>
Got almost zero loss from the second iteration in multilabel classification

Bushra-Aljbawi: <p>Hi everyone,</p>
<p>I’m trying to train a neural network to solve a problem that I framed as a multi-label classification problem. Everything is almost working fine after reading a lot about the problem. However, I got bad results when evaluating the model, and I realized that the training loss is a very low (almost zero) starting from the second iteration in training.</p>
<ul>
<li>Loss function: BCEWithLogitsLoss with a linear output layer</li>
<li>Optimizer: Adam</li>
<li>Note1: The dataset in very sparse as I have over 14k labels and about 10 true labels in every sample. Does this affect the result?</li>
<li>Note2: As I was getting a memory error when loading the whole dataset, I had to train on part of data, save the model and optimizer and re-load them to train on the second part. (I’ve tried all the other solutions in these topics but none of them worked)<br>
Any idea would be much appreciated,</li>
</ul>
<p>Thanks,<br>
Bushra.</p>
Libtorch build error in linux

21fa417e3fb06e56040c: <p>hi  all</p>
<p>following error occurs when build with torch::NoGradGuard<br>
What should I check?<br>
Same code build Windows no error</p>
<pre><code class="lang-auto">torch::NoGradGuard
error: declaration does not declare anything [-fpermissive]
</code></pre>
<pre><code class="lang-auto">build info
complier :  g++ 7.5
</code></pre>
<p>CMakeList.txt</p>
<pre><code class="lang-auto">CMakeList.txt

project(SemanticSegmentation-Libtorch)

cmake_minimum_required (VERSION 3.9)

add_definitions(-D__CUDA_NO_HALF_OPERATORS__)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

include_directories(./include)

if(WIN32)
    include_directories(./Poco-1.9.4)
    link_directories(./Poco-1.9.4/lib/x64)
	set(CMAKE_PREFIX_PATH "D:/local/libtorch/")
	set(OpenCV_DIR "D:/local/opencv/build")
	set(CUDNN_INCLUDE_PATH "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/include")
endif(WIN32)

find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)

if(UNIX)
    find_package(Poco REQUIRED Foundation JSON)
endif(UNIX)

set(SRC_FILE src/cocoapi/cocoNote.cpp
			 src/cocoapi/mask.cpp
             src/cocoapi/mask_api.cpp
             src/DataSet/COCODataSet.cpp
             src/DataSet/COCODetection.cpp
             src/models/segmentation/DeeplabV3.cpp
             src/models/segmentation/SegmentationModel.cpp
             src/models/IntermediateLayerGetter
             src/models/segmentation/SegmentationModel.cpp
             src/models/IntermediateLayerGetter.cpp
             src/models/resnet.cpp
             src/utills/metric_logger.cpp
             src/utills/transforms.cpp
             src/main.cpp
            )

add_executable(${PROJECT_NAME} ${SRC_FILE})
target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )

if(WIN32)
    target_link_libraries(${PROJECT_NAME} "PocoFoundation.lib" )
    target_link_libraries(${PROJECT_NAME} "PocoJSON.lib" )
endif(WIN32)

if(UNIX)
    target_link_libraries(${PROJECT_NAME} Poco::JSON)
    target_link_libraries(${PROJECT_NAME} stdc++fs)
endif(UNIX)

set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 17)
</code></pre>

21fa417e3fb06e56040c: <p>It was just an error due to compiler differences.</p>
<pre><code class="lang-auto">// msvc noerror
torch::NoGradGuard;

// g++ usecase
torch::NoGradGuard nograd;
</code></pre>
<p>thank you all.</p>
Angle difference to range [-np.pi, np.pi]

Yashas: <pre><code class="lang-auto">delta_angle[delta_angle &lt; -np.pi] += 2 * np.pi
delta_angle[delta_angle &gt; np.pi] -= 2 * np.pi
</code></pre>
<p>This seems inefficient. Is there a more direct and efficient way?</p>
Bucketiterator gives variable Batch Size

sgaur: <p>Why Bucket-iterator gives me variable length of input_size.</p>
<pre><code class="lang-auto">&gt; train_iterator,valid_iterator,test_iterator = BucketIterator.splits((train_data , valid_data , test_data),
&gt;                                                                    batch_size = 96 , device ='cuda',repeat=False,
&gt;                                                                    sort=False, sort_within_batch=False,
&gt;                                                                    sort_key=lambda x: len(x.text)
&gt;                                                                    )
</code></pre>
<p>Every time I run below Code it gives me different results:</p>
<pre><code class="lang-auto">train_batch = next(iter(train_iterator))
print(train_batch.text.shape)
print(train_batch.LABEL.shape
</code></pre>
<p><strong>1st time Result are below:</strong><br>
torch.Size([383, 96])<br>
torch.Size([96])</p>
<p><strong>2nd time Result are below:</strong><br>
torch.Size([525, 96])<br>
torch.Size([96])</p>
<p>Also when I am trying to run model</p>
<pre><code class="lang-auto">for index,data  in enumerate(train_iterator):
    optimizer.zero_grad()
    
    prediction = model(data.text).squeeze(1)
    loss = criterion(prediction, data.LABEL)
    break;
</code></pre>
<p>I got error related to sizes of original Label abd prediction Label.<br>
<strong>ValueError: Target size (torch.Size([96])) must be the same as input size (torch.Size([501]))</strong></p>
<p>Batch Size should be constant similar to Target_size but every time it got changed.</p>
<p>Thanks.!!</p>

sgaur: <p>Issue is solved I did not provide Fix_length in Field Function.</p>
<pre><code class="lang-auto">TEXT = Field(sequential=True, tokenize=tokenize, lower=True,stop_words = stopwords,
             include_lengths=True, batch_first=True, fix_length=200)
</code></pre>