akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
No gradients available for affine transforms

username: kekday
cooked: <p>I was looking for some information on affine transforms and came across post on a thread: <a href="https://discuss.pytorch.org/t/affine-transformation-matrix-paramters-conversion/19522/2" class="inline-onebox">Affine transformation matrix paramters conversion</a></p>
<p>When I try to print(x_trans.grad_fn) at the end, it prints None. Can someone help me understand why no gradients are calculated for this operation?</p>

username: ptrblck
cooked: <p>If no parameters in the operations require a gradient, you won’t get a <code>grad_fn</code>.<br>
You could set <code>requires_grad=True</code> e.g. for <code>x</code>, which will give you a valid <code>grad_fn</code> in <code>x_trans</code>.</p>
Combine pytorch nonzero calls

username: Luca_Pamparana
cooked: <p>I have a Pytorch tensor and currently I am getting the indexes with zero values as follows:</p>
<p><code>non_zero_ind = (torch.nonzero(array[:, 4]))</code></p>
<p>Now, I also need to get these indexes if the values at indices 2, 3 are also zeros. So something like:</p>
<p><code>non_zero_ind = (torch.nonzero(array[:, 4])) or (torch.nonzero(array[:, 2])) or (torch.nonzero(array[:, 3]))</code></p>
<p>How can one perform this quite of query on the pytorch tensor?</p>

username: LeviViana
cooked: <p>I’m a little confused by your statement.</p>
<aside class="quote no-group" data-username="Luca_Pamparana" data-post="1" data-topic="46438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_pamparana/48/14310_2.png" class="avatar"> Luca_Pamparana:</div>
<blockquote>
<p>I am getting the indexes <strong>with zero values</strong></p>
</blockquote>
</aside>
<p>By using <code>torch.nonzero</code> you get the indices where the values <strong>are not zero</strong>. So, If you want to get the indices where the values of your array are non-zero in all of the three dimensions, namely 2, 3 and 4, then you could do <code>array[:, 2:5].prod(1).nonzero()</code>.</p>
How to add new format of image(.nii) to datasets.ImageFolder?

username: Madi
cooked: <p>RuntimeError: Found 0 files in subfolders of: D:/Python/python_stuff/datasets/MRI Normal/dev/train<br>
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif</p>

username: Geeks_Sid
cooked: <p>Hi,</p>
<p>It makes more sense in this scenario to write a custom dataloader since nifti is not supported as an extension yet.</p>
<p>Check out PyTorch Custom DataLoader.</p>
<p>I personally use CSV for DataLoading instead.</p>

username: Madi
cooked: <p>Hi<br>
Thanks.<br>
Can we add any format to folder.py at the torchvision/datasets?</p>

username: Geeks_Sid
cooked: <p>Have you checked out pytorch custom dataloader</p>
<p>Go ahead and check <a href="https://forums.fast.ai/t/custom-data-loader-for-3d-data/27290" rel="nofollow noopener">this</a><br>
StackOverflow and forums taught me this.</p>

username: Madi
cooked: <p>Not yet. I had a meeting.<br>
I will check and let you know. Thanks a lot.<br>
Also, I have added the nii format to folder.py, I got another error. it does not work.</p>
Custimize your own optimizer?

username: Daniel_Joseph
cooked: <p>I want to code my own optimizer for the CNN like Particle Swarm Optimization (PSO) and use it instead of the others(Adam, SGD …), how can I start I am stuck.<br>
Any suggestion would be much appreciated.</p>

username: Meltemology
cooked: <p>I am also new to Pytorch but as I understood in this thread</p>
<p><a href="https://discuss.pytorch.org/t/simulated-annealing-custom-optimizer/38609">https://discuss.pytorch.org/t/simulated-annealing-custom-optimizer/38609</a> people implemented custom optimizers.</p>
How to use multigpu pytorch?

username: Daniel_Joseph
cooked: <p>How to use multiple GPU on pytorch, any concrete example such as cifat100 or mnist , just to understand it more.<br>
Thaks in advance</p>

username: LeviViana
cooked: <p>The <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">documentation</a> presents you a detailed tutorial on how it can be done.</p>
<p>You could use <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener"><code>torch.distributed</code></a> as well, which is useful if your GPUs are not located in a single machine.</p>
<p>Here is a very simple snippet for you to get a grasp on how it could be done. I’m using <code>torch.distributed.launch</code> here below, you should save this snippet as a python module (say <code>torch_dist_tuto.py</code>) then run <code>python -m torch.distributed.launch --nproc_per_node=4 torch_dist_tuto.py</code></p>
<pre><code class="lang-python">import torch
import argparse
import os
import torch.distributed as dist

parser = argparse.ArgumentParser()
parser.add_argument("--local_rank", type=int)
args = parser.parse_args()

rank = args.local_rank
size = int(os.environ["WORLD_SIZE"]) if "WORLD_SIZE" in os.environ else 1

dist.init_process_group("gloo", rank=rank, world_size=size)

x = torch.arange(10)[rank::size].sum()

print("Process rank {}, partial result {}".format(rank, x))

dist.reduce(x, dst=0)
if rank == 0: print("Final result:", x)
</code></pre>
<p>Try running this with other values of <code>nproc_per_node</code> and see the magic.</p>

username: Daniel_Joseph
cooked: <p>Thanks, <a class="mention" href="/u/leviviana">@LeviViana</a> I appreciate</p>
Torch.ger never returning in 1.1.0

username: jts
cooked: <p>I’m playing with the Q&amp;A code from Facebook Research and had it working perfectly.  It was running with torch 0.4.0 on a GTX 1080 with cuda 9.0 and Python 3.6.7.  I got a new machine with two RTX 2080 Ti cards.  I tried just updating everything to the latest, torch 1.1.0, cuda 10.2, latest nVidia driver.  The app would hang at the torch.ger call.  I tried doing just CPU (no cuda) and got the same result.  I goes into the function and never returns.</p>
<p>I downgraded torch to 1.0.1.post2 and it works fine with the CPU.  I still need to get all the GPU stuff sorted out to fix the CUDNN_STATUS_EXECUTION_FAILED error but for now I’d just like to figure out why the torch.ger function is not returning.</p>
<p>It’s being passed two tensors of shape [452] of type torch.float32. Comparing the tensors in 1.0.1.post2 vs 1.1.0 in a debugger they are almost identical.  There are a few values that are a tiny bit different but nothing that should break it.</p>
<p>One additional bit of information, the code uses the multiprocessing library.  I read that the pytorch wrapper around it is preferred.  I was unable to switch over because of some missing functions like Finalize.  But just looking at the ger function, it doesn’t seem like multiprocessing is the problem.</p>
<p>Any ideas what I should be looking at?</p>
Runtimeerror: cuda error: out of memory pytorch

username: viswanath660
cooked: <p>I’m getting runtimeerror: cuda error: out of memory pytorch with batchsize over 4</p>
<p>on NVIDIA P100 GPU with 16GB memory.</p>
<p>test_loader = DataLoader(dataset=test_dataset, batch_size=BATCH_SIZE,<br>
shuffle=False, num_workers=8, pin_memory=True)</p>
<pre><code># initialize the ground truth and output tensor
gt = torch.FloatTensor()
gt = gt.cuda()
pred = torch.FloatTensor()
pred = pred.cuda()

# switch to evaluate mode
model.eval()

for i, (inp, target) in enumerate(test_loader):
    target = target.cuda()
    gt = torch.cat((gt, target), 0)
    bs, n_crops, c, h, w = inp.size()
    input_var = torch.autograd.Variable(inp.view(-1, c, h, w).cuda(), volatile=True)
    output = model(input_var)        &lt;=== error thrown here.
</code></pre>
<p>I opened the gif files I am training and closed them also, but it didn’t workout</p>
<p>image = Image.open(image_name)<br>
image1 = image.convert(‘RGB’)<br>
image.close()</p>

username: ptrblck
cooked: <p>Could you lower the batch size to 1 and check, if your code works?<br>
Also, could you check the output of <code>nvidia-smi</code> and see, how much memory your model uses and if other processes are using the GPU?</p>
<p>If your memory footprint is really large, you might try to use <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener">checkpointing</a> to trade compute for memory.</p>

username: viswanath660
cooked: <p>It works with batch size 3. But, for my experiment, I need batch size 8. I am processing x-ray images. Let me try checkpointing.</p>
Gradient becomes nan

username: NKPmedia
cooked: <p>Hello</p>
<p>I implemented an own layer with the following little forward method:</p>
<pre><code class="lang-auto">    def __init__(self, a, b):
        super().__init__()

        self.a = nn.Parameter(torch.Tensor([a]))
        self.a.requires_grad = False
        self.b = nn.Parameter(torch.Tensor([b]))
        self.b.requires_grad = False

        self.i = 0

    def activate_parameter_learning(self):
        self.a.requires_grad = True
        self.b.requires_grad = True
    def forward(self, input: torch.Tensor, mse_error):
        if(torch.abs(mse_error).item() &gt;= 0.0001):
            weight = 1 - 1 / (1 + self.a * torch.exp(self.b / mse_error))
        else:
            weight = 1
        output = input.mul(weight)
        return output
</code></pre>
<p>But after some backprop steps the gradient of a and b becomes Nan.<br>
Do you have any idea how I can find the step in the back propagation where this happens.</p>
<p>Thx</p>
Expected backend cpu but got backend gpu

username: cpieringer
cooked: <p>Hi,</p>
<p>I am new in Pytroch. Implementing a VAE example, I got the error message “expected backend CPU but got backend GPU”. I build the network within a class (nn.Module). I solve the error passing the device as a parameter during the instantiation of the VAE class.</p>
<p>In the forward method, originally I use:</p>
<pre><code class="lang-auto">def forward(self, x):
   if x.dim() &gt; 2:
       x = x.view(-1, 28 * 28)

    batch_size = x.size(0)
    means, log_var = self.encoder(x)
    std = log_var.mul(0.5).exp_()
    eps = torch.randn([batch_size, self.latent_size])
    z = ((eps * std) + means) 
    recon_x = self.decoder(z)

    return recon_x, means, log_var, z
</code></pre>
<p>After passing the device as parameters, I modify eps and z as follow:</p>
<pre><code class="lang-auto">eps = torch.randn([batch_size, self.latent_size]).to(self.device)
z = ((eps * std) + means).to(self.device)
</code></pre>
<p>Although, this way makes the code works correctly in a GPU or CPU. I would like to know if it is a correct practice or if there is the best way to do it.</p>
<p>Best.</p>

username: ptrblck
cooked: <p>Instead of passing the <code>device</code> argument to your model, you could also just use the current <code>device</code> of another tensor, e.g. <code>x</code>:</p>
<pre><code class="lang-python">eps = torch.randn([batch_size, self.latent_size], device=x.device)
</code></pre>

username: cpieringer
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the advice!</p>
Loss.backword() goes wrong

username: genous110
cooked: <p>when I use loss.backward(), I meet the error of “RuntimeError: Function SubBackward0 returned an invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.cuda.DoubleTensor”<br>
, I 'm sure that the type of  variable “loss” is torch.cuda.FloatTensor . Because</p>
<pre><code class="lang-auto">isinstance(loss, torch.cuda.FloatTensor)
</code></pre>
<p>it show True.<br>
Can you help me what’s wrong with the loss.backward().</p>

username: theevann
cooked: <p>The problem is not with the loss Variable, but probably with some other Variable in your model that should be FloatTensor but that is DoubleTensor.</p>
<p>Without more context, I can’t say much.</p>
What happen if I reuse nn.Linear?

username: leolcling
cooked: <p>Hello,</p>
<p>What would happen if I reuse nn.Module? Will the module share the same weights?</p>
<p>A simple model:</p>
<pre><code class="lang-auto">class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()  
        self.linear = nn.Linear(1, 1)

    def forward(self, x):
        output = self.linear(x)
        output = self.linear(output)  # re-using self.linear
        return output
</code></pre>
<p>Thanks</p>

username: MaazJamal
cooked: <p>What do you mean by reuse nn.Module and sharing weights?<br>
What you have done above is reusing nn.Module class, by doing simple python inheritance. You have inherited n.Module class and are now making a derived class that contains all of the methods of nn.Module class as well as any new methods you define.<br>
Sharing of weights is something different and depends on architecture that you are making.</p>

username: leolcling
cooked: <p>What I mean is I instantiated nn.Linear and assigned it to self.linear. In the forward method, I use self.linear twice. Does it mean that I repeated two identical layers here? Sorry for the confusion, just realize it is a bad title…</p>

username: MaazJamal
cooked: <p>yes you have repeated the same identical layers, infact you have used the same layer twice. They will have the same weights because at the end of the day they are the same thing.<br>
However:</p>
<pre><code class="lang-auto">self.linear = nn.Linear(1, 1)
self.linear1 = nn.Linear(1, 1)
...
 def forward(self, x):
     output = self.linear(x)
     output = self.linear1(output)  # re-using self.linear
     return output
</code></pre>
<p>linear1 and linear will have different weights. Even though you used the same nn.Linear(1, 1)</p>
Pytorch Model to C++ Model

username: A.V_Yugesh
cooked: <p>Hello all,</p>
<p>I have a pytorch model of size 13.6 mb ,  I need to convert that into c++ for better performance.</p>
<p>I have followed the below steps but after conversion the model size become 4mb.and If I try running both models with same inputs … the outputs are very strange and mismatching.</p>
<p><strong>here is what I tried:</strong></p>
<p>model = ConvTasNet.load_model(“C:/Users/user/Desktop/dss/dss_code/src/Models/13mb.tar”)<br>
model.eval()<br>
example = torch.rand(1,160)<br>
traced_script_module = torch.jit.trace(model, example)<br>
traced_script_module.save(“m13mb.pt”)</p>
<p><strong>Error is:</strong></p>
<p>TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!</p>
<p>can anyone help in this regard?</p>
<p>Thanks,<br>
Yugesh.</p>
Can I custom transform MNIST images and targets in transforms.Compose?

username: Omkar_Manjrekar
cooked: <p>I am trying to flatten MNIST images and convert labels to one hot encoded vectors in transforms.Compose itself. How can that be done?</p>

username: MariosOreo
cooked: <p>The MNIST dataset from torchvision is in PIL image. So if you want to flatten MNIST images, you should transform the images into tensor format by <strong>transforms.ToTensor()</strong> in <code>transforms.Compose([])</code>. And then you could use DataLoader to load the images, read and flatten batches of them.</p>
<p>In my shallow view, you can not convert the sparse labels into one-hot format in <strong>transforms.Compose</strong>. You could change them into one-hot label per batch by this:</p>
<pre><code class="lang-auto">one_hot = torch.zeros(len(sparse_label), 10)
one_hot = one_hot.scatter_(1, sparse_label.unsqueeze(1), 1)
</code></pre>

username: Omkar_Manjrekar
cooked: <p><a class="mention" href="/u/mariosoreo">@MariosOreo</a> Thanks for your answer.</p>
<p>Yeah I saw that there is no way to get “target labels” inside <strong>transforms.Compose</strong> since the call to it includes “img” only. I was accessing MNIST dataset’s train_data and train_labels. I thought transforms would apply to them as well. My bad. I later found that they are applied only in data loaders.</p>
How to get the probability of prediction in the transfer learning

username: bubu_ka
cooked: <p>Hi I am using the transfer learning tutorial for my classification (two labels) with the following link: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a>.</p>
<p>except data loader part, The code about mode is only changed with following code which change the class numbers:</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)
</code></pre>
<p>I found the prediction of outputs in the following code:</p>
<pre><code class="lang-auto"> with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    print(outputs.shape, outputs)
                    _, preds = torch.max(outputs, 1)
</code></pre>
<p>The output like the following result:</p>
<pre><code class="lang-auto">torch.Size([32, 2]) tensor([[ 3.1194, -1.4058],
        [ 1.4559, -1.3360],
        [ 1.9011, -1.0287],
        [ 0.8083, -1.0037],
        [ 0.7831, -1.1790],
        [ 0.7990, -1.3758],
</code></pre>
<p>I found that the number is not probability of prediction like softmax function in the keras output .</p>
<p>Now I am using 5-folds to predict the test data, and I want to get the probability of each folds<br>
,and average of the sum all folds probabilities.</p>
<p>I don’t know how to change the code for my goal of getting probability of prediction.</p>

username: LeviViana
cooked: <p>This tutorial uses <code>torch.nn.CrossEntropyLoss()</code> as the loss function, which means that you should just run <code>torch.nn.functional.softmax</code> on your outputs, since it learned log-softmax values.</p>

username: bubu_ka
cooked: <p>thanks for your answer.</p>
Setting limits on the predictions

username: iatheia
cooked: <p>I’m training a network on the synthetic data that are a decent but not a perfect approximation of the real observations. The input labels have a particular range, but it’s not uncommon for the predictions for the real observations can frequently be outside of that range (e.g. becoming negative, which would not have a physical meaning), and that may be trowing the other parameters off.</p>
<p>Is there a way to force lower and upper limits on the predicted values?</p>
Pytorch to C++ Conversion

username: YUGESH_A_V
cooked: <p>Hello all,<br>
I have a pytorch model , i need to convert that into c++, i followed the steps in the link [<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a>] . My model is related to Conv.TasNet. The initial pytorch model have size 13.6 mb and c++ model is about 4mb</p>
<p>Note: When i run the models with the same inputs , the outputs are very strange and not all matching. What would be the reason ? can anyone help on this?</p>
<p>here is what i did!!!</p>
<p>model = ConvTasNet.load_model(“C:/Users/user/Desktop/dss/dss_code/src/Models/13mb.tar”)<br>
model.eval()<br>
example = torch.rand(1,160)<br>
traced_script_module = torch.jit.trace(model, example)<br>
traced_script_module.save(“m13mb.pt”)</p>
<p>Thanks<br>
Yugesh.</p>
How can I remove element?

username: dedf21972b190932c756
cooked: <p>Hi ~<br>
I’d like to know how can I remove empty elements.<br>
This is my data setting code:</p>
<pre><code class="lang-auto">fields = [("", None),("text", TEXT),("label", LABEL)]
train_data,valid_data, test_data = data.TabularDataset.splits(
                path = '',
                train = 'four_train.csv',
                validation = 'four_valid.csv',
                test = 'four_test.csv',
                format = 'csv',
                fields = fields,
                skip_header = True
)
</code></pre>
<p>Data type is “torchtext.data.dataset.TabularDataset”<br>
And I run this code for finding NULL</p>
<pre><code class="lang-auto">for i in range(len(train_data)):
    if train_data[i].text[:] == []:
        print(i)
</code></pre>
<p>This is the result<br>
424<br>
499<br>
2186<br>
2727<br>
2757<br>
3063<br>
3086<br>
3158<br>
4508<br>
5035<br>
5420<br>
6005<br>
6799<br>
8815<br>
8840<br>
9846<br>
-----Skip~<br>
I know there are a lot of empty elements.<br>
So, how can I remove it?</p>
<p>The reason why i remove it is that it might cause this error:<br>
“RuntimeError: Length of all samples has to be greater than 0, but found an element in ‘lengths’ that is &lt;= 0”, when I run the code.</p>
PTX JIT compilation failed when running FasterRCNN

username: danthekang
cooked: <p>When I try to train FasterRCNN in newly released torchvision 0.3, I run into a PTX JIT compilation failed error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_rcnn.py", line 124, in &lt;module&gt;
    loss_dict = model(images, targets)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/danielkang92/nuscenes/code/FasterRCNN/model.py", line 19, in forward
    return self.rcnn(images, targets)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 48, in forward
    features = self.backbone(images.tensors)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torchvision/models/_utils.py", line 58, in forward
    x = module(x)
  File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: /pytorch/torch/csrc/jit/fuser/cuda/fused_kernel.cpp:202: a PTX JIT compilation failed
</code></pre>
<p>My environment is as follows :</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 410.72
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] torch                     1.1.0                    pypi_0    pypi
[conda] torchvision               0.3.0                    pypi_0    pypi
</code></pre>
<p>Would appreciate any help on this!</p>

username: Michael_Suo
cooked: <p>Thanks for the report! Can you file an issue on github with the same info and we can track it from there? Thanks</p>

username: danthekang
cooked: <p>Just filled it out here: <a href="https://github.com/pytorch/pytorch/issues/21004" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21004</a><br>
Thanks!</p>
Cuda runtime error (77) : an illegal memory access was encountered

username: Robot_mantou
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.utils.rnn

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.lstm = nn.LSTM(batch_first=True,
                            bidirectional=True,
                            input_size=8,
                            hidden_size=252)

    def forward(self):
        output = torch.cuda.DoubleTensor(1,3,8)
        output, (hn, cn) = self.lstm(output) 
        return output

model = Model().cuda()
output = model()
print(output[0, 0, 0])
</code></pre>
<p>This code arised an cuda runtime error: ‘an illegal memory access was encountered’. But when I changed the hidden dimentions of LSTM to 251, the error is missing. Can anyone give me a favor? Thanks a lot.</p>

username: Robot_mantou
cooked: <p>I have solved the question. Actually, change the DoulbeTensor to FloatTensor the error is missing. Related issue can be found here:<br>
<a href="https://github.com/pytorch/pytorch/issues/2267" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2267</a></p>
About weight in BCEWithLogitsLoss to handle imbalanced samples

username: Elvin_Wang
cooked: <p>I read document of BCEWithLogitsLoss and I find “weight” parameter is different with   CrossEntropLoss’s “weight”. In BCEWithLogitsLoss, “weight” has to be a Tensor of<br>
size nbatch , but in CrossEntropLoss, “weight” is a shape of C (number of classes).   I first not understand “nbatch” meaning, number of batch or batchsize. After trying, I<br>
find it should be batchsize. I give a Tensor with N*C (batchsize * class_weights) size<br>
as “weight” to BCEWithLogitsLoss and the loss function is running without any<br>
problems.</p>
<p>But I am still worried about whether I do this correct or not. So my question is<br>
The “weight” parameter in BCEWithLogitsLoss do the same effort (balance imbalance<br>
sample) as the one in CrossEntropLoss’s, right? If so, why they are different. I think<br>
this may bother some user as me.</p>
<p>I would be quite appreciate if someone could help me. Thanks!</p>
How to change size of filter in torch model

username: skyline
cooked: <p>For example, squezeenet1_1 can only input some specific size of the image. Others sizes will raise an error.can we change the filter size?</p>

username: Geeks_Sid
cooked: <p>Basically the issue arises if your network is not fully convolutional, that is, if it consists of some Linear layers(Fully connected). Try removing those and you can probably get away with any input size you would like.</p>

username: skyline
cooked: <aside class="quote no-group" data-username="Geeks_Sid" data-post="2" data-topic="46317">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geeks_sid/48/10013_2.png" class="avatar"> Geeks_Sid:</div>
<blockquote>
<p>size</p>
</blockquote>
</aside>
<p>But it will change the structure of the net.And how to implement remove/replace a layer in torch code?</p>

username: Geeks_Sid
cooked: <p>No, It would not change the structure of the network.</p>
<p>Anyhow, replacing a layer or removing would definitely change the intermediate structure of the network.</p>
<p>There are a lot of tutorials about removing the layers. <a href="https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648">Follow here</a></p>
RuntimeError: Could not infer dtype of numpy.int64

username: cswangjiawei
cooked: <p>I have a line of code: <code>label_tensor = torch.tensor(self.label [item]).long ()</code>when I train on Google colab, it’s not wrong, but when I train on another server, it gets an error. I am very confused. How to solve this problem?</p>

username: ptrblck
cooked: <p>Are you using the same PyTorch version in both setups?<br>
Also, could you try to use <code>torch.from_numpy(self.label[item])</code>?</p>

username: cswangjiawei
cooked: <p>Thank you very much! I modify it to <code>label_tensor = torch.tensor(int(self.label [item])).long ()</code>,then it is correct.</p>
Trying to repeatedly call .backward() - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

username: HaziqRazali
cooked: <p>I am trying to run the code below but I get the error</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>I don’t see where the inplace operation is because I am able to run the first loop. I am trying to get the output of backprop repeatedly after a forward pass.</p>
<blockquote>
<pre><code>    model_output = self.model(input_image)[0][0].squeeze()
    for box in boxes_0:
        self.model.zero_grad()
        roi = model_output.clone()
        roi = torch.cuda.FloatTensor(model_output.size()).zero_()
        roi[0,box[1]:box[3],box[0]:box[2]] = 1
        roi = roi.unsqueeze(0)
        model_output.backward(gradient=roi,retain_graph=True)
        gradients_as_arr0 = self.gradients.data.cpu().numpy()[0]
        gradients_as_arr0 = np.transpose(gradients_as_arr0, [1,2,0])
        grads.append(gradients_as_arr0)
</code></pre>
</blockquote>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/usr/lib/python3.6/runpy.py”, line 193, in _run_module_as_main<br>
“<strong>main</strong>”, mod_spec)<br>
File “/usr/lib/python3.6/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/home/haziq/openpifpaf_crm_pose/openpifpaf/backprop.py”, line 347, in <br>
main()<br>
File “/home/haziq/openpifpaf_crm_pose/openpifpaf/backprop.py”, line 279, in main<br>
guided_grads0 = GBP.generate_gradients_new(data)<br>
File “/home/haziq/openpifpaf_crm_pose/openpifpaf/backprop.py”, line 190, in generate_gradients_new<br>
model_output.backward(gradient=roi,retain_graph=True)<br>
File “/home/haziq/env_crm/lib/python3.6/site-packages/torch/tensor.py”, line 102, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/haziq/env_crm/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>

username: HaziqRazali
cooked: <p>How do i look for the variable that has been modified by an inplace operation?</p>
Strange timing with jit-compiled module during memory allocation

username: ygabuev
cooked: <p>Hi!</p>
<p>There is some strange behavior that I cannot understand. My setup: Ubuntu 16.04.5, pytorch 1.1.0, Tesla V100 SXM2.</p>
<p>I have a simple C++ module <code>empty_module</code> that does memory allocation on GPU and can be called from python (see example code <a href="https://gist.github.com/ygabuev/406d2ddbdd542725333193844998b2f0" rel="nofollow noopener">here</a>). There is a boolean flag that controls, whether the allocated memory should be freed (which is the correct behavior).</p>
<p>The default execution time on my machine is roughly 1.5ms. When I set the flag to 0 (the memory is not freed up), the execution time falls by roughly 3 orders of magnitude to 12µs, and the memory leaks, which can be viewed in <code>nvidia-smi</code>. However, if after that, without relaunching the python interpreter, I run the same function again with the flag set to 1 (memory is freed up, memory does not leak), the execution time stays at the same low levels as if the momory is not freed.</p>
<p>Perhaps this is the question for Nvidia developers, but I’m new to this and cannot devise an example without pytorch.<br>
How does this happen?</p>

username: ptrblck
cooked: <p>How do you check that the memory is leaking?<br>
<code>nvidia-smi</code> might show used memory, while it is in fact cached memory by PyTorch to avoid expensive re-allocations.</p>

username: ygabuev
cooked: <p>Each time I run the cell with <code>deallocate=False</code>, the memory usage on my device slightly increases. Calling <code>torch.cuda.empty_cache()</code> does not help to release it.</p>
Calculate l2-norm to extract highest K regions of feature maps

username: solsol
cooked: <p>Hi.<br>
I need to calculate L2-norm for image representation tensor of img_fmap with [N, M, 25, 25] shape for each ij spatial location. then select the K highest  for each 25X25 and equal to 1.</p>
<p>img_fmap = img_fmap.view(N, M, 25*25)<br>
vector = torch.sum( img_fmap** 2, dim=2)<br>
topk_idxes = torch.topk(vector, k=self.k)[1]</p>
<p>the result is in [N, M] shape.</p>
<p>Thanks!</p>
How to label a picture and please help me to fix error in my post

username: buinhatquan
cooked: <p>I’am senior and my topic is handwritten digit recognition, but i use my dataset to train, not use MNIST dataset. I’m trying follow to this website <a href="https://nextjournal.com/gkoehler/pytorch-mnist" rel="nofollow noopener">https://nextjournal.com/gkoehler/pytorch-mnist</a>   to train my dataset. Unfortunately, I dont know how to label my image and I have a problem with my code like this. My code and my error below. Thank you everyone!!</p>
<p><a href="https://drive.google.com/drive/folders/1CbkZ7mz2KDrK7MClb5sAOlglY2Z3vlAk?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/drive/folders/1CbkZ7mz2KDrK7MClb5sAOlglY2Z3vlAk?usp=sharing</a></p>

username: ptrblck
cooked: <p>The easiest way would probably be to store each image in the corresponding subfolder and use an <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder" rel="nofollow noopener">ImageFolder</a> to automatically generate the labels.</p>

username: buinhatquan
cooked: <p>Let me try, thank u <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Batch pinverse for vectors

username: Adam_Stevens
cooked: <p>So I am trying to solve for the matrix A in b = Ax. I have a batch of vectors for b and x and I want to get a batch of A matices. I have tried two ways. One uses torch.gels and the other uses pinverse. Neither handles vectors properly with a batch.</p>
<p>This works with single vectors</p>
<p><strong>embed_0 = torch.tensor([[.4,.7,.8]])<br>
embed_9 = torch.tensor([[3,.2,.1]])<br>
M, _ = torch.gels(embed_9, embed_0)</strong></p>
<p>tensor([[0.9302, 0.0620, 0.0310],<br>
[1.6279, 0.1085, 0.0543],<br>
[1.8605, 0.1240, 0.0620]])</p>
<p>However,  if I have batches of vectors it still outputs a single matrix for M</p>
<p><strong>embed_0 = torch.tensor([[.5, .9 ,1.0],[.4,.7,.8]])<br>
embed_9 = torch.tensor([[.55, .77, .2],[3,.2,.1]])<br>
M, _ = torch.gels(embed_9, embed_0)</strong><br>
tensor([[  46.2997,   -7.1800,   -1.0000],<br>
[-127.9999,   20.8000,    3.0000],<br>
[  92.6000,  -14.3600,   -2.0000]])</p>
<p>How can I solve for the A matrix with batches?</p>
Regarding Detach function

username: SHASHANK_KUMAR_MISHR
cooked: <p>Can somebody explain in simple words with illustration what does detach function do in pytorch?</p>

username: LeviViana
cooked: <p>There are basically two things to understand before talking about detach:</p>
<ul>
<li>What is a Computational Graph ?</li>
<li>How PyTorch builds a Dynamic Computational Graph ?</li>
</ul>
<p>I’m assuming that you know the answers to the questions above. So, detach is a method that allows you to manipulate Tensors without expanding the computational graph. For instance:</p>
<pre><code class="lang-python">import torch

x = torch.rand(5).requires_grad_(True)
y = x ** 2 + 1
</code></pre>
<p>Say that you are interested in the sum of the elements of x. Then if you run <code>print(x.sum())</code> PyTorch will create a new computational graph having x as leaf. But if you run <code>print(x.detach().sum())</code> no additional graph will be created.</p>
<p>The most common use of <code>detach</code> I’ve seen happens when you want to transfer PyTorch’s Tensors (having <code>requires_grad == True</code>) to NumPy. You can’t run <code>x.numpy()</code> in the snippet above, you should rather run <code>x.detach().numpy()</code>.</p>

username: SHASHANK_KUMAR_MISHR
cooked: <p>Thanks @  <a href="https://discuss.pytorch.org/u/LeviViana">LeviViana</a></p>
Retrain a model multiple times

username: Mdhvince
cooked: <p>Hi everyone,</p>
<p>I was wondering if we can train a model on day 1, save it, then do another training on day 20 but on additional data without losing what the model has already learned. Is it something we can do with PyTorch? If yes, can someone provide a very simple example, like from saving the model to reload, keeping in mind that we can’t lose what the model has already learned?</p>
<p>I hope I was clear, Thank you.</p>
<p>Medhy</p>

username: LeviViana
cooked: <p>It can be easily done in PyTorch, I recommend you to read the <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">documentation</a>.</p>

username: Mdhvince
cooked: <p>Thank you, I was looking exactly for this part :slight_smile</p>
<h2>Saving &amp; Loading a General Checkpoint for Inference and/or Resuming Training</h2>
Make multiple AIs vote the decision to take

username: heitaum
cooked: <p>hello everyone !</p>
<p>I have trained various  neural networks, often with the same parameters and training environnement, but I often ended up with imperfect AIs which were always slightly different from their peers. Especially when the environnment contained some randomness.</p>
<p>I just had the idea of making different neural networks with the same training, let’s say a dozen, and to make them vote the decision to take.</p>
<p>My hope being that the different AIs, who are right around 90% of the time, would not be wrong all at the same time.</p>
<p>I wanted to know if anyone ever tried anything similar ? is the idea good or just stupid (and if the latter, why ?)</p>
<p>It would take long to train those, so if anyone tried before and can tell me about the result, that would be nice (and save me much time ! ^^)</p>
<p>Have a nice day !</p>

username: JuanFMontesinos
cooked: <p>It’s already invented for machine learning <a href="https://en.wikipedia.org/wiki/AdaBoost" rel="nofollow noopener">https://en.wikipedia.org/wiki/AdaBoost</a>. I haven’t seen works on deep learning but it should be similar.</p>
How to transform the batch in the loss function?

username: nikogamulin
cooked: <p>Hi,</p>
<p>The dependent variable of the custom model is a 24-dimensional vector of continuous values that represent 6 bounding boxes (top, left, bottom right for each bounding box). In the loss function, I would like to compare the distances between the centroids of the represented bounding boxes. Thus, every batch item should be reshaped into 12-dimensional vector as every bounding box should be transformed into a point (y = (top + bottom)/2, x = (left + right)/2).</p>
<p>I tried to do the following:</p>
<pre><code class="lang-auto">input_boxes = list(map(lambda x: list(zip(*[x[i::4] for i in range(4)])), input))
input_points = list(map(lambda x: ((x[0] + x[2])/torch.Tensor(2), (x[1] + x[3])/torch.Tensor(2)), input_boxes))
</code></pre>
<p>This resulted an error: mul(): argument ‘other’ (position 1) must be Tensor, not tuple</p>
<p>Does anyone know how to reshape the batch properly?</p>
<p>Thank you!</p>
<p>Niko</p>

username: nikogamulin
cooked: <p>Upon checking the documentation, I came up with the following function:</p>
<pre><code class="lang-auto">def bb2centroids(source_tensor):
    batch_size, coords_count = list(source_tensor.size())
    bb_count = coords_count/4
    box_coords = torch.functional.split(source_tensor, 1, dim=1)
    centroid_coords = torch.tensor(())
    for i in range(int(bb_count)):
        bb_idx = i * 4
        y1 = box_coords[bb_idx]
        y2 = box_coords[bb_idx + 2]
        x1 = box_coords[bb_idx + 1]
        x2 = box_coords[bb_idx + 3]
        y = (y1 + y2)/2
        x = (x1 + x2)/2
        centroid_coords = torch.cat((points_coords, y), 1)
        centroid_coords = torch.cat((points_coords, x), 1)
    return centroid_coords
</code></pre>
<p>While testing this by manually setting a tensor, it worked fine. When I plugged this in loss function, however, it threw the following error:</p>
<p> in bb2centroids(source_tensor)<br>
12         y = (y1 + y2)/2<br>
13         x = (x1 + x2)/2<br>
—&gt; 14         centroid_coords = torch.cat((points_coords, y), 1)<br>
15         centroid_coords = torch.cat((points_coords, x), 1)<br>
16     return centroid_coords</p>
<p>RuntimeError: Expected object of backend CPU but got backend CUDA for sequence element 1 in sequence argument at position <span class="hashtag">#1</span> ‘tensors’</p>
<p>Does anyone know how to fix this?</p>

username: nikogamulin
cooked: <p>After initializing centroid_coords as FloatTensor, everything worked fine:</p>
<pre><code class="lang-auto">centroid_coords = torch.cuda.FloatTensor(())
</code></pre>
Problems loading list of layers to cuda device

username: baalimago
cooked: <p>Hello!</p>
<p>I’m trying to make “dynamic layering”, so that I can enter parameters when creating a model to easily vary layer/neuron size and try many different settings to find what’s optimal through brute force (cuz math is hard, and takes brainpower). This works well on the CPU! But it doesn’t work when I try to load it onto cuda, for some reason. I’m suspecting the list holding the different layers cannot be loaded to cuda.</p>
<p>Here’s example code of working and not working concepts:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Working(nn.Module):
    def __init__(self):
        super(Working, self).__init__()
        self.inp = nn.Linear(784, 30) 
        self.l0 = nn.Linear(30, 30)
        self.l1 = nn.Linear(30, 30)
        self.l2 = nn.Linear(30, 30)
        self.l3 = nn.Linear(30, 30)
        self.l4 = nn.Linear(30, 10)

    def forward(self, x):
        x = self.inp(x)
        x = self.l0(x)
        x = self.l1(x)
        x = self.l2(x)
        x = self.l3(x)
        x = self.l4(x)

        return F.softmax(x, dim=-1)

class Not_Working(nn.Module):
    def __init__(self, am_l, am_n):
        super(Not_Working, self).__init__()
        self.inp = nn.Linear(784, am_n)  
        self.h_layers = []
        for i in range(am_l):
            o_lay = 10  if i == am_l-1 else am_n
            self.h_layers.append(nn.Linear(am_n, o_lay))

    def forward(self, x):
        x = self.inp(x)
        for layer in self.h_layers:
           x = layer(x) 
        return F.softmax(x, dim=1)

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

#This works
w   = Working().to(device)
x2  = torch.randn(784).to(device)
y2  = w(x2) 

#This doesn't work
n_w = Not_Working(5, 30).to(device)
x   = torch.randn(784).to(device) 
y   = n_w(x) #RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'mat2'
</code></pre>
<p>Am I doing something wrong? According to my own logic, the list of layers should be transferred to cuda using Not_Working(3,30).-&gt;to(device)&lt;- but it doesn’t seem to work. Should I try to modify the .to() function to include lists somehow?</p>

username: ptrblck
cooked: <p>To properly register modules you would have to use <code>nn.ModuleList</code> instead of a plain Python list.<br>
This will also make sure to pass all parameters to the optimizer.</p>

username: baalimago
cooked: <p>Ah, I see. Thank you!</p>
Using torch.no_grad() gives RuntimeError about modifying variable by an inplace operation

username: tarvoc
cooked: <p>I want to train two agents who play a game (in the game theory sense) through gradient ascent. Each agent has a number of parameters identical to their number of strategies. These parameters are initialized with normally distributed random variables. The parameters get translated into a probability distribution by applying a softmax. I am aware of <code>torch.optim</code>, but want to write the gradient ascent step myself, since I later want to investigate modifications to it. The code below works as expected:</p>
<pre><code class="lang-auto">import torch as T

lr = 0.01
std = 0.01
num_strategies1 = 3
num_strategies2 = 5
# Generate random payoff matrices for both players.
payoff_matrix1 = T.randn((num_strategies1, num_strategies2))
payoff_matrix2 = T.randn((num_strategies2, num_strategies1))

params1 = std * T.randn(num_strategies1)
params2 = std * T.randn(num_strategies2)
params1.requires_grad_()
params2.requires_grad_()

for _ in range(10):
  probs1 = T.softmax(params1, dim=0)
  probs2 = T.softmax(params2, dim=0)
  payoff1 = T.dot(probs1, T.matmul(payoff_matrix1, probs2))
  payoff2 = T.dot(probs2, T.matmul(payoff_matrix2, probs1))
  grad1 = T.autograd.grad(payoff1, params1, create_graph=True)[0]
  grad2 = T.autograd.grad(payoff2, params2, create_graph=True)[0]
  params1.data += lr * grad1.data # GRADIENT ASCENT
  params2.data += lr * grad2.data # GRADIENT ASCENT  
</code></pre>
<p>Instead of using <code>.data</code>, I want to use the <code>torch.no_grad()</code> context manager. So I replace the two lines marked with <code>GRADIENT ASCENT</code> with the following code:</p>
<pre><code class="lang-auto">  payoff1.backward(retain_graph=True)
  with T.no_grad():
    params1 += lr * grad1
    params2.grad.zero_()

  payoff2.backward()
  with T.no_grad():
    params2 += lr * grad2
    params1.grad.zero_()
    params2.grad.zero_()     
</code></pre>
<p>However, I now get the following error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>I’ve never had this issue before when using <code>torch.no_grad()</code>. Surprisingly, the issue also disappears if I replace <code>probs1 = T.softmax(params1, dim=0)</code> with a meaningless <code>probs1 = T.sigmoid(params1)</code> (and the same for <code>probs2</code>).</p>
<p>How do I need to change my code such that I can use <code>torch.no_grad()</code>?</p>
How to fine-tuning network for face recognition

username: skyline
cooked: <p>The question is how to fine tuning a network for face recognition.<br>
We with triple-loss as metric. It’s somewhat different from other vision task?<br>
I found some torch model here <a href="http://www.robots.ox.ac.uk/~albanie/pytorch-models.html" rel="nofollow noopener">http://www.robots.ox.ac.uk/~albanie/pytorch-models.html</a></p>
Change batch size (during forward pass)?

username: marchinidavide
cooked: <p>Hi all,</p>
<p>First of all, thank to all of you guys at Pytorch.<br>
It’s a great tool with outstanding documentation!</p>
<p>I came across this problem while working on different “feedings” in seq2seq models (I like “Professor Forcing” approach (<a href="https://arxiv.org/abs/1610.09038" rel="nofollow noopener">paper</a>), but would like skip an expensive GAN).<br>
I have an input tensor with size (T, B, N) and need to output a tensor with size (T, 3*B, N) which is the concatenation of 3 different methods of the forward pass (different encoder-decoder pairs with different feeding mechanism in the decoder part, very similar to the seq2seq tutorial), something like:</p>
<pre><code class="lang-auto">def forward(self, input):
    out1 = self.first_method(input) # (T, B, N)
    out2 = self.second_method(input) # (T, B, N)
    out3 = self.third_method(input) # (T, B, N)
    out = torch.cat((out1, out2, out3), dim=1)
    return out
</code></pre>
<p>My target tensor (used to calculate loss) is also with size (T, 3*B, N) and I use no BatchNorm or any “batch_” dependent layer operation.<br>
What is the expected behaviour of autograd in this case?<br>
I’m not entirely sure this ca work as it would need to backprop the loss from an object with batch size 3B into an input with batchsize B … should this be the case, any recommended trick?</p>
<p>Thanks in advance!</p>

username: dimaxano
cooked: <p>Hi!<br>
The best solution, as usual, is just to check it in practice (whether it works or not) <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But as I understand mechanics, after loss calculation you will get a scalar value that will be backpropagated. And I suppose it shouldn’t care about the difference of input and output batch sizes.</p>