akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using libtorch inference with tensorrt

pizuzadan: <p>I know that using a tensorrt gives better performance then cuda libtorch. I built libtorch from source code with <code>use_tensorrt</code> flag. Will this give me a performance improvement on a GPU inference? Or is it used for another purpose?</p>

ptrblck: <p>Where are you passing this argument when building <code>libtorch</code>?<br>
Are you using the <code>USE_TENSORRT=1</code> env variable for Caffe2 as described <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/contrib/tensorrt/README.md">here</a>?</p>

pizuzadan: <p>I used CMake directly (<code>-DUSE_TENSORRT</code>, C++ package manager). CMake found tensorrt. I got nvonnxparser_static.lib library (windows static build) after compilation and then linked my application with nvinfer and nvonnxparser_static libraries.</p>
How to normalize a float number between [0,1] range?

costa98: <p>I am working on a discriminator which goal is to say if a video that it gets in input is human made or not.<br>
Therefore the output is a value included between 0(false) and 1(true) for each frame; then the final result is going to be the sum of each frame evaluation.<br>
How do I apply the normalization?</p>
<p>And I also have another doubt: should I normalize every frame result and then normalize the sum too or leave the frame results as they are and normalize only the sum?</p>
Loading of Quantized Model

mohit7: <p>I have quantized model and I want to load it in pytorch but I am not able to do it.<br>
After quantisation the definition of model is changing as fusion of BatchNormalization layer is happening.<br>
But when I am loading the model I have previous definition which does not contain fused layer but other layers are there like quant and dequant layer.</p>
<p>Is there a way  to load quantized model in pytorch?</p>

raghuramank100: <p>Hi Mohit,<br>
Can you provide more details/code? You can load/save quantized models by saving a state_dict().  When you perform fusion, make sure you set inplace=True.</p>

mohit7: <p>Hey <a class="mention" href="/u/raghuramank100">@raghuramank100</a> I have saved the model correctly but I want to use it in pytorch so we must know the definition of model then we can load the state_dict from the saved model file.<br>
But what I have is definition of model without the fusion of layer and that’s where the definition of model changing and I can’t load model.</p>
<p>So Do I need to change the model definition according to the fused layers?</p>

jerryzh168: <p>I think the expectation is to have the original model and go through the whole eager mode quantization flow again, and then load from the saved state_dict.</p>

Giang_Dang: <p>Hi mohit7,<br>
Make sure you create the net using previous definition, and let the net go through process that was applied during quantization before (prepare_model, fuse_model, and convert), without rerun the calibration process.<br>
After that you can load the quantized state_dict in. Hope it helps.</p>
MMdnn conversion

Zrufy: <p>i’m using MMdnn for convert Tensorflow model to pytorch when i’m going to load the model i have this error<br>
RuntimeError: Given groups=1, weight of size 9480 1024 1 1, expected input[1, 1, 1, 1024] to have 1024 channels, but got 1 channels instead</p>
<p>any solution for this error?</p>

ptrblck: <p>Your input seems to be created in a channels-last memory layout, while PyTorch expects channels-first tensors in the shape <code>[batch_size, channels, height, width]</code>.<br>
Assuming your current input has the shape <code>[batch_size, height, width, channels]</code> you could permute the input tensor via:</p>
<pre><code class="lang-python">x = x.permute(0, 3, 1, 2)
</code></pre>

Zrufy: <p>I try but the problem is only for my output layer.If i’m going to change i the input i have this error</p>
<pre><code class="lang-auto">x = torch.randn(1,3, 224, 224)
x = x.permute(0, 3, 1, 2)
</code></pre>
<p><code>RuntimeError: Given groups=1, weight of size 32 3 3 3, expected input[1, 224, 4, 225] to have 3 channels, but got 224 channels instead</code></p>

ptrblck: <p>Your <code>x</code> tensor seems to have the correct shape.</p>
<p>I don’t quite understand the use case.<br>
Are you using the model output in another convolution?</p>

Zrufy: <p>I found the solution through your help. It was an error of the badly converted network. Changed a data everything went well!</p>
RuntimeError: ONNX export failed: Couldn't export Python operator qfn

octavianmm: <p>Hi,<br>
I’m trying to convert a pytorch model to onnx, but the conversion gives the following error:</p>
<p>RuntimeError: ONNX export failed: Couldn’t export Python operator qfn</p>
<p>The definition for qfn is:</p>
<pre><code class="lang-auto">class qfn(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, k):
        n = float(2**k - 1)
        out = torch.round(input * n) / n
        return out

    @staticmethod
    def backward(ctx, grad_output):
        grad_input = grad_output.clone()
        return grad_input, None
</code></pre>
Modify ResNet to make compatible with RNN (LSTM)

shakeel608: <p>How the resnet layers can be modified in Pytorch.<br>
Suppose I want to add some more convolutions before the FC layer and want to feed the output of ResNet to the LSTM. That means the Resnet should return the</p>
<p><code> [batchsize, Features, Length]</code></p>
<p>How can we modify it ?</p>

ptrblck: <p>The most flexible way to apply your modifications would be to write a custom model and derive from ResNet as the base class.<br>
This would allow you to add new layers in your custom model (such as the <code>LSTM</code>) and to modify the <code>forward</code> method as you want.<br>
The resnet implementation from <code>torchvision</code> can be found <a href="https://github.com/pytorch/vision/blob/4246abc897aa4547b76bad51357178e5911b803d/torchvision/models/resnet.py">here</a>.</p>

shakeel608: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Is an activation functions always required?

Javier_Rodriguez_Pui: <p>Hi, from the literature it is known that normally in NN it is used a Softmax for classification and a Sigmoid for regression.</p>
<p>My question is always required an activation function at the end. I am trying a U-net to acquire pixel wise depth in a scene from rgb. and I get better results if I don’t use any sigmoid at the end. I was wondering that maybe there is a logical explanation for that.</p>

tom: <p>Chapter 6 of <a href="https://pytorch.org/deep-learning-with-pytorch" rel="nofollow noopener">our book that you can download here</a> has some discussion on this.</p>
<p>There are two main uses of the activations functions</p>
<ul>
<li>In the inner bits of the network, you need to have one to go beyond linear (i.e. to get the ability to approximate arbitrary functions),</li>
<li>at the final layer, the main purpose is slightly different, it is to “force” the domain: softmax ensures you get probability vectors, sigmoids ensure that you get values in [0…1] (probabilities or intensities or whatever),</li>
<li>so while it may be useful to use an activation to move things in the right domain (and sometimes required e.g. if your loss function, like cross entropy, only works on probabilities), there isn’t any such requirement to do so in regression tasks (and in fact, the most classical linear regression doesn’t do anything like that, right?).</li>
</ul>
<p>So you’re good with that. I’d also take a peek at what other <a href="https://github.com/xanderchf/MonoDepth-FPN-PyTorch" rel="nofollow noopener">people do for your problem domain</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Torch installation issue

Aditya1: <p>I created a virtual environment for my project and I’m trying to install <code>torch</code> via <code>pip</code>.</p>
<pre><code class="lang-auto">pip install torch===1.5.1 torchvision===0.6.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>This gives me the error -</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch===1.5.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.5.1
</code></pre>
<p>Some solutions recommend the use of conda, is it necessary? Can I not do it through just <code>pip</code></p>
<p>I thought it was a virtual env issue maybe but it doesn’t work on the actual environment either</p>

thomasjo: <p>You are using an incorrect version specification. Those triple equal signs need to be double equal signs. In other words, you need to use</p>
<pre><code class="lang-auto">pip install torch==1.5.1 torchvision==0.6.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>

Aditya1: <pre><code class="lang-auto">  ERROR: Could not find a version that satisfies the requirement torch==1.5.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.5.1
</code></pre>
<p>I get this error</p>
Create inmemorydataset in PYG

shangguan91: <p><a>https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html</a></p>
<p>I have already load my data, and split into train and valid. but i need to build them inmemory form, so i can implement repo model on my dataset.</p>
<p>bs = 4<br>
train_loader = DataLoader(train_list, batch_size=bs, shuffle=True, num_workers=6)<br>
test_loader = DataLoader(val_list, batch_size=bs, shuffle=False, num_workers=6),</p>
<p>how could i simply turn my data into inmemorydataset. should i Skip download and process?<br>
can i have some code guidance?</p>
KL Divergence Loss For Regression

MUnique: <p>Hello everyone,<br>
How can I use KL divergence loss instead of MSE loss for regression? Let’s say in a batch of 30 samples we have 30 ground truth labels. In the case of MSE utilization, the mean square of these 30 labels would be calculated and backpropagated to the layers. Now I want to fit a gaussian or a GMM to the 30 labels and get the KL divergence between the true and the predicted distribution and use it as the loss function of the network. How can I do so with Pytorch tensors?</p>

ebarsoum: <p>What is the true distribution in this case? You can try something similar to VAE (Variation Auto Encoder), let the network predict the parameters of the distribution in case of Gaussian it is N means and N std’s. Then sample from the output and try to match that with the label and use KL between a predefined Gaussian such as N(0, 1) and your model output.</p>
Total size of Tensor?

Chong_Toby: <p>Hello, I was trying to get the total pixel count for an image tensor.<br>
The only solution I found is torch.Tensor(np.prod(tensor.size())) which isn’t as elegant as I would like it to be.<br>
Is there somewhere in the documentary I overlook that contains a way to directly return the value? If not, will it be useful if I make a PR about this?<br>
Cheers.</p>

ptrblck: <p>Try <code>tensor.nelement()</code>.</p>

Everett_You: <p>If you do not have the tensor, but you have the torch.Size object, you can use <code>torch.Size.numel()</code>. For example <code>torch.Size([3,5]).numel()</code> returns 15.</p>
Neural net for coarse segmentation

Alex_I: <p>I’m looking at doing something like semantic segmentation of images but where I only have pretty coarse labels - roughly, for each 32x32 patch, I know if the answer should be “yes”, “no” or “unknown”.</p>
<p>I’d like to start with a pretrained net that is reasonably well suited for this, and then modify it.</p>
<p>This is a bit like FCN-ResNet101 in the torch hub, but I don’t need any of the stuff that converts the feature maps back up to full resolution (and in my case it would probably both be slow and hurt performance since my labels just aren’t that exact).</p>
<p>I’m not exactly sure what the ideal size of the receptive field for each position of the feature map should be, but I suspect it should be a bit on the small side - around 128x128 to 256x256, based on what I know about the data.  Receptive fields for resnet/resnext-50 and 101 before the top pooling layer are really <em>really</em> huge by comparison, 483 or 1027 square (most of which comes from padding with typical 224x224 size inputs).  In my case I’d like to feed inputs around 1024x1024 and have receptive fields not larger than 256x256 with a 32x32 stride - ideally <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<ol>
<li>
<p>Is there any example in the literature or a well-known dataset of coarse segmentation similar to this?  What would be the state of the art?</p>
</li>
<li>
<p>What is the best network to use as a starting point?  Can I use a pretrained classifier network (eg some of the nice swsl_resnext… models in timm) and just remove the top pooling layer?  Is any specific classifier best for this? (resnext vs efficientnet vs…) Or, should I use a network already intended for segmentation? (in that case I’m a lot less clear on what to remove…)</p>
</li>
<li>
<p>If I use a network with a large receptive field, how should I shrink it?  Should I just remove blocks from the top until the field shrinks enough?</p>
</li>
<li>
<p>What should I do about padding?  I’d really like to not have padding affect the results. I can ignore the output where it’s too close to the edges; but it would be nice if I could instead somehow modify the network so it has no padding.  In that case I’m really not sure what that would do to a pretrained network that expects padding - any tips?</p>
</li>
</ol>
AttributeError: 'builtin_function_or_method' object has no attribute 'cuda'

cuixing158_1: <p>When I use <code>features = torch.FloatTensor(1,512).zero_.cuda()</code>  in the python script, there is an error, why?</p>
<p><code>features  = torch.zeros(1,512).cuda()</code> is ok!</p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="cuixing158_1" data-post="1" data-topic="88476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cuixing158_1/48/12146_2.png" class="avatar"> cuixing158_1:</div>
<blockquote>
<p>features = torch.FloatTensor(1,512).zero_.cuda()</p>
</blockquote>
</aside>
<p><code>.zero_</code> is a method not attribute. You need to call it using parentheses:</p>
<pre><code class="lang-auto">features = torch.FloatTensor(1,512).zero_().cuda()
</code></pre>
<p>Bests</p>
Pytorch jit vs tensorRT

0xFFFFFFFF: <p>has anyone compared the throughput of a model optimized by both jit and tensorRT?</p>

KevNull: <p>It seems to depend on the specific network. The biggest speedup I’ve seen was close to three times as fast as PyTorch. The lowest was about one and a quarter times as fast.</p>

0xFFFFFFFF: <p>Thanks for sharing. That’s not too bad. could you share which models showed such speedup? and perhaps the speedup data on some models you have tried? I would really appreciate any details.<br>
Thanks again.</p>

Rizhao_Cai: <p>TensorRT much faster</p>

alanzhai219: <p>jit and trt are two different things.<br>
our team are looking into pytorch for a long time. jit is front-end while trt is back-end.<br>
Always, jit is from python. it optimizes pytorch codes and tries to merge some  ops before running the forward. If you dig it, you will find jit and eager call the same op set and just little diff.<br>
However, trt is another accelerated engine, which depends on Nvidia’s GPU. trt will fuse the ops as possible as it can. fused kernel can reduce cost of discrete kernel calls.<br>
besides, trt has other exiting features.</p>

baheytharwat: <p>Hi <a class="mention" href="/u/alanzhai219">@alanzhai219</a>,<br>
Is it possible to optimize a model using torch2trt then scripting the optimized model using torch script/trace? If yes, Will I get better optimization?</p>

alanzhai219: <p>It is not recommended. torch2trt is designed to help developers deploy their script/trace model in TensorRT.<br>
In detail, script/trace just interpreters original PyTorch into IR graph and then torch2trt maps and fuses such graph in trt.<br>
I never try the opposite flow. If you succeed, please let me know.<br>
Thanks,<br>
Alan Zhai</p>
Can I train one network with different image scales

luciaL: <p>My work is medical image segmentation, becasuse of its high image dimension. I always train the segmentation network with smaller patches. But this will lost the global info of the organs. So I’d like to use the whole  3D or 2D image to update the network during the last iteration of every epoch. But I am not sure that’s feasible and how to make it happen.</p>

ptrblck: <p>If the original image resolution yields an out of memory error on your device, you could either try to:</p>
<ul>
<li>apply <code>torch.utils.checkpoint</code> to trade compute for memory</li>
<li>use model sharding, if you have more than a single device, and execute parts of the model on separate devices</li>
<li>execute the last iteration on the CPU, since you would usually have more system RAM than GPU memory</li>
<li>try <a href="https://pytorch.org/docs/master/amp.html">mixed precision training</a> for a potentially reduced memory usage and speedup (you would need to install the nightly binaries for this or build from master).</li>
</ul>
Torchvision super(ImageFolder, self).__init__ TypeError

Ying_Wong: <p>I meet a type error that occurred at the internal function of TorchVision when I try to build ImageNet dataloader. The info is as follows:</p>
<p>test_set = datasets.ImageNet(root=test_data_path, split=‘val’, download=None, transform=transf_test)</p>
<p>File “D:\anaconda3\envs\pytorch-15\lib\site-packages\torchvision\datasets\imagenet.py”, line 57, in <strong>init</strong><br>
super(ImageNet, self).<strong>init</strong>(self.split_folder, **kwargs)</p>
<p>File “D:\anaconda3\envs\pytorch-15\lib\site-packages\torchvision\datasets\folder.py”, line 203, in <strong>init</strong><br>
super(ImageFolder, self).<strong>init</strong>(root, loader, IMG_EXTENSIONS, transform=transform, target_transform=target_transform, is_valid_file=is_valid_file)</p>
<p>TypeError: super(type, obj): obj must be an instance or subtype of type</p>
<p>Pytorch version: 1.5, TorchVision version: 0.6, Python: 3.7.7</p>
<p>Anyone knows why? Thanks in advance.</p>
Using cudnnMultiHeadAttnForward for attention

supun: <p>There’s a CuDNNprimitive (<code>cudnnMultiHeadAttnForward</code>, etc) provided for handling multi-head attention. However, upon browsing the PyTorch code I realized that CuDNN API is not used in PyTorch. In fact, I found none of the issues even discuss about this API. I was wondering whether there’s any reason for not using that. Intuitively, I would expect such a “fused” API would have better performance over launching multiple kernels.</p>

ptrblck: <p>We are exploring different options to accelerate the <code>MultiHeadAttention</code> layer and cudnn would be one possible way to go forward.</p>
How to share kernel weight between Conv2d while using different padding/dilation param

11194: <p>for example, I want to keep conv1 and conv2 using the same weights</p>
<p>self.conv1 = nn.Conv2d(3, 3, kernel_size=3, stride=1)<br>
self.conv2 = nn.Conv2d(3, 3, kernel_size=3, stride=1, dilation=2)</p>
Why the CPU utility has reached about 100% when i run libtorch model in VS

Mr.Crab: <p>i change my pytorch model into libtorch one and runs it in VS2017. However, i  found the consumption of GPU is about 1G which is same as pytorch model run in the python. It is strange that the CPU utility has reached about 100% and never decreased until the program is over.<br>
How should i to slove the p[roblem and decrease the pre-consumption of CPU resource?<br>
Can anyone help me? thanks!<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b9dadfac908e152e12b43a031d9de031addf4b3.png" alt="QQ截图20200709100701" data-base62-sha1="1EL4Um4iy1b5nSJ1NoTlVPp7kUr" width="217" height="75"></p>
In what order do dataloader workers do their job?

Jochem_Grietens: <p>Hello,</p>
<p>Hello, i was wondering how the dataloder with num_workers &gt; 0 queu works. I imagine N wokers are created.<br>
I see 2 options:</p>
<ol>
<li>
<p>the program goes through all workers in sequence? This would mean that if one worker is delayed for some reason, the other workers have to wait until this specific worker can deliver the goods. This would also mean that if a worker gets stuck into an infinite loop while fetching its data, the training process is going to be waiting forever.</p>
</li>
<li>
<p>The program takes a batch from whichever worker is done first. This means that if a worker gets into an infinite loop, another worker will always be able to get the training process new data. In this case i am wondering what would happen to the worker in the infinite loop. Does it get killed, does it get sidelined in some way ?</p>
</li>
</ol>
<p>To give some context to my reason for asking this question:<br>
I am doing object detection. And in my data loader i have a transform that does a random crop. However, i don’t want to do random crops with no bounding boxes contained within it. So i wrote a loop inside the transform that does random crops on the same image until it finds a candidate with a bounding box contained in it.</p>
<p>My fear is that there might be cases in which no such crop is ever found ( maybe there is no bounding box in the image). In this case i want to understand how the dataloader/training process would deal with this.</p>
<p>Thanks a lot.</p>

ptrblck: <p>Yes, your first point is correct. If I remember if correctly, the order of the returned batches wasn’t defined some time ago (in <code>0.4</code>?), which could yield a non-deterministic behavior. Based on this code snippet, you can see that <code>worker0</code> would slow down <code>worker1</code> and the output order would still be the expected one.</p>
<pre><code class="lang-python">import torch
from torch.utils.data import Dataset, DataLoader


class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(100)

    def __getitem__(self, index):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info:
            worker_id = worker_info.id
            print('worker_id {} calling with index {}'.format(worker_id, index))
            if worker_id == 0:
                print('slowing down worker0')
                a = 0.
                for idx in range(10000000):
                    a += idx
        x = self.data[index]
        return x

    def __len__(self):
        return len(self.data)


if __name__=='__main__':
    dataset = MyDataset()
    loader = DataLoader(dataset, batch_size=2, num_workers=2, timeout=1)

    for data in loader:
        print(data)
</code></pre>
<p>If you slow down <code>worker0</code> even more, the <code>timeout</code> threshold (passed to the <code>DataLoader</code>) would kick in and raise an error.<br>
Note that the <code>timeout</code> is set to <code>0</code> by default, so that your code could hang, if a worker isn’t returning from your processing logic.</p>
Share Kernel Between Dilated Conv2d Layers

Micah_Goldblum: <p>I would like to perform 2d convolution on the same data using the same kernel but with different dilation rates.  I do I do this kernel sharing?</p>
<p>Would it be best to initialize a kernel and then use that same kernel between multiple conv2d layers using the functional version as in the following?</p><aside class="quote quote-modified" data-post="6" data-topic="13207">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mactarvish/48/3555_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-share-a-convolution-kernels-transpose-to-another-kernel/13207/6">How to share a convolution kernel's transpose to another kernel?</a> 
  </div>
  <blockquote>
    Thank you man. The reason I do 
self.conv2.weight[0,0] = self.conv1.weight[0,0].t()

is that the convolution kernel is a 4d tensor, i.e.: 
[torch.FloatTensor of size output_chanel x input_channel x kernel_height x kernel_width]

So I cannot just use conv2.weight = conv1.weight.t() to transpose all of the kernels. 
But even I replace conv with linear layer, like that: 
import torch
import torch.nn as nn


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        se…
  </blockquote>
</aside>

<p>Or should I make multiple layers with different kernels and then set the kernels equal to each other?</p>

11194: <p>Hi, I have the same question. Do you solve the problem?</p>
Ways to speed up training when models has long for loops

Chao_Feng: <p>I hope this question make sense.  Is tensorflow faster than pytorch in general when the model contains a long <code>for</code> loop of modules? For example, the model I have needs to call a module for ~ 20 times in a for loop at each iteration. I reimplemented in pytorch based on original author’s tensorflow code. Although I achieved similiar accuracy, the tensorflow code is about 4 or 5 times faster than pytorch code. I did some profiling and it seems this is due to the large number of calls to certain modules in each iteration. And my understanding is tensorflow created the graph before training so the <code>for</code> loop does not matter. Is my impression correct? Is there any ways to speed it up? If I somehow changed the for loop into a large module with deeper layers, would it be able to achieve similiar performance to the tensorflow code? Thank you!</p>
Parameter not updating

vainaijr: <p>When I do something like this,</p>
<pre><code class="lang-auto">class Custom(nn.Module):
    def __init__(self):     
        super().__init__() 
        self.threshold = nn.Parameter(torch.randn(1))
    def forward(self, x):
       print(self.threshold)
       return x
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):     
        super().__init__() 
        self.custom = Custom()
    def forward(self, x):
       x = self.custom(x)
       return x
</code></pre>
<p>when I do</p>
<pre><code class="lang-auto">net = Net()
list(net.parameters())
</code></pre>
<p>then it does show the threshold value, plus requires_grad is set as True for it.<br>
but when I do, optimizer.step() later, then the threshold parameter do not update, it stays to one fixed value.</p>

qmeeus: <p>Do you have a loss function? Normally, one should do:</p>
<pre><code class="lang-auto">net = Net()
pred = net(X)
loss = loss_fn(pred, y)
loss.backward()
optim.step()
</code></pre>
<p>If you did not calculate the error, pytorch does not know how to update the parameters…</p>

vainaijr: <p>Hello, thanks for your reply, the parameters described in Net class get updated, but the parameters described in Custom class do not get updated, that is the threshold does not get updated. I am doing the get loss, backprop, optimize part.</p>

qmeeus: <p>Would you have a small working example to reproduce the issue?</p>
Create a dataloader using a list of targets and a list of tensors as data

SandPhoenix: <p>Hello everyone,</p>
<p>So, I am working on a small project and I am kind of stuck for like 2 hours now on a thing that seems simple so I would be very thankful if anyone can help.<br>
So, I have a list of tensors that I called new_images and a list of labels.<br>
I want to create a dataloader using these two.<br>
So, I tried several things like for example a dumb list of tuples but it did not work. It showed an error but I ignored it because I know that this kind of  solution would not work.<br>
I tried creating a class dataset like this:</p>
<pre><code class="lang-auto">class DataSetCos(torch.utils.data.Dataset):
    def __init__(self, nbr_samples, new_images, new_labels, transform=None):
        self.nbr_samples=nbr_samples
        self.labels=new_labels
        self.transform = transform
    def __len__(self):
        return self.nbr_samples
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        sample = new_images[idx]
        if self.transform:
            sample = self.transform(sample)
        if (sample.float().size()[0] == 3):
            return [sample.float(), self.labels[idx]]
</code></pre>
<p>after that I loaded it like this:</p>
<pre><code class="lang-auto">dataset= DataSetCos(nbr_samples=100, new_images=new_images, new_labels=new_labels, transform =None)
dataloader= torch.utils.data.DataLoader(dataset,batch_size=batch_size,shuffle=False)
</code></pre>
<p>but when I try testing my model, I get this error IndexError: list index out of range.</p>

Nikronic: <p>Hi,</p>
<p>It is really hard to say what the exact problem is but it seems data loader is generating indices that is out of bound for your lists. I think changing <code>__len__</code> can help. Try this:</p>
<pre><code class="lang-auto">
    def __len__(self):
        return len(self.new_images)
</code></pre>
<p>Also, can you let me know the shape of <code>new_image</code> and <code>new_labels</code>?</p>
<p>Bests</p>

SandPhoenix: <p>Hi thx for your answer!<br>
I will try doing that.<br>
For the shapes, the new_images is a list of tensors of len 100 and same thing for the new_labels</p>
Check which engine is used for convolution operations

JustasZ: <p>Hello,</p>
<p>I am trying to figure out which engine is being used for convolution operations when I run my model with PyTorch Mobile. I would like to change it to XNNPACK if that isn’t the default. Any help on how to do any of these two things would be appreciated.</p>

xta0: <p>Hi <a class="mention" href="/u/justasz">@JustasZ</a>. If you compile pytorch from the current master branch, the XNNPACK has already been enabled on mobile. But if you want to use cocoapods or  Android maven, please wait for the 1.6.0 release, which will set XNNPACK as the default computation kernel.</p>
How to transfer and load Glove vectors on a machine without internet

abhigenie92: <p>I was working on computer gpu which doesn’t have internet connection. So something like</p>
<pre><code class="lang-auto">TEXT.build_vocab(train_data, 
                 vectors = "glove.6B.50d", 
                 max_size=25000,
                 unk_init = torch.Tensor.normal_)
</code></pre>
<p>gives me an error</p>
<pre><code class="lang-auto">urllib.error.URLError: &lt;urlopen error [Errno 111] Connection refused&gt;
</code></pre>
<p>But I can transfer to it via a usb-drive. How can I download glove vectors from somewhere and configure them to work on pytorch?</p>

fatihbeyhan: <p>were you able to solve this problem?</p>
Writing training data to a log file and plotting training curves

Vijay_Dubey: <p>Hello everyone!!<br>
How can I write my training data like the training loss, training error, validation error in a log file and then plot i the  learning curve to visualize the performance of my model??<br>
Please suggest . Also it would be very helpful, if there is some pre-implemented example related to that.</p>
<p>Thanks and regards</p>

jdhao: <p>Save the stat of each epoch either in numpy array or in a list and save it. When the training process ends, plot the stat saved. I think it is pretty simple.</p>

SKYHOWIE25: <p>You can do it with the tensorboard logger.</p>
<p>For more information, please refer to <a href="https://github.com/TeamHG-Memex/tensorboard_logger" rel="nofollow noopener">https://github.com/TeamHG-Memex/tensorboard_logger</a></p>
<p>I am using it and it works perfectly for me.</p>

tengerye: <p>I think <a href="https://github.com/lanpa/tensorboardX/tree/8a7925b27cc7233c8fa700c11b741725fc1ce2d9" rel="nofollow noopener">tensorboardX</a> is better. You can find tutorial <a href="https://tensorboardx.readthedocs.io/en/latest/index.html" rel="nofollow noopener">here</a>.</p>

arnabsinha: <p>Does TensorboardX work for Windows? I am using Jupyter via Anaconda on Windows OS.</p>
Add Attention to CNNs

shakeel608: <p>I want to add an attention layer to the CNN layers.<br>
Is this okay in Pytorch to add an attention layer  like below</p>
<pre><code class="lang-auto">        input = self.conv8(input)
        input = self.batchnorm8(input)
        input = self.relu(input)
        #Attention Along Frequency and Channel Dimension  #Input Shape is [b_size X Channels X Feature X Time]
        attention_weights = self.get_attention_weights(input)
        input = input * attention_weights
        out = LSTM(input)   #Feeding to RNN then 



def get_attention_weights(self, input):
        #print("attention weights", input.shape)
        input = self.attention(input)
        #print("after attention weights", input.shape)
        input = self.attnpool(input)
        #print("after attention [pool]", input.shape)
        return input
</code></pre>
DDP Learning-Rate

Ilia_Karmanov: <p>I was a bit confused how DDP (with NCCL) reduces gradients and the effect this has on the learning-rate that needs to be set.</p>
<p>Would the below example be a correct way to interpret this → that <strong>DDP and DP should have the same learning-rate if scaled out to the same effective batch-size</strong>?</p>
<p>Assume set contains 80 samples<br>
Single-gpu LR = 0.1<br>
Total-grad-distance = LR * g * (samples/batch-size)</p>
<ol>
<li>
<p>Single-gpu<br>
batch = 8<br>
gradient = 8g/8 = g<br>
total-grad-distance = 0.1 * g * 10 = g</p>
</li>
<li>
<p>DP (2-gpu, 1 node)<br>
batch = 16<br>
gradient = 16g/16 = g<br>
total-grad-distance = 0.1 * g * 5 = 0.5g<br>
 → thus scale LR by 2</p>
</li>
<li>
<p>DDP (2-gpu, 1 node OR 1-gpu, 2 nodes)<br>
batch-per-process = 8<br>
gradient = (8g/8) + (8g/8) / 2 = g<br>
total-grad-distance = 0.1 * g * 5 = 0.5g<br>
 → thus scale LR by 2?</p>
</li>
</ol>
<p>Or does allreduce just <a href="https://pytorch.org/docs/stable/notes/ddp.html#implementation" rel="noopener nofollow ugc">sum the gradients</a> in which case:</p>
<blockquote>
<p>and  <code>ProcessGroup::allreduce()</code>  to sum gradients.</p>
</blockquote>
<ol start="3">
<li>DDP (2-gpu, 1 node OR 1-gpu, 2 nodes)<br>
batch-per-process = 8<br>
gradient = (8g/8) + (8g/8) = 2g<br>
total-grad-distance = 0.1 * 2g* 5 = g<br>
 → thus leave LR the same as single-GPU</li>
</ol>

vfdev-5: <p>If you maintain the same batch size between single GPU and DP/DDP, according to your calculations you do not need to adjust LR ?</p>
<p>PS:<br>
In DDP grads are averaged: <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a></p>
<blockquote>
<p>During the backwards pass, gradients from each node are averaged.</p>
</blockquote>
<p>PPS: <a href="https://arxiv.org/pdf/1706.02677.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1706.02677.pdf</a></p>
<blockquote>
<p>Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.</p>
</blockquote>

mrshenli: <p>More discussions can be found at <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769" class="inline-onebox">Should we split batch_size according to ngpu_per_node when DistributedDataparallel</a></p>
Why the spawn is just running

sharkdeng: <p>Hello, why the spawn cell looks like stuck? This state has been for 30 mins.</p>
<pre><code class="lang-auto">FLAGS = {}

xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/743bb770fbef6e2b532425fbc6871be31bfe3f94.png" data-download-href="https://discuss.pytorch.org/uploads/default/743bb770fbef6e2b532425fbc6871be31bfe3f94" title="Screen Shot 2020-07-08 at 11.14.19 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/743bb770fbef6e2b532425fbc6871be31bfe3f94_2_690x146.png" alt="Screen Shot 2020-07-08 at 11.14.19 PM" data-base62-sha1="gAfknDbMKQSKr4KjlsQ3xTh3zOk" width="690" height="146" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/743bb770fbef6e2b532425fbc6871be31bfe3f94_2_690x146.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/743bb770fbef6e2b532425fbc6871be31bfe3f94_2_1035x219.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/743bb770fbef6e2b532425fbc6871be31bfe3f94.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-07-08 at 11.14.19 PM</span><span class="informations">1190×252 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Add KL div as regularization to CrossEntropyLoss

netaglazer: <p>hi<br>
I use CrossEntropyLoss, and I want to add KL div between labels and prediction as regularization to the loss</p>
<p>how can I do it?<br>
thanks</p>

alpha_omega_alpha: <p>Check out the VAE <code>loss_function</code> in Pytorch’s examples <a href="https://github.com/pytorch/examples/blob/e7870c1fd4706174f52a796521382c9342d4373f/vae/main.py#L73" rel="nofollow noopener">here</a>.</p>
Selective masking when computing the gradient

Eweler: <p>Is it possible to only compute the loss of some model with respect to certain batch elements?</p>
<p>For concrete, let’s say we have some <code>model</code>, with <code>loss = model(x)</code>, where <code>x</code> has size <code>[batch_size, dim]</code>. There is a boolean mask <code>mask</code> with entries which are True at the batch indices where the tensor should be evaluated and False otherwise. Is there a way to allow the model to only compute the gradient for these entries?</p>
<p>While it is possible to compute the loss on unmasked elements only, <code>loss = model(x[mask])</code>, I would like to make use of the loss of the masked elements in <code>x</code>, just not allow these elements to affect the model.</p>

chetan_patil: <p>Consider you have simple classifier whose input size is <code>[20,5]</code> with <code>20</code> being the <code>batch_size</code>. After you forward the input to the model, it will give the output of size <code>[20, num_classes]</code>.<br>
Now, when you calculate the loss using the <code>nn.CrossEntropyLoss()</code> or <code>nn.MSELoss()</code>, you have an argument <code>reduction</code>, if you pass its value as <code>'none'</code>, the resulting loss-tensor would be a vector of size <code>25</code>. Each entry in the <code>loss</code> would be the corresponding loss for each image in the batch.<br>
You can apply the <code>Boolean-mask</code> here and then average it or sum it up.</p>
How do I further train already trained model

Muhammad4hmed: <p>Hi, I’m training effiicient net b0 on custom data set,<br>
I already have weights for models which was trained with 30 epochs.<br>
now what I did is, I loaded the model,</p>
<pre><code class="lang-auto">pth = 'model.pth'
model.load_state_dict(torch.load(pth))
</code></pre>
<p>and started training for 5 epochs.<br>
My question is, will this model be considered for trained for total of 35 epochs or this will overwrite previous ones and be trained for 5 epochs?</p>
<p>if it will overwrite, then kindly suggest me how can I train it further</p>

alpha_omega_alpha: <p>Loading a pretrained model for training will further train the model, for example to fine-tune the model (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#finetuning-from-a-pretrained-model" rel="nofollow noopener">Finetuning from a pretrained model</a>).</p>

Muhammad4hmed: <p>Hi indeed it is training and can be seen through the loss and accuracy that it is further improving.<br>
but on submitting the solution, the accuracy on hidden data is dropped.<br>
on straight 30 epochs it was 87 but now after training for 5 more epochs it is dropped to 83.<br>
is there any parameter I am missing ?</p>
Selection part of file.names as label

shangguan91: <p>Hallo,<br>
I have filenames like abc_def_ghl</p>
<p>i want to select def<br>
what should my code be<br>
if f[5:7] == “def”:<br>
d.y = torch.tensor([0]) ?</p>

alpha_omega_alpha: <p>I would suggest using Python’s <code>str.split()</code> (<a href="https://docs.python.org/3/library/stdtypes.html#str.split" rel="nofollow noopener">Reference</a>):</p>
<pre><code class="lang-auto">filename = "abc_def_ghi"
def = str.split(filename, "_")[1]
</code></pre>
How to retrain a pretrained model loaded from the hub

qmeeus: <p>Hi,</p>
<p>I am trying to retrain a pretrained model (<code>pytorch/fairseq/xlmr.base</code>) loaded from the hub. The examples provided on the documentation focus on fine-tuning and evaluation. What if I want to retrain a pretrained model on additional data? Is there an interface that allows this?</p>
<p>I should specify that the architecture mentioned above is not available in fairseq’s registered architectures and AFAIK, it is not possible to use the utility <code>fairseq-train</code> with a pretrained model from the hub.</p>
<p>Thanks a lot</p>
No module named 'convpoint.knn.lib'

shangguan91: <p>hallo, i could not install the nearest neighbor library, can not found the library in the folder.</p>
<p>The intruction said that i could compile, however there is error as well.<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/aboulch/ConvPoint/tree/master/convpoint" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/9435359?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/aboulch/ConvPoint/tree/master/convpoint" target="_blank" rel="nofollow noopener">aboulch/ConvPoint</a></h3>

<p>Contribute to aboulch/ConvPoint development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Which error message do you get when you try to build the repository?<br>
Note that you might get a faster and better answer, if you create an issue in the linked repository, so that the original authors would have a chance to support you. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

shangguan91: <p>thank you, i solved!!!</p>
Name 'torch' is not defined while doing `torch.load`

vas: <p>Hi,</p>
<p>I am doing <code>torch.load</code> to load my model (I know it’s not recommended but I don’t have another way)</p>
<p>But then I got the following error</p>
<p><code>35         x = torch.cat(x, 1)</code><br>
<code>NameError: name 'torch' is not defined</code></p>
<p>Which is very strange, the training went smoothly, the model code is accessible.<br>
How I can resolve this issue?</p>

alpha_omega_alpha: <p>Maybe try restarting your Python kernel runtime and perform <code>import torch</code> again.</p>

brenth: <p>Hi Vas, have you managed to figure something out? Facing a similar issue.</p>
Slowdown over time using momentum

mzimmer: <p>I have created my own torch.optim.Optimizer class which implements a variant of the FrankWolfe (aka conditional gradient descent) method. While initializing the optimizer, I pass a momentum value 0 &lt;= m &lt;= 1. I recently noticed that the optimizer is becoming slower and slower in each step and I cannot figure out why. I did some timings and figured that it is due to my momentum code. The following is the momentum part of my step() function:</p>
<pre><code class="lang-auto">@torch.no_grad()
def step(self, closure=None):
    for group in self.param_groups:
        for p in group['params']:
            if p.grad is None:
                continue
            d_p = p.grad

            # Add momentum
            momentum = group['momentum']
            if momentum &gt; 0:
                param_state = self.state[p]
                if 'momentum_buffer' not in param_state:
                    param_state['momentum_buffer'] = d_p.detach().clone()
                else:
                    param_state['momentum_buffer'].mul_(momentum).add_(d_p, alpha=1 - momentum)
                    d_p = param_state['momentum_buffer']
...
</code></pre>
<p>In the first iteration of step(), the momentum_buffer is ont set and I hence set it to be the last gradient itself. In later iterations I multiply the old momentum buffer by momentum factor (e.g. 0.9) and add d_p to it, being the p.grad.</p>
<p>What could be the problem here? Do I accumulate history? I am unsure why this would get slower and slower over time, in the beginning it is fairly fast but the runtime is monotonically increasing.</p>

ptrblck: <aside class="quote no-group" data-username="mzimmer" data-post="1" data-topic="88212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mzimmer/48/31510_2.png" class="avatar"> mzimmer:</div>
<blockquote>
<p>What could be the problem here? Do I accumulate history? I am unsure why this would get slower and slower over time, in the beginning it is fairly fast but the runtime is monotonically increasing.</p>
</blockquote>
</aside>
<p>That could be the case. Do you also see an increased memory usage?<br>
If so, could you check, if your momentum buffer has a valid <code>.grad_fn</code> after a couple of iterations?</p>
<p>If so, it could store the computation graph. However, since you have already used the <code>no_grad</code> context manager this should not be the case.</p>
<p>How did you isolate that the momentum calculation is responsible for the slowdown?</p>
Visulaizing Tensorboard in pytorch

887574002: <p>Hi, would you please tell me which package should I install that I can run<br>
<code>from torch.utils.tensorboard import SummaryWriter</code></p>
<p>Or</p>
<p><code>from tensorboard import SummaryWriter </code></p>
<p>to visualize my graph in tensorboard?<br>
I installed a tone of packages and still, these two commands are unknown in my notebook!<br>
Shouldn’t it have been mentioned somewhere in PyTorch tutorials or Docs!</p>

111358: <p><code>from torch.utils.tensorboard import SummaryWriter</code> is right, you can see it here: <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">TORCH.UTILS.TENSORBOARD</a>. Maybe your notebook’s environment is not correct.</p>

887574002: <p>This command is not known in my notebook, despite I have PyTorch installed.</p>

111358: <p>What is your pytorch’s version? It should be higher than 1.2.</p>

887574002: <p>pytorch version 1.5.1</p>

harsha_g: <p>Try checking if you have <code>tensorboard</code> installed.</p>

887574002: <p>I installed tensorboard but still got the error</p>
<pre><code class="lang-auto">ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed```
When i try to use the command

```from torch.utils.tensorboard import SummaryWriter```</code></pre>

111358: <p>Can you import tensorboard in python command line?</p>
Binary file to train network

ML0401: <p>Hello,</p>
<p>I am new to pytorch and I am trying to import binary files (matlab matrices which are images) to train a standard resnet18 with. What is the best way to do this? Is there a way to import binary files with torchvision.datasets or do I have to write my own dataloader?  Is it possible to train the network with these binary files or do I have to imoprt every file and transform it to a tensor on my own?</p>
<p>Thanks in advance.</p>

ptrblck: <p>You could use <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html">scipy.io.loadmat</a> to load the MATLAB files.<br>
Once you have loaded them as numpy arrays, you would have to transform them to tensors via <code>torch.from_numpy()</code>.<br>
I would recommend to write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> for this use case.</p>