akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Segmentation fault when matrix inverted

username: cagatayyildiz
cooked: <p>I’m having a segmentation fault error when I try to invert a 150x150 matrix, as simple as this:</p>
<pre><code class="lang-auto">J = torch.randn([20,150])
torch.inverse(J.t()@J)
</code></pre>
<p>I didn’t try to find out the exact threshold but for example, 120x120 inversion is fine.</p>
<p>Also, I am having this issue on my macOS Sierra (10.12.6) with Python 3.7.3, Clang 4.0.1 and torch 1.3.0. My Ubuntu machine with Python 3.7.4, GCC 7.3.0 and 1.1.0 has no problem inverting even 3k-3k matrices.</p>
<p>Any suggestions?<br>
Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I cannot reproduce this locally with python 3.7 on macos.<br>
Do you know where the segfaults happen exactly? (using gdb or similar tools)<br>
Also, do you see the same problem with <code>torch.inverse(J.t().clone()@J)</code>?<br>
Also how did you installed pytorch?</p>

username: Yaroslav_Bulatov
cooked: <p>Singular matrices expose bugs in numalg algorithms…</p>
<p>Wondering if your call is going through MKL under the hood. I’ve seen crashes in that library for singular matrices which <a href="https://software.intel.com/en-us/forums/intel-distribution-for-python/topic/628049" rel="nofollow noopener">get fixed</a> by setting OMP_NUM_THREADS=1</p>
<p>Dump utility below could be useful to get repro – save the matrix before the crash. Then you can upload the file and wrap it into easy to run repro like <a href="https://github.com/pytorch/pytorch/issues/25978#issue-492018796" rel="nofollow noopener">here</a></p>
<pre><code class="lang-auto">def dump(result, fname):
    """Save result to file. Load as np.genfromtxt(fname). """
    result = to_numpy(result)
    if result.shape == ():  # savetxt has problems with scalars
        result = np.expand_dims(result, 0)
    location = fname
    # special handling for integer datatypes
    if (
            result.dtype == np.uint8 or result.dtype == np.int8 or
            result.dtype == np.uint16 or result.dtype == np.int16 or
            result.dtype == np.uint32 or result.dtype == np.int32 or
            result.dtype == np.uint64 or result.dtype == np.int64
    ):
        np.savetxt(location, X=result, fmt="%d", delimiter=',')
    else:
        np.savetxt(location, X=result, delimiter=',')
    print("Dumping to", location)
</code></pre>

username: cagatayyildiz
cooked: <p>I’m using Anaconda3 on my computer. I don’t really remember how I installed torch but I anyways uninstalled it. When I re-installed it via <code>pip</code>, the error persisted. I once again uninstalled and installed this time via <code>conda</code>. Now, everything seems fine.</p>
<aside class="quote no-group" data-username="cagatayyildiz" data-post="1" data-topic="60899">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cagatayyildiz/48/17682_2.png" class="avatar"> cagatayyildiz:</div>
<blockquote>
<p>torch.inverse(J.t()<span class="mention">@J</span>)</p>
</blockquote>
</aside>
Std::cout printing without tensor value

username: Edwinzero
cooked: <p>I am using mathematical operations in the libtorch C++<br>
However, when I try to make a scalar-tensor product, the cout returns only the shape of the tensor without values.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1cb95f65a49a312a86e3e82b7d3b7e9cc5fcf726.png" alt="image" data-base62-sha1="466wKwvV79Bgi0ou6q5970r8j8W" width="612" height="255"><br>
The corresponding printing info of the last iteration:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f14534530a083c49d701086857e9c70aacafc13.png" alt="image" data-base62-sha1="901CTuehqXyIn7xacG3XB3jddbd" width="493" height="130"></p>
<p>Why this error occurs? Is this a computational error or something else?</p>
<p>Is there any tutorial materials for mathematical operations of libtorch?</p>
Vectorizing a normalizing flow

username: crainone
cooked: <p>Hello, I am quite new to Pytorch and DL in general. I have a flow (an NVP one to be exact) that takes a 2d tensor z in input and transforms it into another 2d tensor \phi, like so</p>
<pre><code class="lang-auto">phi,logJ = the_Flow(z)
</code></pre>
<p>and returns also the logarithm of the Jacobian of the transformation.<br>
I would like to be able to feed a batch of zs (i.e. a 3d tensor) without having to do it sequentially (in a “for” loop) for every single z, and get a batch of \phis and determinants in return. I need to do it because the loss needs to be computed as an average over said batch.<br>
Can I do this without having to redefine the flow to operate on 3d tensors?</p>
Inconsistency in lambda learning rate

username: tremblerz
cooked: <p>So, I had to implement my own Cosine Annealing learning rate because the one given by the PyTorch does the LR warmup in a different way therefore I implemented it myself and put it as <code>lambdaLR</code>. Now the problem is if I print the learning rate from inside my lambda function then it prints the expected value but when I print it using the code <code>optimizer.param_groups[0]["lr"]</code> it is exactly <code>20</code> times lower and even the model actually trains on a learning rate 20 times lower than the expected one. This I can verify by using some other learning rate scheduler instead of the lambda scheduler. What can be possibly wrong? If the issue doesn’t get solved by the aforementioned description then I will try to reproduce it on a toy problem and put the code here.</p>

username: Tony-Y
cooked: <p>If you are interested in LR warmup schedulers, try <a href="https://github.com/Tony-Y/pytorch_warmup" rel="nofollow noopener">pytorch_warmup</a>.</p>
<p><a href="https://colab.research.google.com/drive/1gWGM6jQLmPZABaYwW9O-A4TtFuoG4dh8" rel="nofollow noopener">Colab example</a> | <a href="https://pypi.org/project/pytorch-warmup/" rel="nofollow noopener">PyPI</a></p>

username: AralRalud
cooked: <p>I am having the same issue. Did you find the reason behind this?</p>
Get right accuracy

username: musgan
cooked: <p>Please help me,<br>
I use crossentrpyloss for loss function.  how is the right way to obtain the right accuracy for the loss function that I use?</p>

username: quanguet
cooked: <p>The documentation provides quite details here about <code>CrossEntropyLoss</code>. <a href="https://pytorch.org/docs/stable/nn.html#crossentropyloss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#crossentropyloss</a><br>
It’s better to read it carefully first and give it a try in your case to see how it goes. Keep in mind that the prediction from your model must the <code>logits</code> (don’t use <code>softmax</code> or <code>logsoftmax</code> at the end of your model).</p>
About CUDA VERSION

username: zhtjww
cooked: <pre><code>Found GPU0 GeForce RTX 2070 which requires CUDA_VERSION &gt;= 8000 for
 optimal performance and fast startup time, but your PyTorch was compiled
 with CUDA_VERSION 8000. Please install the correct PyTorch binary
 using instructions from http://pytorch.org
</code></pre>
<p>It seems that I’m using the correct version of pytorch, how this warning happen?</p>

username: quanguet
cooked: <p>Did you try to update your CUDA version?</p>
I want to know what does this code waht to do

username: hao_wang2
cooked: <p>def init_hidden(self, bsz, requires_grad=True):<br>
weight = next(self.parameters())<br>
if self.rnn_type == ‘LSTM’:<br>
return (weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad),<br>
weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad))<br>
else:<br>
return weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad)</p>

username: ptrblck
cooked: <p>This code uses the device and data type of the first parameter (here called <code>weight</code>) to initialize the hidden states as zeros (and the cell state for LSTM) and returns these tensors.</p>

username: dachosen1
cooked: <p>I’m not sure if this is the right place to ask but I’ve been studying LSTM, and I don’t understand how the hidden state can be hyper-parameter in an LSTM. Shouldn’t the hidden state parameter be fixed to the number of outputs x number of layers?</p>

username: hao_wang2
cooked: <p>Thank you very much for your answer, now I understand</p>

username: hao_wang2
cooked: <p>I don’t quite understand your problem, this is just to initialize the parameters of the hidden state, we also need to get the appropriate parameters through training</p>
Tensor.__getitem__ error when change method back

username: lambAmirstan
cooked: <p>I’m working on something similar like<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py" target="_blank" rel="nofollow noopener">NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py</a></h4>
<pre><code class="lang-py">import torch
import tensorrt as trt
from copy import copy
import numpy as np
from .calibration import TensorBatchDataset, DatasetCalibrator, DEFAULT_CALIBRATION_ALGORITHM


# UTILITY FUNCTIONS


def torch_dtype_to_trt(dtype):
    if dtype == torch.int8:
        return trt.int8
    elif dtype == torch.int32:
        return trt.int32
    elif dtype == torch.float16:
        return trt.float16
    elif dtype == torch.float32:
        return trt.float32
    else:
</code></pre>

  This file has been truncated. <a href="https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>replacing some pytorch method/function , do some tracing , then replace them back.</p>
<p>most other function/method is ok besides torch.Tensor.<strong>getitem</strong></p>
<p>here’s the example code</p>
<pre><code class="lang-auto">import torch

a=torch.randint(1,10,(15,))

b=torch.randint(1,10,(35,))

print(a[b]) # OK

print(b[a]) # OK

getitem = torch.Tensor.__getitem__

def new_getitem(self, key):

    print(key)

torch.Tensor.__getitem__ = new_getitem

torch.Tensor.__getitem__ = getitem

print(a[b]) # OK

print(b[a]) # IndexError: too many indices for tensor of dimension 1
</code></pre>
<p>it seems the behavior of <strong>getitem</strong> changed after replacing,  only accept index with larger size, why did this happened and how do I fix it?</p>
<p>thanks</p>
Torchtext Iterator omit the last batch

username: vas
cooked: <p>It seems like Torchtext Iterator doesn’t omit the last batch that usually doesn’t conforms to the batch_size parameter.</p>
<p>If there is a way to omit the last batch from the Iterator?</p>

username: ptrblck
cooked: <p>Are you feeding this <code>Iterator</code> to a <code>DataLoader</code>?<br>
If so, you could set <code>drop_last=True</code>.<br>
I’m not that deeply familiar with Torchtext so let me know, if that would work for you.</p>
NMS in mrcnn during reference stage

username: Michael-J98
cooked: <p>hi, guys~ I’m training my own dataset with mrcnn. But when I tested my model, I found some bboxes overlapped with each other like the picture below. I’m really confused about the principle here, shouldn’t the nms have suppressed these boxes after rpn before classification?  Thanks a lot for any help!<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17ac8b8e721dfe57ae19f369b9c6260bbb0bfb82.png" alt="image" data-base62-sha1="3nqENFyepO4d3I8g2HuGRQHU6YO" width="260" height="136"></p>

username: simaiden
cooked: <p>Those bboxes are of the same class?</p>

username: Michael-J98
cooked: <p>No, they are from different classes, and of course, some of them are bad predictions with relative low scores. <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:">  BTY, I’m not sure, are there two nms during reference stage, one for all proposals in rpn, and the other  only between the same class after classification or what?</p>
Dealing with imbalance dataset for multi-label task

username: Md_Saddam_Hossain
cooked: <p>Hi! I’m newbie<br>
In my case, I’ve 33 labels per samples. The target tensors are like [0,0,1,01,1,1,0,0,0,0,0…]. And the samples for some labels are quite low and some are high. So what will be the best approach to improve the prediction? I would like apply data balancing technique. But so far I found the balancing technique available only for multi-class. I’m grateful to you if you share your best knowledge about regarding my problem or any other idea to improve the performance. Thanks in Advance.</p>
SE Moudle pytorch

username: kungkookie
cooked: <p>I am a pytorch beginner, recently regenerated deeplab v3 + pytorch code, downloaded github jzZhang code in learning, but I want to use this code to train their own data set, and add the attention mechanism into deeplab, I have SE’s tensorflow code but do not know how to write pytorch code, ask for help</p>

username: ptrblck
cooked: <p>Are you stuck at a specific point and need some information how to translate this particular code snippet to PyTorch?</p>

username: kungkookie
cooked: <p>I want to add this SE module to the ASPP to see if it can improve the accuracy of semantic segmentation. The tensorflow code of SE module is as follows:<br>
def Global_Average_Pooling(x):<br>
return global_avg_pool(x, name=‘Global_avg_pooling’)</p>
<p>def Dropout(x, rate, training) :<br>
return tf.layers.dropout(inputs=x, rate=rate, training=training)</p>
<p>def Fully_connected(x, units):<br>
return tf.layers.dense(inputs=x, use_bias=False, units=units)</p>
<p>def Squeeze_excitation_layer(inputs, n_filters):<br>
n_filters=n_filters/16<br>
net = Global_Average_Pooling(inputs)<br>
net = Fully_connected(net, units=n_filters)<br>
net = tf.nn.relu(net)<br>
n_filters=n_filters*16<br>
net = Fully_connected(net, units=n_filters)<br>
net = tf.nn.sigmoid(net)<br>
net = inputs * net<br>
return net<br>
I want to know how to convert this module to pytorch, because I just learned pytorch, and it can be converted to some basic statements, but this is a little difficult for me, I hope to get help</p>

username: Eta_C
cooked: <p>Hey, is it what you want?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/moskomule/senet.pytorch/blob/master/senet/se_module.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/moskomule/senet.pytorch/blob/master/senet/se_module.py" target="_blank" rel="nofollow noopener">moskomule/senet.pytorch/blob/master/senet/se_module.py</a></h4>
<pre><code class="lang-py">from torch import nn


class SELayer(nn.Module):
    def __init__(self, channel, reduction=16):
        super(SELayer, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
            nn.Linear(channel, channel // reduction, bias=False),
            nn.ReLU(inplace=True),
            nn.Linear(channel // reduction, channel, bias=False),
            nn.Sigmoid()
        )

    def forward(self, x):
        b, c, _, _ = x.size()
        y = self.avg_pool(x).view(b, c)
        y = self.fc(y).view(b, c, 1, 1)
        return x * y.expand_as(x)
</code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: kungkookie
cooked: <p>Thank you, it was useful to me</p>
How to cast (not convert) data type of a tensor in C++?

username: ChenYuHo
cooked: <p>I’m implementing a custom Cuda operation that involves typecasting.<br>
What I want to achieve, if written in python, is similar to:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.ones(10).to('cuda')
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')
&gt;&gt;&gt; a.dtype
torch.float32
&gt;&gt;&gt; MyModule.myCustomOp(a, scaling_factor=100)
&gt;&gt;&gt; a
tensor([100, 100, 100, 100, 100, 100, 100, 100, 100, 100], device='cuda:0', dtype=torch.int32)
</code></pre>
<p>a simplified version of my c++ code looks like the following:</p>
<pre><code class="lang-auto">__global__ void my_kernel(float* input, float scaling_factor) {
  auto index = blockIdx.x * blockDim.x + threadIdx.x;
  auto integer = int32_t(round(input[index] * scaling_factor));
  // some other logics, not important here
  int32_t* int_arr = (int32_t*) input;
  int_arr[index] = integer;
}

void myCustomOp(torch::Tensor input, float scaling_factor) {
  my_kernel&lt;&lt;&lt;GRID_DIM, BLOCK_DIM&gt;&gt;&gt;(input.data_ptr&lt;float&gt;(), scaling_factor);
  CUDA_CHECK(cudaDeviceSynchronize());
  // how to cast the type of input to int32_t?
}
</code></pre>
<p>My implementation does not change <code>dtype</code> of the <code>input</code>, so I get a meaningless <code>torch.float32</code> tensor.<br>
An alternative way is to create a new tensor and return it, but I’d like to do this in-place if possible.</p>
<p>Any suggestions, please?</p>
How to use DataParallel with attention layers?

username: tlaurent
cooked: <p>I am looking for clarification on the best way to use DataParallel with attention layers. As an example, MultiheadAttention expects inputs which have shape (L,N,E) where L is the length of the sequence, N is the batchsize, and E is the embedding size. The fact that the batch size is NOT the first dimension leads to problem when using DataParallel. To work around this I am transposing the dimension, see example below:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class AttnParallel(nn.Module):
        
    def __init__(self, dim, num_heads):
        super(AttnParallel,self).__init__()
        self.attn = nn.MultiheadAttention(dim, num_heads, dropout=0, bias=False)
        
    def forward(self, h, mask):
        
        print("h has shape:", h.shape)
        print("mask has shape", mask.shape)
        
        h = h.transpose(0,1).contiguous()
        h = self.attn(h,h,h, key_padding_mask=mask)[0]
        h = h.transpose(0,1).contiguous()
        return h

# create model
dim =4
num_head=2
device = torch.device("cuda")
mod = AttnParallel(dim, num_head)
mod = nn.DataParallel(mod.to(device))

# create data
bsz = 16
L = 5
h = torch.rand(bsz,L,dim)
mask = torch.zeros(bsz,L).bool()
mask[0,1] = True
mask[2,4] = True

# forward
h = mod(h,mask)
</code></pre>
<p>I have a few questions:</p>
<ol>
<li>
<p>My understanding is that when using DataParralel, whatever tensors I feed to the forward() function will be chunked over the first dimension into 8 pieces and fed to 8 replica of my network (assuming 8 GPUs). So in this example, both the h and mask tensor will be chunked into 8 pieces. Eventually, the outputs of the 8 replica are concatenated over the first dimension. Am I understanding this correctly?</p>
</li>
<li>
<p>Is transposing the input the recommended way of dealing with module that expect input whose first dimension is not the batch dimension. Is it recommended to use contiguous() to improve performance, or is that unnecessary?</p>
</li>
<li>
<p>Should it be  nn.DataParallel(mod.to(device)) or nn.DataParallel(mod).to(device)? Both seem to work but the doc says: " The parallelized  <code>module</code>  must have its parameters and buffers on  <code>device_ids[0]</code>  before running this DataParallel module."  So I don’t understand how come nn.DataParallel(mod).to(device) work?</p>
</li>
</ol>
<p>Thanks!</p>

username: Yanli_Zhao
cooked: <ol>
<li>your understand is correct, it will chunk data based on dim=0 in default</li>
<li>I’m not sure transposing it recommended way, but contiguous() will cause copy</li>
<li>I think nn.DataParallel(mod.to(device)) is better</li>
</ol>
CUDA multiprocessing in jupyter notebook is possible?

username: HyoungsungKim
cooked: <p>Is it possible to run CUDA multiprocessing in jupyter notebook?</p>
<p>Currently I am implementing A3C in notebook.</p>
<p>For running multiprocessing with CUDA, <code>mp.set_start_method('spawn')</code> is required.(mp is torch.multiprocessing)</p>
<p>and <code>mp.set_start_method('spawn')</code> is required <code>__name__ == "__main__"</code>.</p>
<p>I have known <code>__name__ == "__main__"</code> is used in .py not in .ipynb</p>
<p>Is there any way instead of using  <code>__name__ == "__main__"</code>?</p>

username: George3d6
cooked: <p>Are you sure the <code>__name__ == "__main__"</code> is required and there’s no easy alternative to use here ? Maybe posting the actual could would help since there might be an easy way to modify it as to not require using <code>set_start_method</code> (E.g. by using DataParallel instead of mp)</p>
<p>That being said, there is a way to run code in a notebook as if it were ran under main:</p>
<pre><code class="lang-auto">exec(compile('your code here', '__main__.py', 'exec'))
</code></pre>
<p>Though I’m not sure if this will actually produce the expected behavior here and I don’t really have a multiple GPU machine handy right now to test.</p>
When I run model on multiple GPUs,register_hook is invalid

username: Asuka-EVA
cooked: <p>I want to save gradients of internal variables through register_hook() or retain_grad().<br>
When I run model on single GPU, it works.<br>
But when I run model on multiple GPUs through wrapping model into nn.DataParallel, I find that it doesn’t work.<br>
Can anyone help me?</p>

username: Yanli_Zhao
cooked: <p>based on comments “In each forward, :attr:<code>module</code> is <strong>replicated</strong> on each device, so any<br>
updates to the running module in <code>forward</code> will be lost. For example,<br>
if :attr:<code>module</code> has a counter attribute that is incremented in each<br>
<code>forward</code>, it will always stay at the initial value because the update<br>
is done on the replicas which are destroyed after <code>forward</code>. However,<br>
:class:<code>~torch.nn.DataParallel</code> guarantees that the replica on<br>
<code>device[0]</code> will have its parameters and buffers sharing storage with<br>
the base parallelized :attr:<code>module</code>. So <strong>in-place</strong> updates to the<br>
parameters or buffers on <code>device[0]</code> will be recorded.”</p>
<p>That means gradients of internal variables can not be updated in multiple GPUs, can only be updated in device[0]. if you want to sync buffers, you can try to use DistributedDataParallel package?</p>
Masking the Attention Weights

username: _joker
cooked: <p>I am trying to implement masking in the “attention-weight” so that I update only the weight of the image label. Like there are 10 classes in CIFAR10. If the image is of class “5”, then I would like to update the weight of only class “5” by making its corresponding weight to 1 and the rest of the weights as “0”.</p>
Text detection in pytorch

username: Nitesh
cooked: <p>Hey I want to build a text detection model. I wanted to know if there are any object detection(like yolo, etc) libraries in Pytorch or do I have to implement these algorithms from scratch. Or is there any better way to do that?</p>

username: simaiden
cooked: <p>Look at</p>
<p><a href="https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn</a></p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/facebookresearch/detectron2" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/16943930?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/facebookresearch/detectron2" target="_blank" rel="nofollow noopener">facebookresearch/detectron2</a></h3>

<p>Detectron2 is FAIR's next-generation platform for object detection and segmentation. - facebookresearch/detectron2</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Nitesh
cooked: <p>Hey thanks for the help, but what about YOLO?</p>
How to remember the order of the data coming from a dataloader?

username: PytorchBeginner
cooked: <p>The topic may be confusing. In each epoch, I need to call tqdm two times. Let me explain the situation in detail.</p>
<pre><code class="lang-auto">for data in tqdm(self.dataloader['train'],leave=False,total=len(self.dataloader['train'])):
    self.input=data[0].to(self.device)
    scale1_code_vectors=AFunction(self.input)
    all_scale1_code_vectors.append(scale1_code_vectors.data.cpu())

all_scale1_code_vectors=torch.cat(all_scale1_code_vectors,dim=0).to(self.device)
return all_scale1_code_vectors
</code></pre>
<pre><code class="lang-auto">AllTargetDistribution=SoftAssign(all_scale1_code_vectors)
</code></pre>
<p>As indicated above, I use self.dataloader[‘train’] to generate all_scale1_code_vectors. Then I need to generate code_vectors in a batch-wise manner like:</p>
<pre><code class="lang-auto">for data in tqdm(...)
    batch_code_vectors=AFunction(self.input)
    batchTargetDistribution=SoftAssign(batch_code)
    batchTargetCorresponding=AllTargetDistribution(batchsize*ind:batchsize*(ind+1))
    loss=criterion(batchTargetDistribution,batchTargetCorresponding)
</code></pre>
<p>That is, I need to guarantee the input generated in the second tqdm is same as the one of the first time. Currently, I just set shuffle=False in dataset setting, but this is not a good idea (may overfitting). So when shuffle is set as True, how to use dataloader[‘train’] two times in one epoch to guarantee the correspondance of iterative data?</p>

username: ayalaa2
cooked: <p>I would suggest the use of a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="nofollow noopener"><strong>Sampler</strong></a>. This is an optional argument when initializing a <strong>DataLoader</strong>.</p>
<p>In general, it’s expected to yield sample indices to the dataloader. A very straight forward example would be something like this:</p>
<pre><code class="lang-auto">class CustomSampler(Sampler):
    def __init__(self, data_size):
        self.samples = [i for i in range(data_size)]
    
    def __len__(self):
        return len(self.samples)
    
    def __iter__(self):
        for sample in self.samples:
            yield sample
</code></pre>
<p>Here it simply generates a list of all possible indices and yields them 1 at a time in order. What you could do is randomly shuffle the list initially and use the <em>same</em> sampler when you wish to preserve order. Here would be an example:</p>
<pre><code class="lang-auto">class CustomSampler(Sampler):
    def __init__(self, data_size):
        self.samples = [i for i in range(data_size)]
        random.shuffle(self.samples)
    
    def __len__(self):
        return len(self.samples)
    
    def __iter__(self):
        for sample in self.samples:
            yield sample
</code></pre>
<p>Notice that the shuffling is done upon initialization. So if you created two dataloaders using the same instance of <strong>CustomSampler</strong>, the order of samples will be the same. I would recommend reading the <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">doc</a> for more info</p>
Decoded output dimension not the same as the input in AutoEncoder

username: jitesh
cooked: <p>I am building a Custom Autoencoder to train on a dataset. My model is as follows</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder,self).__init__()

        self.encoder = nn.Sequential(
        nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=128,out_channels=256,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=256,out_channels=512,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=512,out_channels=1024,kernel_size=5,stride=2),
        nn.ReLU(inplace=True)
        )

        self.decoder = nn.Sequential(
        nn.ConvTranspose2d(in_channels=1024,out_channels=512,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=512,out_channels=256,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=256,out_channels=128,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=32,out_channels=3,kernel_size=3,stride=1),
        nn.ReLU(inplace=True)
        )

    
    def forward(self,x):
        x = self.encoder(x)
        print(x.shape)
        x = self.decoder(x)
        return x



def unit_test():
    num_minibatch = 16
    img = torch.randn(num_minibatch, 3, 512, 640).cuda(0)
    model = AutoEncoder().cuda()
    model = nn.DataParallel(model)
    output = model(img)
    print(output.shape)

if __name__ == '__main__':
    unit_test()
</code></pre>
<p>As you can see, my input dimension is (3, 512, 640) but my output after passing it through the decoder is (3, 507, 635). Am I missing something while adding the Conv2Dtranspose layers ?</p>
<p>Any help would be appreciated. Thanks</p>

username: ptrblck
cooked: <p>Based on your input shape and layer configs, you would need to set <code>output_padding=1</code> in the first and third <code>nn.ConvTranspose2d</code> layer to get the same output shape for your input.</p>

username: jitesh
cooked: <p>Thanks a lot, it worked.</p>
Linear Regression models

username: Aww
cooked: <p>Hello guys. I am trying to build a linear regression application that is taking 20 inputs and gives 4 outputs. I tested with different custom models and none seem to be able to fit the data. Can you guys recomend a model that is able to fit/overfit most linear regression problems (like boston pricing)? I can’t get other data because that’s all I have and I have around 60k samples (so my data is big enought).</p>
CNN shows no learning

username: 1337mathster
cooked: <pre><code class="lang-auto">import time as t
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
from random import randint
import numpy as np

data = np.load('cvd.npy',allow_pickle=True)




class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.mp = nn.MaxPool2d(2)
        self.fc = nn.Linear(1620, 2)
        

    def forward(self, x):
        in_size = x.size(0)
        x = F.relu(self.mp(self.conv1(x)))
        x = F.relu(self.mp(self.conv2(x)))
        x = x.view(in_size, -1)  # flatten the tensor
        x = self.fc(x)
        
        return F.log_softmax(x,dim = 1)

net = Net()

optimizer = optim.Adam(net.parameters(), lr=(1.0e-2))
counter = 0
losses = []
for epoch in range(1):
   err =  []
   for d in data:
      x,y = d 
      x = x/255 # normalizing the i/p image
      optimizer.zero_grad()
      X = torch.Tensor([[x]]) # a 1x50x50 tensor
      Y = torch.Tensor([[y]]) # a 1x1 tensor
      Y = Y.long()
      output = net(X)
      loss = F.nll_loss(output, Y[0])
      losses.append(float(loss))
      counter += 1
      err.append(loss)
      if counter%20 == 0: # backprop every 20 iters
         k = torch.stack(err).mean()
         counter = 0
         err = []
         k.backward()
         optimizer.step()
         print(loss)
</code></pre>
<p>I wrote this cnn to solve the cats v dogs dataset. The loss remains more or less constant for @ ~0.7.<br>
This is my first crack at CNNs. I’m just training in a batch size of one initially. You can find the numpy file here -&gt; <a href="https://drive.google.com/file/d/1WF_Bti7x2K93AmUrIje8RGVmrBco71_S/view?usp=sharing" rel="nofollow noopener">https://drive.google.com/file/d/1WF_Bti7x2K93AmUrIje8RGVmrBco71_S/view?usp=sharing</a></p>

username: Dipam_Vasani
cooked: <p>Hey, I have a question. I don’t quite understand what you are backpropagating every 20 iterations. Do you want a batch size of 20? Also, not related to the question but why not normalize the input before training itself? Try changing your model around a bit. Try changing the learning rate as well. It’s okay if your loss goes to infinity. At least, you’ll know something is happening.</p>

username: 1337mathster
cooked: <p>The 20 iterations thing was just to mimic using a batch of 20.<br>
I’ve messed around with the learning rates and still don’t seem to make any progress.<br>
So I’m still kind of confused on where the issue is.</p>

username: Dipam_Vasani
cooked: <p>Check the gradients and weights, are they updating? This is fairly common in deep learning. Your error won’t decrease at all. Try a different architecture, kernel size. For image classification you might need more layers, Try batch normalization to see if that helps.</p>
`requires_grad` or `no_grad` in prediction phase

username: chari8
cooked: <p>Based on the official tutorial, during prediction (after training and evaluation phase), we are supposed to do something like</p>
<pre><code class="lang-python">model.eval()
with torch.no_grad():
    # run prediction
</code></pre>
<p>But let’s assume that we want to use state_dict extracted from trained model.<br>
In this case, we usually do</p>
<pre><code class="lang-python">model.load_state_dict(torch.load('path/to/state_dict'))
model.to(device)
model.eval()
with torch.no_grad():
  # run prediction
</code></pre>
<p>My question is, if there is no need for further training, is it possible to set <code>requires_grad=False</code> to all layers in model and run prediction without <code>torch.no_grad()</code>, or will there be any difference?</p>
<pre><code class="lang-python">model.load_state_dict(torch.load('path/to/state_dict'))
for param in mode.parameters():
    param.requires_grad = False
model.to(device)
model.eval()
# run prediction
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>There will be a small difference in most cases I think.<br>
If you set every <code>requires_grad</code> to <code>False</code>. After each op, it will check whether the output requires gradients.<br>
If you set the <code>no_grad</code> mode, then it won’t even try.</p>

username: chari8
cooked: <p>Hi, albanD!<br>
Thanks for the reply.<br>
I will use no_grad mode. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Rishik_Mourya
cooked: <p>If you have a prediction function as-</p>
<pre><code class="lang-python">def predict(features):
    with torch.no_grad():
        return model(features)
</code></pre>
<p>Then it will always call this context-manager method, which in turn might affect inference performance.</p>
<p>So after training your model, I think it makes more sense to use “<code>torch.set_grad_enabled(False)</code>”, function since it disables any gradient computation globally, so then you can simple create your prediction function as-</p>
<pre><code class="lang-python">def predict(features):
    return model(features)
</code></pre>
<p>Am I right <a class="mention" href="/u/alband">@albanD</a>?</p>

username: albanD
cooked: <p>If you only ever want to do inference you can yes.<br>
The context manager is nice because it restores the previous state and so is better when you use it to test in your training loop for example.</p>
Gradient of loss with respect to parameters

username: Egon_Schl
cooked: <p>Hey everyone,</p>
<p>for my high school project I want to give a talk about neural networks, and connecting it to our calculus class. I now want to implement a small example in python and found pytorch for it.</p>
<p>Now I want to compute the gradient of the loss function wrt to the parameters of my model f(w).</p>
<p>The gradient of the loss function f(x,y) wrt to parameters w, where x is the input and y is the target, now has only the dimension of x, and not y, can anybody help me with that and maybe show me a way to fix it?<br>
Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am a bit confused by “f(x,y) wrt to parameters w” Does that mean that <code>f</code> also takes “w” as input but it is not explicit (like a model created with an nn.Module in pytorch)?</p>
<p>Also I don’t get this “now has only the dimension of x, and not y”, could you clarify please?</p>

username: Egon_Schl
cooked: <p>Hey,</p>
<p>so let’s assume I just want linear regression,</p>
<p>input: x, output: y<br>
y = f(x,W) = x*W</p>
<p>loss(f(x,w), y)</p>
<p>now  dloss(x,y) / dw has dimension |x| and not |x|+1 what I thought it should be</p>
<p>maybe I’m just confusing something</p>

username: albanD
cooked: <p>Ok.<br>
So you don’t actually want the gradient for <code>x</code>, only <code>W</code> right?</p>
<p>I think the <code>+1</code> difference in the size comes from the bias term that is often added in linear regression that you are missing when you do x*W only.<br>
You can either have another parameter of size 1 <code>b</code> and compute <code>x*W + b</code>. And now you get gradient for both W and b (this is the recommended way to do with NN-based libraries like pytorch).</p>
<p>Another approach that is used in more classical ML, is to add one more element to <code>x</code> in your function and have W be of size |x|+1:<br>
<code>extended_x = torch.cat([x, torch.tensor([1.])], dim=0)</code> and then <code>y = extended_x * W</code>.</p>
<p>Hope this helps !</p>

username: Egon_Schl
cooked: <p>Hey, one more question:</p>
<p>so I’m now using torch.autograd.grad(output=criterion(y, x), model.parameters()) and somehow the dimension is still X-1 (so w/o bias in your example, I just assumed x0=1 and have the bias included inside the W, sorry)… any tip?</p>

username: albanD
cooked: <p>That would depend how and when you add this new feature.<br>
Can you share your code?<br>
If you use the model version, you most likely have 2 paremeters? One of size X-1 and one of size 1.</p>
How to truly fix a network with BN layers in training?

username: Zhaoyi-Yan
cooked: <p>Suppose I use a pretrained network as the backbone, then extra layers are add afterwards to form a new model.<br>
I only train the parameters of extra layers. Several epochs later, I investigate the model parameters of two trained models. All the parameters are the same in the fixed backbone, however, it outputs different backbone features during evaluation.  And the differences come out after the first BN layer, although the parameters are the same. I realize that <code>running_var</code> and <code>running_mean</code> changes during training, which is the main reason.</p>
<p>If I want to keep the running statistics of pretrianed backbone, and I DO NOT want it to be changed when training (Truly fix the model with BN layers), and when testing I also need the running statistics kept the same as the pretrianed ones, then how should I do ?</p>

username: SimonW
cooked: <p>then you should just set the backbone in eval mode.</p>
Training ImageNet- VGG16

username: abhyantrika
cooked: <p>I am trying to train(kind of finetuning) the VGG16 network.<br>
I am using a custom set of CNN filters and am trying to retrain the final dense classification layer only. I am using the default settings provided in<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/imagenet/main.py</a></h4>
<pre><code class="lang-py">import argparse
import os
import shutil
import time

import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.distributed as dist
import torch.optim
import torch.utils.data
import torch.utils.data.distributed
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torchvision.models as models

model_names = sorted(name for name in models.__dict__
    if name.islower() and not name.startswith("__")
    and callable(models.__dict__[name]))
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Where I have modified model creation from lines 73-88, to import the model with my CNN filters. I have also made sure that “requires_grad=True” parameter is enabled only for the dense layer.</p>
<p>This is my result after around 22 epochs,<br>
Epoch: [22][4510/5005] Time 1.438 (2.538) Data 0.001 (1.757) Loss 6.9070 (6.9070) Prec@1 0.000 (0.103) Prec@5 0.391 (0.520)</p>
<p>As you can see prec@1 is Zero!</p>
<p>I suspect there is something wrong in the data loading and labeling of the imagenet dataset.<br>
I downloaded and extracted the ILSVRC2012 file to obtain train,test and eval.</p>
<p>The folder names inside train/ directory corresponds to some kind of synset id (“n02084071”).<br>
I am assuming the pytorch loaders use this as the class label.</p>
<p>The eval/ folder does not come with the same set of folders, instead comes with a .txt file that maps the images to numerical classes (1-1000). So I wrote a script to create folders accordingly for each images using the numerical class labels, using the data/metadata.mat file (this file has all the info &amp; minute details)</p>
<p>Why is my precision zero, even after 22 epochs?<br>
Please do inform if my data processing methods are wrong or feel free to point out any other errors.<br>
<b> Please help !</b></p>

username: Giocobon
cooked: <p>Hi <a class="mention" href="/u/abhyantrika">@abhyantrika</a>. Have you been able to solve the issue? I have the same problem. My guess is that the pretrained model uses labels in different orders than the one you have in the train/val datasets.</p>
Checking my understanding of detach method

username: jwillette
cooked: <p>If I had a network which output two values <code>a</code> and <code>b</code></p>
<pre><code class="lang-auto">a, b = model(x)
loss = ((a - y) ** 2).mean() # part one
loss += (a.detach() * b).mean() # part two
</code></pre>
<p>In part one, the mean squared error would only have an effect on the parameters with respect to the output variable <code>a</code></p>
<p>In part two of the loss function, it would only affect the parameters with respect to the output variable <code>b</code></p>
<p>So in effect I can define a loss function which depends on an output variable, but where the network will only optimize for the attached variable. Is this correct, or did I miss something?</p>
Comparison of torch.optim

username: FelipeVW
cooked: <p>Hi,</p>
<p>The torch.optim has a wide an rich selection of both common and more exotique optimizers. I am wondering which optimizers to pick for an object segmentation project.</p>
<p>I was reading through this paper: S. Ruder - An overview of gradient descent optimization algorithms.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.2.9/images/icons/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://arxiv.org/abs/1609.04747" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://arxiv.org/abs/1609.04747" target="_blank" rel="nofollow noopener">An overview of gradient descent optimization algorithms</a></h3>

<p>Gradient descent optimization algorithms, while increasingly popular, are
often used as black-box optimizers, as practical explanations of their
strengths and weaknesses are hard to come by. This article aims to provide the
reader with intuitions...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
And I was thinking if something similar has been published with updated guidelines on more modern optimizers like AdamW, SparseAdam, etc.<br>
In other words, which optimizer should I take?</p>
<p>Thank you for your time</p>
Creating graph dataset for node classifications from csv file having user-user relationship

username: AsmaAdil
cooked: <p>I am new in this field. I am following PyTorch geometric tutorial for creating my own graph dataset but<br>
These tutorials are not enough for basic understanding. Can anyone suggest me something good for basic understanding. I want to perform node classification on user-user graph . and also need to build this graph first (very important) so it can be used by pytorch geometry lib for node classification<br>
<a href="https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8" rel="nofollow noopener">https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8</a><br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html" rel="nofollow noopener">https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html</a></p>
How to fill values of tensor by mask in C++ API?

username: Lobanov
cooked: <p>Hello! I’m trying to google an answer for a few hours but can’t find it. So asking</p>
<p>How to implement this code in C++:</p>
<pre><code class="lang-auto">a = torch.zeros((10, 4))
b = torch.ones((4, 4))
mask = torch.tensor([0, 1, 0, 0, 1, 0, 1, 1, 0, 0], dtype='bool')
a[mask] = b
</code></pre>
<p>I am trying like this but it returns error:</p>
<pre><code class="lang-auto">  at::Tensor a = at::zeros({10, 4});
  at::Tensor b = at::ones({4, 4});
  bool _mask[]{false, true, false, false, true, false, true, true, false, false};
  at::Tensor mask = at::from_blob(_mask, {10}, torch::kBool);
</code></pre>
<p>first:</p>
<pre><code class="lang-auto">a.masked_fill(mask, b);
</code></pre>
<p>second:</p>
<pre><code class="lang-auto">a.index_fill(0, mask.nonzero(), b);
</code></pre>
<p>Thanks for reply!</p>

username: yf225
cooked: <p>Does the following work?</p>
<pre><code class="lang-cpp">auto a = torch::zeros({10, 4});
auto b = torch::ones({4, 4});
auto mask = torch::tensor({0, 1, 0, 0, 1, 0, 1, 1, 0, 0}, torch::dtype(torch::kBool));
a.index_put_({mask}, b);
</code></pre>

username: Lobanov
cooked: <p>Yes, it works, thank you very much!</p>