akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom training

Bondurant: <p>Hi!<br>
I want to make my training process with simpe CNN (on MNIST): first iteration - forward on training set, backward on 80% of training set where loss value was highest on first, at the second iteration, a forward pass is made for those 80% from the previous backward pass, 80% of examples with the highest loss value are taken from them, these 80% of 80% are used for the backward pass; etc. At every fifth iteration, we start with the entire dataset.<br>
Any ideas how to implement this?<br>
Thank you in advance.</p>

ptrblck: <p>You could use an unreduced loss (via passing <code>reduction='none' to the criterion) and sort the loss values (and store the sort indices). Once they are sorted you could slice them, reduce the loss, call </code>backward` on it, and update the parameters.<br>
In the next step you could reuse the sort indices from the previous run and repeat the procedure using your schedule.</p>
When applying FloatFunctional sequentially. Error

Andrew_Yang: <p>I am doing int8 quantization and I need to exchanged the mul operation of pytorch.</p>
<pre><code class="lang-auto">answer = a_tensor * 0.2 * b_tensor
</code></pre>
<p>I tried to replace the multiplication operations like the below with FloatFunctional’s.</p>
<pre><code class="lang-auto">self.ff = nn.quantized.FloatFunctional()

d = self.ff.mul_scalar(a_tensor, 0.2)
answer = self.ff.mul(d, b_tensor)
</code></pre>
<p>But, when calls the torch.jit.trace()<br>
I got the exception below.</p>
<pre><code class="lang-auto">    answer = self.ff.mul(d, b_tensor)
  File "/root/.pyenv/versions/3.7.1/lib/python3.7/site-packages/torch/nn/quantized/modules/functional_modules.py", line 160, in mul
    r = ops.quantized.mul(x, y, scale=self.scale, zero_point=self.zero_point)
RuntimeError: Mul operands should have same data type.
</code></pre>
<p>I printed out the dtype.<br>
print("### d.dtype", a.dtype)<br>
print("### b_tensor.dtype", b_tensor.dtype)</p>
<p>I got the below.<br>
### d.dtype torch.quint8<br>
### b_tensor.dtype torch.float32</p>
<p>Any good solution for this situation?</p>

jerryzh168: <p>first, you will need to use one FloatFunctional instance for each invocation.</p>
<p>although the reason for the error is that <code>b_tensor</code> is not quantized, you’ll need to add a quantstub after b_tensor to quantize it before feeding it to floatfunctional.mul</p>
Nn.linear with multi-dimensional channel datas

peepeepoopoo: <p>I was trying to implement an MLP layer that takes a 3-dimensional data, but only process data on one axis only (so the other two dimensions are considered channels in this case). I’ve done a quick look around on the documentation page and couldn’t seem to find anything similar to what I want. Granted, I could just call nn.linear separately but that will greatly slow down the layer since they won’t be parallelized. If anyone can point toward something that might meet my need that would be awesome!</p>

ptrblck: <p>Could you post a small pseudo code, please?<br>
<code>nn.Linear</code> accepts additional dimensions as <code>[batch_size, *, in_features]</code>, so you might be able to directly use it.</p>

peepeepoopoo: <p>Thanks, that should work out.</p>

peepeepoopoo: <p>Actually ran into a bit of issue again, it seems like nn.linear doesn’t like higher than 2 dimension input:</p>
<p>Here is the pseudo code:</p>
<p>input_matrix[ X x Y x Z x Channels]</p>
<p>linear_layer = nn.linear( in_channel, out_channel)</p>
<p>output_matrix = linear_layer(input_matrix)</p>

ptrblck: <p>It seems to work for me in <code>1.7.0.dev20200830</code>:</p>
<pre><code class="lang-python">x, y, z, channels = 2, 3, 4, 5
input = torch.randn((x, y, z, channels))
lin = nn.Linear(channels, 6)
out = lin(input)
print(out.shape)
&gt; torch.Size([2, 3, 4, 6])
</code></pre>

peepeepoopoo: <p>Aha I screwed up the batch-normalization part instead,thanks for the reply thou! (Gotah be a little more careful going through the errors haha)</p>
Learn++ algorithm

ratul_hasan125: <p>it  possible to implement learn++ algorithm with  pytorch<br>
link <a href="https://www.researchgate.net/publication/3421419_Learn_An_incremental_learning_algorithm_for_supervised_neural_networks" rel="noopener nofollow ugc">https://www.researchgate.net/publication/3421419_Learn_An_incremental_learning_algorithm_for_supervised_neural_networks</a></p>

albanD: <p>Hi,</p>
<p>You might want to rewrite your question to make it clearer what you’re trying to accomplish and maybe link reference papers that present what you’re trying to do.</p>

ratul_hasan125: <p>it possible to implement learn++ algorithm with pytorch<br>
link <a href="https://www.researchgate.net/publication/3421419_Learn_An_incremental_learning_algorithm_for_supervised_neural_networks" rel="noopener nofollow ugc">https://www.researchgate.net/publication/3421419_Learn_An_incremental_learning_algorithm_for_supervised_neural_networks</a></p>

albanD: <p>I am not familiar with this algorithm but you can most likely build the neural network using pytorch yes.<br>
You will have to implement the custom training by hand I’m afraid though.</p>
Data augmentation in pytorch!

edshkim98: <p>Hello everyone,<br>
I am trying to develop a deep learning network and here I am planning to use Data augmentation.<br>
In this augmentation, it adds random noise e.g. np.random.randn().<br>
My question is,<br>
if I use above method in pytorch, then would the dataset be updated with new random noise for every<br>
epoch? Hence, there will be no same dataset repeatedly trained.<br>
If not, how does data augmentation work in pytorch? Does it also use my original dataset too for training?</p>
<p>Thank you</p>

Jeremy_Tavrisov: <p>Depends on how you implement it. If you you have a dataset and you put the augmentation step in <strong>get_item</strong> then that would happen every batch so it would be unique across epochs.</p>

KarthikR: <p>Hi,</p>
<p>There is a good amount of discussion in this <a href="https://discuss.pytorch.org/t/data-augmentation-in-pytorch/7925">thread</a></p>

edshkim98: <p>Thank you for all of your answers!<br>
<a class="mention" href="/u/jeremy_tavrisov">@Jeremy_Tavrisov</a>, so in that case, would my model ever use original dataset(e.g. before transformation)?<br>
I want to also ask you then wouldn’t that prevent accuracy or performance to increase, as no dataset is repeatedly trained throughout epochs?<br>
How can I transform my dataset so that it does not augment my dataset for every epoch differently?</p>

Jeremy_Tavrisov: <p>Again this is dependent on your implementation. I think the on-the-fly augmentation generally works better. No dataset is repeatedly trained means less for it to overfit on. Augmentation is generally small perturbations of the original set.</p>
<p>If you wanted to you could augment ahead of time before training so that you have 1 set of transformed samples or design your augmentations such that they are not random across epochs.</p>

edshkim98: <p>Thanks for the help, I will try!!</p>
The purpose of unbind

willprice: <p>What is the purpose of <code>torch.unbind</code>? Can I not just iterate over the tensor itself indexing the the dimension I want to slice? I don’t really understand why it exists.<br>
Thanks!</p>

albanD: <p>Hi,</p>
<p>Yes unbind is similar to creating a list of slices of the Tensor.<br>
Unbind is going to be slightly faster than creating this list and doing the slicing by hand.</p>
Difference between setting tensor to device and setting dtype to cuda.FloatTensor

enterthevoidf22: <p>hi,</p>
<p>so i saw some posts about difference between setting torch.cuda.FloatTensor and settint tensor.to(device=‘cuda’)</p>
<p>i’m still a bit confused. are they completely interchangeable commands?</p>
<p>is there a difference between performing a computation on gpu and moving a tensor to gpu memory?<br>
i mean, is there a case where i want to utilize the gpu for speed but still memorize the tensor on cpu?</p>
<p>maybe someone can point me in the right direction</p>

ptrblck: <p>Changing the type to <code>torch.cuda.FloatTensor</code> would not only push the tensor to the default GPU but would also potentially transform the data type.<br>
The <code>to('cuda:id')</code> operation would only transform the tensor to the specified device as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([1])
print(x.type())
&gt; torch.LongTensor

y = x.type(torch.cuda.FloatTensor)
print(y, y.type())
&gt; tensor([1.], device='cuda:0') torch.cuda.FloatTensor

z = x.to('cuda')
print(z, z.type())
&gt; tensor([1], device='cuda:0') torch.cuda.LongTensor
</code></pre>
<aside class="quote no-group" data-username="enterthevoidf22" data-post="1" data-topic="98658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enterthevoidf22/48/29338_2.png" class="avatar"> enterthevoidf22:</div>
<blockquote>
<p>is there a case where i want to utilize the gpu for speed but still memorize the tensor on cpu?</p>
</blockquote>
</aside>
<p>That’s not possible, since the GPU operates on the data stored on the GPU.<br>
(Note that unified memory would still copy the data to the device if needed.)</p>
Understanding PyTorch native mixed precision

JoakimHaurum: <p>HI all,</p>
<p>I am currently working with PyTorchs native mixed precision, and I am uncertain on how some elements are handled udner the hood. I tried reading the scripts in torch.cuda.amp, but they did not answer my questions.</p>
<p>How does PyTorch handle casting of specifc modules, such as Batch Norm, under the hood? I know that NVIDIA’s APEX library has different levels of mixed precision (see: <a href="https://nvidia.github.io/apex/amp.html" rel="noopener nofollow ugc">https://nvidia.github.io/apex/amp.html</a>) where Batch Norm can be explictely stated to be float32.</p>
<p>Is this also the case for PyTorch native amp, or does it simply convert everything to Flaot16. If so, how would I go about converting my batch norm layers to Float32? This also pertains to how to properly handle custom normalization layers such as Inplace Activated BatchNorm (<a href="https://github.com/mapillary/inplace_abn" rel="noopener nofollow ugc">https://github.com/mapillary/inplace_abn</a>) which should be in Float32 for the best performance.</p>
<p>Hope you cna help me figure this out.</p>
<p>Best regards,<br>
Joakim</p>

seungjun: <p>PyTorch has <a href="https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float16" rel="noopener nofollow ugc">a list of operations</a> that can autocast to fp16. The operations not listed here will remain in fp32. Batch normalization will stay in fp32 when you use <code>amp.autocast()</code>.</p>
<p>PyTorch native amp is similar to apex level O1. A more detailed explanation of <a class="mention" href="/u/mcarilli">@mcarilli</a> can be found <a href="https://github.com/NVIDIA/apex/issues/818#issuecomment-629357106" rel="noopener nofollow ugc">here</a>.</p>

JoakimHaurum: <p>Hi <a class="mention" href="/u/seungjun">@seungjun</a>, thank you very much for your response, it answer my question fully <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">.</p>
Data Augmentation & Visualization example with Detectron2

saimanoj18: <p>Hi,</p>
<p>I am able to get the Detectron2 work on custom dataset for instance segmentation, exactly following the Google Colab tutorial, by registering the custom dataset.</p>
<p>Using the Detectron2 framework -</p>
<ol>
<li>I would like to perform data augmentation on both images and annotations for MaskRCNN application. I would like the way of randomly selecting a transform from a list of transforms that PyTorch offers for data augmentation</li>
<li>I also want to visualize some sample augmented images+annotations (as in Google Colab tutorial)</li>
</ol>
<p>I read the detectron2 docs but I was not able to understand them.</p>
<p>Can anyone point me to some learning resources or reference code, so that I can understand and adapt it ?</p>
<p>Any help will be greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The <a href="https://github.com/facebookresearch/detectron2/blob/master/docs/tutorials/augmentation.md">Data Augmentation docs</a> give you an example how to use these transformations. Would that help? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to visualise the impact of input variables on the output

ykukkim: <p>Hi all,</p>
<p>I developed an LSTM model, and I am trying to see how my inputs affect the results. I have tried to use saliency, as it is used for CNN.</p>
<p>Which I am now quite confused, the values of saliency is based on the gradient back-propagation of the output of the model, for instance, in CNN saliency can be used as heat-map to represent which part of the image is prominent. However, this is not what I want.</p>
<p>Or have I misunderstood saliency?</p>
Initial input to the decoder in a Encoder-Decoder RNN

Kong: <p>I am working on future frame prediction using an encoder-decoder RNN. I read online that the input of the decoder is supposed to be the output at the previous timestep. What about at time t=0? In the figure below, should the input to the decoder at the 1st timestep be the frame x2 or the output of the encoder or a 0 vector?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/675aa1b94ac18b82e28844680261536664c6cdd2.png" data-download-href="https://discuss.pytorch.org/uploads/default/675aa1b94ac18b82e28844680261536664c6cdd2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/675aa1b94ac18b82e28844680261536664c6cdd2.png" alt="image" data-base62-sha1="eKjm8uZuQqpPBhDii5uoHFnLErg" width="690" height="254" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1124×415 8.57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Using custom dataset training on Pytorch and export to onnx

ZedChang: <p>Hi, I’m a new hand to PyTorch and onnx, here is my problem:<br>
I used custom dataset (csv file) to train a model on PyTorch and export to onnx model, which were used on visual studio. Now I can train and get good testing accuracy on Pytorch, but when I exported it to onnx model and used it on visual studio to test same dataset, it got low accuracy. Please help me to figure out my problem, thanks a lot!</p>
<p>Here is my code:</p>
Using Linear Layers to adjust output size

dllacer: <p>Hi everyone,</p>
<p>I have been working on an Autoencoder and I have a problem with the MaxPooling outputs: if the input size is odd (put <code>20x11</code>), the output will be <code>10x5</code>. So, when I want to recover the original size performing the deconvolution I get <code>20x10</code>, but I need <code>20x11</code>.</p>
<p>Mi idea is to put a Linear layer at the output of the Decoder to adjust the size, but I don’t know if it can negatively affect the learning of the model.</p>
<p>My question is: Is there any way to achieve the desired output size without adding a Linear layer? Like setting some parameter of the Conv2d, MaxPool or ConvTranspose2d layers or something like that</p>
<p>Thank you very much!</p>

KarthikR: <p>I believe there is a way to give the desired output size, to the ConvTranspose2d function. However, you need to play with the Kernel and Stride parameters, to make sure your input size to the function can be shaped to the output size you want. Also, you may want to look into how this helps your downstream task.</p>
<p>input = torch.randn(1, 16, 11, 11)<br>
upsample = nn.ConvTranspose2d(16, 16, <strong>kernel_size=4, stride=2</strong>, padding=1)</p>
<p>mPool = nn.MaxPool2d((2, 2), stride=(2, 2))<br>
print('Input Shape: ', input.shape)<br>
input1 = mPool(input)<br>
print('Shape after Max Pool: ', input1.shape)<br>
output = upsample(input1)<br>
print('Shape after upsample: ', output.size())<br>
output = upsample(input1, output_size=input.size())<br>
print('Shape after upsample, giving input size: ', output.size())</p>

dllacer: <p>Thank you very much, this is very useful!</p>
Memory leak in libtorch even after using torch::NoGradGuard no_grad;

Kumar_Govindam: <p>There is a memory leak in libtorch and its occupies lot of space. Here is my code,</p>
<pre><code class="lang-auto">#include    &lt;unistd.h&gt;
#include    &lt;iostream&gt;

#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/utils.h&gt;

#include    "fReader.h"
//#include &lt;cuda_profiler_api.h&gt;

int main (int argc, char *argv[])
{
   // each of the fReader, fReader1 and  fReader2 occupies 700MB. 
    void    *fReader = NULL;
    void    *fReader1 = NULL;
    void    *fReader2 = NULL;
        
    {
        //torch::NoGradGuard no_grad;
        torch::NoGradGuard no_grad_guard;

        fReader = videoReader::Init ( url1);
        fReader1 = videoReader::Init ( url2 );
        fReader2 = videoReader::Init ( url3) ;
    }
      //// decoding only for one fReader for example
        pOutData        = 0;
        nResult = videoReader::GetVideoFrame (pReader, &amp;pOutData);
        auto options = torch::TensorOptions().device(torch::kCUDA, 0);    
        auto input_tensor = torch::from_blob(pOutData, {1, 640, 360, 3}, options);

    } while (1);
  
    return 0;
}

</code></pre>
<p>Here, I am trying to reach camera feed through fReader, which is initializing the camera feed. Its similar to OpenCV, except that it gives output in cuda memory. When I initialise the camera feed without the libtorch library, each of the camera feed takes 200 MB, but initialising the camera feed with libtorch, it takes around 700MB for each initialisation fReader1, fReder2 and fReader3.<br>
So,<br>
GPU memory consumption when initialising the camera feed without libtorch : 600 (3 x 200) MB<br>
GPU memory consumption when initialising the camera feed with libtorch : 2100 (3 x 700) MB</p>
<p>Note that the camera feed gives out as cuda array, which I am converting to torch using  torch::from_blob().</p>
<p>Even when I am using  torch::NoGradGuard no_grad_guard; there is no change in memory consumption. I am no idea how there is extra memory consumption for each initialisation as every library loaded only once as shared library.</p>
Sampling positive values

cndcnd: <p>I have a tensor with positive and negative values, I would like to have a new tensor of the same size but only with positive values, whose distribution is the same as that of the positive values of the original tensor. In other words, the new tensor should be constructed by sampling only the positive values from the original tensor.</p>
<p>Something like that, but more “pythonic”:</p>
<pre><code>     oldact = copy.deepcopy(act)
     for no in range(z.lrsize[lr]):
        for si in range(BATCHSIZE):
           getout = False
           while getout == False:
              xi = random.randrange(BATCHSIZE)
              if oldact[no][xi] &gt; 0:
                 act   [no][si] = oldact[no][xi]
                 getout = True</code></pre>
Data Augmentation Tools for image classification

alexcruz0202: <p>Hello, everyone.<br>
I’d like to know tools or packages except default torch augmentation tools, such as albumentations, …</p>

a_d: <p>you could augment the data using the transforms pipeline available in Torchvision. You create a transorm(a set of operations an image would go through in the pipeline ) as follows -</p>
<pre><code class="lang-auto">transforms = transforms.Compose([
                              transforms.Resize((350, 350)),
                              transforms.RandomHorizontalFlip(),
                              transforms.RandomRotation(degrees=20),
                              transforms.RandomAffine(degrees=20),
                              transforms.ColorJitter(hue=0.5, saturation=.01),
                              # Convert the image to tensor,
                              transforms.ToTensor()
                                ])
</code></pre>
<p>to get a Dataloader containing both modified and the original images (hence augmenting the dataset)  do the following -</p>
<pre><code class="lang-auto">def getConcatedDataset(root_dir):

    image_transformations = transforms.Compose([
        transforms.Resize((600, 600)),
        transforms.RandomHorizontalFlip(),
        transforms.RandomRotation(degrees=20),
        transforms.RandomAffine(degrees=20),
        transforms.ColorJitter(hue=0.5, saturation=.01),
        # Convert the image to tensor,
        transforms.ToTensor(),
        #transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

    minimum_transformations = transforms.Compose([
        transforms.ToTensor(),
        #transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

    dataset1 = ImageFolder(root_dir, transform=minimum_transformations)
    dataset2 = ImageFolder(root_dir, image_transformations)
    return ConcatDataset([dataset1, dataset2])
</code></pre>
<p>This way, you can create multiple transformation pipelines and append them into one big Dataloader.</p>

alexcruz0202: <p>thanks for your help.<br>
Do you know external tools like Imgaug, albumentations?</p>

a_d: <p>Nope, Do not know any of them…</p>
LU Kernel Pivoting Definition

kzheng9: <p>Hi,</p>
<p>Please let me know if this is not the right place to post this! I am having some trouble understanding the documentation for <code>torch.lu</code> here:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.lu.html#torch.lu" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.lu.html#torch.lu" target="_blank" rel="noopener nofollow ugc">torch.lu — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Specifically, how is the returned <code>pivots</code> tensor defined? I thought that it would be the index of the pivot for each row during partial pivoting, but this is obviously not the case since the example given in the docs has [3,3,3] for the pivots. I could not find any additional documentation on this, so if someone has any ideas, please let me know!</p>
<p>Thanks!</p>
DataLoader transpose axis for text?

Mehdi: <p>Hello,</p>
<p>I’d like to use the DataLoader and Dataset classes to train a NLP model. These classes are pretty useful, but when the dataloader returns a batch, it concats them with batch dim first. Is there a specific way to have the time axis first or should I just transpose the axis ?</p>
<p>Thanks  (:</p>

Panwen_Wang: <p>It’s transposed because:</p>
<blockquote>
<p>the  <code>DataLoader</code>  loads the data in a systematic way such that we stack data vertically instead of horizontally.This is particularly useful for flowing batches of tensors as tensors stack vertically (i.e. in the first dimension) to form batches.<br>
(<a href="https://towardsdatascience.com/building-efficient-custom-datasets-in-pytorch-2563b946fd9f" class="inline-onebox" rel="noopener nofollow ugc">Building Efficient Custom Datasets in PyTorch | by Syafiq Kamarul Azman | Towards Data Science</a>)</p>
</blockquote>
<p>And this is caused by the <code>default_collate</code> function. You can use the <code>default_convert</code> function as <code>collate_fn</code> to keep the axis:</p>
<pre><code class="lang-auto">from torch.utils.data._utils.collate import default_convert
from torch.utils.data import DataLoader

data = DataLoader(&lt;dataset&gt;, collate_fn=default_convert)
</code></pre>
Maximize the loss of non-target classes

_joker: <p>I am trying to design an ensemble loss function for an autoencoder to minimize the loss of the target class as well as maximize the loss of the non-target class.</p>
<p>My current loss function works fine with just minimizing loss <code>target_loss</code> but starts giving <code>nan</code> values when I add the <code>non_target_loss</code> term to the overall loss.</p>
<pre><code class="lang-auto">target_loss = MSE(input, output) + sparsity_term
non_target_loss = MSE(input, output_hat) + sparsity_term_hat

# I subtract because the second term has to be maximized.
loss = target_loss - (Coeff)*non_target_loss.
</code></pre>
<p>I am unsure about my design, any suggestions for a better design?</p>

KFrank: <p>Hi Inferno!</p>
<aside class="quote no-group quote-modified" data-username="_joker" data-post="1" data-topic="98602" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p>I am trying to … maximize the loss of the non-target class.<br>
…<br>
starts giving <code>nan</code> values when I add the <code>non_target_loss</code> term</p>
<pre><code class="lang-auto">target_loss = MSE(input, output) + sparsity_term
non_target_loss = MSE(input, output_hat) + sparsity_term_hat

# I subtract because the second term has to be maximized.
loss = target_loss - (Coeff)*non_target_loss.
</code></pre>
</blockquote>
</aside>
<p>This is to be expected.  <code>mse_loss()</code> is bounded below (by zero), so<br>
minimizing it won’t cause anything to diverge – the most you can do<br>
is drive it to zero.</p>
<p>However, because <code>mse_loss()</code> is unbounded above, your non-target<br>
term, basically <code>-mse_loss</code>, is unbounded below, so minimizing the<br>
non-target term can and will diverge, hence the <code>nan</code>s.</p>
<p>Assuming that your basic approach makes sense, you could consider<br>
running your <code>non_target_loss</code> through something like a <code>sigmoid()</code><br>
before subtracting it from your total <code>loss</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = target_loss - (Coeff) * torch.nn.functional.sigmoid (non_target_loss)
</code></pre>
<p>This total <code>loss</code> will still penalize your model for matching the non-target<br>
class.  <code>non_target_loss</code> can still become arbitrarily large, but, because<br>
it is “softly” clipped by the <code>sigmoid()</code>, it won’t ever cause your total<br>
<code>loss</code> to become arbitrarily negative, and your training (almost certainly)<br>
won’t diverge.</p>
<p>Best.</p>
<p>K. Frank</p>

_joker: <p><a class="mention" href="/u/kfrank">@KFrank</a> That’s a nice way out. It works now. Thanks a lot.</p>
How to load CUB-200-2011 in pytorch?

SAI_VARSHITTHA: <p>I am trying to do fine-grained image classification for CUB-200-2011 dataset, couldn’t figure out the correct way for loading the data. I’m using pre-trained ImageNet ResNet-101 for training by freezing all layers except last two layers. Suggest a way to for data loading in Pytorch.</p>

ptrblck: <p>It seems that <a href="https://github.com/TDeVries/cub2011_dataset">this repository</a> has implemented a custom <code>Dataset</code> for CUB-200-2011.</p>

SAI_VARSHITTHA: <p>That shows following problem,<br>
<a href="https://discuss.pytorch.org/t/file-not-found-or-corrupted-while-loading-cub-200-2011-dataset/98609">https://discuss.pytorch.org/t/file-not-found-or-corrupted-while-loading-cub-200-2011-dataset/98609</a>.<br>
How do I resolve my error?</p>
File not found or corrupted while loading CUB-200-2011 dataset

SAI_VARSHITTHA: <p>I am facing the following problem,<br>
<a href="https://stackoverflow.com/questions/64253628/file-not-found-or-corrupted-while-loading-cub-200-2011-data-set" rel="noopener nofollow ugc">https://stackoverflow.com/questions/64253628/file-not-found-or-corrupted-while-loading-cub-200-2011-data-set</a><br>
What is the error in my code and how to correct it ?</p>
Indexing 3d tensor along its last axis

Lucas_Zanini: <p>Hello,</p>
<p>Given a 3d tensor <code>x</code> and a 2d Long tensor  <code>idx</code>, I want to compute the <code>out</code> tensor defined as</p>
<p><code>out[i, j] = x[i, j, idx[i, j]]</code></p>
<p>Any idea how I could do that efficiently?</p>
<p>Thanks</p>

Lucas_Zanini: <p>Found the solution. For anyone interested:</p>
<pre><code class="lang-python">idx = idx.unsqueeze(2)
out = x.gather(2, idx)
out = out.squeeze(2)
</code></pre>
Metric for similarity between two point clouds

edshkim98: <p>Hello everyone,<br>
I am currently doing a project related to using point cloud in pytorch. My ultimate goal is that using the point cloud, I want to find an another point cloud that has <strong>similar features</strong> (not like pixel-wise). I saw some posts about using chamfer distance or cosine similarity. Could someone please recommend me some good ways of measuring similarities between two point clouds? Thank you!</p>
Updating weights of WeightedRandomSampler b/w epochs

wmpauli: <p>I would like to update WeightedRandomSampler.weights between epochs, e.g. for curriculum learning. A look at its class implementation suggests that I can just update the class field “weights” in b/w epochs, because the <code>__iter__</code> method simply draws from the multinomial distribution - with self.weight as input. Is that correct, or are their broader implications that I’m missing, e.g. something in the Sampler class?</p>

ptrblck: <p>It might work if you manipulate the <code>weights</code> attribute, however I also think you shouldn’t see any performance regression if you create a new <code>sampler</code> and a new <code>DataLoader</code> after the epoch, as it should be relatively cheap compared to the iterations.</p>

wmpauli: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, you’re right on both accounts. Updating the <code>weights</code> attribute worked, and I could have just created a new <code>DataLoader</code> and sampler.</p>
Given size of the image is larger than the input size of the LeNet5 architecture input. How can I reduce the size of the image so that the image can be sent into the architecture

chamanthi_aki: <p>hey, i got some error in my project, please help me .</p>
<p><a href="https://jovian.ml/chamschamanthi36/mini-pro-error/v/1&amp;cellId=9" rel="noopener nofollow ugc">here is the code snippet </a></p>
<p><a href="https://jovian.ml/chamschamanthi36/mini-pro-error" rel="noopener nofollow ugc">here is the entire notebook</a></p>

mathematics: <p>I think , you could use <code>transforms</code> and  <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/transforms.html</a> and add it to your all dataset.</p>

chamanthi_aki: <p>i already did that<br>
train_tfms =tt.Compose([tt.Compose([<br>
tt.Resize(256),<br>
tt.CenterCrop(224),<br>
tt.ToTensor(),<br>
tt.Normalize(mean=0.1817, std=0.1797)<br>
])])<br>
valid_tfms=tt.Compose([tt.Compose([<br>
tt.Resize(256),<br>
tt.CenterCrop(224),<br>
tt.ToTensor(),<br>
tt.Normalize(mean=0.1817, std=0.1797)<br>
])])</p>

mathematics: <p>May be, You shouldn’t  put <code>input</code> and <code>labels</code> to  <code>torch.autograd.Variable</code> because  autograd is handle by default and also <code>Variable</code> is depreciated now. inputs are already changed to <code>tensor</code> by transforms, so need not to worry that.<br>
you have this error <code>RuntimeError: Given groups=1, weight of size [6, 1, 5, 5], expected input[100, 3, 224, 224] to have 1 channels, but got 3 channels instead</code><br>
that means your input should take (3,224,224) but you may be passing 256</p>

chamanthi_aki: <p>hey, then again i got error as<br>
RuntimeError: mat1 dim 1 must match mat2 dim 0</p>

mathematics: <p>I think your problem is on <code>conv2 to linear</code>, seen on notebook you have used<br>
<code> self.fc1 = torch.nn.Linear(16*5*5, 120)</code> which may become incompatible when given <code>1x224x224</code> or <code>1x256x256</code> images.</p>
<p>For input of [1, 1, 224,224] ,  you should use <code>16*54*54</code> which worked.</p>
<p>In your forward method you could print what is output of last conv2-maxpool layer, and tried to initialize dimension with that</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = torch.nn.functional.relu(self.conv1(x))  
        x = self.max_pool_1(x) 
        x = torch.nn.functional.relu(self.conv2(x))
        x = self.max_pool_2(x)
        print(x.shape)
        x = x.view(-1, 16*54*54)
        # x = x.view(-1, x.shape[0])
        x = torch.nn.functional.relu(self.fc1(x))
        x = torch.nn.functional.relu(self.fc2(x))
        x = self.fc3(x)
</code></pre>
<p>you could always use dummy variable to test it  like below and try to print where is error coming from.</p>
<pre><code class="lang-auto">rand = torch.randn([1,1,224,224])
net.forward(rand)
</code></pre>
<p>Hope this works.</p>
Initialize parameters in a torch.autograd.Function

hanzCV: <p>Hi, I want to initialize my <code>class(torch.autograd.Function)</code>, with some variables, however, I could not see any example like that. When I try to initialize the object with <code> __init__()</code>, the <code>forward pass</code> cannot find those variables. Is it possible to do so?</p>
<p>Bellow is the only example I could find, where the class MyRelu doesn’t have any initial paramateres.</p>
<pre><code class="lang-auto">import torch

class MyReLU(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """
   #how can I initialize the class with some variables here?
   #I've tried this without success
   #  def __init__(ctx, alpha=0.5, beta=0.5, gamma=1.0):
   #    ctx.alpha = alpha
   #    ctx.beta = beta
   #    ctx.gama = gama
    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input &lt; 0] = 0
        return grad_input


</code></pre>

albanD: <p>Hi,</p>
<p>No this is not possible.<br>
We don’t really use <code>Function</code> as class but more as a convenient way to store the pair of functions for forward and backward.<br>
You can pass these arguments directly to the forward though. (and return None for their grad in the backward).</p>
TypeError: __init__() missing 1 required positional argument: 'vgg_name'

chamanthi_aki: <p>cfg = {<br>
‘VGG11’: [64, ‘M’, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG13’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG16’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, ‘M’, 512, 512, 512, ‘M’, 512, 512, 512, ‘M’],<br>
‘VGG19’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, 256, ‘M’, 512, 512, 512, 512, ‘M’, 512, 512, 512, 512, ‘M’],<br>
}</p>
<p>class VGG(nn.Module):<br>
def <strong>init</strong>(self, vgg_name):<br>
super(VGG, self).<strong>init</strong>()<br>
self.features = self._make_layers(cfg[vgg_name])<br>
self.classifier = nn.Linear(512, 10)<br>
self._initialize_weight()</p>
<pre><code>def forward(self, x):
    out = self.features(x)
    out = out.view(out.size(0), -1)
    out = self.classifier(out)
    return out

def _make_layers(self, cfg):
    layers = []
    in_channels = 3
    for x in cfg:
        if x == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                       nn.BatchNorm2d(x),
                       nn.ReLU(inplace=True)]
            in_channels = x
    layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
    return nn.Sequential(*layers)

def _initialize_weight(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            # xavier is used in VGG's paper
            nn.init.xavier_normal_(m.weight.data)
            if m.bias is not None:
                m.bias.data.zero_()
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()
        elif isinstance(m, nn.Linear):
            m.weight.data.normal_(0, 0.01)
            m.bias.data.zero_()
</code></pre>
<p>net = VGG()<br>
net.cuda()</p>
<p>this my code and I got error as:</p>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 net = VGG()<br>
2 net.cuda()</p>
<p>TypeError: <strong>init</strong>() missing 1 required positional argument: ‘vgg_name’</p>
<p>can anyone help me?</p>

KarthikR: <p>You need to pass value to the vgg_name variable used in the ‘init’ function.</p>
<p>Try like below:<br>
class VGG(nn.Module):<br>
def  <strong>init</strong> (self, cfg, <strong>vgg_name</strong>):<br>
super(VGG, self). <strong>init</strong> ()<br>
self.features = self._make_layers(cfg[vgg_name])<br>
…<br>
…</p>
<p>Then:<br>
net = VGG(cfg, ‘<strong>VGG1</strong>1’)</p>

chamanthi_aki: <p>wow, it worked,thanks karthik <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Pytorch LSTM and offsetting data for time-series models

krishnab: <p>Hello folks.<br>
I was looking at an implementation of the <a href="https://github.com/zhykoties/TimeSeries" rel="noopener nofollow ugc">DeepAR</a> model for time-series prediction. The model uses an LSTM and takes in 168 hours of data to predict the next 24 hours of data–in other words training on 7 days of data to predict the 8th day. They also stride the time series by 1 day or 24 hours, so each window is 192 (168 + 24) timesteps long, but incremented by a rolling window of 24 hours.</p>
<p>Now the one confusing thing I found was that in setting up the data array for training. The folks who created the repo setup the input array as a tensor with the first dimension as the number of time-series window, the second dimension as sequence length, and the third dimension as some additional covariates, so [number_of_windows, 192, number_of_covariates]. <strong>HOWEVER, and here is the tricky thing, the first value in every time series window is begins with a 0 and only runs for 191 steps. for example windows look like.</strong></p>
<pre><code class="lang-auto">window      | sequence                   | covariates    |
    1       | 0, 1, 2, 3, 4...191        | 1, 9, 1       |
    2       | 0, 5, 6, 8, 9, ...         | 1, 9, 2       |
    3       | 0, 10, 3, 8, 12, ...       | 1, 9, 3       |
</code></pre>
<p>I asked the developers in a github issue, and they said the following:</p>
<blockquote>
<p>At every time step, z_t is given and the goal is to predict z_{t+1}. In this way, if the window size is 192, we actually need 193 timesteps to cover both given and labels. Therefore, we let the first timestep in the window be 0. There are other setups too.</p>
</blockquote>
<p>So this makes sense, but I have not seen this setup mentioned anywhere before. So is this the right way to setup this kind of prediction model where z_t is predicting z_{t+1}, or is there a more consistent and common way to do this. Is there any documentation about this kind of setup. Any help is appreciated.</p>
Invalid shape error in pad_pack_sequence

saruftw: <p>I’m trying to implement a Pyramidal Bi-LSTM module. The forward function calls the following code</p>
<pre><code class="lang-auto"># input to pack_padded_sequence is torch.Size([618, 16, 512])
x = pack_padded_sequence(x, x_len, batch_first=False, enforce_sorted=False) 
out, _ = self.blstm(x) # out.data.size() = torch.Size([8632, 256])
outputs, output_lens = pad_packed_sequence(out, batch_first=False) # this errors out
</code></pre>
<p>The lstm executes fine but it throws the following error on padding back…</p>
<pre><code class="lang-auto">
&lt;ipython-input-10-0412449191e5&gt; in forward(self, x, x_len)
     17     out, _ = self.blstm(x)
     18     print("pblstm output: ", out.data.size())
---&gt; 19     outputs, output_lens = pad_packed_sequence(out, batch_first=False)
     20     # out = out.transpose(0, 1)
     21     # return out, hidden

/usr/local/lib/python3.6/dist-packages/torch/nn/utils/rnn.py in pad_packed_sequence(sequence, batch_first, padding_value, total_length)
    327         max_seq_length = total_length
    328     padded_output, lengths = _VF._pad_packed_sequence(
--&gt; 329         sequence.data, sequence.batch_sizes, batch_first, padding_value, max_seq_length)
    330     unsorted_indices = sequence.unsorted_indices
    331     if unsorted_indices is not None:

RuntimeError: shape '[156, 8, 256]' is invalid for input of size 114688

</code></pre>
<p>I don’t understand where the shape [156, 8, 256] comes from and what does 114688 represent. I’d appreciate any help thanks.</p>

Xenia-W: <p>Maybe you can check if your x and x_len is matching.<br>
I’ve met this error today, it’s because my x is [64,50,300] , but the x_len has an element of 51 (bigger than 50).</p>

PeterV: <p>Thanks a lot for your answer. It really helps me.</p>
Torch.det annoying warning

JuanFMontesinos: <pre><code class="lang-auto">   WARNING batched routines are designed for small sizes. It might be better to use the
   Native/Hybrid classical routines if you want good performance.
</code></pre>
<p>How to disable this warning produced by <code>torch.det</code>?<br>
Have seen some issues related to that in github but no practical solution</p>
Weird optimizer behaviour

Sudarshan_Babu: <p>why is this code not crashing</p>
<pre><code class="lang-auto">import torch
x=2.0
y =2.0
param  =  torch.nn.Parameter(torch.tensor(x))
opt = torch.optim.SGD([param],lr =.1)
loss = torch.nn.Parameter(torch.tensor(y))
loss.backward()
opt.step()
print('done')
</code></pre>
<p>Should not the optimizer complain saying param was not used in the computation ?<br>
Thanks in advance!</p>

albanD: <p>Hi,</p>
<p>Optimizers don’t do that in general no. They will ignore Tensors whose <code>.grad</code> field is <code>None</code> and use the gradient that is there if there is one.</p>
Cuda runtime error (801) multiple workers

Fibbi: <p>I have a simple setup where I want to employ multiple workers in a dataloader. A sample code looks like:</p>
<pre><code class="lang-python">import torch
import torch.utils.data as Data
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

X = torch.rand((10000,200)).to(device)
y = torch.rand((10000,1)).to(device)


dataset = Data.TensorDataset(X,y)

loader = Data.DataLoader(
    dataset=dataset, 
    batch_size=20,  
    shuffle=True, num_workers=2)


for i,(X_batch,y_batch) in enumerate(loader):
    continue
</code></pre>
<p>But I get error message:</p>
<pre><code class="lang-python">RuntimeError: cuda runtime error (801) : operation not supported at C:\w\1\s\tmp_conda_3.7_100118\conda\conda-bld\pytorch_1579082551706\work\torch/csrc/generic/StorageSharing.cpp:245
</code></pre>
<p>Can anyone help me understand this problem?<br>
(It only happens for <code>num_workers &gt;0</code>)</p>

ptrblck: <p>You are most likely recreating the CUDA context and would have to change the startmethod as described <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing">here</a>.</p>

granth_jain: <p>Hi,</p>
<p>I am getting the same error and I am using set_start_method(‘spawn’).</p>
<p>Could you please help a check what is wrong with this code.</p>
<aside class="quote quote-modified" data-post="1" data-topic="97254">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/granth_jain/48/27523_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-801-operation-not-supported-at-torch-csrc-generic-storagesharing-cpp-247/97254">RuntimeError: cuda runtime error (801) : operation not supported at ..\torch/csrc/generic/StorageSharing.cpp:247</a> 
  </div>
  <blockquote>
    Hi, 
I am trying to modify my A3C code from cpu device version to cuda version. 
However I am facing above error. 
I have already set the start method to ''spawn" 

from torch.multiprocessing import Process, set_start_method 
try: 
set_start_method(‘spawn’) 
except: 
pass 

and send the model to device 

device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”) 
model = ActorCritic(params).to(device) 

and running the workers by: 
            for rank in range(params.num_processes):
…
  </blockquote>
</aside>