akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Read file line by line

slavavs: <p>I used the excellent genfromtxt function to test.<br>
But on real data, my file is very large and a memory error occurs.<br>
How can I read a file line by line into an array, so that I can use the [;] section?</p>
<pre><code class="lang-auto">0;1;0;1;7;5;5;5.75;7;5;5;2.75;7;5;5;5.25;0;0;4;3;0;0;3;0;0;0;0;0;0;0;2;9

0;1;0;1;7;5;5;5.50;7;5;5;2.75;7;5;5;3.75;0;0;4;0;0;0;0;2;0;0;2;5;0;0;2;0
...
</code></pre>

ptrblck: <p>What do you mean by <code>[;]</code> section?<br>
If you want to read the file in chunks, you could probably use <a href="https://stackoverflow.com/a/519653" rel="nofollow noopener">this method</a>.<br>
Alternatively, you could use <code>pd.read_csv</code> with the <a href="http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html" rel="nofollow noopener">chunksize</a> argument.</p>
Batch Renormalization implementation in THCUNN

Nabarun_Goswami: <p>Hi,</p>
<p>I have been trying to implement the batch renormalization. Implementing in python by creating a custom Module results in much higher memory usage and time consumption. So I created a BatchReNorm function in cuda, following the same pattern as BatchNorm in THCUNN.</p>
<p>I am able to successfully build and get the updates and the clipping working perfectly, however, I am not sure regarding the stop_gradient mentioned in the batch renorm paper.</p>
<p>I create the parameters <code>r</code> and <code>d</code> in the <code>BatchReNormalizationUpdateOutput_kernel</code> as below:<br>
<code>Acctype r = 0;</code><br>
<code>Acctype d = 0;</code><br>
I then clamp the values using  <code>THCNumerics&lt;Acctype&gt;::lt</code> and  <code>THCNumerics&lt;Acctype&gt;::gt</code></p>
<p>I do not pass them to the <code>BatchReNormalizationBackward_kernel</code></p>
<p>Does that ensure that the gradients don’t flow through them?</p>
<p>Any insight would be really helpful.</p>
<p>Regards<br>
Nabarun</p>

dhpollack: <p>Did you ever solve this?</p>

Nabarun_Goswami: <p>Hi,</p>
<p>This approach works.</p>
<p>Regards<br>
Nabarun</p>

arc144: <p>Could you make your implementation public available? It would help a lot folks like me who don’t know how to mess directly with cuda. Thanks!</p>

catosphere: <p>Hello everyone<br>
I am also interested in trying to change BatchNorm1d with an implementation of Batch Renormalization … does anyone have implemented it please ? <a class="mention" href="/u/nabarun_goswami">@Nabarun_Goswami</a> would you share yours please ?</p>
Stochastic Pooling and Stochastic Dropout

shivangi: <p>Can someone give an example on how can we implement Stochastic Pooling and Stochastic Dropout in Pytorch.</p>
<p>References:<br>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1301.3557.pdf" target="_blank" rel="nofollow noopener">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1301.3557.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1301.3557.pdf" target="_blank" rel="nofollow noopener">1301.3557.pdf</a></h3>

<p class="filesize">1076.88 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<a href="http://mipal.snu.ac.kr/images/1/16/Dropout_ACCV2016.pdf" class="onebox" target="_blank" rel="nofollow noopener">http://mipal.snu.ac.kr/images/1/16/Dropout_ACCV2016.pdf</a></p>

ptrblck: <p>Have you started an implementation already and got stuck at some point?</p>

shivangi: <p>Not yet.<br>
I don’t have clue on how to approach.<br>
Can you suggest something?</p>
Grayscale images for resenet and deeplabv3

Nishanth_Sasankan: <p>How can I modify a resnet or VGG network to use grayscale images. I am loading the network the following way</p>
<pre><code class="lang-auto">m=torchvision.models.segmentation.fcn_resnet50(pretrained=False, progress=True, num_classes=2, aux_loss=None)
</code></pre>
<p>Is there some way I can tweak this model after loading it?</p>
<p>Thanks</p>

Juan_Sensio: <p>You can always define a custom resnet and change the first layer to adapt for your input shape. For example, with resnet18:</p>
<pre><code class="lang-auto">import torch 
import torchvision

resnet = torchvision.models.resnet18()

input = torch.randn((16,3,244,244))
output = resnet(input)
print(output.shape)

# this fails becasue resnet expects 3 channels

#input = torch.randn((16,1,244,244))
#output = resnet(input)
#print(output.shape)

import torch.nn as nn

class MyResNet(nn.Module):

    def __init__(self, in_channels=1):
        super(MyResNet, self).__init__()

        # bring resnet
        self.model = torchvision.models.resnet18()

        # original definition of the first layer on the renset class
        # self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
        
        # your case
        self.model.conv1 = nn.Conv2d(in_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)

    def forward(self, x):
        return self.model(x)

my_resnet = MyResNet()

# Now this works just fine !

input = torch.randn((16,1,244,244))
output = my_resnet(input)
print(output.shape)
</code></pre>
<p>Now, you could try the same for your model. If you want to dive in on how this models are built, I suggest you to go to <a href="https://github.com/pytorch/vision/tree/master/torchvision/models" rel="nofollow noopener">https://github.com/pytorch/vision/tree/master/torchvision/models</a>. You can learn a lot on how to build networks just by going over the different models.</p>
<p>I hope this helps !</p>
Optimizer with line search

yewang: <p>Is there any optimizer in pytorch that currently supports line search? I want to solve an optimization problem where being able to guarantee that the error always goes down is necessary. Last time I checked, LBFGS for example, in theory supports line search, but it’s not implemented yet in pytorch. Is there any optimizer that supports line search?</p>

palimboa: <p>I would also be very interested in this. Have you made any progress?</p>

hjmshi: <p>Making a shameless self plug, but if you’re still interested in this, I’ve <em>just</em> made public my implementation of L-BFGS which contains both a backtracking line search (with or without cubic interpolation) and weak Wolfe line search. Take a look here: <a href="https://github.com/hjmshi/PyTorch-LBFGS" rel="nofollow noopener">https://github.com/hjmshi/PyTorch-LBFGS</a>. Any feedback would be appreciated!</p>

quanhua92: <p>FYI, An implementation of L-BFGS with Strong Wolfe line search is merged in <a href="https://github.com/pytorch/pytorch/pull/8824" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/8824</a></p>
Loading ADE20K dataset

WeiQin_Chuah: <p>Hi,</p>
<p>I am trying to train PSPNet (Pyramid Scene Parsing Network) for semantic segmentation. I have created the model but I am having difficulties in loading the dataset as I only have experience loading images with folder names as labels for image classification.</p>
<p>Will appreciate some help here to guide me on how to load training images and ground truth labels in PyTorch.</p>
<p>Thank you</p>

Balamurali_M: <p>I am assuming there is a folder which contains sub-folders image and mask. The filenames are same. This is a template you can modify it and use. All the best.</p>
<pre><code class="lang-auto">import torch.utils.data as data

class DataLoaderSegmentation(data.Dataset):
    def __init__(self, folder_path):
        super(DataLoaderSegmentation, self).__init__()
        self.img_files = glob.glob(os.path.join(folder_path,'image','*.png')
        self.mask_files = []
        for img_path in img_files:
             self.mask_files.append(os.path.join(folder_path,'mask',os.path.basename(img_path)) 

    def __getitem__(self, index):
            img_path = self.img_files[index]
            mask_path = self.mask_files[index]
            data = use opencv or pil read image using img_path
            label =use opencv or pil read label  using mask_path
            return torch.from_numpy(data).float(), torch.from_numpy(label).float()

    def __len__(self):
        return len(self.img_files)

</code></pre>
<p>I have already this in other thread. But I don’t know how to share that url. See if it helps</p>
How to use nn.DataParallel to split a list of elements into several sublists

Jiaming_Shen: <p>In the <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">Data Parallelism Tutorial</a>, authors show how to train a model using multi-GPU when each training batch is a tensor. My question is how to extend it <strong>when each batch is a python list</strong>. To give a concrete example, I slightly modify the <em>RandomDataset</em> class and <em>Model</em> in the above tutorial as follows.</p>
<pre><code class="lang-auto"># Parameters and DataLoaders
input_size = 5
output_size = 2
batch_size = 30
data_size = 100

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return [self.data[index] for i in range(12)]  # This line is changed

    def __len__(self):
        return self.len

class Model(nn.Module):
    # Our model

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)

    def forward(self, input_list):
        output = [self.fc(ele) for ele in input_list]  # This line is changed
        print("\tIn Model: length of input_list", len(input_list))

        return output
</code></pre>
<p>Previously, one batch is simply a tensor of size [30, 5] and if we have 3 GPUs, it can be automatically splitted into 3 tensors of size [10, 5] and the “input” of model is a tensor of size [10,5]. This is good.</p>
<p>Now, one batch is a list of 12 tensors, each tensor is of size [30, 5]. I want the DataParallel to split this list into 3 sublists, Each sublist contains 4 tensors and each tensor is of size [30, 5]. It seems the default nn.DataParallel cannot do this and I am wondering how I can achieve this goal.</p>
<p>I put the full script/notebook here: <a href="https://gist.github.com/mickeystroller/d428af55e3d2afb6b79eb888b139ba31" rel="nofollow noopener">https://gist.github.com/mickeystroller/d428af55e3d2afb6b79eb888b139ba31</a></p>
<p>P.S.:Just to clarify, in this toy example, I know we can simply create a new dimension and stack this list of 12 tensors. However, for my real application, each training batch will be a list of complex objects instead of a list of tensors and thus I do need the DataParallel to split a list into several sublists.</p>
<p>Thanks for your help and time.</p>
A vectorized sliding window: clever tensor reshaping/expansion

MBle: <p>Hi all,</p>
<p>Let’s say I have a tensor, with dims NxHxWxC, for this example 1x4x4x1. I’m trying to do some reshaping of the tensor with a sliding window. This window has a kernel size of k=3, which slides over the W and H dims. I want something like this</p>
<p>t = torch.Tensor([[<br>
[1, 2, 3, 4],<br>
[5, 6, 7, 8],<br>
[9, 10, 11, 12],<br>
[13, 14, 15, 16],<br>
]])</p>
<p>=&gt;</p>
<p>[[<br>
[1, 2, 3, 5, 6, 7, 9, 10, 11], # i.e. the top-left corner with k=3<br>
[2, 3, 4, 6, 7, 8, 10, 11, 12], # one step to the left<br>
…,<br>
[6,7,8,10,11,12,14,15,16] # bottom-right corner with k=3<br>
]]</p>
<p>The output dim should be N x ? x k **2 x C. If k=3, for this example, it should be 1x6x9x1.</p>
<p>Is this somehow possible, in an efficient way?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>You can compute it by slicing, flattening and stacking in a simple way. The only drawback is it would be parallelized…</p>

LeviViana: <p>This way:</p>
<pre><code class="lang-python">import torch
from torch.nn import functional as f
x = torch.arange(1, 17).view(4, 4).float()
x = x[None, None, :, :]
f.unfold(x, kernel_size=2)
</code></pre>
<p>It prints:</p>
<pre><code class="lang-bash">tensor([[[ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
         [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
         [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
         [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.]]])
</code></pre>

MBle: <aside class="quote no-group" data-username="LeviViana" data-post="3" data-topic="48644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>x = torch.arange(1, 17).view(4, 4).float() x = x[None, None, :, :] f.unfold(x, kernel_size=2)</p>
</blockquote>
</aside>
<p>Thanks! This is was I needed!</p>
Image channels concatenation

aayush: <p>Can anyone tell me how can i concatenate three greyscale images(1 channels each) into a 3 channel image so that it can be fed into standard convnets?<br>
(i.e. the greyscale images are of dimesions[60,60,1] and i want them to be of [60,60,3])</p>

LeviViana: <p>If <code>img</code> is your image, then <code>torch.cat((img, img, img), 2)</code> will work. However, maybe this is not what you really want to do. If you could refit your model to receive greyscale images, it could improve the performance of your model.</p>

aayush: <p>I did as you said but i am getting the following error.</p>
<p>TypeError: expected Tensor as element 0 in argument 0, but got tuple</p>

LeviViana: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/46035ec5764ed5ece7ca3bf5b9829c554d5c69eb.png" alt="image" data-base62-sha1="9ZmDTOE28GV0Gjb4DV66jjLV75h" width="368" height="102"></p>
<p>You must be missing something. Could you send a snippet so that I can reproduce this error ?</p>
Why do we need backpropagation?

vainaijr: <p>Does an approach exist by which we train without back-propagation.<br>
How to remove back-propagation ?</p>

sai_tharun: <p>it doesnt make sense to train a model without backpropagation because the model has to adjust the weights and bias to make the model perform better and better.</p>
<p>As of my knowledge there is no other approach than doing backpropagation</p>

JuanFMontesinos: <p>There are some networks called capsule networks which uses routing by agreement but I still have to fully read those papers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

LeviViana: <p><a href="https://en.wikipedia.org/wiki/Neuroevolution" rel="nofollow noopener">Neuroevolution</a> has proven to be a good alternative in some cases.</p>
Why the output size of a pretrained model is always 1000?

marsggbo: <p>I have trained Resnet50 provided by torchvision.models on my own dataset. And because my dataset has 100 classes, I modified the <code>fc</code> layer, and everything goes fine. After finishing the training, I use <code>torch.save(best_model, path)</code> to save the best model.</p>
<p>Now I want to load the best model to predict on the test set, So I load the model structure as <code>model = torch.load(path)</code>, and print the model as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5af2b2495a9c0228e52895ad220e82ceac564151.png" data-download-href="https://discuss.pytorch.org/uploads/default/5af2b2495a9c0228e52895ad220e82ceac564151" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5af2b2495a9c0228e52895ad220e82ceac564151.png" alt="image" data-base62-sha1="cYyWIrV48qXbG28oyPIP4WL9xol" width="690" height="409" data-dominant-color="0F100F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">799×474 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can see that the out_features of fc layer is 100, but the size of output became 1000, is it weird? How to solve this problem?</p>

Oli: <p>Edit: Looking closer, I see that your model is correct. Are you sure that the model you are printing is the model m2 used for the input?</p>
<p>Hi there. Hmm, can you show us how you changed the model? Or print the model after you’ve changed it?</p>
<p>When I do <code>resnet.fc = nn.Linear(self.basenet.fc.in_features, n_out_features)</code> it works fine after saving/loading</p>
<p>My code -&gt;</p>
<pre><code class="lang-auto">n_features = 100
resnet = models.resnet18()
resnet.fc = nn.Linear(resnet.fc.in_features, n_features)
torch.save(resnet, 'model.pth')
model = torch.load('model.pth')

inp = torch.randn(1, 3, 224, 224)
out = model(inp)
print(out.size()) -&gt; torch.Size([1, 100])
</code></pre>

marsggbo: <p>Thanks for your reply. My code is as follows:</p>
<pre><code class="lang-python">
m2 = torchvision.models.resnet50(pretrained=True)
m2.fc.out_features = 100

# after training
...
torch.save(m2, 'best.pt')

</code></pre>
<p>Then I use ipython to load the model:</p>
<pre><code class="lang-auto">[1] import torch

[2] m2 = torch.load('best.pt')

[3] m2
the model architecture is shown as above figure.

[4] m2(torch.rand(1,3,224,224).cuda(0)).shape
</code></pre>
<p>but the result of [4] is <code>torch.Size([1, 1000])</code></p>

sai_tharun: <pre><code class="lang-auto">m2 = torchvision.models.resnet50(pretrained=True)
in_ftr  = m2.fc.in_features
out_ftr = 100
m2.fc = nn.Linear(in_ftr,out_ftr,bias=True)
</code></pre>
<p>Hope this helps!</p>
Getting wrong output size from nn.Unfold for input size ([1, 1, 1, 15])

CarlosMendoza: <p>Hello,</p>
<p>I am trying to use nn.Unfold to build a customized non-linear (max) convolution over 1-D tensors. This is my code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

xx = torch.Tensor([4,5,7,6,4,3,4,2,0,2,4,5,6,7,8])
xx_as4D = xx.view(1,1,1,-1)
unfold = nn.Unfold(kernel_size=(1,4), dilation=(1,1), padding=(1,3), stride=(1,1))
xx_unf = unfold(xx_as4D)

print(f'xx_unf.shape = {xx_unf.shape}')
print(f'xx_as4D.shape = {xx_as4D.shape}')
</code></pre>
<p>and the output</p>
<pre><code class="lang-auto">
xx_unf.shape = torch.Size([1, 4, 54])
xx.shape = torch.Size([15])
xx_as4D.shape = torch.Size([1, 1, 1, 15])
</code></pre>
<p>Following the <a href="https://pytorch.org/docs/stable/nn.html#unfold" rel="nofollow noopener">docs</a>, I was expecting that the output of <code>unfold()</code> would be of size <code>[1,4,18]</code>, but I am getting the size <code>[1,4,54]</code>. Could some please help me figure out where is my flaw?</p>
<p>In addition, if you have suggestions or references about how to accomplish my goal in an efficient way on Pytorch, those will be welcome! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks.</p>

CarlosMendoza: <p>I found the error! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> … the right padding is <code>padding=(0,3)</code>.</p>
Why Each Step of Recurrent Network Has Its Own Workspace? C++

fulltopic: <p>Hi,</p>
<p>I am reading caffe2 c++ codes: operators/rnn/recurrent_network_op.h.<br>
In RecurrentNetworkOp::DoRunWIthType, the Op created a collection of workspaces, one for each t. Why each time step needs unique workspace? Seemed the hidden_state and cell_state and other recurrent states (both forward pass and backward pass) have been assigned enough storage for states of all the time steps?</p>
<p>Thank you very much!</p>

fulltopic: <p>Maybe for parallel execution and storage of output of internal operator of each step.</p>
Is it necessary to use a lr_schedulre when I choose an adaptive optimizer

Komoriii_Sen: <p>Hello my friends. I am using adabound, an adaptive lr learning algorithm to train my CNN model. Is there any negative effect to use the lr_scheduler with an adaptive learning algorithm? I prefer not to use the lr_scheduler in this situation.<br>
My network is still being training. If any interesting results appear I shall show here.</p>

bit_comedian: <p>You have no need to use lr_schedulers with adaptive optimizers (e.g adabound), as these optimizers change learning rate themselves.</p>
Torch.fft zero-padding and shifting

Carsten_Ditzel: <p>Do I understand correctly, that I have to do both zero-padding as well as fftshift operations manually prior and post torch.fft invocation? I cannot find an appropriate arguments for passing on the call-site.</p>
<p>Also is by convention the first FFT always performed along a certain direction? Because I cant seem to specify the axis along which the operation is performed.</p>
<p>Help is appreciated</p>
How to avoid overflow computing the graient of pow?

notacode: <p>Hi everyone,</p>
<p>I am trying to implement a function that takes a 4-D <code>Tensor</code> as input and raises its absolute value to a power <code>alpha</code> preserving the sign. The trick is that <code>alpha</code> is a trainable parameter but when the gradient is computed w.r.t. <code>alpha</code>, I get <code>nan</code> which is probably due to overflow.</p>
<p>I have tried using <code>autograd.Function</code> and <code>nn.Module</code> but no luck in fixing the issue so far. Here is my code with <code>autograd.Function</code>:</p>
<pre><code class="lang-auto">a = torch.tensor(2.,  requires_grad=True)
class Power(torch.autograd.Function):
  @staticmethod
  def forward(ctx, x, alpha):
    result = x.sign() * torch.abs(x) ** alpha
    ctx.save_for_backward(result)
    return result
  

  @staticmethod
  def backward(ctx, grad_output):
    result, = ctx.saved_tensors
    return result*torch.log(result.abs()+1e-6), None 


power = Power.apply
x = torch.randn(N, D_in, device='cpu', dtype=torch.float)
out = torch.sum(power(x, a))
out.backward()
out.grad # none here
</code></pre>
<p>and using <code>nn.Module</code>:</p>
<pre><code class="lang-auto">class Power(nn.Module):
  def __init__(self, alpha=2.):
    super(Power, self).__init__()
    self.alpha = nn.Parameter(torch.tensor(alpha))
  
  def forward(self, x):
    return x.sign()*torch.abs(x)**self.alpha
</code></pre>
<p>Is there any way to properly fix the function to avoid overflow / <code>nan</code>'s? Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>In your custom Function, since you compute the gradient wrt alpha, the not-None result should be second one no?<br>
To ensure the problem is overflow, I would add some prints to find exactly which op does that in your custom Function. To detect nan in aTensor, you can do <code>t.ne(t).any()</code>.</p>

notacode: <p>Hi, thanks for your response! In general, it should the output order in <code>backward</code> follow the order of inputs? If in <code>forward</code> I had <code>forward(ctx, alpha, x)</code> then <code>backward</code> should return <code>grad_wrt_alpha, grad_x</code> in that order?</p>

albanD: <p>yes the order follows the order of the arguments of the forward.</p>
Input dimension

Kai_Qu1: <p>Can someone in this example the input has to be 4-dimensional?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/147a5a8dc85adfc969df318c839f62a677abf2be.png" data-download-href="https://discuss.pytorch.org/uploads/default/147a5a8dc85adfc969df318c839f62a677abf2be" title="28%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/147a5a8dc85adfc969df318c839f62a677abf2be_2_460x500.png" alt="28%20PM" data-base62-sha1="2V9GQqqjEw89sngKFjSq3fWDn0q" width="460" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/147a5a8dc85adfc969df318c839f62a677abf2be_2_460x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/147a5a8dc85adfc969df318c839f62a677abf2be_2_690x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/147a5a8dc85adfc969df318c839f62a677abf2be_2_920x1000.png 2x" data-dominant-color="333334"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">28%20PM</span><span class="informations">1088×1182 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is an error message saying that the input has to be 4-dimensional if my input is anything but 4-dimensional…</p>
<p>Thanks a lot!</p>

Kai_Qu1: <p>the blocked lines are:</p>
<pre><code>    x = self.fc3(x)
    return x
</code></pre>
<p>sorry about that…</p>

alorentemur: <p>The Conv2d class only takes 4 dimensional inputs that’s why.</p>
<p>The inputs should be formated this way :<br>
[batch_size, channels, height, width] if your want your forward to work.</p>
<p>For more information see the doc <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
<a href="https://pytorch.org/docs/stable/nn.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a></p>
Gradients with respect to input for RNN

OlhaRomaniuk: <p>I am trying to calculate gradients of output with respect to input of a network that contains recurrent layers.<br>
I create input variable with <code>requires_grad = True</code>, run forward pass and backpropagation, but the gradient with respect to input is None. The same approach works fine for convolutional networks, the input is a leaf variable, so I don’t think that <code>register_hook</code> is the way to go.<br>
Here is some minimal example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import numpy as np

sequence_length = 97
print(sequence_length)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.gru = nn.GRU(input_size=5,hidden_size=5, num_layers=1, batch_first=True)
        self.linear =nn.Linear(5,1)

    def forward(self, x, h0):
        h0 = h0.permute(1,0,2)
        out,h = self.gru(x,h0)
        x = out[:,-1,:]
        x = self.linear(x)
        x = torch.sigmoid(x)
        return x

model = Net()
model =model.cuda()

optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = torch.nn.BCELoss()

print("model")

data = Variable(torch.normal(torch.zeros(4, sequence_length, 5), torch.ones(4, sequence_length,5)),requires_grad= True)
data = data.cuda()

label = torch.from_numpy(np.array([1,0,1,0]).astype('float32'))
label = label.cuda()

print("data")

for epoch in range(1000):
    h0 = Variable(torch.randn(4,1,5).cuda())
    output = model(data, h0)
    print(output.shape)
    print(output)
    loss = criterion(output, label)
    print("loss ", float(loss))

    optimizer.zero_grad()
    loss.backward()
    print(data.grad)

    optimizer.step()
    event = 'Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, 1 * len(data),
                                                                     1,
                                                                     100. * 1 / 1,
                                                                     float(loss) / 1)
    print(event)

</code></pre>
<p>What is the best way to get the gradients with respect to input?</p>

abhayraw: <p>Hey!<br>
Following <a href="https://discuss.pytorch.org/t/tensor-to-device-changes-is-leaf-causing-cant-optimize-a-non-leaf-tensor/37659">this post</a>, you would need to detach your tensor on the gpu so that it becomes a leaf node in the compute graph. Gradients can only be computed wrt. a leaf node in the graph.</p>
<p><a href="https://gist.github.com/abhayraw1/b9b530d276a23f953d3fd22a59277496" rel="nofollow noopener">Here</a> is the gist of the modified code. Hope it helps</p>
Pretrained model on CityScapes?

M_S: <p>Hi,<br>
I want to run a segmentation network on CityScapes, but I don’t want to train a model, I want to use a trained model from a checkpoint (don’t care which model as long as the accuracy is good).<br>
I noticed CityScapes was recently added to torchvision datasets, and many new models were added as well.<br>
However there’s nothing on the docs yet… so is it possible to import a trained model on CityScapes in torchvision?</p>
Pytorch running on GPU, memory anomalies

pybald: <p>This issue is closed. It was just a bug when initializing the instances.</p>
<p>We created a model with 15 layers, FP32, each layer size 12K (so weights are 4x12Kx12K = 576MB). So, the model size itself is 576*15= 8.6GB and there are activations, input layers etc. The size of the model alone cannot fit into an 8GB GPU.</p>
<p>Yet Pytorch runs fine on the GPU using only about 4.7GB while training (checking nvidia smi or torch memory allocated). We also printed the model.state_dict() and it shows the layers as we defined them.</p>
<p>We also noticed that the CPU process is spiking on virtual memory going up to 50 GB. Any explanations? Thanks.</p>

albanD: <p>The very high virtual memory is expected and always there.</p>
<p>How big is the model when you save its state_dict to disk?</p>

pybald: <p>We should have tried the checkpoint save. Just printing the model state_dict info was misleading. It seems we found that it is a bug. It was a bug … the actual model may have been smaller due to the way it was initialized. Sorry about that … will close this issue. Thanks!</p>
How to create a custom Dataset and DataLoader with reference table

jkonan: <p>I have a reference.csv file that has three columns: Type, Class, and Path. Here are the first 5 example rows:</p>
<pre><code class="lang-auto">    "Type","Class","Path"
    "train","A","./path1/001.jpg"
    "train","A","./path2/002.jpg"
    "test","C","./path3/003.jpg"
    "train","B","./path4/001.jpg"
    "test","B","./path5/002.jpg"
    ...
</code></pre>
<p>In a more viewer-friendly format:</p>
<pre><code class="lang-auto">    |----------------------|------------------|------------------|
    |         Type         |       Class      |       Path       |
    |----------------------|------------------|------------------|
    |        train         |         A        | ./path1/001.jpg  |
    |----------------------|------------------|------------------|
    |        train         |         A        | ./path2/002.jpg  |
    |----------------------|------------------|------------------|
    |        train         |         C        | ./path3/003.jpg  |
    |----------------------|------------------|------------------|
    |        test          |         B        | ./path4/001.jpg  |
    |----------------------|------------------|------------------|
    |        test          |         B        | ./path5/002.jpg  |
    |----------------------|------------------|------------------|
</code></pre>
<p>I want to create a Dataset class (torch.utils.data.Dataset) to read the images so I can use the DataLoader (torch.utils.data.DataLoader).</p>
<p>What is the correct way to create a custom Dataset and DataLoader using a reference table?</p>
Why to() failed?

fr1140m: <p>I define a model, and use model.to(DEVICE) assign to GPU. but i always get errors below:</p>
<p><strong>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</strong></p>
<p>This is my codes:<br>
class BranchNet(nn.Module):<br>
def <strong>init</strong>(self, dropout=DROPOUT, num_classes=NUM_CLASSES):<br>
super(BranchNet, self).<strong>init</strong>()<br>
…</p>
<p>DEVICE = torch.device(“cuda:2”)</p>
<p>model = BranchNet(dropout=dropout, num_classes=NUM_CLASSES)<br>
model = model.to(DEVICE)<br>
image = image.to(DEVICE)<br>
label = label.to(DEVICE)</p>
<p>…</p>

albanD: <p>Hi,</p>
<p>The error is most likely not with the <code>.to()</code> op. It’s just that cuda is asynchronous and so errors will point to the wrong line. Run with <code>CUDA_LAUNCH_BLOCKING=1</code> to make sure the error points to the right line.</p>

Balamurali_M: <p>As far as I can see. This is nothing to do with to() method. In general, pytorch expects both the model and the input data to be of the same data type i.e float in our case. And if you are using GPU both the model weight and the input should be moved to gpu. Else the error mentioned will occur. As the first step check for the architecture class.</p>

fr1140m: <p>Thank you!</p>
<p>CUDA_LAUNCH_BLOCKING is a ENV VARIABLE ?</p>

fr1140m: <aside class="quote no-group" data-username="fr1140m" data-post="1" data-topic="47467">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fr1140m/48/11216_2.png" class="avatar"> fr1140m:</div>
<blockquote>
<p>model = model.to(DEVICE)<br>
image = image.to(DEVICE)<br>
label = label.to(DEVICE)</p>
</blockquote>
</aside>
<p>I have moved sample and model to same device already.</p>

SimonW: <p>you likely did something wrong in your module code, e.g., not properly register parts as submodules.</p>
<p><a class="mention" href="/u/alband">@albanD</a> CUDA is async, but out-of-kernel checks are not, since they are done without looking at the data contained. So CUDA_LAUNCH_BLOCKING won’t change things.</p>

albanD: <p>Ho right I read it too fast <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Which line exactly causes the issue? I should help you know which module is to blame as simon said.</p>
Create a customised recurrent neural network

Allan_Jie: <p>I guess this question should be asked but I couldn’t find it.</p>
<p>I want to create a customized recurrent neural network, for example, a different LSTM.<br>
My thought is to create another class <code>NewRNNModule</code> which extends the <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L27" rel="nofollow noopener">RNNBase</a> class.</p>
<p>But I think I also have to implement everything similar to the <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L378" rel="nofollow noopener">LSTM</a> class.</p>
<p>Is it the only way or there is any easy way to do so?</p>
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1532576128691/work/aten/src/THC/THCCachingAllocator.cpp line=507 error=3 : initialization error

shuo_yang: <p>Traceback (most recent call last):<br>
File “GeResult_ensemble.py”, line 70, in <br>
GeResult()<br>
File “GeResult_ensemble.py”, line 56, in GeResult<br>
for (Input_O, Input_H, visit_tensor, anos) in Dataloader:<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 336, in <strong>next</strong><br>
return self._process_next_batch(batch)<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 357, in _process_next_batch<br>
raise batch.exc_type(batch.exc_msg)<br>
RuntimeError: Traceback (most recent call last):<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 106, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 106, in <br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/media/RT_4TB/clj/2019Baidu-XJTU_URFC/2019BaiduXJTU/Dataloader/MultiModal_BDXJTU2019.py”, line 186, in <strong>getitem</strong><br>
Img_O = self.normal(Img)<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/transforms.py”, line 60, in <strong>call</strong><br>
img = t(img)<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/transforms.py”, line 163, in <strong>call</strong><br>
return F.normalize(tensor, self.mean, self.std, self.inplace)<br>
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/functional.py”, line 206, in normalize<br>
mean = torch.tensor(mean, dtype=torch.float32)<br>
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1532576128691/work/aten/src/THC/THCCachingAllocator.cpp:507</p>

ptrblck: <p>Are you using multiprocessing (multiple workers in your <code>DataLoader</code>) and CUDA tensors?<br>
If so, <a href="https://discuss.pytorch.org/t/a-call-to-torch-cuda-is-available-makes-an-unrelated-multi-processing-computation-crash/4075/2">this post</a> might be the answer.</p>
How to Use a tensor as an Index to Parameter in Pytorch?

Rafael_R: <p>Hi,</p>
<p>I have a scenario in which I have a 5128 dimensional parameter.</p>
<p>I also have a [16, 1] dimensional variable.</p>
<p>What I want to do?</p>
<p>Use the elements of the 16 x 1 dimensional variable as index to get the elements of the 5128 dimensional tensor.</p>
<p>In Python I would have done it like: [parameter[i] for i in variable], but here I want to tune the parameters, so I need to do it in a way gradients flow.</p>
<p>How can I achieve that? Any help is welcome!</p>

JuanFMontesinos: <p>You can do that and it will work wereas the dimensional variable is fixed. If you expect to make a learnable vector which chooses parameters it will not work.</p>
<p>Aditionally, consider that those elements which aren’t “chosen” will does not backpropagate anything. You can always go for other ways like attention rather than this.</p>

Rafael_R: <p>learnable vector for choosing parameters is not required…</p>
<p>could you share a snippet for this?</p>
Can a CNN learn if the inputs have a lot of zeros?

himat: <p>I have images that contain a lot of zero pixel values (25% of the image).</p>
<p>Will these prevent the CNN from learning? Since when the convolutional kernels are convolved with these zero regions, the outputs will be zero after relu activation, and so nothing will get backpropped later on since the error will be multiplied with this 0 value.</p>
<p>Is there a way around this?</p>

sonnguyen: <p>It should work fine. The weights relevant to the non zero pixels still learn.  MNIST dataset is also sparse and CNN still works very well there.</p>
Efficient way of finding Cross Entropy

adarsh-kr: <p>I have a list, which stores <code>N</code> tensors of dimension <code>B*C</code>, where <code>B</code> is the batch size and <code>C</code> is the total classes possible.   <code>[Y_1, Y_2, Y_3,....Y_N]</code>.  I also have a target list which stores the <code>N</code> target labels, each of dimension <code>B*1</code>.</p>
<p>I want to calculate the cross entropy of each individual batch with its corresponding target in the target list. I can write a loop to do that, but I was wondering if there’s more effective way of achieving this.</p>
Why is always the numer of input channels 3

blackbirdbarber: <p>For <code>nn.Conv2D</code>, why we have always input channel set to 3:</p>
<p>Examples:</p>
<pre><code class="lang-auto">AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

SqueezeNet(
  (features): Sequential(
    (0): Conv2d(3, 96, kernel_size=(7, 7), stride=(2, 2))

VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

DenseNet(
  (features): Sequential(
    (conv0): Conv2d(3, 96, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

Inception3(
  (Conv2d_1a_3x3): BasicConv2d(
    (conv): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)
</code></pre>
<p>And so on you get the point?</p>

Oli: <p>It’s because an image that is in color has three channels, RGB - or Red Green Blue. Most networks operate on color images</p>

blackbirdbarber: <p>Is this the same for <a href="https://en.wikipedia.org/wiki/CMY_color_model" rel="nofollow noopener">CMY</a>?</p>

Oli: <p>I’ve never done it, but I guess that if you want to use another colorspace, such as CMY, you could still use the network architecture but would have to retrain the networks yourself. Or, was this what you were wondering?</p>

blackbirdbarber: <p>Thanks <a class="mention" href="/u/oli">@Oli</a>, I don’t plan to work on CMY, at the moment, just checking if this would be possible. NFQ. (no further questions).</p>

mftoml84: <p>Images can also be converted to greyscale and then the number of input channels will be 1.</p>
Learning CNN and getting below error

mftoml84: <p>Hi,<br>
I know below error is discussed many times but i am still not able to figure out.<br>
The error i am getting is :<br>
RuntimeError: Given groups=1, weight of size [12, 6, 5, 5], expected input[100, 1, 32, 32] to have 6 channels, but got 1 channels instead</p>
<p>below is my code:</p>
<p>class Network(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Network, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels=1, out_channels= 6, kernel_size=5)<br>
self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)</p>
<pre><code>    self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
    self.fc2 = nn.Linear(in_features=120, out_features=60)
    self.out = nn.Linear(in_features=60, out_features=10)

def forward(self, t):
    #input layer
    t = t

    #hidden conv layer1
    x = F.relu(self.conv1(t))
    x = F.max_pool2d(x,kernel_size= 2, stride=2)

    # hidden conv layer2
    x = F.relu(self.conv2(t))
    x = F.max_pool2d(t,kernel_size= 2, stride=2)

    #hidden linear layer
    x = x.view(-1, 12 * 4 * 4)
    x = F.relu(self.fc1(x))
    x = F.relu(x)

    #hidden linear layer
    x = self.fc2(x)
    x = F.relu(x)

    x = self.out(x)
    #return F.log_softmax(x, dim=1)
    return x
</code></pre>
<p>I am converting the RGB image to grayscale by using below lines:</p>
<p>transform = transforms.Compose(<br>
[<strong>torchvision.transforms.Grayscale</strong>(num_output_channels=1), transforms.ToTensor(),<br>
transforms.Normalize([0.5],[0.5])])</p>
<p>Not sure where the problem is, can someone point it out to me?</p>

ptrblck: <p>You are passing the input tensor <code>t</code> again to <code>self.conv2</code> instead of <code>x</code>.</p>

mftoml84: <p>Thank you, silly mistake…</p>
Evaluate all submodules for GPU use

Adrian_Tovar_Lopez: <p>Hello everyone. I have a question. I’ll be grateful for any help<br>
I have a network:</p>
<pre><code class="lang-auto">class Net(nn.Module):

	def __init__(self):
		super(Net,self).__init__();
		self.Charts = nn.ModuleList([net() for i in range(10)]) 
#net is a module I defined before


def forward(self,x):
		x = torch.cat([self.Charts[i](x) for i in range(10)],1)
        	return x
</code></pre>
<p>But I want to evaluate all the modules at the same time instead of going trough the loop: <code>[self.Charts[i](x) for i in range(10)]</code>. Otherwise running in the GPU doesn’t improve performance</p>
<p>Anyone know how to do that?<br>
(Also I know 10 submodules is not enough to see an improvement by running it in GPU but this is only an example, I have way more submodules in my actuala code).</p>
Custom DataLoader (not Dataset)

Andreas_Bloch: <p>I’m sampling minibatches of sequences and I want to store them in a tensor of dimension:</p>
<p>(num_sequences, max_sequence_length)</p>
<p>where max_sequence_length is the length of the longest sequence. Sequences shorter than max_sequence_length will be padded with an appropriate padding value.</p>
<p>Now, ideally I’d like to do the padding already in the DataLoader (as I can parallelize that step in the workers on the CPUs) instead of doing it in the network’s forward function. The Dataset class however only provides the <strong>getitem</strong>() function. In that function I yet don’t know the length of the longest sequence that will be in the minibatch.</p>
<p>Do you know about a tutorial to write a custom DataLoader that would could do the dimension adaptions to a batch?</p>

Andreas_Bloch: <p>I’ve found the solution, all you need to do is to implement a custom function called:</p>
<p>collate_fn</p>
<p>It takes a batch (list) of examples returned by the Dataset <strong>getitem</strong>() function, and should return a batch as desired by the network (or next consumer).</p>
What is the problem with my code?

smani: <p>The following is my code:</p>
<pre><code> index = torch.tensor([3])
 A = torch.tensor([  [0.3, 0.15, 0.1, 0.1, 0.05]], requires_grad=True)
 B = torch.gather(A, 1, index.view(-1,1))
 loss = (A - B.expand_as(A)) + 0.2
 print('Loss = \n', loss)

 mask = torch.ones_like(loss)
 mask = mask.scatter_(1, index.view(-1,1), 0)
 loss = loss*mask
 print('Loss After mask = \n', loss)

 loss = loss.sum()
 loss.backward()
 print('Input and grad:')
 print(A)
 print(A.grad)
</code></pre>
<p>The output is:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/af3db027008b98c584253f1b3be0a0df26945f8e.png" data-download-href="https://discuss.pytorch.org/uploads/default/af3db027008b98c584253f1b3be0a0df26945f8e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af3db027008b98c584253f1b3be0a0df26945f8e_2_690x132.png" alt="image" data-base62-sha1="p0fIQZs6ozPhchXPkH3lZv47bem" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af3db027008b98c584253f1b3be0a0df26945f8e_2_690x132.png, https://discuss.pytorch.org/uploads/default/original/2X/a/af3db027008b98c584253f1b3be0a0df26945f8e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/af3db027008b98c584253f1b3be0a0df26945f8e.png 2x" data-dominant-color="3B1B32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">700×134 48.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I multiply the loss with a mask where some values are zero. In that case I expect the gradient for that position also should be zero. But I dont understand when I am getting -4 as the gradient at that position.</p>

a3VonG: <p>I’m not sure what the objective was of the code but the problem is in your loss function:</p>
<pre><code class="lang-auto"> loss = (A - B.expand_as(A)) + 0.2
</code></pre>
<p>Imagine you want to minimize this then you would take A as negative as possible. Minimizing this loss thus corresponds to making the elements of A as negative as possible irrespective of the values of B.</p>
<p>Setting the loss to zero thus does not mean in this case that the gradient will be zero as the loss function can go below zero.</p>
<p>You will see that if you use:</p>
<pre><code class="lang-auto"> loss = (A - B.expand_as(A))**2 + 0.2
</code></pre>
<p>Then the gradient will be zero, you can also omit the 0.2 as it will not influence the gradients in any way.</p>