akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: cuda runtime error (9) when implementing Kalman Filter

CDhere: <p>Hi, I’m implementing a baseline tracker using Kalman Filter and bbox matching. I implemented my Kalman Filter with many cuda tensor broadcastings going on. However, when I run inference, the script always crashes <strong>after</strong> a few frames (weirdly, not at the 0-th frame), and I got all sorts of “configuration errors” during debugging. The most frequent one looks like the following:</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (9) : invalid configuration argument at /pytorch/aten/src/THC/THCTensorMathCompare.cuh:82
</code></pre>
<p>when I’m trying to determine if a detection confidence is below a threshold, using <code>bbox[d, 4] &lt; thresh</code>.</p>
<p>I also tried to use pdb to poke into the pipeline to see if I got any tensor shapes wrong. However, at some break point when I got the error message above generated by some expression, if I enter that same expression again, the right tensor came out without error (exactly the same expression!). This behavior really confused me as I cannot find where my bug is.</p>
<p>Any help or suggestions will be appreciated! Thanks!</p>

ptrblck: <p>Did you write a custom CUDA extension or are you using plain PyTorch methods?<br>
Could you post your code so that we could take a look at it, please?</p>
CVAE: Kernel size can't be greater than actual input size

sajaldash: <p>I am trying to run a Convolutional VAE using the following model. My input images are single channel images of size 32x32. The input tensor size is [32, 1, 32, 32] since the batch size is 32. Why am I getting this error, the example code is supposed to run on 3-channel 32x32 images [32, 3, 32, 32]. I have a total of 499 images. How do I fix this?</p>
<p>The model’s specification:<br>
DCVAE(<br>
(encoder): Sequential(<br>
(0): Conv2d(1, 32, kernel_size=(4, 4), stride=(2, 2))<br>
(1): LeakyReLU(negative_slope=0.2)<br>
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2))<br>
(3): LeakyReLU(negative_slope=0.2)<br>
(4): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2))<br>
(5): LeakyReLU(negative_slope=0.2)<br>
(6): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2))<br>
(7): LeakyReLU(negative_slope=0.2)<br>
(8): Flatten()<br>
)<br>
(encoder_mean): Linear(in_features=1024, out_features=32, bias=True)<br>
(encoder_logvar): Linear(in_features=1024, out_features=32, bias=True)<br>
(fc): Linear(in_features=32, out_features=1024, bias=True)<br>
(decoder): Sequential(<br>
(0): UnFlatten()<br>
(1): ConvTranspose2d(1024, 128, kernel_size=(5, 5), stride=(2, 2))<br>
(2): ReLU()<br>
(3): ConvTranspose2d(128, 64, kernel_size=(5, 5), stride=(2, 2))<br>
(4): ReLU()<br>
(5): ConvTranspose2d(64, 32, kernel_size=(6, 6), stride=(2, 2))<br>
(6): ReLU()<br>
(7): ConvTranspose2d(32, 1, kernel_size=(6, 6), stride=(2, 2))<br>
(8): Sigmoid()<br>
)<br>
)</p>
<p>The error message:<br>
RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (4 x 4). Kernel size can’t be greater than actual input size</p>

ptrblck: <p>As the error message claims, your output activation is too small for the last conv layer.<br>
Inline are the output activation shapes:</p>
<pre><code class="lang-python">(encoder): Sequential(
(0): Conv2d(1, 32, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 32, 15, 15]
(1): LeakyReLU(negative_slope=0.2)
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 64, 6, 6]
(3): LeakyReLU(negative_slope=0.2)
(4): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 128, 2, 2]
(5): LeakyReLU(negative_slope=0.2)
(6): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2)) # ERROR: 2x2 is too small for a 4x4 kernel
(7): LeakyReLU(negative_slope=0.2)
(8): Flatten()
)
</code></pre>
<p>You could either increase the input shape or decrease the kernel shapes.<br>
Alternatively you could also add padding to the preceding layers, so that the spatial size won’t be decreased so much.</p>
How turn a torch audio to binary wav without saving to disk

quality: <p>I need to send some audio I use a torch based text to speech to generate as a JSON. I cannot find a way to turn it into a base64 without the following awful code:</p>
<pre><code class="lang-auto">torchaudio.save('C:/Work/test_images/temp.wav', wav_array, sampling_rate)
wav_file = open('C:/Work/test_images/temp.wav', 'rb')
</code></pre>
<p>any ideas how to clean this up? there doesn’t appear to be any built-in functions with torch to do this, but I’m fairly new so would appreciate any help.</p>

ptrblck: <p>I’m not deeply familiar with <code>torchaudio</code>, but would it be possible to use an <code>io.BytesIO</code> object for temporal storage instead of a file work?</p>
Extract list of sequences from tensor of padded sequences

aRI0U: <p>Hi,<br>
I have a tensor containing padded sequences, and another one containing the lengths of all those sequences. What is the most efficient way to retrieve a list of those sequences?</p>
<p>For example, assuming I have:</p>
<pre><code class="lang-auto">seq = tensor([
    [3, 2, 5, 0, 0],
    [2, 1, 8, 5, 1],
    [9, 0, 0, 0, 0]
])
lengths = tensor([3, 5, 1])
</code></pre>
<p>what is the fastest way to return <code>[tensor([3,2,5]), tensor([2,1,8,5,1]), tensor([9])]</code>?</p>
<p>For now, the less bad solution I found is to do:</p>
<pre><code class="lang-auto">slices = torch.stack((lengths, lengths.max()-lengths), dim=1).view(-1)
return torch.split(seq.view(-1), slices.tolist())[0::2]
</code></pre>
<p>but it is a really ugly (and probably slow) solution</p>
<p>Thanks,<br>
Alain</p>

ptrblck: <p>A one-liner would be:</p>
<pre><code class="lang-python">result = [s[:l] for s, l in zip(seq, lengths)]
</code></pre>
<p>Depending on the shapes you are using, my approach might be faster.<br>
For the current workload I see:</p>
<pre><code class="lang-python"># your approach
41.5 µs ± 2.36 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# mine
26.4 µs ± 1.84 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>
Loss not change

Tshzzz: <p>I try to train a network to do object detection. When I train the network after 2~3epochs ,the loss will never change.</p>
<pre><code class="lang-auto">class my_test_net(nn.Module):
    input_size = 300

    def __init__(self):
        super(SSD300, self).__init__()

        # model
        self.base = self.VGG16()
        self.norm_1 = nn.BatchNorm2d(512)
        #self.norm4 = L2Norm(512,20)
        
        self.conv5_1 = nn.Conv2d(512, 512, kernel_size=3, padding=1, dilation=1)        
        self.conv5_2 = nn.Conv2d(512, 512, kernel_size=3, padding=1, dilation=1)
        self.conv5_3 = nn.Conv2d(512, 512, kernel_size=3, padding=1, dilation=1)
        
        self.norm_2 = nn.BatchNorm2d(512)
        
        self.conv6 = nn.Conv2d(512, 1024, kernel_size=3, padding=6, dilation=6)  
        self.conv7 = nn.Conv2d(1024, 1024, kernel_size=1)
        
        self.norm_3 = nn.BatchNorm2d(1024)
        self.fc = nn.Linear(1024*6*6, 50*(21+4))
        torch.nn.init.xavier_uniform(self.fc.weight)
        #self.dropout = torch.nn.functional.dropout()
        
    def forward(self,x):
        output = self.base(x)
        output = self.norm_1(output)
        #print(output.size())
        
        output = F.max_pool2d(output, kernel_size=2, stride=2, ceil_mode=True)
        
        output = F.relu(self.conv5_1(output))
        output = F.relu(self.conv5_2(output))
        output = F.relu(self.conv5_3(output))
        output = F.max_pool2d(output, kernel_size=3, padding=1, stride=1, ceil_mode=True)
        
        output = F.relu(self.conv6(self.norm_2(output)))
        output = F.relu(self.conv7(output))
        output = F.max_pool2d(output, kernel_size=2, padding=1, stride=1)
        
        output = self.norm_3(output)
        
        output = output.view(-1,1024*6*6)
        
        output = F.sigmoid(self.fc(output))
        output = torch.nn.functional.dropout(output,training = True)
        
        
        output = output.view(-1,50,21+4)
        
        return output
</code></pre>
<p>[1] batch_loss = 1.79685 epoch_loss = 59.99467<br>
[2] batch_loss = 2.68952 epoch_loss = 54.00942<br>
[3] batch_loss = 1.10251 epoch_loss = 54.00942<br>
[4] batch_loss = 4.84961 epoch_loss = 54.00942:<br>
[5] batch_loss = 4.32710 epoch_loss = 54.00942</p>

richard: <p>Have you checked your gradients? It’s possible that they’re 0 (and have vanished).</p>

mukul54: <p><a class="mention" href="/u/tshzzz">@Tshzzz</a> did you solve this issue? I am facing the same issue with SSD.</p>
Removing nn.BatchNorm2d() improved my model

Idan_Uri: <p>TD;DR:<br>
Removing nn.BatchNorm2d() after the conv2d layer improved my model and I don’t know why.</p>
<p>More info:<br>
I’m using <a href="https://github.com/milesial/Pytorch-UNet/tree/master/unet" rel="nofollow noopener">this implementation of Unet</a> with a 256,257 image with 14 channels (The stft of a signal throgh 8 microphones) and I’m trying to categorize each pixel in the 256x257 to a certain angle (number between 0 and 12).</p>
<p>I’m using nn.CrossEntropyLoss. Before I removed the batchnorm the loss on the validation was going down in small steps with large volatility but the resulats weren’t good enough. After I googled a little bit why it’s happend I found this <a href="https://stats.stackexchange.com/questions/425610/why-massive-random-spikes-of-validation-loss" rel="nofollow noopener">forum</a> with a simillar problem saying removing batchnorm improved his model.</p>
<p>It improved the resulats on my model greatly but I have no idea why. Does someone have any explanation?</p>

Kushaj: <p>Because batchnorm is not the best normalization technique for your task. Batch norm tries to normalize all your images in a batch to same mean and std, whereas in case of segmentation you want different statistics for each image. Use InstanceNorm instead.</p>

ptrblck: <p>Additionally to batchnorm being a “bad fit” for your use case, you should check your current batch size, as batchnorm layers usually perform poorly with a small batch size.</p>
Custom backward as a Jacobian vector product

kealexanderwang: <p>To implement a custom pytorch Function, we need to implement a <code>forward</code> and a <code>backward</code> function. However, if the input is a tensor and the output is a tensor, the <code>backward</code> function will return the Jacobian explicitly. On the other hand the autograd engine never forms the Jacobian explicitly and uses Jacobian vector products instead.</p>
<p>In my use-case, the Jacobian of my custom Function is a very large tensor. Is there any way to implement <code>backward</code> as a Jacobian vector product instead? If not, are there plans to make this a possibility in the future?</p>

aarrushi: <p>Have you tried invoking the backward function with the feature vector i.e. backward(x) instead of backward()?</p>
<p>I wrote a custom class which inherits autograd Function and the backward call defined as “def backward(ctx, grad_output):” . It returns the J*grad_out which is of same dimension as grad_out.</p>
May be problem wrapping main code in if name=="__main__" OR setting num_workers

vamshi888: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d95a3553eab3dc60982b55e3019cae2e428ced2b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d95a3553eab3dc60982b55e3019cae2e428ced2b" title="Screenshot (51)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d95a3553eab3dc60982b55e3019cae2e428ced2b_2_690x388.png" alt="Screenshot (51)" data-base62-sha1="v0MSEv4wMrijFeqSfwAVvyjDFDt" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d95a3553eab3dc60982b55e3019cae2e428ced2b_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d95a3553eab3dc60982b55e3019cae2e428ced2b_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d95a3553eab3dc60982b55e3019cae2e428ced2b_2_1380x776.png 2x" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (51)</span><span class="informations">1920×1080 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
here is code attachement: <a href="https://github.com/kkraoj/leafnet/blob/master/model.py" class="inline-onebox" rel="noopener nofollow ugc">leafnet/model.py at master · kkraoj/leafnet · GitHub</a></p>

vamshi888: <p>someone please have a look and suggest your solutions</p>

ptrblck: <p>Did you wrap your code in the if-clause protection?<br>
Also, is your code working fine using <code>num_workers=0</code>?</p>
torchtext.datasets.IMDB

pascal_notsawo: <p>By default, the following code split the IMDB dataset with ratio of 0.5 ( train_data =25000 and test_data=25000).</p>
<p><strong>train_data, test_data = torchtext.datasets.IMDB.splits(TEXT, LABEL)</strong></p>
<p>I want to split it with ratio of .0.8 ( train_data =40000 and test_data=10000). How I can process?</p>
Should I send scalar tensor to GPU?

liangbright: <p>Hello,</p>
<p>Please read the two implementations, which one is preferred for GPU?<br>
general question: should I send scalar to GPU ?</p>
<pre><code class="lang-python">class Net1(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer('tau', torch.tensor(0.5))
        
    def forward(self, x):
        return self.tau*x
#------------------------------------------------------------------
class Net2(nn.Module):
    def __init__(self):
        super().__init__()
        self.tau=0.5
        
    def forward(self, x):
        return self.tau*x
#------------------------------------------------------------------
model=Net1()
model.to('cuda')
#or
model=Net2()
model.to('cuda')
# will self.tau be sent to GPU 'automatically' and then do tau*x ?</code></pre>

Kushaj: <p>When you move model to GPU then all the computation happens on the GPU. So <code>self.tau</code> is also moved to GPU automatically.</p>

liangbright: <aside class="quote no-group" data-username="liangbright" data-post="1" data-topic="82763">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liangbright/48/4986_2.png" class="avatar"> liangbright:</div>
<blockquote>
<p><code>self.tau=0.5</code></p>
</blockquote>
</aside>
<p>self.tau=0.5  will be moved to GPU automatically?</p>

googlebot: <p>IIRC, "Scalar"s are handled in specialized ops in c++, so they probably just end up as arguments to cuda kernel functions. And cuda automatically copies kernel arguments (pointers &amp; scalars) to gpu.<br>
So <em>maybe</em> scalars are marginally faster than buffers, not sure.</p>

Kushaj: <p>Consider this. When you move your model to GPU, all rhe weights of the model are moved to GPU. Now when you have to do some computation on these weights that has to be done on the GPU because GPU-CPU-GPU transfers are expensive. So all the code of your model in turn runs on rhe GPU.</p>
PyTorch tensor to fortran-style ordering

spaceinvader: <p>I have already asked a similar question, but I’ll try to reformulate it.</p>
<p>My question now is how to reorder PyTorch tensor data in Fortran-style order like np.reshape(order=‘F’) in numpy?</p>
<p>I need column-major ordering to use CUDA functions from cusparse library.</p>

rdroste: <p>How about this?</p>
<pre><code class="lang-auto">t = torch.Tensor(2,3)
f = t.numpy().reshape(2,3,order='F')
</code></pre>

spaceinvader: <p><a class="mention" href="/u/rdroste">@rdroste</a><br>
First of all, this function is not differentiable.<br>
Secondly, it needs data to be transfered from CPU to GPU and back. This operation is very slow.</p>

colesbury: <p>It’s a few more lines in PyTorch, but this will work if your input is C-style contiguous:</p>
<pre><code class="lang-auto">t = torch.randn(3, 5)
t = t.t().contiguous().t()
print(t.shape)  # torch.Size([3, 5])
print(t.stride())  # (1, 3)
</code></pre>

rdroste: <p><a class="mention" href="/u/spaceinvader">@spaceinvader</a> You are right, the numpy solution will be slow and non-differentiable.</p>
<p><a class="mention" href="/u/colesbury">@colesbury</a> How about reversing the stride manually to perform less operations like this:</p>
<pre><code class="lang-auto">t = torch.randn(3, 5)
t.set_(t.storage(), t.storage_offset(), t.size(), tuple(reversed(t.stride())))
</code></pre>
<p>This should have the same effect as <code>t = t.t().contiguous().t()</code> but is potentially faster.</p>

Zhongyu_Jiang: <p>How about this?<br>
vp=torch.randn(1,204)<br>
vp1=vp.view(68,-1)#<span class="hashtag">#torch</span>.size([68,3])<br>
vp2=vp1.t()#<span class="hashtag">#torch</span>.size([3,68])<br>
then vp2 is the F order version of vp</p>
pytorch split dataset

xiaomo: <p>How does pytorch divide the data set, instead of reading the corresponding train and test files. For example, divide data.csv into train and test data sets.</p>
<p>I used data.split in torchtext to divide the data set, and the following error is displayed at train_data, val_data = dataset.split ()：</p>
<pre><code class="lang-auto">payloads = pd.read_excel('data/payloads.xlsx')
print('加载数据中...')
text = data.Field(sequential=True, lower=True, tokenize=get3gram)
label = data.Field(sequential=False)

fields=[('index', None), ('payload', text), ('is_malicious', label), ('injection_type', None)]
dataset = data.Dataset(es_420,fields)
train_data, val_data = dataset.split()
</code></pre>
<p>KeyError                                  Traceback (most recent call last)<br>
D:\anaconda\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)<br>
2645             try:<br>
-&gt; 2646                 return self._engine.get_loc(key)<br>
2647             except KeyError:</p>
<p>pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()</p>
<p>pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()</p>
<p>pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()</p>
<p>pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()</p>
<p>KeyError: 1914</p>
How to know if GPU memory being utilised is actual or is there a memory leak

bing: <p>I have 3 Tesla V100(16 GB). I am doing transfer learning using efficeint net (63 Million parameters) on images of (512,512) with a batch size of 20.<br>
My GPU memory utilisation is below -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95d5f482adddcff5c60f0c2a63572869c0d4fd5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/95d5f482adddcff5c60f0c2a63572869c0d4fd5e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95d5f482adddcff5c60f0c2a63572869c0d4fd5e_2_690x325.png" alt="image" data-base62-sha1="lnvxt5tEJGoSKpn64kNJfe4ssBo" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95d5f482adddcff5c60f0c2a63572869c0d4fd5e_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95d5f482adddcff5c60f0c2a63572869c0d4fd5e_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95d5f482adddcff5c60f0c2a63572869c0d4fd5e.png 2x" data-dominant-color="F9F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1198×566 57.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, it has almost filled up all the 3 GPUs(almost 80%).<br>
My question is is there any theoretical way of calculating that the utilisation of GPU memory being shown is what is required by the model at a certain image and batch size or is there a memory leak in my GPU?</p>
Problem with CrossEntropyLoss

dllacer: <p>Hi everyone!</p>
<p>I’m trying to use Cross Entropy as my loss function, but there’s something wrong with dimensions of my output or target that I can’t notice.</p>
<p>Te output of my net is a single channel image with dimension [batch_size, 1, 320, 180]<br>
The target is a single channel image with dimension [batch_size, 1, 320, 180]<br>
Each pixel have values in the range [0,1].</p>
<p>Now I want to feed the CrossEntropyLoss function with them, but I don’t know which are the dimensions they have to have to be accepted by the function, so I’m getting this error:</p>
<p>RuntimeError: expected scalar type Long but found Float</p>
<p>I solved it by passing labels like: label.long()<br>
(I don’t know if this is correct)</p>
<p>But I achieved the loss function to accept my labels.</p>
<p>The problem comes when the loss is always 0 and it does not train</p>
<p>Any idea which could be the problem?</p>

ptrblck: <p>For a binary segmentation use case, you could simply use <code>nn.BCEWithLogitsLoss</code>, which would accept the shapes you are currently providing.</p>
<p><code>nn.CrossEntropyLoss</code> is used for a multi-class segmentation use case. It can also be used for a binary classification, but would require your output to have the shape <code>[batch_size, 2, height, width]</code> and the target to have the shape <code>[batch_size, height, width]</code> containing the class indices in the range <code>[0, 1]</code>.</p>

dllacer: <p>Thanks for your answer, it helped me a lot.</p>
<p>It worked for me with <code>nn.BCEWithLogitsLoss</code>, but I would like to make it work with <code>nn.CrossEntropyLoss</code> also.</p>
<p>In my problem, the goal is to make the network output look as much as the target as possible. That makes me think that my problem is not a classification but a regression problem.<br>
So, when I try to set the dimension of the network output like <code>[batch_size, C, height, width]</code> (being C = number of classes), I cannot set the number of classes because I don’t have them.</p>
<p>How should I use the <code>nn.CrossEntropyLoss</code> for a regression problem?</p>

ptrblck: <aside class="quote no-group" data-username="dllacer" data-post="3" data-topic="82739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/50afbb/48.png" class="avatar"> dllacer:</div>
<blockquote>
<p>How should I use the <code>nn.CrossEntropyLoss</code> for a regression problem?</p>
</blockquote>
</aside>
<p>That won’t be possible, since you would need to provide class indices to calculate the cross entropy.<br>
Based on “the goal is to make the network output look as much as the target as possible”, it seems indeed that you might be working on a regression problem, so you could use e.g. <code>nn.MSELoss</code> instead.</p>
How to fix Dimension out of range (expected to be in range of [-1, 0], but got 1) error

nlam: <p>Hi, getting an error during model training that im not sure how to interpret. data_tensor contains 4 numbers such as [a, b, c, d] and result is a tensor containing either -1 or 1(what im trying to predict). My code is below. Thank you for your help</p>
<pre><code class="lang-auto">class LogReg(nn.Module):
    def __init__(self):
        super(LogReg, self).__init__()
        self.lin1 = nn.Linear(4, 2)
        self.sig1 = nn.Sigmoid()
    
    def forward(self, x):
        x = self.lin1(x)
        y = self.sig1(x)
        return y

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = LogReg()
model = model.to(device)

criterion = nn.CrossEntropyLoss()

learning_rate = 0.01
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)

def train(num_epoch):
    model.train()
    
    for epoch in range(num_epoch):
        for i in range(len(dataset)):
            
            data_tensor, result = dataset[i]
            data_tensor, result = data_tensor.to(device), result.to(device)
            
            optimizer.zero_grad()
            
            output = model(data_tensor)
            loss = criterion(output, result)
            loss.backward()
            optimizer.step()
            
    
    print("DONE TRAINING for", epoch, "out of", num_epoch)   
</code></pre>

crcrpar: <p>I assume that the task is binary classification, then I would use <a href="https://pytorch.org/docs/stable/nn.html?highlight=bceloss#torch.nn.BCELoss" rel="nofollow noopener"><code>nn.BCELoss</code></a>, set <code>self.lin1</code> as <code>nn.Linear(4, 1)</code>, and classes as 0 or 1, not -1 or 1.</p>

nlam: <p>alright thank you, let me try that out</p>
Visualize Images while Training at Full Resolution

Maria_Fernanda_De_La: <p>I would like to visualize CIFAR10 images from the tensors during training at full resolution? Even with un-normalization, they look pixelated.</p>

crcrpar: <p>CIFAR10 images are tiny. They have only 1024 pixels per channel. You can check the original images at <a href="https://www.cs.toronto.edu/~kriz/cifar.html" rel="nofollow noopener">https://www.cs.toronto.edu/~kriz/cifar.html</a>.</p>
Wide resnet not found

nicill: <p>Hi everyone,</p>
<p>I am trying to use the pre-trained models in pytorch and I am encountering problems with wide_resnet.</p>
<p>The part of the code that is suposed to access it is:</p>
<blockquote>
<p>elif featEx==“resnext”:<br>
self.featEx = torchModels.resnext101_32x8d(pretrained=True)<br>
elif featEx==“wideresnet”:<br>
self.featEx = torchModels.wide_resnet101_2(pretrained=True)</p>
</blockquote>
<p>I have managed to get all the other models to work (I have left in the code for resnext as an example), but when I try to use it with the wide_resnet I get:</p>
<p>AttributeError: module ‘torchvision.models’ has no attribute ‘wide_resnet101_2’</p>
<p>I am thinking this is possibly a pytorch version problem but have been trying to update to the latest version without apparent success.</p>

ptrblck: <p>Which <code>torchvision</code> version are you using?<br>
You can get it via <code>print(torchvision.__version__)</code>.<br>
This model was added ~10 months ago.</p>

nicill: <p>Thanks for your answer.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="78171">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>print(torchvision.__version__)</code> .</p>
</blockquote>
</aside>
<p>My torchvision version is:</p>
<p>0.3.0</p>

ptrblck: <p><code>0.3.0</code> doesn’t contain this model as seen <a href="https://github.com/pytorch/vision/blob/v0.3.0/torchvision/models/resnet.py">here</a> and is too old.<br>
I would recommend to update to the latest stable version (<code>0.6.0</code>), which will contain this model as well as other new features and bug fixes.</p>
<p>Once you are about to update <code>torchvision</code> also update PyTorch, as I assume you might be using an older PyTorch version as well.</p>

nicill: <p>Just for completeness, the answer was totally correct.Thanks a lot.</p>
<p>After finding a newer version of pytorch and torchvision, the model works fine with the above code.</p>
Non-leaf variables as a module's parameters

mfkasim: <p>I wonder how to set up an NN module whose parameters are non-leaf variables.<br>
What I would like to do is something like below:</p>
<pre><code>class NNTest(torch.nn.Module):
    def __init__(self):
        super(NNTest, self).__init__()
        self.module = torch.nn.Linear(2, 3, bias=False)
    
    def forward(self, x):
        return self.module(x)

nn = NNTest()
p = torch.nn.Parameter(torch.randn(2, 3))
d = p**2
nn.module.weight = d  # raises a TypeError here
# nn.module.weight.data = d.data  # this is not what I want
</code></pre>
<p>The problem of the last line is that it does not propagate the gradient to <code>p</code> and I would like the gradient to propagate beyond the Module’s weight.<br>
Any help will be appreciated! Thanks!</p>

albanD: <p>You can do this by first deleting the original one and then assigning your non-leaf Tensor:</p>
<pre><code class="lang-python">del nn.module.weight
nn.module.weight = d
</code></pre>

jaks19: <p>Is there a way to do this <code>del</code> call on the parameters dynamically, without having to write the nested object name <code>nn.module.weight</code>?</p>
<p>Example: Say the module has N total parameter sets, and I want to delete all of them, and cannot know what they will be named pre run, because I am writing a general wrapper for any torch.nn.Module.</p>
<p>I do not mind reading the names at runtime and dealing with those, but I just cannot write less general code with their names explicitly typed out.</p>
<p>For recursive set and get, <a href="https://stackoverflow.com/questions/31174295/getattr-and-setattr-on-nested-subobjects-chained-properties" rel="nofollow noopener">this stackoverflow post</a> was helpful, and also I could use the .children() or .modules() functions. But I am stuck on deletion.</p>
<p>Thanks!</p>

jaks19: <p>Update: I managed to use python’s <code>exec</code> function to do <code>exec(f'del {model_name}.{module_name}')</code> where <code>module_name</code> is one of the names from my model’s <code>named_parameters()</code>, but would appreciate any less hacky way to do it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>Another way to do it is described in this other post: <a href="https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/9" class="inline-onebox">How does one have the parameters of a model NOT BE LEAFS?</a></p>
Filter tensors based on other tensor

hamidriasat: <p>I want to filter tensors. I have two tensors.<br>
tensorA has dimensions [ 264 , 2 , 3] and tensorB has dimensions [264]. tensorB contains data like 0,1,0,0,0,1 …  i want to filter tensorA on the basis of tensorB such that output tensor has dimensions [ 264 , 3] (from two indexes ( 0 and 1) of dim = 1  select either 0 or 1 based on tensorB).<br>
Basically select index of  dim = 1 of tensorA on the basis of tensor 2.<br>
I tried using gather function but it says input and index dimensions are not same. i have tried using for loop but that is slowing my model a lot so i need fast way which works fast on gpu.</p>

futscdav: <p>You have to expand the dims of the indexing tensor for gather, or at least I’m not aware of a way to circumvent that. This should work</p>
<pre><code class="lang-auto">a = torch.rand((264, 2, 3))
b = torch.randint(0, 2, (264,))
i = torch.stack(3*[b], dim=1).view(-1, 1, 3)
r = a.gather(1, i).squeeze()
</code></pre>
Torch.utils.bottleneck keeps on running

bing: <p>I am trying to profile my DL model to see the bottleneck. On using the following statement to profile it -<br>
<code>python -m torch.utils.bottleneck mycode.py</code></p>
<p>It keeps on running and running and not printing the expected profile summary. Below is the output-</p>
<pre><code class="lang-auto">Loaded pretrained weights for efficientnet-b3
Let's use 3 GPUs!
1536
	Epoch: 1 	Training Loss: 1.091230 	Validation Loss: 1.110597

Valid Accuracy: 37% (203/539)
Validation loss decreased (inf --&gt; 1.110597).  Saving model ...
Epoch: 2 	Training Loss: 1.092819 	Validation Loss: 1.108480

Valid Accuracy: 38% (206/539)
Validation loss decreased (1.110597 --&gt; 1.108480).  Saving model ...
Running your script with the autograd profiler...
Loaded pretrained weights for efficientnet-b3
Let's use 3 GPUs!
1536
Epoch: 1 	Training Loss: 1.091230 	Validation Loss: 1.110597

Valid Accuracy: 37% (203/539)
Validation loss decreased (inf --&gt; 1.110597).  Saving model ...
Epoch: 2 	Training Loss: 1.092819 	Validation Loss: 1.108480
</code></pre>
Autograd through torch.distributions.MultivariateNormal not working

nierth: <p>Hi,</p>
<p>I was trying out some code related to reinforcement learning (TD3) when I thought that I could make the code a little bit nicer by replacing a few lines. However, this broke the entire autograd update and I am just wondering why. Below is the actor class with the forward pass that breaks the code.</p>
<pre><code class="lang-auto">class Actor(nn.Module):
   def __init__(self, state_dim, action_dim, action_std):
       super(Actor, self).__init__()

       self.net = nn.Sequential(
                       nn.Linear(state_dim, 256),
                       nn.ReLU(),
                       nn.Linear(256, 256),
                       nn.ReLU(),
                       nn.Linear(256, action_dim),
                       nn.Tanh())

       self.action_std = torch.full((action_dim,), action_std).to(device)

   def forward(self, state):
       action_mean = self.net(state)
       return action_mean + torch.randn_like(action_mean) * self.action_std
</code></pre>
<p>works fine, but</p>
<pre><code class="lang-auto">class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, action_std):
        super(Actor, self).__init__()

        self.net = nn.Sequential(
                        nn.Linear(state_dim, 256),
                        nn.ReLU(),
                        nn.Linear(256, 256),
                        nn.ReLU(),
                        nn.Linear(256, action_dim),
                        nn.Tanh())

        self.action_var = torch.full((action_dim,), action_std*action_std).to(device)
        self.cov_mat = torch.diag(self.action_var).to(device)

    def forward(self, state):
        action_mean = self.net(state)
        dist = torch.distributions.MultivariateNormal(action_mean, self.cov_mat)
        return dist.sample()
</code></pre>
<p>fails and I have no clue why.</p>

googlebot: <p>replace sample() with rsample(), documentation describes reparametrizable sampling I believe.</p>

nierth: <p>thanks a lot, that fixed it!</p>
Size mismatch when running locally but not on Colab

maroov: <p>Hi</p>
<p>I’ve been running my code in Colab for a couple months and now I’m trying to execute the same code on a local file. However, I’m getting this:</p>
<p><code>RuntimeError: size mismatch, m1: [16 x 3], m2: [8192 x 3] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283</code></p>
<p>On a line that runs <strong>perfectly fine</strong> on Colab (m2 is 3x8192). I have checked my versions and everything is up to date. I can and will probably track this down and transpose the tensor so that there is no missmatch, but I just don’t get why the tensor shape changes when using the same in Colab notebooks and in a local .py file.</p>
<p>Any ideas?</p>
<p>EDIT: Adding traceback in case it helps</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "SFA_Single_Feat.py", line 242, in &lt;module&gt;
    zt = sfa(xt)
  File "/home/maroov/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/maroov/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/maroov/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/maroov/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/maroov/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1610, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [16 x 3], m2: [8192 x 3] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283
</code></pre>

alx: <p>go on colab, export the libraries you are using and their versions with:</p>
<pre><code class="lang-auto">pip freeze &gt; requirements.txt
</code></pre>
<p>From there, pip install the libraries in the txt locally.</p>
<p>Clear GPU cache, re-run and it should work if its library related.</p>

maroov: <p>Okay so in the end it was not related to libraries, it was just a typo in a parameter (yay!). It is working now.</p>
Pass parameter to next layer

VolhaL: <p>Hi</p>
<p>How can I pass weights from the previous layer to the next layer?<br>
Namely, I want to do dynamic parameter passing</p>
<p>I tried this strategy, but it doesn’t work. I get error ValueError: optimizer got an empty parameter list</p>
<pre><code>def __init__(self):
    self.register_parameter('weight', None)
def reset_parameters(self, input):
    self.weight = nn.Parameter(input)    
def forward(self, input):
    if self.weight is None:
        self.reset_parameters(input)
    return self.weight</code></pre>

Kushaj: <p>You would have to create the optimizer for it to work. After you created the optimizer for the first time, then you cannot add new parameters to the optimizer without creating a new optimizer.</p>

Caruso: <p>You can manually set the weights parameters equal to the weights from another layer (if the shapes are the same!):</p>
<pre><code class="lang-auto">l1 = nn.Linear(5, 5)
l2 = nn.Linear(5, 5)

l2.weight = l1.weight
l2.bias= l1.bias
</code></pre>
<p>The gradients will be calculated for both weight parameters together!</p>
How do you properly clone a tensor without affecting training?

darylnak: <p>I am trying to create a custom loss function. I have some tensor <code>x</code> and I need to make a duplicate so I can manipulate the values without affecting the original tensor and whatever computation that goes on in the background. When I am done manipulating the copy, I perform <code>log_softmax(x_copy)</code>, use <code>gather(...)</code> to select one element in each row that are relevant for my loss, then compute the loss by returning the <code>mean()</code> of those “gathered” elements. Right now, my model is not performing as well as I would expect, so I wanted to make sure that I am duplicating <code>x</code> properly. Here are some methods I have tried:</p>
<pre><code class="lang-auto">x_copy  = torch.tensor(x)
x_copy  = torch.empty_like(x).copy_(x)
x_copy = x.clone().detach() # This seems to be preferred?
</code></pre>
<p>I was told by someone that <code>clone()</code> will destroy the gradients of <code>x</code> (?), so I was recommended that I create a new tensor with the same dimensions as <code>x</code>, then copy the values over to keep the two tensors entirely separate. Is this recommendation correct and am I duplicating the tensor properly?</p>

Kushaj: <aside class="quote no-group" data-username="darylnak" data-post="1" data-topic="82721">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a87d85/48.png" class="avatar"> darylnak:</div>
<blockquote>
<p>I was told by someone that <code>clone()</code> will destroy the gradients of <code>x</code> (?)</p>
</blockquote>
</aside>
<p>It is not true. Check this example.</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.tensor(1., requires_grad=True)
b = 2*a
b.backward()

a.grad
# 2.

a_ = a.clone()

a.grad
# 2. 
</code></pre>
<p>Also, I think you need to do a backward pass through the tensor that you cloned. So you should not use <code>.detach()</code> as that sets the <code>requires_grad=False</code> of the returned tensor.</p>

googlebot: <p>copy_ still does backprop from x_copy to x (as you can see with x_copy.grad_fn being CopyBackwards)</p>
<p>detach() is opposite, and <em>probably</em> does what you want (x - trainable graph branch, x_copy - auxiliary non-trainable w.r.t. x)</p>
<p>clone() in addition to detach() is only nessesary if detached branch does inplace operations on x_copy, otherwise any operation, e.g. x.detach() * z allocates new memory and initial cloning has no effect</p>
How ResNets and DenseNets speedup the training

aditya_raj: <p>Please Help!!<br>
I am back again with a silly question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Please help me to understand—</p>
<p>How ResNets and DenseNets accelerate the speed of training of the deep networks?</p>

Kushaj: <p>They have fewer parameters compared to VGG model. Also, they are just better architectures meaning they can generalize faster.</p>
Gradient or Activation information from Saved Optimizer's State Dict

Kale-ab_Tessera: <p>I have saved the weights and optimizers state dict following from - <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>.</p>
<p>Can I get gradient or activation information from the optimizer’s state dict? What other information can I get about the network behaviour from the optimizers state dict?</p>
<p>I do have the weights from the model’s state dict, but I am trying to get other information such as activations or gradients.</p>

Kale-ab_Tessera: <p>I understand that the optimizer’s state dict has a <code>state</code> variable, which contains optimization state and this is different for each optimizer.</p>
<p>I am trying to do this accross the optims - Adam, Adagrad, SGD, SGD with momentum and Rmsprop. Do I simply get the state info and use that to compare optimizers e.g. <code>state['exp_avg']</code> (from Adam) and compare it to <code>state['sum']</code> (from Adagrad)?</p>

Kushaj: <aside class="quote no-group" data-username="Kale-ab_Tessera" data-post="2" data-topic="82826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kale-ab_tessera/48/11407_2.png" class="avatar"> Kale-ab_Tessera:</div>
<blockquote>
<p>Do I simply get the state info and use that to compare optimizers e.g. <code>state['exp_avg']</code> (from Adam) and compare it to <code>state['sum']</code> (from Adagrad)?</p>
</blockquote>
</aside>
<p>This will not work. Both these states carry different meanings specific to the optimizer. Also, these values are largely dependent on the training process.</p>
<aside class="quote no-group" data-username="Kale-ab_Tessera" data-post="1" data-topic="82826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kale-ab_tessera/48/11407_2.png" class="avatar"> Kale-ab_Tessera:</div>
<blockquote>
<p>Can I get gradient or activation information from the optimizer’s state dict?</p>
</blockquote>
</aside>
<p>No. This information is stored in the weights of the model, as <code>weight.grad</code>.</p>
Difference between CNN and Autoencoder

Mike2004: <p>Hi, can someone explain me, the difference between a CNN and an autoencoder?<br>
Thanks.</p>

Kushaj: <p>CNN is used to process an image i.e. you take an image and pass it through a CNN. For example, you pass <em>3x224x224</em> size images to the CNN (3 = num_channels (RGB), 224x224 = height x width of image). At the end of a CNN you get something like <em>512x7x7</em> size feature map (which is also like RGB image except it has 512 channels instead of 3). Rest everything is same, so you are going from 3 channels to 512 channels and also along the way you are reducing your image size from 224x224 to 7x7.</p>
<p>Why we are doing that? The main idea is that the 512 channels produced at the end would carry important information/features about the input image which would allow us to do image recognition, object detection and many more things.</p>
<p>Autoencoders. Your aim to learn the input data distribution. If we have to extend the CNN case from above. After getting the 512x7x7 feature map, we can now repeat the process in inverse i.e. go from 512x7x7 -&gt; 2x224x224. And you have got your autoencoder. So autoencoder in short does something as <code>3x224x224 -&gt; 512x7x7 -&gt; 3x224x224</code>.</p>
<p>Note: It is an oversimplified example.</p>
<p>Why we are doing this? Because using this method allows the autoencoder to learn important information about the input distribution, thus allowing you to reconstruct the input image. (This is different from GAN, an autoencoder cannot generate new samples it can only reconstruct the information it is trained on).</p>

Mike2004: <p>Thank you for the detailed explaination. I just wanted to know more about this.</p>
How to debug your code when using dataparallel

ray342659093: <p>Normally, it does not print out in which line the error raised after using nn.DataParallel. So, how do you guys to locate the bug in your code when using nn.DataParallel.</p>

WERush: <p>It is strange. I did not encounter this kind of things. Can you give the detailed error information.</p>

ray342659093: <blockquote>
<p>Traceback (most recent call last):<br>
File “test_fusion.py”, line 139, in <br>
rst = eval_video((i, data, label))<br>
File “test_fusion.py”, line 123, in eval_video<br>
rst = net(input_var)<br>
File “/home/rusu5516/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 224, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/rusu5516/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py”, line 60, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/home/rusu5516/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py”, line 70, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/home/rusu5516/.local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 67, in parallel_apply<br>
raise output<br>
RuntimeError: invalid argument 3: sizes do not match at /pytorch/torch/lib/THC/generated/…/generic/THCTensorMathPointwise.cu:217</p>
</blockquote>
<p>like this error message. I have no idea in which layer of my model raise this error. but if I do not use DataParallel It would show in which line of my code produce this error</p>

Andrei_Pokrovsky: <p>Try disabling parallel and debugging single-treaded first.</p>

arushi_019: <p>I am also using nn.DataParallel and getting expected shape errors. In my case, my model gives OOM errors when I run it on single GPU or CPU. Therefore, I tried to use nn.DataParallel but I cannot debug where the expected shape error is coming from. How to approach in this case?</p>