akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Polynomial evaluation by Horner rule?

username: f3ba
cooked: <p>Is there a polynomial evaluation function in PyTorch? Similar to <a href="https://www.tensorflow.org/api_docs/python/tf/math/polyval" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/math/polyval</a> in Tensorflow, that automatically broadcasts over tensors?</p>

username: anantguptadbl
cooked: <p>You can have something like this</p>
<pre><code class="lang-auto"># POLYVAL in PYTORCH
coeffs=torch.FloatTensor([1.,2.,1.,1.])
x=torch.FloatTensor([3.0])

def getPolyVal(x,coeffs):
    curVal=0
    for curValIndex in range(len(coeffs)-1):
        curVal=(curVal+coeffs[curValIndex])*x[0]
    return(curVal+coeffs[len(coeffs)-1])

getPolyVal(x,coeffs)
</code></pre>

username: f3ba
cooked: <aside class="quote no-group" data-username="anantguptadbl" data-post="2" data-topic="67124">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anantguptadbl/48/17784_2.png" class="avatar"> anantguptadbl:</div>
<blockquote>
<pre><code class="lang-auto"># POLYVAL in PYTORCH
coeffs=torch.FloatTensor([1.,2.,1.,1.])
x=torch.FloatTensor([3.0])

def getPolyVal(x,coeffs):
    curVal=0
    for curValIndex in range(len(coeffs)-1):
        curVal=(curVal+coeffs[curValIndex])*x[0]
    return(curVal+coeffs[len(coeffs)-1])

getPolyVal(x,coeffs)
</code></pre>
</blockquote>
</aside>
<p>Thanks.</p>
<p>Any tricks I can do to make this as fast as possible?</p>
Time cost on data load and traning

username: Mabinogiysk
cooked: <p>I have done some experiments about the time cost on data load and model training.</p>
<ol>
<li>
<p>Load data with torch.utils.data.DataLoader and train model.<br>
Time cost on dataload:  14.0 ms<br>
Time cost on training(dataload+model traning): 1.845 ms</p>
</li>
<li>
<p>Almost all time cost on data.to(device) during loading data, so I tried another way to load data. I read data to a queue in other process, then I collected those data into a batch and executed data.to(device) in other thread and push them to a queue. In main thread, I access to those data by q.get().</p>
</li>
</ol>
<p>Time cost on dataload:  &lt; 0.1 ms<br>
Time cost on training(dataload+model traning): 1.885 ms</p>
<p>My question is, why dose method 2 cost more time on training?<br>
I guess it is because the new thread slow down the main thread, but torch.utils.data.DataLoader load data with multithreading as well.</p>

username: ptrblck
cooked: <p>Could you post the code you’ve used to benchmark the data loading?<br>
How large is each <code>data</code> tensor you push to the device?<br>
Based on your description it seems the <code>to()</code> call is quite expensive and I would like to make sure you are indeed timing the right operations.<br>
Note that CUDA operations are asynchronous, so you would need to synchronize via <code>torch.cuda.synchronize()</code> before starting and stopping the timer.<br>
Otherwise some operations might create a synchronization point and thus collect the execution time of the currently executed async calls.</p>

username: Mabinogiysk
cooked: <p>data size: [16, 3, 473, 473]</p>
<p>In method 1:</p>
<pre><code class="lang-auto">t0 = time.time()
batch = next(trainloader)
images, labels, _, _ = batch
images = images.to(device)
labels = labels.long().to(device)

t.append(time.time() - t0)
if i_iter % 20 == 0:
    print('batch read time: ', sum(t[-20:]) / 20)

###
    ....
    model forward and backward
    ....
###
t1.append(time.time() - t0)
if i_iter % 20 == 0:
    print('model train time: ', sum(t1[-20:]) / 20)

</code></pre>
<p>In method 2：</p>
<pre><code class="lang-auto">t0 = time.time()
batch = q.get() # .to() has been executed in other thread before
images, labels, _, _ = batch

t.append(time.time() - t0)
if i_iter % 20 == 0:
    print('batch read time: ', sum(t[-20:]) / 20)

###
    ....
    model forward and backward
    ....
###
t1.append(time.time() - t0)
if i_iter % 20 == 0:
    print('model train time: ', sum(t1[-20:]) / 20)
</code></pre>
<p>I have not ues <code>torch.cuda.synchronize()</code>  before starting and stopping the timer.</p>
Assiging a non contiguous slice of a tensor to another tensor

username: antspy
cooked: <p>I have a two dimensional tensor A and a list of indices b. I want to assign the sub-matrix defined by the indices in b to some other values, contains in a tensor c.</p>
<p>If the indices b are contiguous, this is easy. For example suppose b contains indices from 0 to 3, then I can do:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.zeros(5, 5)
&gt;&gt;&gt; c = torch.ones(3, 3)
&gt;&gt;&gt; a[0:3, 0:3] = c
&gt;&gt;&gt; a

 1  1  1  0  0
 1  1  1  0  0
 1  1  1  0  0
 0  0  0  0  0
 0  0  0  0  0
[torch.FloatTensor of size 5x5]
</code></pre>
<p>which is what I want. But what if the indices in b are not contiguous? That is, what is b = [0, 2, 4] or b=[0,2,3]?<br>
Doing a[b, b] returns something that is not the sub-matrix I thought. Right now I am doing</p>
<pre><code class="lang-auto">&gt;&gt;&gt; b = [0, 2, 4]
&gt;&gt;&gt; for i in range(len(b)):
...  for j in range(len(b)):
...   a[b[i], b[j]] = c[i, j]
... 
&gt;&gt;&gt; a

 1  0  1  0  1
 0  0  0  0  0
 1  0  1  0  1
 0  0  0  0  0
 1  0  1  0  1
[torch.FloatTensor of size 5x5]
</code></pre>
<p>But is way too slow. Is there a better way?</p>

username: ptrblck
cooked: <p>The the indices evenly spaced?<br>
If so, you could try:</p>
<pre><code class="lang-auto">a = torch.zeros(5, 5)
a[::2, ::2] = 1
print(a)
&gt; tensor([[ 1.,  0.,  1.,  0.,  1.],
          [ 0.,  0.,  0.,  0.,  0.],
          [ 1.,  0.,  1.,  0.,  1.],
          [ 0.,  0.,  0.,  0.,  0.],
          [ 1.,  0.,  1.,  0.,  1.]])
</code></pre>

username: antspy
cooked: <p>Hi,</p>
<p>no they are not - they are random indices</p>

username: livenletdie_gvs
cooked: <p>How about index_copy_, index_put_ or scatter? They all take index copying of data.</p>

username: antspy
cooked: <aside class="quote no-group" data-username="livenletdie_gvs" data-post="4" data-topic="18899">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f08c70/48.png" class="avatar"> livenletdie_gvs:</div>
<blockquote>
<p>index_copy_</p>
</blockquote>
</aside>
<p>If I am not mistaken, index_copy only works with one dimension. Here I want to slice across two dimensions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: justusschock
cooked: <p>You could flatten the tensor and recalculate the indices and afterwards reshape it again</p>

username: noep
cooked: <p>So if you want to select the submatrix given rows and columns, with possibly non continuous indexes and not evenly spaced, the clearest way I found is generating the indexes using <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.ix_.html" rel="nofollow noopener">np.ix_</a>.</p>
<pre><code class="lang-auto">sub_matrix = matrix[np.ix_(wanted_rows, wanted_columns)]
</code></pre>
<p>Sorry for bump, I think I found a good solution and this poped as first google result.</p>
Conv-like linear layer

username: anima
cooked: <p>Consider convolutional layers. They create filters and pass them through an input, according to its stride and padding parameters. The filter size is the kernel size.<br>
Now, consider a new form of linear layer, like the convolutional layer. Each node is connected to a specific area in the image, like the kernel. Nodes look at sensing regions with ‘stride’ apart. I would like to create such a linear layer, not fully connected and very similar to a convolutional layer. How could I do it? Are there a library you could refer me to? Thank you,</p>

username: tom
cooked: <p>Use <code>F.fold</code> to get the image as the patches just like for convolutional layers (it will be of shape <code>batch x number of patches x size of a patch</code>, then you implement your operation, and <code>F.unfold</code> to get back. The documentation (either for the functional or the modular interface) has an example for convolution.</p>
<p>Best regards</p>
<p>Thomas</p>

username: anima
cooked: <p>fold/unfold could be a start if I was trying to pass patches through the same linear layer, however patches have to pass through different weights, unlike cnn where filter weights are shared. Do  you have any further ideas?<br>
Thank you for your reply,</p>

username: tom
cooked: <p>This forum can be less full service than you might expect, but:</p>
<p><code>unfold/fold</code> will get you the patches. If you then reshape to <code>batch x (number_of_patches * patch_size)</code> you can apply <code>linear</code> with different weights.<br>
The technical term for this type of layer is “locally connected layer” by the way.</p>
<p>Best regards</p>
<p>Thomas</p>
Loading datasets in Pytroch using Google Colab

username: SurajSubramanian
cooked: <p>Loading datasets in Google Colab takes more time than being done locally in Jupyter Notebook</p>
<pre><code class="lang-auto">
|-- datasets
|   |-- train_folder
|   |   |-- 00
|   |   |    |-- 0
|   |   |    |   |-- file2161.jpg
|   |   |    |   |-- file2162.jpg
|   |   |    |   |-- file2163.jpg
|   |   |    |   |-- file2164.jpg
|   |   |      
|   |-- test_folder
|   |   |-- 01
|   |   |    |-- 1
|   |   |    |   |-- file1161.jpg
|   |   |    |   |-- file1162.jpg
|   |   |    |   |-- file1163.jpg
|   |   |    |   |-- file1164.jpg
|   |   |  

</code></pre>
<p>This is the organization of my dataset.</p>
<pre><code class="lang-auto">class SmileDataset(Dataset):
    def __init__(self, data_root):
        self.samples = ImageFolder(data_root)

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx1):
        return self.samples[idx1]

ds_rude = SmileDataset('../mindnotix/smile-detection-master/datasets/train_folder/00')
ds_smile = SmileDataset('../mindnotix/smile-detection-master/datasets/train_folder/01')
trans1 = transforms.ToTensor()
ds_rude  = [trans1(img) for img,l in ds_rude]
ds_smile = [trans1(img) for img,l in ds_smile]
ds_rude, ds_smile = torch.stack(ds_rude), torch.stack(ds_smile)
</code></pre>
<p>I used the above code to load the dataset using Pytorch.<br>
When I run the code locally in Jupyter Notebook, it runs within a few seconds since there are only 2000 images in both the train and test folders.<br>
But when I run it in Google Colab, it takes several minutes.</p>
<p>Is there anyway to speed up the data-loading in Google Colab.<br>
I did change the runtime to GPU, but that doesn’t speed it up.</p>
<p>Thanks in Advance !</p>
How can I stack tensors like this?

username: 111226
cooked: <p>I’m trying to stack tensors with different size. But only dimension 2 is different while other dimensions are all same. Is there any way to do this?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7aed3abcffab1e5225a3d0b0a1914f22f7a865e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7aed3abcffab1e5225a3d0b0a1914f22f7a865e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7aed3abcffab1e5225a3d0b0a1914f22f7a865e.png" alt="image" data-base62-sha1="qcVWoy3p70yco7RLMaHwDF0BuXA" width="690" height="406" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">922×543 15.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>torch.cat</code> should work:</p>
<pre><code class="lang-python">a = torch.randn(5, 5, 1)
b = torch.randn(5, 5, 4)
c = torch.cat((a, b), dim=2)
print(c.shape)
&gt; torch.Size([5, 5, 5])
</code></pre>

username: 111226
cooked: <p>thanks bro, you saved me <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
Fine tune only parameters in optimizer

username: shaunlipy
cooked: <p>Hi, I am wondering when fine tune only part of parameters, what would happen if we only append parameters (to be trained) in the optimizer, but don’t set other parameters’ parameter.requires_grad = False.</p>
<p>Normally we would have:</p>
<pre><code class="lang-auto">model.feature1.requires_grad = False # Not training
model.feature2.requires_grad = True # to be trained

optimizer = Adam(model.feature2.parameters(), ...)
</code></pre>
<p>However, what if I do:</p>
<pre><code class="lang-auto">model.feature1.requires_grad = True # Not training
model.feature2.requires_grad = True # to be trained

optimizer = Adam(model.feature2.parameters(), ...)
</code></pre>
<p>Would this be ok?</p>

username: ptrblck
cooked: <p>We had a similar discussion <a href="https://discuss.pytorch.org/t/passing-params-to-an-optimizer/63191/2">here</a>.<br>
I would recommend to explicitly define how your training routine should work and thus would recommend to first approach.</p>

username: shaunlipy
cooked: <p>I see. Yeah, I saw your reply elsewhere: the “edge” case when having weight decay, if we append all parameters into the optimizer. But not sure if certain parameters are not included in the optimizer at all, would the “edge” case still happen. Thanks</p>

username: ptrblck
cooked: <p>If your current optimizer doesn’t have access to some parameters, it won’t be able to update them.<br>
However, gradients would still be accumulated in these “unused” parameters, so if you decide after a while to create a new optimizer and pass all parameters, your training might blow up.</p>
Context management in libtorch

username: Erissonleo
cooked: <p>In pytorch, we can use with torch.no_grad() to realize context management. Is there a similar method in Libtorch in C++ ? Thanks!</p>

username: ptrblck
cooked: <p><code>NoGradGuard</code> should do the same as <code>with torch.no_grad()</code> as explained <a href="https://discuss.pytorch.org/t/how-does-nogradguard-works-in-cpp/34960">here</a>.</p>

username: Erissonleo
cooked: <p>Thanks a lot, I will try it.</p>
What is the difference between b and e?

username: DKNT
cooked: <p>In the autograd doc <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.is_leaf" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.1 documentation</a>, I found the following examples:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/724b05baf93fa25725bc9cf4e5a4a134d10b3431.png" data-download-href="https://discuss.pytorch.org/uploads/default/724b05baf93fa25725bc9cf4e5a4a134d10b3431" title="Screen Shot 2020-01-19 at 2.59.22 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/724b05baf93fa25725bc9cf4e5a4a134d10b3431_2_690x381.png" alt="Screen Shot 2020-01-19 at 2.59.22 AM" data-base62-sha1="gj5ajrpPHlGTOeFyl1YtzFKwWhH" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/724b05baf93fa25725bc9cf4e5a4a134d10b3431_2_690x381.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/724b05baf93fa25725bc9cf4e5a4a134d10b3431_2_1035x571.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/724b05baf93fa25725bc9cf4e5a4a134d10b3431_2_1380x762.png 2x" data-dominant-color="ECEDF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-01-19 at 2.59.22 AM</span><span class="informations">3812×2108 497 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I wonder what is the difference between b and e here? Why is e not a Tensor created by the operation that casts a cpu Tensor into a cuda Tensor, just like b?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The last operation on <code>e</code> is the inplace operation <code>.requires_grad_()</code>, which will create the leaf.<br>
In <code>b</code> you are creating a leaf tensor first, but are applying another operation on it (<code>.cuda()</code>).</p>
Can I use model.apply with argument in function?

username: hadaev8
cooked: <p>Something like</p>
<pre><code>def set_dropout(m, p):
    if isinstance(m, nn.Dropout2d):
        m.p = p
netG.apply(set_dropout(0.1))</code></pre>

username: ptrblck
cooked: <p>Yes, you can use a <code>lambda</code> for this:</p>
<pre><code class="lang-python">def set_dropout(m, p):
    if isinstance(m, nn.Dropout):
        m.p = p

model.apply(lambda m: set_dropout(m, 0.1))
</code></pre>
<p>Note that you are checking for <code>nn.Dopout2d</code>, while the “vanilla dropout” is <code>nn.Dropout</code>.<br>
If you are using <code>nn.Dropout2d</code> in your model, please ignore this sentence. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How could I remove certain layer in torchvision.models

username: coincheung
cooked: <p>I would like to use the torchvision models without some of their layers. For example, I may need to use the model without its last nn.Linear layer, but I stiil need the remain part to work as the original torchvision.models. How could I do it ?</p>

username: jmandivarapu1
cooked: <p>Oneway to do it would be pull the source code from pytorch and change the layers as you want.</p>
<p>Eg: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py</a><br>
pull the resent and change resnet as you need</p>

username: pawandeep_singh
cooked: <p>The code given below is to download pre-trained Resnet152 and use it till second last layer.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models
from torch.autograd import Variable

# Pretrained resnet152 model till second last layer as feature extraction.
resnet152 = models.resnet152(pretrained=True)
modules=list(resnet152.children())[:-1]
resnet152=nn.Sequential(*modules)
for p in resnet152.parameters():
    p.requires_grad = False

# Get resnet features for random image_
img = torch.Tensor(3, 224, 224).normal_() # random image
img = torch.unsqueeze(img, 0)  # Add dimension 0 to tensor  
img_var = Variable(img) # assign it to a variable
features_var = resnet152(img_var) # get the output from the last hidden layer of the pretrained resnet
features = features_var.data # get the tensor out of the variable
</code></pre>
<p>For more variants of pretrained models go to <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">Models</a>.</p>

username: ToughMind
cooked: <aside class="quote no-group" data-username="pawandeep_singh" data-post="3" data-topic="23303">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pawandeep_singh/48/6288_2.png" class="avatar"> pawandeep_singh:</div>
<blockquote>
<p>for p in resnet152.parameters(): p.requires_grad = False</p>
</blockquote>
</aside>
<p>I tried and got error:</p>
<pre data-code-wrap="python"><code class="lang-python">TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple. 
</code></pre>
<p>Please help, thanks!</p>
Implementing Excitation Backprop

username: thesemicolonguy
cooked: <p>I’m implementing excitation backprop (mwp and c-mwp). I’ve read and understood the paper quite well. I’m not able to figure out an efficient way to write a backward pass function for pre-trained models like vgg, inception and resnet.</p>
<p>One way is to store all activations and weights during the forward pass, and then do the backward (top-down) calculations outside the model. Although this will work, I think there must be some better way to do this instead of manually storing the weights and activations. Since they are already available in the model, I don’t see any point in storing them outside the model.</p>
<p>Any help would be appreciated, thanks!</p>

username: yulongwang12
cooked: <p>I implemented excitation backprop and other visual saliency methods in <a href="https://github.com/yulongwang12/visual-attribution" rel="nofollow noopener">https://github.com/yulongwang12/visual-attribution</a> (although they are based on pytorch 0.2 :-/ ). Hope it helps</p>
How to pick the diag values for selected dimensions?

username: AlexisW
cooked: <p>Suppose I have a tensor of [10, 2, 2]. I want to pick the diag vector of dim 1 and 2, i.e., make the tensor to [10, 2]. Is there an easy way to do so? The <code>.diag</code> method doesn’t seem to work unless I iterate the 10 instances.</p>

username: Eta_C
cooked: <p>Compared with using <code>for</code>, I have a faster implementation.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

n, k = 10, 2
x = torch.rand(n, k, k)
y = x.reshape(n, -1)[:, 0::(k + 1)]

for i in range(10):
    assert (y[i] == torch.diag(x[i])).all()
print("finish")
</code></pre>
<p>The following code will also work.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

n, k = 10, 2
x = torch.rand(n, k, k)
y = x[torch.eye(k).repeat(10, 1, 1).bool()].reshape(n, -1)

for i in range(10):
    assert (y[i] == torch.diag(x[i])).all()
print("finish")
</code></pre>
[ImageNet]A question about Data Augmentation

username: cclover
cooked: <p>The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random <strong>224 × 224</strong> patches (and their horizontal reflections) from the<br>
256×256 images and training our network on these extracted patches4. This increases the size of our<br>
training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have<br>
forced us to use much smaller networks. At test time, the network makes a prediction by extracting<br>
five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal<br>
reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax<br>
layer on the ten patches.</p>
<p>I do not know why make the input images are <strong>224x224x3</strong><br>
<strong>The passage is from "ImageNet Classification with Deep Convolutional</strong><br>
<strong>Neural Networks "of 4.1</strong></p>

username: ptrblck
cooked: <p>My best guess is that this shape is somehow the “standard” for ImageNet-related models.<br>
AlexNet used it and a lot of models adapted the shape.</p>
<p>The models were originally written in the sense that the input shape had to be constant, which would also explain a specific shape which “just works”.<br>
Today you would most likely find model implementations with adaptive pooling layers, which allow different input shapes.</p>
VAE example: UserWarning on `binary_cross_entropy`

username: gon1332
cooked: <p>I’m using torch of version 1.3.1+cpu and torchvision version of 0.4.2+cpu.</p>
<p>I’m using the VAE example in master ( <a href="https://github.com/pytorch/examples/commit/0c1654d6913f77f09c0505fb284d977d89c17c1a" rel="nofollow noopener">0c1654d</a>) but the interpreter results in this warning:</p>
<pre><code class="lang-auto">/home/gon1332/Development/Training/ML/learning-data-augmentation/model.py:147: UserWarning: Using a target size (torch.Size([128, 784])) that is different to the input size (torch.Size([128, 20])) is deprecated. Please ensure they have the same size.
  BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')
Traceback (most recent call last):
  File "main.py", line 38, in &lt;module&gt;
    main()
  File "main.py", line 34, in main
    ...
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')
  File "-/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2058, in binary_cross_entropy
    "!= input nelement ({})".format(target.numel(), input.numel()))
ValueError: Target and input must have the same number of elements. target nelement (100352) != input nelement (2560)
</code></pre>
<p>x and x_recon have the below sizes:</p>
<pre><code class="lang-auto">x_recon.size() = torch.Size([128, 20])
x.size() = torch.Size([128, 1, 28, 28]) and with x.view(-1, 128) torch.Size([128, 784])
</code></pre>

username: ptrblck
cooked: <p>The printed shapes correspond to the shape mismatch mentioned in the error message.<br>
How would you like to compute the binary cross entropy with a different number of samples?</p>
<p>Did you change anything in the example code?<br>
I just retried it with the latest nightly build and it seems to work without any errors.</p>
How to incorporate a dataset in json format to dataloader?

username: TinfoilHat0
cooked: <p>I’ve an image dataset given as bunch of json files. Each key is a class label and the corresponding value is a subset of images that have this label, represented as a list of integers.</p>
<p>How should I pre-process and save this data such that I can use DataLoader to fetch batches of it during training?</p>

username: ptrblck
cooked: <p>Based on your description it seems you have multiple json files, where each file might contain the same class indices as keys with the corresponding image samples.<br>
If that’s the case, indexing into these file would work using the class label, which would have the drawback of a possibly complicated calculation of the sample index.</p>
<p>You could iterate your json files once and store each image in a subfolder corresponding to the class index. Using this approach you could simply get all the image paths and index them directly to lazily load the data inside your <code>Dataset</code>.</p>
Pytorch 1.1 load model from pytorch 0.4

username: sofzh
cooked: <p>Hey guys, Recently I met a problem.<br>
The case is I want to load a model with pytorch 1.1 , the model is trained by pytorch 0.4. I cannot make it to load the model correctly.<br>
Can anybody meet this problem before, how to solve this problem?<br>
Thx for ur kindly help~</p>

username: ptrblck
cooked: <p>Do you get a specific error message?<br>
<code>0.4.0</code> is quite old by now, so loading the model might not work out of the box, but we would need some kind of error in order to help debugging.</p>

username: sofzh
cooked: <p>Sry for missing the error message.<br>
Here is the meaasge goes:   “‘LogSoftmax’ object has no attribute ‘_state_dict_hooks’”</p>

username: ptrblck
cooked: <p>Are you trying to load the <code>state_dict</code> or the model via <code>torch.load('model.pyt')</code> when getting this error?</p>

username: sofzh
cooked: <p>Yes. I load model weights from ‘***.pth’ , then when I replace ‘module.’ with ‘’ , this error occured .</p>

username: ptrblck
cooked: <p>The replacement of <code>.module</code> points to loading a <code>state_dict</code> from a <code>nn.DataParallel</code> model and should most likely be unrelated to the error.<br>
Do you see any values in your loaded <code>state_dict</code> for the <code>LogSoftmax</code> layer?</p>
How to extract just one (random) batch from a data loader?

username: f3ba
cooked: <p>I constructed a data loader like this:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', transform=data_transforms, train=True, download=True),
    batch_size=batch_size, shuffle=True)
</code></pre>
<p>Now I want to extract one batch. I tried <code>next(train_loader)</code> but it doesn’t work. What can I do?</p>

username: f3ba
cooked: <p>Nevermind, found the answer here!</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/1917" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/1917" target="_blank" rel="nofollow noopener">Get a single batch from DataLoader without iterating</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-06-26" data-time="19:35:08" data-timezone="UTC">07:35PM - 26 Jun 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-06-26" data-time="20:05:30" data-timezone="UTC">08:05PM - 26 Jun 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/narendasan" target="_blank" rel="nofollow noopener">
          <img alt="narendasan" src="https://avatars2.githubusercontent.com/u/1790613?v=4" class="onebox-avatar-inline" width="20" height="20">
          narendasan
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Is it possible to get a single batch from a DataLoader? Currently, I setup a for loop and return a batch...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Should I use Module as parameter container (with no forward method)?

username: f3ba
cooked: <p>I need to define a class <code>Model</code> that contains my model parameters, but for which it makes no sense to define a <code>forward</code> method. Instead, I’ll define other modules which can read the parameters from <code>Model</code> and which then define their <code>forward</code> methods.</p>
<p>Is it natural that <code>Model</code> inherits from <code>Module</code> in this case (even though it doesn’t implement <code>forward</code>)?</p>
<p>Inheriting from <code>Module</code> helps, because it means that the parameters are tracked automatically and I can compose several <code>Model</code> components together easily.</p>
<p>What do you think?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think you can yes. Note that you can also make this module a Submodule of your other modules that do the forward.</p>

username: f3ba
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="67104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Note that you can also make this module a Submodule of your other modules that do the forward.</p>
</blockquote>
</aside>
<p>If by this you mean that module <code>A</code> contains the model module as an attribute, then yes, that’s what I have in mind.</p>
Why cannot the defined Variable be computed by nueral network?

username: 999999999
cooked: <p><strong>the  neural network is defined as</strong><br>
class networka(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(networka,self).<strong>init</strong>()<br>
self.network=nn.Sequential(<br>
nn.Linear(19,11),<br>
nn.ReLU(True),<br>
nn.Linear(11,3),<br>
nn.Softmax()<br>
)<br>
def forward(self, x):<br>
x=self.network(x)<br>
return x<br>
A=networka()</p>
<p><strong>This code works</strong></p>
<p>z = Variable(torch.randn(19,19))<br>
A(z)</p>
<p><strong>But this doesn’t</strong></p>
<p>X_data=(np.random.rand(19,19))<br>
X=Variable(torch.from_numpy(X_data))<br>
A(X)</p>
<p>Could anyone tell me why?</p>

username: ptrblck
cooked: <p>You should get an error message:</p>
<pre><code class="lang-python">RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm
</code></pre>
<p>which points to a type mismatch in an operation.</p>
<p>Numpy uses float64 by default, while PyTorch uses float32.<br>
You could either transform the data to float32 via <code>X = X.float()</code>, which would be the usual use case or alternatively you could transform the model parameters to float64 via <code>model = model.double</code>.</p>
<p>Also, <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code>, so don’t use them.</p>

username: 999999999
cooked: <p>Thank you very much for your help!</p>
<p>ptrblck via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a>于2020年1月19日 周日01:58写道：</p>
How to create embeddings for POS tags?

username: stuck
cooked: <p>Hi,</p>
<p>I want to create embeddings for POS tags and append them to my word embeddings. I use pre-trained Glove embeddings for words and I am wondering is there any way to create embeddings for POS tags instead of using one-hot vectors for them?</p>
<p>How can I do that?<br>
Can I train the Glove for only POS tags; if yes, how?</p>
<p>Any help would be appreciated,</p>
<p>Have a nice day!</p>
Transform resize not working

username: Saewon_Yang
cooked: <pre><code>    transform = transforms.Compose([transforms.Resize(224),
                                    transforms.ToTensor(),
                                    transforms.Normalize(mean = [0.5, 0.5, 0.5],
                                                         std = [0.5, 0.5, 0.5])])


    train_dataset = torchvision.datasets.ImageFolder(root=DATASET_PATH + '/train/train_data', transform=transform)

    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=2)



    print("Found {} number of mini-batches".format(len(train_loader)))

    t0 = time.time()
    total_step = len(train_loader)
    for epoch in range(nb_epoch):
        
        avg_loss = 0.0
        
        for i, (images, lables) in enumerate(train_loader):
            images = images.to(device)
            labels = lables.to(device)
</code></pre>
<p>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 475 in dimension 2 at /pytorch/aten/src/TH/generic/THTensorMath.cpp:3616</p>
<p>I tried to apply transform Resize and loaded during training and this error came up. It says that image sizes are different. How is this happening even though I resized all the images in the same size??<br>
Any advice would be welcome!</p>

username: vabh
cooked: <p>If your images are of different sizes and not squares, the smaller edge is matched to the given size, which means that in this case there is no guarantee that all images have all edges of the same size.<br>
See here: <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize</a></p>

username: Saewon_Yang
cooked: <p>Thanks for your reply. Then what else can I do when I load datasets?<br>
I’m working on image retrieval problem. So the image sizes are all different.<br>
for example, there are 400x800, 600x800, 700x400, 400x300 and many more.<br>
How can I decide the right size for the network??</p>

username: vabh
cooked: <p>I should’ve mentioned that you can create the transform as <code>transforms.Resize((224, 224))</code>. If you pass a tuple all images will have the same height and width.<br>
This issue comes from the dataloader rather than the network itself. When the dataloader creates the batches it expects all tensors to have the same shape.</p>

username: ronghui
cooked: <p>it’s not working for me</p>

username: kc_dharma
cooked: <p>you can do transforms.Resize((224, 224))</p>
Adam optimizer exp_avg_sq turns to infinity

username: sun_stealer
cooked: <p>After upgrading from 1.1.0 to 1.3.1 I started getting some weird problems during training related to ADAM optimizer.<br>
Essentially after a certain point, the model would stop training, as if learning rate was set to zero.<br>
Looking into Adam optimizer state revealed that exponential moving average of the gradient square is infinite for some tensors:</p>
<p>state[‘exp_avg’]<br>
tensor([[[[-1.7468e-03, -1.6368e-03, -1.2466e-03,  …, -1.4461e-03, -1.6768e-03, -9.0883e-04], …</p>
<p>st[‘exp_avg_sq’]<br>
tensor([[[[inf, inf, inf,  …, inf, inf, inf], [inf, inf, inf,  …, inf, inf, inf], [inf, inf, inf,  …, inf, inf, inf] …</p>
<p>I have never seen this problem in previous versions of PyTorch. Is this a bug or a numerical problem with my training procedure?</p>
<p>It might be related to this fix: <a href="https://github.com/pytorch/pytorch/pull/23737/files/b90bd775cf538381d7f5dc0327eb59937b2cca35#diff-c82300e6ba86ad06720f2bb4ecc658bd" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/23737/files/b90bd775cf538381d7f5dc0327eb59937b2cca35#diff-c82300e6ba86ad06720f2bb4ecc658bd</a><br>
EDIT: no, probably not related</p>

username: sun_stealer
cooked: <p>I managed to fix this by clipping the gradients with                             <code>torch.nn.utils.clip_grad_norm_(self.actor_critic.parameters(), self.cfg.max_grad_norm)</code></p>
<p>Unfortunately this slows down the code, and I didn’t have to do it before. Weird.</p>
Integrating LEAF benchmark datasets to PyTorch

username: TinfoilHat0
cooked: <p>I’ve been trying <em>simulate</em> federated learning (FL) with Pytorch.<br>
For FL, there’s a benchmark called <a href="https://leaf.cmu.edu/" rel="nofollow noopener">Leaf</a> which contains some datasets that are particularly suitable for non-iid data partition setting  that arises in FL context.</p>
<p>Is there an easy way to integrate the datasets from that benchmark to PyTorch? As far as I can see, they’re currently not available via torchvision datasets.</p>

username: ptrblck
cooked: <p>If you want to preload the complete dataset, you could pass it to a <code>TensorDataset</code>.<br>
On the other hand, if you are dealing with image data, which is stored as separate image files in folders corresponding to the classes, you could use <code>ImageFolder</code>.</p>
<p>Depending on the dataset and how each sample is stored, you might write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 7 7, but got 3-dimensional input of size [3, 224, 224] instead

username: kabilan
cooked: <pre><code class="lang-auto">from torch.autograd import Variable
import torch.nn.functional as F
from PIL import Image
data_transforms = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])
])


My code:
image='dudth.jpg'
image = Image.open(image).convert('RGB')
image=data_transforms(image)
image.unsqueeze(dim=0)
imgblob = Variable(image)
torch.no_grad()
predict = F.softmax(model_ft(imgblob))
print(predict)
</code></pre>

username: Eta_C
cooked: <p><code>image.unsqueeze(dim=0)</code> -&gt; <code>image = image.unsqueeze(dim=0)</code><br>
It is not an in-place operator.</p>
Torch.cuda.is_available() always returns false

username: MrLebovsky
cooked: <p>Hello there! I’m have video card Nvidia GeForce 650M.<br>
Also i have I have the following configuration:</p>
<pre><code class="lang-auto">Python version: 3.7.5
CUDA runtime version: 9.0.176
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip] numpy==1.17.4
[pip] torch==1.1.0
[pip] torch-nightly==1.2.0.dev20190723
[pip] torchfile==0.1.0
[pip] torchvision==0.3.0
[conda] blas                      1.0                         mkl    defaults
[conda] cuda92                    1.0                           0    pytorch
[conda] mkl                       2019.0                   pypi_0    pypi
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.15           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.1.0           py3.7_cuda90_cudnn7_1    pytorch
[conda] torch-nightly             1.2.0.dev20190723          pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.3.0               py37_cu90_1    pytorch
</code></pre>
<p>And finnaly  <code>torch.cuda.is_available()</code>  returns false. Looks like my video card is too old and is no longer supported?<br>
How can I solve this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Indeed, your card is not supported anymore in the binaries.<br>
You can try to compile pytorch from source to be able to use it.</p>

username: MrLebovsky
cooked: <p>So how i can do that?<br>
What version of libs I should be compile from sources?</p>

username: albanD
cooked: <p>You can find instructions <a href="https://github.com/pytorch/pytorch#from-source">on github</a> where you can find both the general and windows specific instructions.<br>
You should be able to compile from master.</p>

username: MrLebovsky
cooked: <p>But if assemble the library from the master branch, how will it differ from the ready-made compiled solutions?</p>

username: albanD
cooked: <p>Each GPU version requires different compiled code. So the binaries we provide do not support all the GPUs because that would make the binaries too big (they are already 500MB+). So we do not provide the least popular versions.<br>
But if you compile from source, it will detect your gpu automatically and make the compiled code for your GPU.</p>

username: MrLebovsky
cooked: <p>Finally after building from sources I have the following error:</p>
<blockquote>
<p>CUDA error: no kernel image is available for execution on the device</p>
</blockquote>

username: ptrblck
cooked: <p>Could you post the log of your build here, please?</p>

username: MrLebovsky
cooked: <p>The topic is no longer relevant. In order to solve the last problem it was necessary to rebuild the torch and set the TORCH_CUDA_ARCH_LIST parametr into setup.py. Thanks!</p>
Binary classification using datasets with different importance

username: a_c
cooked: <p>Hello,</p>
<p>I am built a NN to separate signal from background events. For the signal training I have a single dataset of 20000 events. For the background training I have instead 5 different sets, with a different number of events for each each set.<br>
I managed successfully to train and implement  a NN (based on MSELoss or BCELoss) when I concatenate the 5 background sets into a single set/tensor.</p>
<p>My problem is, however, that I need to treat the 5 different sets with 5 different weights which are associated to the “importance” (i.e. absolute normalization) of each set. To be clear: the set <span class="hashtag">#1</span> contains 1000 events but they are 10 times more important than those from set <span class="hashtag">#2</span> which contains 5000 events.</p>
<p>How can I take this “importance weight” into account using the standard torch losses?</p>
<p>Thanks</p>
<p>Andrea</p>

username: ptrblck
cooked: <p>You could use unreduced losses (reduction=‘none’), calculate the weight tensor for each element of your current batch and multiply it with the per-sample loss.<br>
<code>nn.BCEW(WithLogits)Loss</code> povides a <code>weight</code> argument, which can be used alternatively.</p>
How to use TensorRT backend to accelerate TorchScript

username: BobChen
cooked: <p>Can we directly use TensorRT to accelerate TorchScript without ONNX?</p>

username: ptrblck
cooked: <p>You could try to use <a href="https://github.com/NVIDIA-AI-IOT/torch2trt">tensor2trt</a> to directly use TensorRT with the Python API. However, as far as I know, the supported methods might be limited, so you would have to check, if your current model uses only supported layers.</p>
<p>A bit unrelated to your question, but might also be interesting: you could have a look ar <a href="https://devblogs.nvidia.com/how-to-deploy-real-time-text-to-speech-applications-on-gpus-using-tensorrt/">this blogpost</a> to see how to deploy Tacotron2 and Waveglow to TensorRT7.</p>

username: BobChen
cooked: <p>I will check it, thank you very much:)</p>
Expectation Maximization (EM) algo demo in Pytorch

username: DuaneNielsen
cooked: <p>Hi guys,</p>
<p>Here are some demos of how to implement the EM algorithm in Pytorch.</p>
<p>If anyone has also done this, would be super interested to see different takes on it.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/DuaneNielsen/45ac9f3a70cb6091a38984c35e372e6a" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/DuaneNielsen/45ac9f3a70cb6091a38984c35e372e6a" target="_blank" rel="nofollow noopener">https://gist.github.com/DuaneNielsen/45ac9f3a70cb6091a38984c35e372e6a</a></h4>
<h5>em_algo_2D.py</h5>
<pre><code class="Python">import torch
from torch import tensor
from torch.distributions.multivariate_normal import MultivariateNormal
import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse
import matplotlib.patches as mpatches
from math import acos, degrees
from matplotlib.pyplot import cm

"""</code></pre>
This file has been truncated. <a href="https://gist.github.com/DuaneNielsen/45ac9f3a70cb6091a38984c35e372e6a" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>1D example</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/DuaneNielsen/8c5bde8d35a46d60640d0579d913dcff" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/DuaneNielsen/8c5bde8d35a46d60640d0579d913dcff" target="_blank" rel="nofollow noopener">https://gist.github.com/DuaneNielsen/8c5bde8d35a46d60640d0579d913dcff</a></h4>
<h5>em_algo.py</h5>
<pre><code class="Python">import torch
from torch.distributions.normal import Normal
import matplotlib.pyplot as plt

"""
EM algo demo, in pytorch

"""

n = 40 # must be even number</code></pre>
This file has been truncated. <a href="https://gist.github.com/DuaneNielsen/8c5bde8d35a46d60640d0579d913dcff" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Is there a way to calculate Hessian with respect to activations for a small network like this?

username: gebrahimi
cooked: <p>Hi,</p>
<p>I need to calculate the hessian with respect to activations (intermediate values) for a small network like this? Is there a way to do that using pytorch?</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(2,2)
    self.s1 = nn.Sigmoid()
    self.fc2 = nn.Linear(2,2)
    self.s2 = nn.Sigmoid()
    self.fc1.weight = torch.nn.Parameter(torch.Tensor([[0.15,0.2],[0.250,0.30]]))
    self.fc1.bias = torch.nn.Parameter(torch.Tensor([0.35]))
    self.fc2.weight = torch.nn.Parameter(torch.Tensor([[0.4,0.45],[0.5,0.55]]))
    self.fc2.bias = torch.nn.Parameter(torch.Tensor([0.6]))
    
def forward(self, x):
    x= self.fc1(x)
    x = self.s1(x)
    x= self.fc2(x)
    x = self.s2(x)
    return x

net = Net()

data = torch.Tensor([0.05,0.1])
out = net(data)
target = torch.Tensor([0.01,0.99])  # a dummy target, for example
criterion = nn.MSELoss()
loss = criterion(out, target); loss</code></pre>