akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DCGAN outputs ordered garbage

Topsoil: <p>I have a dcgan training on a dataset of living room images of 256 * 256 * 3 size. But for some reason, even after 10 epochs, after some uniform grayscale at the beginning, its output looks like this:<br>
<a href="https://imgur.com/a/ObzqTrc" class="onebox" target="_blank" rel="noopener nofollow ugc">https://imgur.com/a/ObzqTrc</a></p>
<p>Losses (sampled each batch) look like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e3ca46b24553de462311bafc8628f9cd44e14cf.png" alt="Losses" data-base62-sha1="8SzwLYEuaxDrH0TrJQIvRHZBJcH" width="432" height="288"></p>
<p>Code is available at:<br>
<a href="https://github.com/Grubzerusernameisavailable/DCGAN-pytorch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/Grubzerusernameisavailable/DCGAN-pytorch</a></p>
<p>My guess is that something is wrong in my training code(Since i transfered from TF2.0 recently) and not completly sure that it is correct</p>

ptrblck: <p>I guess the <code>view</code> operations will interleave the image pixels, if you try to swap the dimensions with them.<br>
Check which shape each loaded sample has in your <code>ImagesDataset</code>.<br>
If the sample has the shape <code>[height, weight, channels]</code>, you should call <code>sample = sample.permute(2, 0, 1)</code> instead of <code>sample = sample.view(3,256,256)</code>.<br>
The same applies for the output of your model:</p>
<pre><code class="lang-python">test_image = gen(noise).cpu().detach()[0].view(256,256,3)
</code></pre>
<p>Which is most likely wrong, as PyTorch uses the layout <code>[batch_size, channels, height, width]</code>.</p>
Multiple Bounding Boxes in single image for training

Kartik_Chopra: <p>I am trying to train an YOLO Network with my own data. However, a single image can have multiple bounding boxes that belong to the same class. For instance, one of the text files is</p>
<p>0 0.262346 0.498264 0.002315 0.004630<br>
0 0.293210 0.516204 0.000772 0.001157<br>
0 0.343750 0.679977 0.003086 0.009259<br>
0 0.360725 0.621528 0.001543 0.001157<br>
0 0.437114 0.666667 0.001543 0.001157<br>
0 0.455633 0.214699 0.003086 0.004630<br>
0 0.516204 0.245370 0.002315 0.003472</p>
<p>Is there a way to feed such a structure in the network or is it one bounding box per image?</p>
Numerical instability? pytorch v1.4.0

sbelharbi: <p>hi,<br>
in the following code, i expect some sum to be exactly 1. but it is ALMOST 1. this happens on cpu as on gpu. it is more likely to be an issue of rounding. any idea  why this happens? and how to fix it? thanks</p>
<p><em>code</em>:</p>
<pre><code class="lang-python">import sys


import numpy as np
import torch
from torch.distributions.uniform import Uniform

low_b = 1e-10
upp_b = 0.95 / 9.
nbr = 10
n = 100

# for reproducibility
seed = 0
torch.manual_seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
# end reproducibility

print("Pytorch version:{}".format(torch.__version__))
print("Python version:{}".format(sys.version))

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("Device {}".format(device))

target = torch.zeros((n, nbr), device=device, dtype=torch.float32,
                     requires_grad=False)
dist_unif = Uniform(torch.tensor([low_b]), torch.tensor([upp_b]))

print("Start issue:")

for i in range(n):
    smooth = dist_unif.sample((nbr, )).squeeze()  # sample random numbers
    smooth[2] = 0.  # set position 2 to 0.
    target[i, :] = smooth
    target[i, 2] = 1. - smooth.sum()  # replace position 2 by: 1 - sum(all)
    # now, the sum(all) is expected to be 1. but it is not the case.
    # the sum is ALMOST 1. Numerical instability?
    print("SUM {}: {:.16f}".format(i, target[i].sum()))
    assert target[i].sum() == 1., "sum {:.16f} is not 1.".format(
        target[i].sum())
</code></pre>
<p><em>output CPU:</em></p>
<pre><code class="lang-auto">$ python issues.py 
Pytorch version:1.4.0+cu100
Python version:3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55) 
[GCC 7.3.0]
Device cpu
Start issue:
SUM 0: 0.9999999403953552
Traceback (most recent call last):
  File "issues.py", line 44, in &lt;module&gt;
    target[i].sum())
AssertionError: sum 0.9999999403953552 is not 1.
</code></pre>
<p><em>output GPU:</em></p>
<pre><code class="lang-auto">$ python issues.py 
Pytorch version:1.4.0+cu100
Python version:3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55) 
[GCC 7.3.0]
Device cuda:0
Start issue:
SUM 0: 1.0000001192092896
Traceback (most recent call last):
  File "issues.py", line 44, in &lt;module&gt;
    target[i].sum())
AssertionError: sum 1.0000001192092896 is not 1.
</code></pre>

albanD: <p>Hi,</p>
<p>You might want to check the <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">wikipedia</a> article about float numbers.<br>
The short answer is that single precision float have up to 6/7 digits of precision. Anything after that is noise.<br>
So here 1 - your_number &lt; 1e-6 means that from the point of view of single precision floats, they are the exact same number. You won’t be able to get more precision with them.</p>
<p>If you needs more precision, you can use double precision floats that will give you ~12 digits of precision.</p>

sbelharbi: <p>hi,<br>
thanks for the quick/helpful reply.</p>
Train with fp8 and fp16

Mason-Qin: <p>hi!<br>
I was attempting to train some part of my model with fp8 and fp16, and i’d like to ask:<br>
is there a way to achieve the 1-5-2 fp8 in some calculation, such as plus or mul?<br>
normally the fp16 is 1-5-10, is it possible to make it 1-6-9 or 1-4-11?<br>
thanks!</p>

ptrblck: <p>I don’t think there is an easy way of changing the exponent and mantissa of the underlying values.<br>
You could of course try to simulate these number formats, but I’m not sure how much work that would be.</p>

Mason-Qin: <p>yeah I found it difficult to change the exp and mantissa. now I tried to simulate the errors.<br>
Thanks anyway</p>

Mason-Qin: <p>hi,and I have one more question<br>
is there a easy way to extract the [neg, exponent, mantissa] of a torch.tensor in cuda? just like in numpy can do view() and some bitwise operation, which cannot work on GPU device.</p>

ptrblck: <p>If you have the float data in your CUDA method, you could use e.g. <code>frexpf</code> or some bit operations on the bits. There should be some examples, how this would work in C++/C on StackOverflow.</p>
Detach recurrent connections in RNN

alsuhr: <p>Is it possible to detach the recurrent connections of a <code>torch.nn.RNN</code> when calling <code>forward</code> on it?</p>
<p>I could do this with a <code>torch.nn.RNNCell</code>, like this:</p>
<pre><code class="lang-auto">rnn = nn.RNNCell(x, h)
input = torch.randn(l, b, x)
hx = torch.randn(b, h)
output = list()
for i in range(l):
    hx = rnn(input[i], hx).detach()
    output.append(hx)
</code></pre>
<p>Is there a way to do this without having to explicitly iterate over the sequence? I anticipate that doing it with an <code>RNNCell</code> will make the forward pass much slower.</p>
<p>I could also do it with <code>torch.nn.RNN</code>, but would still have to iterate over the sequence.</p>
<p>Thanks!</p>
<p>(Not sure if this is best for the autograd topic, I guess it could probably fall under NLP as well).</p>

albanD: <p>Hi,</p>
<p>It feels like if you detach the recurent connection, it’s not really an rnn anymore right?</p>
<p>I don’t think you can modify the fused cells <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> the whole point of them is that they are optimized because they do a fixed thing.</p>

alsuhr: <p>It would still be an RNN in the forward pass, but the gradients wouldn’t propagate recurrently.</p>
<p>A colleague suggested a clever way to do this (using <code>RNN</code>, not <code>RNNCell</code>):</p>
<ol>
<li>Forward pass with the RNN as usual and get the sequence of hidden states <code>hidden</code> (of size <code>b x l x h</code>)</li>
<li>Transform <code>hidden</code> to be size <code>bl x h</code> and detach; repeat the input <code>l</code> times to create a new batch of size <code>bl</code> so your input has size <code>bl x 1 x X</code> and <code>hidden</code> is of size <code>bl x 1 x h</code> (basically, sequences of length 1)</li>
<li>Call the same RNN again with these two tensors as input instead, and take the output of this call as the final output which will be used to compute the gradients.</li>
</ol>
<p>It does require two forward passes instead of one, but is still taking advantage of the efficiency of the <code>RNN</code> forward pass.</p>

albanD: <p>I guess you will have to try it out to see which one is faster in practice.<br>
But if you have to do two foward passes, to use the RNN, maybe the RNNCell will still be faster.</p>
Optimizing tensor operations by replacing for loops

Felix_Labelle: <p>I’m trying to fuse and discretize (in terms of image size) segmentation masks. The inputs are m masks that are 1024 x 1024 (&lt;m x 1024 x 1024&gt;) and the output should be &lt;1 x 64 x 64&gt;.</p>
<p>Each mask corresponds to a segmented object, with a corresponding id (up to 91 using MsCoco labels) and these labels are stored in a tensor of size m. To “fuse” the masks, each mask is multiplied by its id and then summed together (using a for loop). I’ve tried doing this matrix multiplication, but have not been able to make it work properly.</p>
<p>To reduce the size of the fused mask, an appropriately sized window is taken and the most common element in a window is kept. I’m unsure how to vectorize this operation either. Any help would be appreciated! This is the current version of the code:</p>
<pre><code class="lang-auto">def createFeatureMaps(masks, num_x_windows = 64, num_y_windows = 64):
    feature_maps = []
    for current_mask in masks:
        # load stored tensors.
        recovered_masks = deserializeTensor(current_mask["masks"])
        recovered_classes = deserializeTensor(current_mask["classes"])
        # Convert boolean tensor to binary
        raw_masks = recovered_masks.long()
       # Create fused mask
        fused_masks = torch.zeros((1,recovered_masks.size(1),recovered_masks.size(2))).long().cuda()
        
        # Converting masks to correct class ID
        # Is there a vectorized way of doing this?
        for i in range(len(recovered_classes)):
            fused_masks += raw_masks[i:i+1] * recovered_classes[i]

        merged_masks = torch.zeros((num_x_windows,num_y_windows))
        window_size_x = int(recovered_masks.size(1)/num_x_windows)
        window_size_y = int(recovered_masks.size(2)/num_y_windows)
        # Here is the other operation that should be vectorized. It runs windows over the fused mask and gets the most common value (mode)
        for x in range(num_x_windows):
            for y in range(num_y_windows):
                mask_window = fused_masks[0,window_size_x*x:window_size_x*(x+1),window_size_y*y:window_size_y*(y+1)]
                mode =  torch.mode(mask_window)
                merged_masks[x,y] = torch.mode(mask_window.flatten()).values
        #paths.append(os.path.join(current_mask['dir_id'],'matterport_skybox_images' ,current_mask['scan_id']))
        #output_masks.append(merged_masks)
        del(current_mask["masks"])
        del(current_mask["boxes"])
        current_mask["feature_maps"] = serializeTensor(merged_masks)
        feature_maps.append(current_mask)
    return feature_maps
</code></pre>
Torchvision suggestion, refactoring weight initialization: common method ._initialize_weights()

xvdp: <p>suggestion for refactoring the canonical weight initialization methods for all networks.</p>
<p>Use case, working in AutoML, Ive been I reinitializing networks as I dont care about the learned weights, just the hyperparameters im working on learning; reloading model and passing into cuda is very slow, so I simply reinitalize them which seems to be much faster;</p>
<p>I noticed in the models that ship with torchvision, that  googlenet and mnasnet have an <code>._initialize_weights() </code> method; alexnet and shufflenetv2 utilize the nn.Conv2d or whichever module initialization, and the other have initializations that are inlined in the class <code>.__init__() </code>method.<br>
Refactoring it into a standard method <code>._initialize_weights()</code> would be a tad cleaner than it is now.<br>
I supposed it could go into nn.Module, but since every net has small bespoke differences, maybe just like you have them in googlenet makes more sense.<br>
Thanks.</p>

xvdp: <p>or I can simply output a cloned/detached state dict and use that instead.<br>
Never mind, I suppose - still I would have a single init function</p>
RAM Issues during training

kangtinglee: <p>I’m experiencing severe memory leaks in my DDPG algorithm found <a href="https://github.com/kangtinglee/CarRacing" rel="nofollow noopener">here</a>.</p>
<p>Even after the first couple episodes, my RAM usage goes up by almost a whole gb.</p>
<p>I’ve tried shutting off different parts of my code such as putting no_grad() wherever possible, limiting my replay buffer to only 500 entries which will definitely not reach the memory usages I am seeing, and just not training the agent at all and only doing inference.</p>
<p>I’ve also tried using other versions of torch such as 1.3 but the result is the same.</p>
<p>Please help, any direction is appreciated. I understand that this is not a debugging session, but I appreciate any help at all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>A quick glance at the code does not reveal anything.<br>
Given how your replay buffer is implemented, it is expected to grow for a bit until it is full.</p>
<p>Could you be clearer about what you tried and the effect?<br>
Did you mean that when you don’t train, you don’t see this behavior?</p>

kangtinglee: <p>Thanks for the very swift reply.</p>
<p>What I’m trying to say is that my replay bufffer has no chance to accumulate to a size that it will even become a problem before my computer runs out of memory. How I determined that my replay buffer is not the problem is by limiting its capacity to only a few hundred episodes, since I’m only storing (96,96) 2D arrays in it, a few hundred of those can’t have caused the memory problem.</p>
<p>When I don’t train, I still continue to see this behaviour. By not training I mean that I comment out all parts of the code that attempts to do any form of backprop or loss estimation, so the only time the network is being invoked is when I do the forward pass.</p>

albanD: <ul>
<li>And if you replace the forward pass by a just returning a random output?</li>
<li>And if you replace all the calls to the gym simulator to just return a random value?</li>
</ul>

kangtinglee: <p>Hello alban,</p>
<p>Thanks for helping me with my problem! The problem went away with a clean install of PyTorch and my cuda/cudnn dependencies!</p>

albanD: <p>Awesome ! Thats a simple enough solution !</p>
Embedding dimension error

111242: <p>I am trying to embed one-hot vectors, each with length 10000 and a corresponding label (0 or 1). With mini-batch size 10, the dimension of the input to my feedforward neural network model is 10 x 10000.</p>
<p>I am trying to embed this input with <code>nn.Embedding(10000, 150)</code>.</p>
<p>So, after this embedding layer, my data becomes 10 x 10000 x 150. I am doing binary classification, so the output dimension is 10 x 10000 X 2. My label is of dimension 10 x 2. Either (1, 0) or (0, 1).</p>
<p>So, I get <code>ValueError: Target size (torch.Size([10, 2])) must be the same as input size (torch.Size([10, 10000, 2]))</code></p>
<p>How can I make this model to work? Each vector of 10000 length corresponds to either a 0 or 1 for its label.</p>

ecdrid: <p>The error has to deal with the output layer not the embedding layer imo</p>

111242: <p>Thanks for the reply. How would you suggest me to edit the final layer? What should the output dimension be?</p>
<p>Since the number of rows of my input data is 10000, I am not sure how the output layer does not include that number.</p>
<p>Here are the dimensions after each layer:</p>
<pre><code class="lang-auto">
Dimension of input: torch.Size([10, 10000])
Dimension after embedding: torch.Size([10, 10000, 150])
Dimension after layer1: torch.Size([10, 10000, 512])
Dimension after layer2: torch.Size([10, 10000, 512])
Dimension after final layer: torch.Size([10, 10000, 2])
</code></pre>

anantguptadbl: <p>You need to add a Linear + Softmax layer on the output of the embedding layer.Here is a working example.</p>
<pre><code class="lang-auto">import os
import numpy as np
import torch
from torch import nn
from torch.autograd import Variable

class embedder(nn.Module):
  def __init__(self):
    super(embedder,self).__init__()
    self.embeddings=nn.Embedding(10000,150)
    self.l1=nn.Linear(150,1)
    self.sigm=nn.Sigmoid()

  def forward(self,x):
    x=self.embeddings(x)
    x=self.l1(x)
    x=self.sigm(x)
    return(x)

model=embedder()
criterion=nn.BCELoss()
optimizer=torch.optim.Adam(model.parameters(),lr=0.001)

inputIndices=np.random.randint(10000,size=(10000))
inputData=np.zeros((10000,10000))
for i,x in enumerate(inputIndices):
  inputData[i,x]=1
inputTarget=np.random.randint(2,size=(10000))

batchSize=10
epochs=10
numBatches=int(inputData.shape[0]/batchSize)

for curEpoch in range(epochs):
  totalLoss=0
  for curBatch in range(numBatches):
    model.zero_grad()
    X=Variable(torch.from_numpy(inputIndices[curBatch*batchSize:(curBatch+1)*batchSize].reshape(1,batchSize))).type(torch.long)
    Y=Variable(torch.from_numpy(inputTarget[curBatch*batchSize:(curBatch+1)*batchSize].astype(np.float32)))
    modelOutput=model(X)
    loss=criterion(modelOutput,Y)
    loss.backward()
    optimizer.step()
    totalLoss=totalLoss + loss.item()
  print("Epoch {0} Loss {1}".format(curEpoch,totalLoss))
</code></pre>
<p>Hope this helps</p>

ecdrid: <p>Yep as Anant said already, you have to do that;<br>
Note Embedding layers just acts as a lookup table, they are moreover quite similar to one-hot-encoding as well</p>
Why does TorchVision's GoogLeNet implementation have more parameters than that in the original paper?

mnslarcher: <p>If I run</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.googlenet(pretrained=False)
params_count = sum(p.numel() for p in model.parameters() if p.requires_grad)
print("GoogLeNet number of trainable parameters: {}".format(params_count))
</code></pre>
<p>I get</p>
<blockquote>
<p>GoogLeNet number of trainable parameters: 13004888</p>
</blockquote>
<p>But, from the original paper, the number of parameters should be 6.7977 millions.</p>
<p>What I’m missing? is TorchVision implementing a different version of GoogLeNet respect to the original paper?</p>
<p>Thanks,<br>
Mario</p>

ptrblck: <p>When you skip the <code>aux</code> layers, you should get the ~6 million parameters.<br>
It seems <a href="https://arxiv.org/pdf/1409.4842.pdf">Table 1</a> of the original paper does the same.</p>

mnslarcher: <p>Thanks for the explanation ptrblck <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:">!</p>
Pretrained models cuda out of memory error

murali_perumalla: <p>hi, i am working with vgg16 pretrained model. when train vgg16 model by freezing all layers there is no error. but when i freeze only few layers and tried to train i got cuda out of memory error.  i also tried by removing last 7 layers of vgg16 and freeze the remaining layers i got cuda out of memory error . can anyone explain me why i got out memory in case freezing only certain layers …?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/tom">@tom</a></p>

ahmed: <p>Not sure if this is relevent, but what batch size are you using? Have you tried reducing it?</p>

murali_perumalla: <p>batch size is 64 , may be it is due to less gpu memory i think i have 6gb nvidia GTX1060 graphic card.   so i tried the same program in colab it worked.</p>

ahmed: <p>Yes, i think that is the cause of your issue.</p>
Training Fast-RCNN with torchvision model Faster-RCNN

fatihbaltaci: <p>Hi,</p>
<p>I want to train Fast-RCNN with pre-computed selective search proposals using <a href="https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn" rel="nofollow noopener">Faster-RCNN</a> model stated in torchvision.</p>
<p>Which modifications should I do for Faster-RCNN model?</p>
<p>Thanks.</p>
How to setup numerical warning?

ElleryL: <p>Consider</p>
<blockquote>
<p>torch.Tensor([torch.exp(torch.Tensor([10000]))])</p>
</blockquote>
<p>it gives</p>
<blockquote>
<p>tensor([inf.])</p>
</blockquote>
<p>without raising any warning or error. During the training, many time I end up with nan value; and have to debug all the way to figure when did the numerical issue occurs.</p>
<p>Is there anyway to fix it</p>
<p>I’m using pytorch 0.4.0 on Mac</p>
<p>(Both jupyter notebook and pycharm doesn’t give any numerical warning)</p>

justusschock: <p>A dirty hack would be to use <a href="https://pytorch.org/docs/master/torch.html#torch.clamp" rel="nofollow noopener">torch.clamp</a> with the maximum possible value for your dtype</p>

ElleryL: <p>But I kind of prefer to raise the warning and error because in many cases; overflow encounter because of the bad implementation ; clamp the value might silence the potential bad implementation and leads to unexpected results even make it harder to debug later</p>

untrix: <p>Has this been addressed in pytorch? I would like to see such a feature as well. Takes too long to identify location of the overflow in code, especially in complex models.</p>

ptrblck: <p>Checking each value for an overflow could yield bad general performance.<br>
To debug an invalid value, you could use <code>torch.autograd.detect_anomaly</code>.</p>
Line 94, in compute_kernel kernel_input = (tiled_x - tiled_y).pow(2).mean(2) / float(dim) RuntimeError: expected backend CUDA and dtype Float but got backend CPU and dtype Float

sam101: <p>My code is showing up this error:</p>
<p>def compute_kernel(self, x, y):<br>
x_size = x.size(0)<br>
y_size = y.size(0)<br>
dim = x.size(1)<br>
x = x.unsqueeze(1)  # (x_size, 1, dim)<br>
y = y.unsqueeze(0)  # (1, y_size, dim)<br>
tiled_x = x.expand(x_size, y_size, dim)<br>
tiled_y = y.expand(x_size, y_size, dim)<br>
kernel_input = (tiled_x - tiled_y).pow(2).mean(2) / float(dim)<br>
return torch.exp(-kernel_input)  # (x_size, y_size)</p>
<p>def compute_mmd(self, x, y):<br>
x_kernel = self.compute_kernel(x, x)<br>
y_kernel = self.compute_kernel(y, y)<br>
xy_kernel = self.compute_kernel(x, y)<br>
mmd = x_kernel.mean() + y_kernel.mean() - 2 * xy_kernel.mean()<br>
return mmd</p>

ptrblck: <p>Could you check the <code>device</code> of <code>x</code> and <code>y</code> and thus also <code>tiled_x</code> and <code>tiled_y</code>?<br>
I assume a tensor might still be on the CPU, while the other might be on the GPU.</p>

sam101: <p>How can i  make them use the gpu</p>

sam101: <p>It worked i sent x and y and tiled_x and tiled_y to device</p>
Unable to getPixels(), pixel access is not supported on Config#HARDWARE bitmaps

vortana_say: <p>I am looking at the demo android app. <a href="https://github.com/pytorch/android-demo-app" rel="nofollow noopener">https://github.com/pytorch/android-demo-app</a></p>
<p>When I run the app in android version 10. There’s an error produce when this code below run.</p>
<p><strong>Code</strong><br>
inputTensor = TensorImageUtils.bitmapToFloat32Tensor(<br>
bitmap,<br>
TensorImageUtils.TORCHVISION_NORM_MEAN_RGB,<br>
TensorImageUtils.TORCHVISION_NORM_STD_RGB<br>
)</p>
<p><strong>Error</strong><br>
2020-02-22 11:24:29.690 19276-19594/com.say.dogbreedfinder E/TAG: Error during image analysis<br>
java.lang.IllegalStateException: unable to getPixels(), pixel access is not supported on Config#HARDWARE bitmaps<br>
at android.graphics.Bitmap.checkHardware(Bitmap.java:423)<br>
at android.graphics.Bitmap.getPixels(Bitmap.java:1935)<br>
at org.pytorch.torchvision.TensorImageUtils.bitmapToFloatBuffer(TensorImageUtils.java:67)<br>
at org.pytorch.torchvision.TensorImageUtils.bitmapToFloat32Tensor(TensorImageUtils.java:108)<br>
at org.pytorch.torchvision.TensorImageUtils.bitmapToFloat32Tensor(TensorImageUtils.java:34)</p>
<p>Note: if I run on android 9, it’s working fine.</p>
<p>Anyone know what could be the issue and how we can resolve this ?</p>

vortana_say: <p>I found a work around is by config Bitmap to not use the HARDWARE.</p>
<pre><code>        val mutableBitmap = bitmap.copy(Bitmap.Config.RGBA_F16, true)

        inputTensor = TensorImageUtils.bitmapToFloat32Tensor(
            mutableBitmap,
            TensorImageUtils.TORCHVISION_NORM_MEAN_RGB,
            TensorImageUtils.TORCHVISION_NORM_STD_RGB
        )
</code></pre>
<p>Even though, this work around work, I would like to know if we can preserve the HARDWARE config since it’s provide more benefits with later Android version.</p>
Naming .pth checkpoints using PyTorch Ignite ModelCheckpoint handler

xen0f0n: <p>I use the ModelCheckpoint handler in Ignite to save the model using these lines</p>
<pre><code class="lang-auto">ckpt_handler = ModelCheckpoint(f'./experiments/{log_num}/checkpoints', filename_prefix='epoch_', n_saved=None, create_dir=True)
trainer.add_event_handler(Events.EPOCH_COMPLETED(every=2), ckpt_handler, to_save)
</code></pre>
<p>but the .pth filename follow this pattern --&gt; epoch__checkpoint_344.pth (344 refers to iterations).</p>
<p>I would to include the number of epoch into the filename… Any idea how’s that done?</p>

vfdev-5: <p><a class="mention" href="/u/xen0f0n">@xen0f0n</a> it can be done by adding a <code>global_step_transform=lambda e, _: e.state.epoch</code></p>
<pre><code class="lang-python">checkpointer = ModelCheckpoint(*args, **kwargs, global_step_transform=lambda e, _: e.state.epoch)
</code></pre>

xen0f0n: <p>Thank you <a class="mention" href="/u/vfdev-5">@vfdev-5</a>! Not sure how this works… I tried accessing trainer.state.epoch (before trainer.run) and state was None. Is this what that lambda does? Accessing trainer engine state after trainer.run?</p>

vfdev-5: <p>Yes, state at the begining of the training is None as it is not defined.<br>
When you attached</p>
<pre><code class="lang-auto">trainer.add_event_handler(Events.EPOCH_COMPLETED(every=2), ckpt_handler, to_save)
</code></pre>
<p>once every 2 epoch, <code>ckpt_handler</code> is triggered to save what to save. Its argument <code>global_step_transform</code> is an optional callable that</p>
<blockquote>
<p>global step transform function to output a desired global step. Input of the function is (engine, event_name). Output of function should be an integer. Default is None, global_step based on attached engine. If provided, uses function output as global_step.</p>
</blockquote>
<p>So it is just executed during the training, where trainer’s state is initialized and can be used.</p>
<p>Please, see the docs for info : <a href="https://pytorch.org/ignite/handlers.html#ignite.handlers.Checkpoint" class="inline-onebox" rel="noopener nofollow ugc">ignite.handlers — PyTorch-Ignite v0.4.13 Documentation</a></p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
My model predicts 0 for all instances

Geet_Shingi: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(30, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.sigmoid(x)
        return x
</code></pre>
<pre><code class="lang-auto">bobs_model = Net()
alices_model = Net()
bobs_optimizer = optim.Adam(bobs_model.parameters(), lr=args.lr)
alices_optimizer = optim.Adam(alices_model.parameters(), lr=args.lr)
</code></pre>
<pre><code class="lang-auto">models = [bobs_model, alices_model]
optimizers = [bobs_optimizer, alices_optimizer]
</code></pre>
<pre><code class="lang-auto">def update(data, target, model, optimizer):
    model.send(data.location)
    optimizer.zero_grad()
    prediction = model(data)
    print(prediction.view(-1))
    loss = F.binary_cross_entropy(prediction.view(-1), target)
    loss.backward()
    optimizer.step()
    return model

def train():
    for data_index in range(len(remote_dataset[0])-1):
        for remote_index in range(len(compute_nodes)):
            data, target = remote_dataset[remote_index][data_index]
            models[remote_index] = update(data, target, models[remote_index], optimizers[remote_index])
        for model in models:
            model.get()
        return utils.federated_avg({
            "bob": models[0],
            "alice": models[1]
        })
</code></pre>
<pre><code class="lang-auto">def test(federated_model):
    correct = 0
    total = 0
    federated_model.eval()
    test_loss = 0
    for data, target in test_loader:
        output = federated_model(data)
        test_loss += F.binary_cross_entropy(output.view(-1), target, reduction='sum').item()
        predection = output.data.max(1, keepdim=True)[1]
        # outputs = federated_model(data)
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += (predicted == target).sum().item()

    print('Accuracy of the network: %f %%' % (100 * correct / total))   
    test_loss /= len(test_loader.dataset)
    print('Test set: Average loss: {:.4f}'.format(test_loss))
</code></pre>
<pre><code class="lang-auto">for epoch in range(args.epochs):
    start_time = time.time()
    print(f"Epoch Number {epoch + 1}")
    federated_model = train()
    model = federated_model
    test(federated_model)
    total_time = time.time() - start_time
    print('Communication time over the network', round(total_time, 2), 's\n')
</code></pre>
Does removing the tensor to CPU break the computation graph?

Abdelpakey: <p>In case you have a series of tensor operation let say you have these dummy operations</p>
<p>a=Operation (t1) ====&gt; on cuda<br>
b= Operation (a) ====&gt; on cuda<br>
c= Operation (b)====&gt; on cuda</p>
<p>d= Operation (removing tensor c to cpu)</p>
<p>convert d to cuda again<br>
.<br>
.<br>
.</p>
<p>Does this style break the computation graph, I’m assuming it doesn’t, not sure though?</p>
<p>If it does is there any workaround ?</p>
<p>The reason why I’m asking because I have a problem with this Pytorch Function<br>
torch.Tensor.svd(X_bar.cpu().T, some=True,compute_uv=True)<br>
or<br>
torch.svd(X_bar.cpu().T, some=True,compute_uv=True)</p>
<p>It always throws an error as below every a couple of Epochs.</p>
<p>svd_cpu: the updating process of SBDSDC did not converge (error: 17)</p>
A2C with Pytorch

Gaurav_Jain: <p>I am not sure if this is the correct forum to post queries related to algorithm implementation correctness, but since I have the implementation in PyTorch, I am giving it a try.</p>
<p>I tried implementing the Policy Gradient A2C algorithm in PyTorch, using the <a href="https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail" rel="nofollow noopener">this</a> and <a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noopener">this</a> as references. I also followed the OpenAI Baselines Tensorflow implementation.</p>
<p>After spending more than a week now on trying to figure out why the agents are not learning even after 5-6 hours of training (which itself is suspicious because with Vectorized implementations for A2C, it should start learning within 30 minutes), I have finally given up.</p>
<p>I am using the PongNoFrameskip-v4 vectorized environment and have the implementation <a href="https://github.com/gauravjain14/rl-implementations-pytorch/blob/master/a2c_pytorchv2.ipynb" rel="nofollow noopener">here</a>. I would really appreciate if someone could either help me identify any potentially obvious errors or guide me to the appropriate forums for such type of questions.</p>
<p>Thanks</p>
VAE's: Residual CNN as encoder

Kaouthar_AARIZOU: <p>Hello everyone, I rather encounter a theoretical issu with VAE.<br>
I aim to <strong>maximize</strong> image-patches similarity across single input image scales, and I want to implement an unsupervised VAE  for this task.<br>
My first question would be: Is it “theoretically correct” to create residual connexion in the VAEs CNN encoder, connecting <strong>each patch</strong> (from each scale) directly with the <strong>latent space</strong> ? And then try to get the patch-similarity distribution?</p>
<p>Thank you</p>
Using transforms.Normalize on a single tensor

lc5415: <p>Hello, I’m relatively new to PyTorch, I want to to apply Instance Normalization to my images. I am wondering if I can use Normalize as a stand alone function, without needing a tranforms.Compose() or any of the sort. Here is how I am currently trying to implement it:</p>
<pre><code class="lang-auto">img = someTensor # dimensions [1x224x224] I am using grayscale images
mean, std = img.mean(), img.std()
normal  = Normalize(mean,std)
normal(tensor = img)
</code></pre>
<p>I am getting the following error:<br>
<strong>IndexError: too many indices for tensor of dimension 0</strong></p>
<p>For now, I will apply element-wise normalisation (i.e. x-mu/sigma) but I would like to make it work with the torch functions too!</p>
<p>I would really appreciate if anyone could help me clarify how all this works!</p>
<p>Thanks in advance!</p>

ptrblck: <p>You should unsqueeze the statistics, as you are currently passing scalar values:</p>
<pre><code class="lang-python">normal = transforms.Normalize(mean.unsqueeze(0), std.unsqueeze(0))
</code></pre>
<p>This should avoid the error, which is caused by <code>mean[:, None, None]</code> inside <code>Normalize</code>.</p>

lc5415: <p>That makes sense, thank you so much for the speedy reply!</p>
How to copy a model?

cndcnd: <pre><code class="lang-auto">def Scramble_acton(net, ort, inputs, dpinda, hocmat):

   # copies net in ort
   for lr in range(0,6):
      for bi in range(0,BATCHSIZE):
         for no in range(0,int(lrsize[lr][0])):
            if lrsize[lr][1] == 1:
               ort.nx[lr][bi][no] = net.nx[lr][bi][no]
            if lrsize[lr][1] != 1:
               for xi in range(0,int(lrsize[lr][1])):
                  for yi in range(0,int(lrsize[lr][1])):
                     ort.nx[lr][bi][no][xi][yi] = \
                     net.nx[lr][bi][no][xi][yi]
</code></pre>
<p>Hi guys,<br>
I pass to this function two pytorch neural network models (net and ort), then copy all output values from one (net) to the other (ort). In the process, the memory allocation increases (which I would not expect to happen) and, after some cycle, the program crashes. I tried to use copy.deepcopy to make the copy, but I got the message “Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol”.<br>
Thank you,<br>
Alex</p>
<p>This is the model definition:</p>
<pre><code class="lang-auto">class Net(nn.Module):

   def __init__(self, x):

      super(Net, self).__init__()
      self.pool =  nn.MaxPool2d (2,2)
      self.conv1 = nn.Conv2d ( 3,10,5)
      self.conv2 = nn.Conv2d (10,16,5)
      self.fc3 =   nn.Linear (16*5*5,120)
      self.fc4 =   nn.Linear (120,84)
      self.fc5 =   nn.Linear (84,10)

      lrsize[0][0] =  3;      lrsize[0][1] =32
      lrsize[1][0] = 10;      lrsize[1][1] =14
      lrsize[2][0] = 16*5*5;  lrsize[2][1] = 1
      lrsize[3][0] = 120;     lrsize[3][1] = 1
      lrsize[4][0] =  84;     lrsize[4][1] = 1
      lrsize[5][0] =  10;     lrsize[5][1] = 1

      self.nx = np.empty(6, dtype=np.object)
      self.nl = np.empty(6, dtype=np.object)
      self.nl[1] = self.conv1
      self.nl[2] = self.conv2
      self.nl[3] = self.fc3
      self.nl[4] = self.fc4
      self.nl[5] = self.fc5
      self.forward(x,0,0)

   def forward(self, x0, opt, sitnx):

      #Tanhact # torch.sigmoid
      xx = self.nx; xx[0] = x0
      if opt == 1: xx = sitnx
      self.nx[1] = self.pool(Tanhact(self.conv1(xx[0])))
      self.nx[2] = self.pool(Tanhact(self.conv2(xx[1])))
      self.nx[2] = self.nx[2].view(-1, 16*5*5)
      xt = self.fc3(xx[2]); self.nx[3] = Tanhact(SLOPETP*xt)
      xt = self.fc4(xx[3]); self.nx[4] = Tanhact(SLOPETP*xt)
      xt = self.fc5(xx[4]); self.nx[5] = Tanhact(SLOPETP*xt)
</code></pre>
<p>nx[] contains the layer values,<br>
BATCHSIZE = 25</p>
<p>I come from C, I just wanted to make a copy of the tensor /array, but something strange is happening here.</p>
<p><a href="https://i.stack.imgur.com/EB2fu.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/EB2fu.png" alt="call stack between 3rd and 4th for" width="657" height="425"></a></p>
Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same

HemaZ: <p>hello everyone.</p>
<p>i just keep getting this error whenever i try to train my model</p>
<blockquote>
<p>RuntimeError: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same</p>
</blockquote>
<p>you can find my code here</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/HemaZ/MNIST/blob/master/pytorch_mnist.ipynb">
  <header class="source">

      <a href="https://github.com/HemaZ/MNIST/blob/master/pytorch_mnist.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/HemaZ/MNIST/blob/master/pytorch_mnist.ipynb" target="_blank" rel="noopener nofollow ugc">HemaZ/MNIST/blob/master/pytorch_mnist.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import torch\n",
    "import torch.optim as optim\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import argparse\n",
    "from torchvision import datasets,transforms\n",
    "from torch.utils.data import Dataset, DataLoader,random_split\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
</code></pre>



  This file has been truncated. <a href="https://github.com/HemaZ/MNIST/blob/master/pytorch_mnist.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>thank you</p>

RicCu: <p>Try moving the data to torch.float, i.e <code>data.to(device=device, dtype=torch.float)</code>.</p>

JuanFMontesinos: <p>cast your input to float type by doing<br>
input=input.float() before calling the forward pass<br>
(I’m not sure if u should call this method after or before allocating tensor to cuda, before seems better.)</p>

HemaZ: <p>after i tried this i got another error</p>
<blockquote>
<p>RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [64]</p>
</blockquote>

JuanFMontesinos: <p>it seems you have to input 3d tensors, like x[i,j,k] but you are inputing a tensor of dim 64 ??</p>

HemaZ: <p>yes that was a mistake I swapped (output, target) in the loss function.<br>
thank you</p>

jan_liao: <p>i meet the same error ，if some somewhere not use cuda</p>
Convert rgb image vector into grey scale image

cbd: <p>I read the rgb image in tensor lets say “rgb_image” with dimension [1,3,256,256]. How can i convert it into the grey scale image?</p>

KFrank: <p>Hi cbd!</p>
<aside class="quote no-group" data-username="cbd" data-post="1" data-topic="70660" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>I read the rgb image in tensor lets say “rgb_image” with dimension [1,3,256,256]. How can i convert it into the grey scale image?</p>
</blockquote>
</aside>
<p>The most standard way:</p>
<pre data-code-wrap="python"><code class="lang-python">grayscale_image = rgb_image.mean (dim = 1)
</code></pre>
<p><code>grayscale_image</code> will now be of shape <code>torch.Size([1, 256, 256])</code>.</p>
<p>The idea is that “3” dimension of <code>rgb_image</code> consists of the three<br>
color channels, r, g, and b.  You take their mean (or sum them, if<br>
you prefer) to get the “brightness” of the pixel in question, that is,<br>
its grayscale level.</p>
<p>Best.</p>
<p>K. Frank</p>
A weird zero_grad problem

cedricchan: <p>I set <code>requires_grad=False</code> for some parameters, and their grad is zero. However, when I run <code>optim.step()</code>, those parameters still update their values.</p>
<p>This is my code</p>
<pre><code class="lang-auto">for p in self.G.enc_layers.parameters():
    requires_grad_flag = p.requires_grad
    p.requires_grad = False
z_rec = self.G(img_fake, mode='enc')
gzr_loss = F.mse_loss(z_rec[-1], zs_a[-1])

import pdb
pdb.set_trace()
# the problem appears in the following three lines
self.optim_G.zero_grad()
(self.lambda_gzr * gzr_loss).backward(retain_graph=True)
self.optim_G.step()

for p in self.G.enc_layers.parameters():
    p.requires_grad = requires_grad_flag
</code></pre>
<p>The picture below shows the values during debug.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd30a2084a5ef381115a223fcb2223cc7efde130.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cd30a2084a5ef381115a223fcb2223cc7efde130" title="WX20200222-111234@2x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd30a2084a5ef381115a223fcb2223cc7efde130_2_690x439.jpeg" alt="WX20200222-111234@2x" data-base62-sha1="thc4M7cins7jc35SuaVxKKX31dK" width="690" height="439" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd30a2084a5ef381115a223fcb2223cc7efde130_2_690x439.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd30a2084a5ef381115a223fcb2223cc7efde130_2_1035x658.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd30a2084a5ef381115a223fcb2223cc7efde130_2_1380x878.jpeg 2x" data-dominant-color="211F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20200222-111234@2x</span><span class="informations">2188×1394 501 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If these parameters were updated previously using an optimizer with running estimates, changing the <code>requires_grad</code> attribute to <code>False</code> might still result in updates due to the running estimates of the optimizer.<br>
If that’s the case, you might need to set these gradients to None, so that they will be skipped, as shown <a href="https://github.com/pytorch/pytorch/blob/8291e06f8fb14a09e6c3b7a740a182250069242c/torch/optim/adam.py#L64">here</a> for Adam.</p>

cedricchan: <p>It works. Thanks a lot.</p>
Weight normalization - setting g constant

Maria_Fernanda_De_La: <p>Hello,</p>
<p>I want to do some experiments with weight normalization and I was wondering if it would be possible to set the g in weight normalization as a constant or a different function of time in PyTorch.</p>
<p>Thank you in advance!</p>

ptrblck: <p>I think the easiest way could be to write your own <code>WeightNorm</code> class by copy-pasting <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py#L38-L39">this code</a> and manipulate the creation of <code>g</code>.</p>
Why does torch.zeroes(10,10,10).to("cuda" or "gpu") return a tensor full of nans?

Dylan_Yung: <p>yea why does to.(DEVICE) with DEVICE = “gpu” or "cuda return a tensor full of nans?</p>

ptrblck: <p><code>"gpu"</code> shouldn’t be a valid argument to the <code>.to()</code> operation.<br>
Could you please post a small code snippet to reproduce this issue and also post information about your setup, i.e. PyTorch, CUDA, version, NVIDIA driver, etc.?</p>
How to quickly assign values like this

ZeratuuLL: <p>Let’s say I have three tensors a, b, c and three hyperparameters bs, L, V</p>
<p>tensor <code>a</code> : dtype torch.long, size (bs, L), all values between 0 and V-1<br>
tensor <code>b</code> : dtype torch.float, size (bs, L), all values between 0 and 1<br>
tensor <code>c</code> : drype torch.float, size (bs, V), initialized as all 0</p>
<p>Now I want to assign values to c based on a and b like this</p>
<pre><code class="lang-auto">for i in range(bs):
    for j in range(L):
        c[i, a[i, j] ] += b[i, j]
</code></pre>
<p>In the final row I use <code>+=</code> because there can be duplicate values in each row of <code>a</code>. That’s also why I am not sure if I can use <code>scatter_</code>…</p>
<p>Does anybody know if there is an API that can do this efficiently? Like with one for loop or even no for loop?</p>
<p>Thank you very much!</p>

ptrblck: <p><code>tensor.pu_</code> should work.<br>
Note that you would have to calculate the linear indices for it as shown here:</p>
<pre><code class="lang-python"># Setup
bs, L, V = 2, 3, 4

# Your approach
a = torch.randint(0, V, (bs, L))
b = torch.empty(bs, L).uniform_(0, 1)
c = torch.zeros(bs, V)

for i in range(bs):
    for j in range(L):
        c[i, a[i, j] ] += b[i, j]


# put_ approach
d = torch.zeros(bs, V)
# calculate linear index
lin_index = a + (torch.arange(a.size(0)) * d.size(1)).unsqueeze(1)
d.put_(lin_index, b, accumulate=True)

# Compare
print((c == d).all())
&gt; tensor(True)
</code></pre>