akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert [0,1] range data to [-1, 1]

bibekx: <p>I have data that is Minmax scaled i.e. in the range [0,1]<br>
For some purpose I need this data to be Standard scaled, i.e. in range [-1, 1]<br>
But, I do not have access to data before it was Minmax scaled</p>
<p>Can I simply so the operation:</p>
<blockquote>
<p>x = (x - mean) / std</p>
</blockquote>
<p>where,<br>
mean = mean of data in range [0,1]<br>
std = standard deviation of data in range [0,1]</p>
<p>I mean, if I do this operation for all data points in range [0,1] now,<br>
Will this be equivalent to doing Standard Scaling to original data (before it was minmax scaled)?</p>

bibekx: <p>Seems like I was totally wrong to understand these concepts:<br>
I did a little bit more study, Could you verify these:</p>
<p><strong>1. Standard Scaling</strong></p>
<blockquote>
<p>x = x - mean / std</p>
</blockquote>
<p>makes the data have a zero mean and unit variance and standard deviation</p>
<p><strong>2. Minmax scaling</strong></p>
<blockquote>
<p>makes the data have a range between min and max</p>
</blockquote>
<p><strong>3. Z-score normalization</strong></p>
<p>same as standard scaling</p>
<p><strong>4. [0, 1] scaling</strong></p>
<p>same as minmax scaling but with the min max range set to 0 and 1 respectively</p>
<p>And it turns out that, standard scaling the minmax scaled data is equivalent to standard scaling the original data</p>

Nikronic: <p>Hi,</p>
<p>Yes, you are right.<br>
Actually, it is the common way to first scale data to <code>[0, 1]</code>, then compute mean and std to get to z-score.</p>
<p>But something I would like to mention is that using z-score does not necessrilty convert your data to <code>[-1, 1]</code>. See below post please:</p>
<aside class="quote" data-post="28" data-topic="21730">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/understanding-transform-normalize/21730/28">Understanding transform.Normalize( )</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi, 
There is main difference here. If you use mean=0.5 and std=0.5, your output value will be between [-1, 1] based on the normalization formula : (x - mean) / std which also called z-score. This score can be out of [-1, 1] when used mean and std of dataset, for instance ImageNet. 
The definition says that we need to use population mean and std but it is usually unavailable, sample mean/std can be used as an estimation.
  </blockquote>
</aside>

<p>Bests</p>
What does affine do in nn.Conv2d(...)?

filleloup: <p>Hi!<br>
I would appreciate it if you could give me a detailed explanation of what affine does in nn.Conv2d() or nn.BatchNorm2d().<br>
Thank you!</p>

googlebot: <p>It is just scale &amp; shift: y = x*w+b, for batch norm it is done channelwise, i.e.: x[B,C,H,W] * w[1,C,1,1]+b[1,C,1,1].</p>
<p>Conv operations don’t have this functionality, as kernel and bias parameters implicitly do scale &amp; shift.</p>

filleloup: <p>Thank you for responding!</p>
It is recommended to use source Tensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True)

Anna_yah: <p>I changed <code>torch.tensor(x)</code> to <code>torch.tensor(x).clone().Detach() </code>but the problem is not solved.<br>
Do you know what I am doing wrong here?<br>
Thanks in advance!</p>
<pre><code class="lang-auto">for epoch in range(num_epochs): 
    outputs = []
    outputs = torch.tensor(outputs, requires_grad=True)
    outputs= outputs.clone().detach().cuda()
    for fold in range(0, len(training_data), 5): # we take 5 images 
        xtrain = training_data[fold : fold+5]
        xtrain = torch.tensor(xtrain, requires_grad=True).clone().detach().float().cuda() 
        xtrain = xtrain.view(5, 3, 120, 120, 120) 
        # Clear gradients
        #optimizer.zero_grad() 
        # Forward propagation
        optimizer.zero_grad() 
        v = model(xtrain)
        v = torch.tensor(v, requires_grad=True).clone().detach()
        outputs = torch.cat((outputs,v),dim=0)
        # Calculate softmax and ross entropy loss
    targets = torch.Tensor(targets).clone().detach()
    labels = targets.cuda()
    outputs = torch.tensor(outputs,  requires_grad=True) 
    _, predicted = torch.max(outputs, 1) #prendre valeur maximale [0.96  0.04] ==&gt; 0 (position de classe)
    accuracy = accuracyCalc(predicted, targets)
    labels = labels.long() 
    labels=labels.view(-1) 
    loss = nn.CrossEntropyLoss()
    loss = loss(outputs, labels)    
    # Calculating gradients
    loss.backward()
    # Update parameters
    optimizer.step()
    loss_list_train.append(loss.clone()) 
    accuracy_list_train.append(accuracy/100)
    np.save('Datasets/brats/accuracy_list_train.npy', np.array(accuracy_list_train))
    np.save('Datasets/brats/loss_list_train.npy', np.array(loss_list_train)) 
    print('Iteration: {}/{}  Loss: {}  Accuracy: {} %'.format(epoch+1,  num_epochs, loss.clone(), accuracy))
print('Model training  : Finished')
</code></pre>
<p>result :</p>
<pre><code class="lang-auto">UserWarning: To copy construct from a tensor, it is recommended to use source
Tensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True),
 rather than torch.tensor(sourceTensor)
</code></pre>

ptrblck: <p>The warning points to wrapping a tensor in <code>torch.tensor</code>, which is not recommended.<br>
Instead of <code>torch.tensor(outputs)</code> use <code>outputs.clone().detach()</code> or the same with <code>.requires_grad_(True)</code>, if necessary.</p>

Anna_yah: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Can't reproduce Faster RCNN Resnet50FPN mAP for coco2017

kostas1515: <p>Hello I am using FasterRCNN model with the pretrained RESNET50 FPN backbone for 91 classes.<br>
For my project I am using many models and one of them is YOLOv3 and because of this I chose to have the coco2017 dataset on YOLO format (txt file with the format &lt; class:[0,79] &gt;, &lt; xc:[0,1] &gt;, &lt; yc:[0,1] &gt;, &lt; w:[0,1] &gt;, &lt; h:[0,1] &gt; bounding box per line.)<br>
When I try to compare Faster RCNN performance I am converting the labels from 80 to 91 and the coordinates to xmin xmax ymin ymax. After this I am training Faster RCNN with the pretrained 91 Renset50FPN backbone with a learning rate 0.001, SGD with weight decay=0.0005 and momentum=0 for 13 epochs. i am reducing the learning rate by a factor of 10 in epoch 8 and 11 but that does not make a difference.<br>
This is my result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53259c7a98fbd2c381bbe2797aa5ec313bc9e469.png" alt="vanilla_lr0.001" data-base62-sha1="bRyd82BEGDFXZ76fG8Xbeg3NlyN" width="480" height="360"><br>
I have seen that the baseline for FasterRCNN on coco2017 is 36% which is 3.4 percent bigger than my results.<br>
I cannot understand why there is this difference. Do you have any ideas?<br>
Thank you</p>
Linear layer on a 2D time series data

sathvik_udupa: <p>For a particular application, I am porting the code from keras to pytorch. The input is of the size [ bs x timesteps x features ], lstm output is [ bs x time step x hidden ]. Now I want to reduce this to [ bs x time step x out_features](time distributed layer on keras)<br>
Using linear,</p>
<p>nn.Linear(in_features=hidden, out_features=out_features)</p>
<p>Is this the right way to do this if I want to preserve time information or do I need to reshape the data using contiguous in any way to achieve it?</p>
<p>Any help appreciated.</p>

sathvik_udupa: <p>For anyone who might need help on this in the future, the linear layer as mentioned in the post works correctly. Using MSE loss on such data, works normally while using a cross entrophy loss, there might be an issue.<br>
On a time step x feats data, I did not find a way to state the dim to apply softmax on. Instead you can specify it in log softmax and then use nllloss. (you get the  same loss overall with proper softmax norm)</p>
Modifying a model's layer weights via state_dict

capoling: <p>Hello!</p>
<p>I am trying to zero out some filter weights of a pytorch model before and after training. Once located the correct layers and filters, I go ahead and replace that precise key in the OrderedDictionary that is state_dict with a value of torch.zeros(correct size). By changing the value in the state_dict, am I satisfactorily changing the whole model, making it ready for training with my intended change (in other words, does the change propagate also to model.parameters() or anything that is use in train.py)? If not, what’s the best way of doing so.</p>
<p>Thanks a lot!</p>

ptrblck: <p>If you don’t create a deepcopy of the <code>state_dict</code>, it should work:</p>
<pre><code class="lang-python">model = models.resnet18()
sd = model.state_dict()
sd['fc.weight'].zero_()
print(model.fc.weight)
&gt; Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
</code></pre>
<p>However, the better approach would be probably to zero out the parameters directly in the model by wrapping it in a <code>with torch.no_grad()</code> statement and manipulating the parameters as you wish.</p>

capoling: <p>Thank you for the quick response! If I were to use the <code>with torch.no_grad()</code>, how would I change the model parameters directly as you mentioned? For instance, if I had a tensor where I want to zero out the weights of the filter at index 34 in it, how would I use torch.no_grad() and model.parameters() to do so? Thanks in advance!</p>

ptrblck: <p>This code snippet should work:</p>
<pre><code class="lang-python">model = models.resnet18()
with torch.no_grad():
    model.conv1.weight[34].zero_()
print(model.conv1.weight[33:35])
</code></pre>

Maxime_G: <p>modifying by reference doesn’t work for me, this method worked:</p><aside class="quote" data-post="2" data-topic="12907">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zeakey/48/4341_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-change-weight-after-traning-nueralnetwork/12907/2">How to change weight after traning nueralNetwork</a> 
  </div>
  <blockquote>
    You can get the parameters by state_dict = model.state_dit(), and state_dict will hold all the trainable parameters. 
You can do whatever changes you want to the content of the dict. 
At last you just use model.load_state_dict(state_dict) to load all the updated state_dict back to the model.
  </blockquote>
</aside>
How to make the data of each batch have different sizes

laizewei: <p>libtorch 1.6.0 win10 x64 .</p>
<p>I wrote an OCR model of the dataset.The word is encoded with different lengths as the label input.How to make the data of each batch have different sizes?<br>
example:<br>
data:123.png datasize:[batchsize,3,180,32] ,label: 123,labelsize:[batchsize,3]<br>
data:3234.png datasize:[batchsize,3,180,32] ,label: 3234,lablesize:[batchsize,4]<br>
[batchsize,3]!=[batchsize,4]<br>
The dataset in Pytorch supports different sizes, but libtorch does not.</p>
How to sum all the diagonals of a matrix in parallel

omer_sahban: <p>Normally we can sum over any dimention of a tensor, like this:<br>
1+2+3<br>
4+5+6<br>
7+8+9</p>
<p>My question is how to sum all of the diagonals efficiently:<br>
0+0+3<br>
0+2+6<br>
1+5+9<br>
4+8+0<br>
7+0+0</p>
<p>An equivalent question would be how to roll a tensor like this:<br>
1 2 3<br>
4 5 6<br>
7 8 9<br>
↓<br>
1 2 3<br>
5 6 4<br>
9 7 8</p>
<p>And here is a naive solution:</p>
<pre><code class="lang-auto">a = torch.arange(1, 10).view(3,3)
b = torch.zeros_like(a)
for i in range(3)
 b[i] = a[i].roll(shifts=i, dims=0)
</code></pre>
<p>Obviously this will be slow because of the for loop, how should I get rid of the for loop?</p>

RoySadaka: <blockquote>
<p>EDIT:<br>
see my other reply below for possible solution.</p>
</blockquote>
<p>Interesting question!<br>
Still thinking if it’s possible to get rid of the for loop, ill update if an idea comes up (currently testing it with einsum function but no solution so far)</p>
<p>In case ‘for loop’ will stay, I belive using sum+diagonal will be more readable:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.arange(1, 10).view(3,3)
b = torch.zeros(5) # 5 diagonals for 3x3 due to (2*3-1), for nxn, there are (2*n-1) diagonals
for i in range(5):
     b[i] = torch.sum(torch.diagonal(a, offset=2-i))
</code></pre>
<p>Would be nice if torch.diagonal could get a list of offsets <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>output from code above:<br>
<code>tensor([ 3.,  8., 15., 12.,  7.])</code></p>
<p>Roy</p>

RoySadaka: <p>Managed to solve it via conv2d:</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

dim = 3
num_diagonls = 2*dim-1
# need to unsqueeze twice for use in conv2d
x = torch.rand(dim, dim).unsqueeze(0).unsqueeze(0)

print('x:')
print(x)

expected = torch.zeros(num_diagonls)
for i in range(num_diagonls):
    expected[i] = torch.sum(torch.diagonal(x[0][0], offset=dim-1-i))

print('expected diagonal sums:')
print(expected)

# need to unsqueeze twice for use in conv2d
w = torch.eye(dim).unsqueeze(0).unsqueeze(0)
# from after conv2d result, extract inner inner dim, then take the middle column
result = F.conv2d(x, w, padding=num_diagonls//2)[0][0][:, num_diagonls//2]
print('result diagonal sums:')
print(result)
</code></pre>
<p>outputs:</p>
<pre><code class="lang-auto">x:
tensor([[[[0.5106, 0.9985, 0.6385],
          [0.5529, 0.5057, 0.7685],
          [0.0656, 0.4694, 0.0604]]]])
expected diagonal sums:
tensor([0.6385, 1.7670, 1.0767, 1.0223, 0.0656])
result diagonal sums:
tensor([0.6385, 1.7670, 1.0767, 1.0223, 0.0656])
</code></pre>
<p>Roy</p>
Fix model parameters in PyTorch

Ohm: <p>Hi, How can I update my model parameters while I am keeping the original one unchanged?</p>
<p>Fix_net = model()<br>
Fix_net = Fix_net.named_parameters()<br>
NotFix_net_par = dict(Fix_net)</p>
<p>net = model()<br>
net.load_state_dict(NotFix_net_par)<br>
…</p>
<p>Now the Fix_net changed as well. However, I would like to update the NotFix_net_par, but keep Fix_net  unchangedged</p>

ptrblck: <p>You could use <code>copy.deepcopy</code> fo create <code>NotFix_net_par</code>.</p>
Extending a precompiled library

nore: <p>I’m trying to use a precompiled library for custom implementations of standard routines. I’ve been looking on the internet but have not had much luck finding documentation. Does anyone know how to start/ where to look for more information?</p>

ptrblck: <p>What kind of library are you looking for and how does PyTorch come into play here? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you explain your use case a bit more, please?</p>

nore: <p>I am building library to execute deep learning models with limited memory. This library has custom implementations of standard routines(forward_pass, etc) written in C++. I have attempted to extend the files as explained from <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this tutorial</a>. However, I have been having issues with dependencies between the library and Pytorch. As an alternate approach, I’m trying to utilize the library precompiled. I know that Pytorch uses the cuDNN library for CUDA. Is it possible to replicate that process for a custom library? I hope that shed some light on my question.</p>
<p>Thank you for your help!</p>

ptrblck: <p>You should be able to add your library to <code>cmake</code> as explained <a href="https://stackoverflow.com/questions/29368026/how-to-add-prebuilt-static-library-in-project-using-cmake">here</a>.</p>
Number of Classes for Faster RCNN Multi-Class Object Detection

sarmientoj24: <p>I have a dictionary of classes numbered from 1-19 (starts with 1).<br>
What should be the num_classes for the Faster RCNN predictor?<br>
Should it be 20 (num of classes + 1) or still just 19?</p>

ptrblck: <p>Generally the class indices start at 0, so if you are dealing with 19 classes, you would use indices in <code>[0, 18]</code> unless you want to reserve class index 0 for a specific class such as the background class.</p>
How to convert Keras' "same" padding for Conv2DTranspose in Pytorch

Frank_Martin: <p>Hi,<br>
I have this piece of code from keras</p>
<pre><code class="lang-auto">tf.layers.conv2d_transpose(fcn8, filters=512,
    kernel_size=4, strides=(2, 2), padding='SAME', name="fcn9")
</code></pre>
<p>that I would like to convert to Pytorch. In keras, for the input image of size(4,4), it would yield the image of size (8,8). Without the “same” padding in Pytorch, the operation would give a (10,10) image. Is there anyway I can model this operation?</p>

Abhilash_Srivastava: <p>From what I know, Pytorch doesn’t support this as an inbuilt option, TensorFlow does. Checkout this <a href="https://github.com/pytorch/pytorch/issues/3867#issuecomment-507010696" rel="noopener nofollow ugc">discussion </a> which mentions how dynamic loading makes it hard.<br>
However, there could be ways to hack it by combining asymmtric padding layers with conv2d layers. I wouldn’t bother doing it, unless super useful and just go with the inbuilt padding options. More discussion <a href="https://discuss.pytorch.org/t/is-asymmetric-padding-of-style-same-available-in-pytorch/4354/8">here</a>.</p>
Which network's weights are backpropagated?

hyuntae: <p>Suppose there are three networks.<br>
When the input is I<br>
First network(f) is encoder : f(I) = a<br>
Second network(g) is decoder: g(a) = b<br>
Third network(h) is classifier: h(b) = c</p>
<p>My loss function is composed with only a and b like <strong>loss = torch.norm(b, p=1)</strong>.</p>
<p>My question is when I implement loss.backward() with my loss function 'torch.norm(b,  p=1)<br>
which network’s weights are updated?</p>
<p>I think because the loss is composed with the output of f and g,  the first and the second network should be updated and <strong>the third network should not be updated.</strong><br>
But when I implement my code, the third network was also be updated…<br>
I think although I did <strong>third_optimizer.step()</strong>, the grad for the third network should be zero.</p>

KFrank: <p>Hi Hyuntae!</p>
<aside class="quote no-group" data-username="hyuntae" data-post="1" data-topic="101186" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyuntae/48/15469_2.png" class="avatar"> hyuntae:</div>
<blockquote>
<p>But when I implement my code, the third network was also be updated…<br>
I think although I did <strong>third_optimizer.step()</strong>, the grad for the third network should be zero.</p>
</blockquote>
</aside>
<p>If you use weight decay, your optimizer will update the weights even<br>
if their gradients are zero.</p>
<p>(Also, you can easily check whether the gradients are zero.)</p>
<p>Best.</p>
<p>K. Frank</p>

hyuntae: <p>Oh! Thanks <a class="mention" href="/u/kfrank">@KFrank</a> !!</p>
<p>I’m really sorry for ignoring you, can you tell me how I can check the gradient?<br>
Actually, I wrote a code for checking gradient by looking at the change of weight like</p>
<pre><code>         print(self.classifier.fc[0].weight[0:1])
         loss.backward()
         print(self.classifier.fc[0].weight[0:1])
</code></pre>
<p>But it is too coarse code.<br>
I would really appreciate it if you could let me know.</p>
<p>Thanks…!</p>

KFrank: <p>Hi Hyuntae!</p>
<aside class="quote no-group" data-username="hyuntae" data-post="3" data-topic="101186" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyuntae/48/15469_2.png" class="avatar"> hyuntae:</div>
<blockquote>
<p>can you tell me how I can check the gradient?</p>
</blockquote>
</aside>
<p>Look at the <code>grad</code> property of the tensor.  So in your case:</p>
<pre data-code-wrap="python"><code class="lang-python">self.classifier.fc[0].weight[0:1].grad
</code></pre>
<p>If the gradient hasn’t been created yet (for example, if <code>.backward()</code><br>
hasn’t been called) <code>.grad</code> will be <code>None</code>.  Otherwise, <code>.grad</code> will be<br>
a tensor, possibly zero.</p>
<p>Best.</p>
<p>K. Frank</p>

hyuntae: <p>Thanks!</p>
<p>It really helped me a lot of help.</p>
What does .new() mean from Pytorch 0.4

JamesDickens: <p>I am porting some code from Pytorch 0.4 to 1.7, and in the docs (and similarly in the code I am porting) I come across:</p>
<blockquote>
<p>input = torch.empty((2,3), dtype=torch.int64)<br>
input.new(input.size())</p>
</blockquote>
<p>I’m not sure what input.new does, considering input is an empty array of shape (2,3) and dtype torch.int64. Any insights appreciated.</p>
Trouble with indexing with ConcatDataset

nianen88: <p>Hello, I’m brand new to Pytorch and machine learning in general and I had a problem when using ConcatDataset with the MNIST dataset. I’m trying to copy certain parts of the MNIST train set and change them and then add them into the dataset. My original MNIST train_set is size 60000, and after I apply transforms.ToTensor(), my train set consists of 60000 tuples with form (image, label). However, when I try new_set = ConcatDataset(datset),<br>
where datset = [train_set, modified_tuple] and modified_tuple is one tuple of form (image, label), my new_set doesn’t count the tuple as one item and so my new_set has size 60002 instead of 60001. Does anyone have any advice on how to fix this problem? I would like my new_set to be size 60001 where my modified tuple doesn’t become 2 separate items in my new_set.<br>
Thank you!</p>

ptrblck: <p>Try to pass the <code>tuple</code> in a list as seen here:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(root='data', transform=transforms.ToTensor())
print(len(dataset))
&gt; 60000
sample = dataset[0]

concat_dataset = torch.utils.data.ConcatDataset([dataset, [sample]])
print(len(concat_dataset))
&gt; 60001
print(concat_dataset[60000] == sample)
&gt; True
</code></pre>

nianen88: <p>Thank you so much for the response! I was able to find a different solution, but this way is more concise.</p>
Adding row,column numbers to make_grid of tensorboard

TinfoilHat0: <p>I’m using tensorboard’s</p>
<p><code>make_grid</code></p>
<p>functionality to make grid of bunch of images, and then display them as seen in the image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe8a5e574d9fb7b2ffc403ed77a0652474ed911b.png" alt="grid_of_images" data-base62-sha1="AjLJiJxBE8S6ab0KRM5nI2R7jej" width="384" height="417"></p>
<p>I’d like to ask if it’s possible to add row and column numbers to a grid so I can refer individual images easily when discussing with others.</p>

ptrblck: <p>Are you sure <code>tensorboard</code> has a <code>make_grid</code> method and you are not using the one from <a href="https://pytorch.org/docs/stable/torchvision/utils.html"><code>torchvision.utils.make_grid</code></a>?<br>
In the latter case, you can specify the number of rows via <code>nrow</code>.</p>

TinfoilHat0: <p>yes, you’re right. I mean torcvision’s make grid.</p>
<p>I know I can specify the number of rows like that but i’d rather want to add indexes by the sides of grid to<br>
to highlight a particular image in the grid.</p>
<p>Like I’d like to see 1 2 … nrows at the top of the grid, and 1 2 … ncol at the left side of the grid for example.</p>
RuntimeError: mat1 dim 1 must match mat2 dim 0 3DCNN

Anna_yah: <p>Iused pytorch to create 3DCNN<br>
My code works with 2 layers of conv. But does not work with 3 layers of conv.<br>
input image size:120<em>120</em>120<br>
Result: RuntimeError: mat1 dim 1 must match mat2 dim 0<br>
I think the problem is with this line <code>out = self.fc1(out)</code><br>
can you help me please ?<br>
thank you in advance!</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__() # héritage
        
        self.conv_layer1 = self._conv_layer_set(3, 32) #The first dimension of Pytorch Convolution should always be
                                                        #the the number of channels (3)
        self.conv_layer2 = self._conv_layer_set(32, 64) 
        self.conv_layer3 = self._conv_layer_set(64, 128)
        self.fc1 = nn.Linear(128*28*28*28, 2) # couche linéaire entièrement connectée,
        self.fc2 = nn.Linear(2809856, num_classes) ###### couche linéaire entièrement connectée 
        self.relu = nn.LeakyReLU()
        self.batch=nn.BatchNorm1d(2)
        self.drop=nn.Dropout(p=0.15, inplace = True)   
        
    def _conv_layer_set(self, in_c, out_c):
        conv_layer = nn.Sequential(
        nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0),
        nn.LeakyReLU(),
        nn.MaxPool3d((2, 2, 2)),
        )
        return conv_layer
    

    def forward(self, x):
        # Set 1
        out = self.conv_layer1(x)
        out = self.conv_layer2(out)
        out = self.conv_layer3(out)
        #out = out.view(out.size(0), -1) #regler l'output pour devenir l'input de Flly conncted layer 
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.relu(out)
        out = self.batch(out) # batchnormalization pour 
        out = self.drop(out)
        out = F.softmax(out, dim=1) # softmax dim = 1 puisque 1 sig. batch size  )
        return out
</code></pre>

ptrblck: <p>Could you add a <code>print</code> statement before the usage of <code>fc1</code> and check the shape of the activation tensor?</p>
<pre><code class="lang-python">print(out.shape)
out = self.fc1(out)
</code></pre>
<p>I guess that <code>out.shape</code> doesn’t match the expected input features as <code>128*28*28*28</code> so that you might have to adjust this value in <code>self.fc1</code>.</p>

Anna_yah: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
DataParallel() takes more time

gnadaf: <p>If I use DataParallel() in tranformer model (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a>)</p>
<p>The time for each epoch increases as we increase the number of GPUs,</p>
<p>Example: time for one epoch<br>
1 GPU = 42 sec - 44 sec<br>
4 GPUs = 175 sec - 180 sec</p>
<p>Why performance is decreasing if we increase the number of GPUs, is it a drawback in DataParallel()?</p>

ptrblck: <p><code>nn.DataParallel</code> creates model replicas in each forward pass and thus needs to broadcast a lot of parameters. We generally recommend to use <code>DistributedDataParallel</code> with a single process per device for the fastest approach(also on a single node with multiple GPUs).</p>

Abhilash_Srivastava: <p><a class="mention" href="/u/gnadaf">@gnadaf</a> were you able to actually run the model by just adding <code>nn.DataParallel(model)</code>?<br>
I can run my model just fine w/o DataParallel, but w/ DataParallel the training doesn’t proceed (kind of freezes).</p>

gnadaf: <p><a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a><br>
Yes, I am able to run my code, we need to add just nn.DataParallel(model),</p>

gnadaf: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Does it mean nn.DataParallel() has drawback?<br>
Is there no advantages of using nn.DataParalllel?</p>

Abhilash_Srivastava: <p>nn.DataParallel certainly has advantages and it should speed up your training in some cases (try with a simple CNN + FC model).<br>
However, as ptrblck mentioned the major disadvantage of <code>nn.DataParallel</code> is that it creates model replicas in each forward pass and thus needs to broadcast a lot of parameters.<br>
Since most transformer models are huge (w/ millions of parameters), the advantage of increased speed (due to data parallelism) is overshadowed by the reduction in speed due to broadcasting.<br>
Hence, <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">DistributedDataParallel</a> is the recommended way.</p>
Dead SSD amidst training

NTL: <p>Last night I put my model on training. A transformer model for Lyft5 Kaggle competition. Lo and behold, the next morning my SSD (Biostar m500 1TB) is dead (I mean dead as a brick and sent back to use warranty). The dataloader had memory leak despite my efforts to tune workers and batch size. I don’t know what happened. Whether it was swap memory overload or checkpoint model saving that lead to this event, I have no idea.<br>
I was wondering if any of you guys had such an experience. Thank you.<br>
PC spec:<br>
core i9 10980xe<br>
64gb of RAM<br>
RTX 2080TI</p>
What is the correct way to validation for DistributedDataParallel

Sangwon_Jake: <p>I’m trying to train multi gpu DistributedDataParallel.</p>
<pre><code class="lang-auto"> for epoch in range(args.start_epoch, args.num_epoch):
        
        train(train_loader, model, scheduler, optimizer, epoch, args)

        if (epoch + 1) % 1 == 0:
            print('test args...',args)
            validation(valid_dataset, model, epoch, args)

        state = {
            'epoch': epoch,
            'parser': args,
            'state_dict': get_state_dict(model)
        }

        torch.save(
            state,
            os.path.join(
                args.save_folder,
                args.dataset,
                args.network,
                "checkpoint_{}.pth".format(epoch)))
</code></pre>
<p>this code give some error</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/jake/venv/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/home/jake/Gits/EfficientDet.Pytorch/train.py", line 341, in main_worker
    test(valid_dataset, model, epoch, args)
  File "/home/jake/Gits/EfficientDet.Pytorch/train.py", line 164, in test
    evaluate(dataset, model)
  File "/home/jake/Gits/EfficientDet.Pytorch/eval.py", line 190, in evaluate
    generator, retinanet, score_threshold=score_threshold, max_detections=max_detections, save_path=save_path)
  File "/home/jake/Gits/EfficientDet.Pytorch/eval.py", line 103, in _get_detections
    2, 0, 1).cuda().float().unsqueeze(dim=0))
  File "/home/jake/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jake/venv/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/jake/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jake/Gits/EfficientDet.Pytorch/models/efficientdet.py", line 61, in forward
    inputs, annotations = inputs
ValueError: not enough values to unpack (expected 2, got 1)
</code></pre>
<p>seems like it run gpu0 and gpu1 and try to validation one by one.<br>
but some how second one does not recognized as validation then gives this error</p>
Backpropagation with one-hot encoded feature

cliffplaysdrums: <p>I’m trying to one-hot encode a feature in my network, but I’m not sure how to properly update the weights or if this is a good approach at all. Any feedback is appreciated.</p>
<p>My current approach:</p>
<p>A helper class for the encoded feature:</p>
<pre><code class="lang-python">class Encoder(torch.nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.fc = torch.nn.Linear(in_features=num_classes, out_features=1)

    def forward(self, x):
        return self.fc(x.float())
</code></pre>
<p>My actual network’s forward method assumes the first feature is to be encoded (pclass is the name of the feature &amp; <code>self.pclass_encoder</code> is an instance of the above helper class):</p>
<pre><code class="lang-python">def forward(self, x):
    real_features = x[:, 1:]
    encoded_features = torch.nn.functional.one_hot(x[:, 0].long() - 1)
    sig = torch.nn.Sigmoid()
    pclass_out = self.pclass_encoder.forward(encoded_features)
    x = torch.cat((pclass_out, real_features), 1)
    x = self.fc1(x)
    x = sig(self.out(x))
    return x
</code></pre>
<p>In my training loop, I hoped that the gradient would propagate by just adding an extra optimizer for the helper class and stepping it like I do for the main network.</p>
<pre><code class="lang-python">optimzer = torch.optim.Adam(model.parameters(), lr=0.01)
encoder_optimizer = torch.optim.Adam(pclass_encoder.parameters(), lr=.01)

EPOCHS = 4000

for i in range(EPOCHS):
    predictions = model.forward(features)
    loss = loss_fn(predictions, labels)

    optimzer.zero_grad()
    encoder_optimizer.zero_grad()
    loss.backward()
    optimzer.step()
    encoder_optimizer.step()
</code></pre>
<p>The <code>weight</code> attribute of the encoded layer does appear to be updating.</p>

hash-ir: <p>To me, the entire thing seems to be end-to-end trainable. Have you tried using only the model optimizer and checking if the weights of <code>Encoder</code> fully connected layer are updated or not?</p>

cliffplaysdrums: <p>Good call! Originally I only used the one optimizer, but I got sidetracked by my troubleshooting and forgot to revisit that.</p>
<p>The weights of <code>Encoder</code> do in fact get updated with just the model optimizer. This is what I was hoping for and had assumed that the model graph would include anything through which my input tensors were passed.</p>
Best way to resuse oversized pinned buffer

iiSeymour: <p>I’m trying to improve the performance of my PyTorch code by using a preallocated page locked tensor to copy all my results from the GPU into. The results from the GPU will vary in size in the first dimension so I created a pinned buffer large enough to handle the max sized result.</p>
<p>What is the most performant way to copy the undersized results in the pinned memory and then to numpy arrays of their original size?</p>
<pre><code class="lang-auto">import torch

channels = 8
maxsize = 10

# create a page locked buffer on the cpu of masize in first dimension                                                                                                                                                                
buf = torch.zeros([maxsize, channels], pin_memory=True)

# create smaller tensors on the gpu                                                                                                                                                                       
data = [
    torch.empty((i, channels), device="cuda").fill_(i)
    for i in range(1, 6)
]

model = lambda x: x
results = []

for res in data:

    # run dummy model                                                                                                                                                                                     
    res = model(res)

    # best way to async copy the smaller res tensor from the gpu?                                                                                                                                         
    buf[:res.shape[0], :] = res

    # check buffer looks correct                                                                                                                                                                          
    print(buf[:, 0], buf.data_ptr())

    # do something with the correct sized results                                                                                                                                                         
    results.append(buf[:res.shape[0], :].clone().numpy())


# check results look correct                                                                                                                                                                              
for res in results:
    print(res[:, 0])
</code></pre>
<pre><code class="lang-auto">tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([2., 2., 0., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([3., 3., 3., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([4., 4., 4., 4., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([5., 5., 5., 5., 5., 0., 0., 0., 0., 0.]) 140124862021632
[1.]
[2. 2.]
[3. 3. 3.]
[4. 4. 4. 4.]
[5. 5. 5. 5. 5.]
</code></pre>

googlebot: <p>in theory,<br>
buf.narrow(0,0,res.shape[0]).copy_(res,non_blocking=True)</p>
How to call MSE_loss

Omroth: <p>This looks like it should be right to me:</p>
<pre><code class="lang-auto">torch::Tensor loss = torch::mse_loss(prediction, desired_prediction.detach(), torch::nn::functional::MSELossFuncOptions(torch::kNone));
    loss.backward();
</code></pre>
<p>But I’m getting</p>
<blockquote>
<p>error: cannot convert ‘torch::nn::functional::MSELossFuncOptions {aka torch::nn::MSELossOptions}’ to ‘int64_t {aka long int}’ for argument ‘3’ to ‘at::Tensor at::mse_loss(const at::Tensor&amp;, const at::Tensor&amp;, int64_t)</p>
</blockquote>
<p>I’m sure it’s a simple thing - any idea what’s wrong?</p>

vaasudev96: <p>Changing <code>torch::mse_loss</code> to <code>torch::nn::functional::mse_loss</code> should make it work.</p>
<p>mse_loss is defined under <code>torch::nn::functional</code> namespace.</p>
<p><a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1nn_1_1functional_1aba6a5341298f43163c046b3dd50dc6ce.html#function-torch-nn-functional-mse-loss" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/cppdocs/api/function_namespacetorch_1_1nn_1_1functional_1aba6a5341298f43163c046b3dd50dc6ce.html#function-torch-nn-functional-mse-loss</a></p>

offset-null1: <p>It gives me an error saying :</p>
<blockquote>
<p>error: ‘functional’ is not a namespace-name</p>
</blockquote>
<p>And if I go to its definition it gets me to the thrust API. What could be wrong?</p>
Different performance in identical architectures

Aditya1: <p>I have re-implemented ResNet 18. I verified if the architecture is correct by printing my model and the actual pytorch resent implementation. However my resnet model does not seem to learn, what could be the reason for this eventhough printing the model shows that the archs are identical?</p>

ptrblck: <p>Printing the model via <code>print(model)</code> shows only all created modules, not the <code>forward</code> implementation and thus neither how the modules are used.<br>
Also all functional calls are missing from the output, which could be used in the <code>forward</code>.<br>
You could either compare the source codes directly (your vs. torchvision implementation) or load the same <code>state_dict</code> into both models and compare the output of each layer to narrow down where a difference might be coming from.</p>

Shisho_Sama: <p>it could be much easier to help if you could provide a minimal repreducible example.<br>
apart from what ptrblk said, you could be missing the proper weight initialization. check that as well.</p>

Aditya1: <p>I just knew the first answer would come from ptrblck!</p>
<p>Thanks for the insight, the problem was in my forward block. Naive of me to not consider that!</p>
Geometric mean of model weights

en_tetteh: <p>How do I compute geometric mean of the weights and biases in a federated learning settings?<br>
I’m able to compute the arithmetic mean, only.</p>

KFrank: <p>Hi en!</p>
<aside class="quote no-group quote-modified" data-username="en_tetteh" data-post="1" data-topic="101095" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/en_tetteh/48/30070_2.png" class="avatar"> en_tetteh:</div>
<blockquote>
<p>How do I compute geometric mean?</p>
</blockquote>
</aside>
<p>You can calculate the geometric mean directly.  Suppose that you<br>
have the weights and/or biases whose geometric mean you want<br>
in a single tensor, <code>w</code>.  Then:</p>
<pre data-code-wrap="python"><code class="lang-python">geom_mean = w.prod()**(1 / w.numel())
</code></pre>
<p>Note, it might be preferable to perform the calculation in log-space:</p>
<pre data-code-wrap="python"><code class="lang-python">geom_mean = w.log().mean().exp()
</code></pre>
<p>(The geometric mean only makes sense – for some definition of<br>
“makes sense” – if the values are all strictly positive.  As you can<br>
see, both the fractional power in the direct-space calculation and<br>
the <code>log()</code> in the log-space calculation will fail for negative values.)</p>
<p>Best.</p>
<p>K. Frank</p>

en_tetteh: <p>Thanks for your response Frank. I tried your approach, but it didn’t work for my case.<br>
Instead of the mean computation below, I want the geom.</p>
<pre><code class="lang-auto">        model_dict[k] = th.stack([h_models[i].state_dict()[k].float()*(n*h_lens[i]/total) for i in range(len(h_models))], 0).mean(0)```

code source: (https://towardsdatascience.com/preserving-data-privacy-in-deep-learning-part-3-ae2103c40c22)</code></pre>
Pytorch lightning validation_epoch_end error

nivesh_gadipudi: <pre><code class="lang-auto">def validation_epoch_end(self, outputs):
    	val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
    	log = {'avg_val_loss': val_loss}
    	return {'log': log, 'val_loss': log}
</code></pre>
<p>When i am using the above method in pytorh lightning it throws me an error <code>TypeError: cannot unpack non-iterable NoneType object</code></p>
<p>This is happening before the last step of validation data. I can see that the r<code>eturn {'log': log, 'val_loss': log}</code> is where the error is but couldnt actually find out whats wrong</p>

Abhilash_Srivastava: <aside class="quote no-group" data-username="nivesh_gadipudi" data-post="1" data-topic="101169">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivesh_gadipudi/48/38188_2.png" class="avatar"> nivesh_gadipudi:</div>
<blockquote>
<p><code>[x['val_loss'] for x in outputs]</code></p>
</blockquote>
</aside>
<p>Check if this part is returning the results as expected.</p>

nivesh_gadipudi: <pre><code class="lang-auto">    def validation_epoch_end(self, outputs):
    	val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
    	log = {'avg_val_loss': val_loss}
    	return {'val_loss': val_loss, 'log': log}
</code></pre>
<p>I used it instead like this and now its working, thanks <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a></p>
Why are some vanilla RNNs initiliazed with a hidden state with a sequence_length=1 for mnist image classification

Thabang_Lukhetho: <p>I came across several examples of classifying MNIST digit using a RNN, what it the reason to initialize the hidden state with a sequence_length=1? If you were doing 1 step ahead prediction of a video frame prediction, how would you initialize it?</p>
<pre><code class="lang-auto">def init_hidden(self, x, device=None): # input 4D tensor: (batch size, channels, width, height)
    # initialize the hidden and cell state to zero
    # vectors:(number of layer, sequence length, number of hidden nodes)
    if (self.bidirectional):
        h0 = torch.zeros(2*self.n_layers, 1, self.n_hidden)
    else:
        h0 = torch.zeros(self.n_layers,  1, self.n_hidden)

    if device is not None:
        h0 = h0.to(device)
    self.hidden = h0
</code></pre>
<p>The input is usually represented as</p>
<pre><code class="lang-auto">inputs = inputs.view(batch_size*image_height, 1, image_width)
</code></pre>
<p>In this above example are the images passed columns-wise? Is there another way to represent inputs images in RNN? And how does it related to how one initialize the hidden state?</p>
Standard deviation cannot calculate same

yoko: <p>method1<br>
mu_x = nn.functional.avg_pool2d(x, window_size, 1, padding=1)<br>
sigma_x = nn.functional.avg_pool2d((x  - mu_x)**2, window_size, 1, padding=1)<br>
print(“sigma_x1=”,torch.max(sigma_x))<br>
print(“sigma_x1=”,torch.min(sigma_x))</p>
<p>method2<br>
mu_x = nn.functional.avg_pool2d(x, window_size, 1, padding=1)<br>
sigma_x = nn.functional.avg_pool2d(x<strong>2, window_size, 1, padding=1)-mu_x</strong>2<br>
print(“sigma_x2=”,torch.max(sigma_x))<br>
print(“sigma_x2=”,torch.min(sigma_x))</p>
<p>I input same gray scaled image. but result seems diferrent</p>
<p>output<br>
(‘sigma_x1=’, tensor(5441.0488, device=‘cuda:0’))<br>
(‘sigma_x1=’, tensor(0., device=‘cuda:0’))</p>
<p>(‘sigma_x2=’, tensor(16055.5547, device=‘cuda:0’))<br>
(‘sigma_x2=’, tensor(-0.0078, device=‘cuda:0’))</p>
<p>Do you have any idea what’s wrong?<br>
and standard deviation cannot be minus. so I guess something wrong with method 2</p>
Doing operations on different parts of a tensor in parallel

antspy: <p>Hi,</p>
<p>assume I have a tensor, and I want to apply a certain function to buckets of its elements. For example, assume I have the tensor</p>
<pre><code>&gt;&gt;&gt; A 
 0
 2
 0
 2
[torch.FloatTensor of size 4x1]
</code></pre>
<p>and I want to compute the mean for every “bucket” of two elements, and replace it, like so:</p>
<pre><code>&gt;&gt;&gt; for idx in range(0, 4, 2):
    A[idx:idx+2] = torch.mean(A[idx:idx+2] )

&gt;&gt;&gt; A 
 1
 1
 1
 1
[torch.FloatTensor of size 4x1]
</code></pre>
<p>The issue is that for loop may be very slow, since it has to execute A.numel()/2 times. Is there any way to make it parallelizable, so that it runs in parallel over multiple buckets at the same time?</p>
<p>Note: The example with the mean is just to clarify what I meant, the actual function is slower and more complicated.</p>
<p>Thank you!</p>

smth: <p>you can use <code>torch.unfold</code> to compute your <code>bucket</code> tensor, and then you can compute the mean for each bucket.</p>
<p><a href="http://pytorch.org/docs/master/tensors.html?highlight=unfold#torch.Tensor.unfold" class="onebox" target="_blank">http://pytorch.org/docs/master/tensors.html?highlight=unfold#torch.Tensor.unfold</a></p>

antspy: <p>Thank you, that worked well and sped it up more than 10x! Cheers <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

antspy: <p>Actually, in my case the buckets don’t overlap, so I think it would be better to just use</p>
<pre><code>tensor.view(-1, bucket_size)
</code></pre>
<p>which avoids creating a new tensor like unfold would. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

mu2: <p><a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/antspy">@antspy</a>  I’m sorry I don’t understand the answer. I could bucket the tensor using unfold or view, but how do I apply the function (mean in this case) in parallel?</p>

ptrblck: <p>After creating the “buckets” or windows using <code>unfold</code>, you could flatten the elements in this bucket and apply e.g. <code>torch.mean</code> on this dimension.</p>