akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch row by row element-wise multiplication

hezi_S: <p>I’m wondering how to achieve the multiplication like this in Pytorch:</p>
<p>x = [[a, b], [c, d]]<br>
y = [[1, 2], [3, 4]]</p>
<p>result = [[a<em>1, a</em>2, b<em>1, b</em>2], [c<em>3, c</em>4, d<em>3, d</em>4]]</p>
<p>Is ther anyway to avoid doing it row by row in loop? Thanks a lot!</p>

harsha_g: <p>result = torch.repeat_interleave(x, 2, dim=1) * y.repeat(1, 2)</p>

hezi_S: <aside class="quote no-group" data-username="harsha_g" data-post="2" data-topic="84386">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/48.png" class="avatar"> harsha_g:</div>
<blockquote>
<p>torch.repeat_interleave(x, 2, dim=1) * y.repeat(1, 2)</p>
</blockquote>
</aside>
<p>Exactly! Thank you so much!</p>
Tensor is not conforming the model device

silvester: <p>I added the following function to my model architecture to reshape my parameter vector <code>L_vec</code> inside the model to a triangular shape matrix. However I need to map manually the device of matrix to the device of L_vec.<br>
I am not sure what could be the reason.</p>
<pre><code class="lang-auto">def train_tensor(self, L_vec, n): 
        tri_mat = torch.zeros((n, n)).to(L_vec.device) 
        tril_indices = torch.tril_indices(row=n, col=n, offset=0)
        tri_mat[tril_indices[0], tril_indices[1]] = L_vec.squeeze(1)
        return tri_mat
</code></pre>

ptrblck: <p>Do you see any errors and if so, could you post the error message please?<br>
Or does <code>L_vec.device</code> not give the desired device?</p>

silvester: <p>I wonder why I need manually map the device of <code>tri_mat</code> to the device <code>L_vec</code> while they are located in the same model.</p>
<p>.</p>

ptrblck: <p>You are recreating <code>tri_mat</code> in the <code>forward</code> method, so this tensor is not part of the model but just a temporary tensor, so it won’t use any specific device.<br>
Also note, that a module doesn’t necessarily belong to a single device, since you can use model sharding etc.</p>
How to use the nn.TransformerEncoder

jaemin: <p>Hello.<br>
I made the Transformer encoder to replace RNN encoder. It seems like training well because CE loss continuously decreases but accuracy is the problem. I think this model can’t find  token because the prediction shows ‘E’ of ‘I’ after the end of sentence.<br>
For example,<br>
<strong>Prediction:</strong><br>
AND TN TIOK TOOA D NH AHEULD TOT TORLYW AHE E WN TNDTVT TF TOU  TOC LNEON  ONSHETL TOT TNTEOE TAOAENNYAN N TYST AASL D AH TITTE TH   TEALEED TERWAICE TN WN TN THE SASH D  TORTENN E EEEEEEEEEEEEEEEEEEEEEEEEEIEEEEIEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE<br>
<strong>Answer:</strong><br>
AND IF LORD FREDERICK SHOULD NOT FOLLOW THERE IS AN END OF YOUR SUSPICIONS I SHALL NOT EASILY PREVAIL UPON MISS MILNER TO LEAVE TOWN REPLIED HE WHILE IT IS IN THE HIGHEST FASHION</p>
<p>I simply use a nn.TransformerEncoder module, and replace</p>
<pre><code class="lang-auto">x = nn.utils.rnn.pack_padded_sequence(x, output_lengths)
x, h_state = self.rnn(x)
x, _ = nn.utils.rnn.pad_packed_sequence(x)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">x = self.encoder(src=x)
</code></pre>
<p>The input tensor “x” is shape like (Sequence_length, Batch_size, Feature_dim)<br>
Is there any additional method to use nn.TransformerEncoder ?</p>
Differences between eval() and train() modes

NIkolaStaykov: <p>Which modules are affected by the modes except BatchNorm and Dropout? I was wondering in which cases the two modes are interchangeable.</p>

ptrblck: <p>The <code>train()</code> and <code>eval()</code> call change the internal <code>self.training</code> flag, so you could <code>grep</code> for it in the source folder (<code>grep -r self.training</code>).<br>
Currently it seems these modules are affected by it in the PyTorch core:</p>
<ul>
<li>Quantization modules</li>
<li>Dropout</li>
<li>InstanceNorm</li>
<li>BatchNorm</li>
<li>RNN (probably only if using cudnn, as different implementations will be called)</li>
<li>RReLU</li>
</ul>
<p>Besides that all custom <code>nn.Module</code>s might of course use this flag internally.</p>
Decompose Segmentation

eranb: <p>I have an image and I was doing segmentation on the image. let’s say for example a human body image. Now I want to decompose the image by the segments of the segmentation that will only show me one part, for example, I have segmented every part of the human body I want to show only the hand or only the elbow.<br>
Can someone guide me for a tutorial or explain to me how to do it.<br>
The file I have:<br>
AnnotationROI.acsv<br>
Segmentation.seg.nrrd<br>
Image.nrrd</p>
<p>In the end, I want to do data augmentation on the segmented image and after that to decompose it to 3 images (what is segmented).</p>

ptrblck: <p>How should the segmented parts be visualized?<br>
It should be possible to use a filtering approach, by comparing the pixel values of the output segmentation to a specific class index, and get the min/max indices via:</p>
<pre><code class="lang-python">idx = (pred == class_idx).nonzero() 
</code></pre>
<p>However, you would usually cut a rectangular segment, which might and most likely will include other classes as well. E.g. the “arm class” might be inside the patch of a segmented “hand class”.</p>

eranb: <p>Thank you!<br>
I only remove the nonzero()</p>

eranb: <p>Now I want to go back and project it on the real image, or to reconstruct only the segment image form real, what is the easy way to do that?</p>
<p>my real image:  (slice=78, h=633, w=788)</p>
<p>after I do idx = (pred == class_idx).nonzero() I get:<br>
3 ndarray with 4293160.<br>
Not I get in trouble to project it on the real image and reconstruct only the part of the segment, but with saving the pixel value.</p>
How to relabel a pretrained fasterrcnn model with new class labels

Aiswariya_Sugavanam: <p>How to relabel a pretrained fasterrcnn model with new class labels</p>

ptrblck: <p>You could change the number of classes and retrain the model as described in <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">this tutorial</a>.<br>
Let me know, if I misunderstood your question and you would like to relabel the classes in a different way.</p>
Manually set gradient of the last layer with binary output

Homa: <p>Hello<br>
I am working with binary data in the generative network and want to  generate binary value in the last layer  .<br>
I have added a layer noisy rectified linear unit  with :<br>
max(     0 ,  x + N(0 ,  sigmoid(x)     )<br>
in the last layer  in order to convert values to (0,1)   .</p>
<p>I dont know do how set gradient for this layer and add to back propagation?<br>
please help me.</p>

ptrblck: <p>How did you define the <code>N</code> method?<br>
If I’m not mistaken, this line of code would not normalize the output to <code>[0, 1]</code>, but would clip it at 0 and add probably noise to all positive values?</p>
<p>If <code>N</code> is implemented using PyTorch operations, Autograd would probably be able to create the backward pass for you.</p>

Homa: <p>thanks for reply.<br>
N is the normal distribution .<br>
oooh I forget to say that<br>
if max( 0 , x + N(0 , sigmoid(x) ) &gt; 0   output is 1 else output is 0 .<br>
For back propagation,I take the gradient of this function to be 0 when output &lt;= 0  and 1 when output&gt;0 .</p>
<p>I have found a way for add this function and gradient to backprob, but dont know that is right or not.<br>
I used this link : <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>
<p>and changed relu  and gradient to  my function  and its gradient.</p>

ptrblck: <p>A custom <code>autograd.Function</code> should work.<br>
You could use <code>gradcheck</code> to verify the gradients, if it’s applicable for your method.</p>

Homa: <p>thanks. I will search to use from 'gradcheck '.</p>
Rescale bounding boxes relatively to original image sizes

evgeniititov: <p>Hi,</p>
<p>I hope everyone’s doing great. Could you please give me a hand with the following problem. I believe there’s no such function in pytorch, so I am trying to implement it manually.</p>
<p>Context: I collect a batch of frames and push them to GPU, keep them there. Then, I have a couple of networks (YOLOs). In order for this to work, I need to resize images on GPU, to the size YOLO expects keeping aspect ratio (theres padding that fills the rest of the image with grey colour). This was done in this ticket (<a href="https://discuss.pytorch.org/t/resize-images-torch-tensor-already-loaded-on-gpu-keeping-their-aspect-ratio/83172/9" class="inline-onebox">Resize images (torch.Tensor) already loaded on GPU keeping their aspect ratio - #9 by evgeniititov</a>).</p>
<p>What I need to do now: resize bounding box coordinates so that they are relative to the original size of the frames, not the resized one given to the network for detection. This is how I am doing it now:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b18fa2c579c470c6b5da91a26976aced3503024.png" data-download-href="https://discuss.pytorch.org/uploads/default/5b18fa2c579c470c6b5da91a26976aced3503024" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b18fa2c579c470c6b5da91a26976aced3503024.png" alt="image" data-base62-sha1="cZSXIV5vWXOV6pbpu3GBJgT99xq" width="634" height="500" data-dominant-color="2D2E27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">994×783 49.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Unfortunately, it doesn’t pass the assert statements. My new_top value is usually &lt; 0, new_right can go beyond the right edge.<br>
detections - is a dictionary, where key - index of the image in the batch (1 for images), and values - list of lists. Each nested list is a detection predicted by the network.</p>
<p>If anyone could please give me a hand with it. Spent a lot of time on it, no success.</p>
<p>Thanks in advance.<br>
Eugene</p>

evgeniititov: <p>Just in case anyone ever finds this - the code above works fine. The issue was when the network detects objects, sometimes their coordinates happen to be in the padded area. As a result, when you’re trying to rescale the boxes, they go beyond edges of the original frames.</p>
<p>Does anyone know why this might be happening? I had to write a simple condition like this to catch it. Works fine now.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/702b0accc895edc97f58576a522320d86f78a53e.png" alt="image" data-base62-sha1="g0hH2jEX9oLENnYhgLWH1TpESpE" width="474" height="168"></p>
How to compare the dis-/ similarity between images?

Array: <p>I have a data set which looks like following:</p>
<pre><code class="lang-auto">Image_1    Image_2    Image_3
  A           B         C
 ...         ...       ...
</code></pre>
<p>If image A is similar to B, it will be assigned with label 1, otherwise with label 0.</p>
<p>I first use pre-trained resnet18 to extract features for each RGB image and get a 1000 dimensional vector. Then I build a deep network and use triplet as loss function to train model, there is a part of my code:</p>
<pre><code class="lang-auto">class Network(torch.nn.Module):
    def __init__(self, n_feature = 1000, n_hidden_1,n_output = 10):
        super(Network, self).__init__()
        self.net = torch.nn.Sequential(
                        torch.nn.Linear(n_feature, n_hidden_1),
                        torch.nn.BatchNorm1d(n_hidden_1),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_1, n_output)
                        )
    def forward(self, x):
        x = self.net(x)
        return x
</code></pre>
<p>Training step:</p>
<pre><code class="lang-auto">for step, (batch_anchor, batch_positive, batch_negative )in enumerate(train_loader):
    optimizer.zero_grad()
    anchor_out = model(batch_anchor)
    positive_out = model(batch_positive)
    negative_out = model(batch_negative)
    loss = loss_func(anchor_out, positive_out, negative_out)
    loss.backward()
    optimizer.step()
</code></pre>
<p>where I define loss function and optimiser with:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr=0.002)
loss_func = torch.nn.TripletMarginLoss()
</code></pre>
<p>After the training process is done, I test this network with validation set:</p>
<pre><code class="lang-auto">with torch.no_grad():
    anchor_out_val = model(val_data_anchor).numpy()
    positive_out_val = model(val_data_positive).numpy()
    negative_out_val = model(val_data_negative).numpy()
</code></pre>
<p>Now I use L2 Norm to measure similarity and assign labels, this works very well in validation set and I got accuracy 80% measured by accuracy_score from sklearn. But when I try it with test set, I only get 50% accuracy. Maybe someone could tell me why? Is the metric to measure similarity not good? or maybe it is the problem from network?</p>

ptrblck: <p>If I understand the issue correctly, you are seeing a good training and validation accuracy, but the test accuracy is a lot worse.<br>
Could your training procedure have leaked the validation data somehow into the training, so that its accuracy might be biased?<br>
Also, are you processing the training and validation data in a different way compared to the processing of the test set? Are all dataset splits generated randomly from the same data pool or were they sampled differently?</p>
Two instances of conv sharing same weights

s_n: <p>Is it possible that two instances of a convolutional layer in my <strong>init</strong> method can share same set of weights?</p>
<p>Ex:<br>
self.conv1 = nn.Conv2d(…)</p>
<p>self.conv2 = MycustomConvFunction(…)</p>
<p>So I want self.conv1 and self.conv2 to share same set of weights.<br>
Actually I want self.conv2 for inference and self.conv1 for training.</p>

ptrblck: <p>The most elegant way would probably be the functional API, which would only create a single <code>weight</code> parameter and just use it when it’s needed.<br>
Alternatively, you could assign the <code>weight</code> parameter to your modules as described <a href="https://discuss.pytorch.org/t/how-to-use-shared-weights-in-different-layers-of-a-model/71263/2">here</a>.</p>

s_n: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I saw the link that you attached. Will I have to every time make the weights equal for two instances in the forward method?<br>
Or I can just equate the weights of two nn.Linear instances once and they will share same storage location and weight values for all the epochs.</p>
<p>For example</p>
<p>class testModule(nn.Module):</p>
<pre><code>def __init__(self):
    super(testModule, self).__init__()
    self.fc1 = nn.Linear(5, 10, bias=True)
    self.fc2 = MyLinearLayerModel(10, 10, bias=False)
def forward(self, x, p=False):
    if p = True:
         x = self.fc1(x)
   else:
        self.fc2.weight.data  = self.fc1.weight.data
         x = self.fc2(x)
    return x
</code></pre>
<p>Or I can just do “self.fc2.weight.data  = self.fc1.weight.data” once inside <strong>init</strong>?</p>
How weights are used for making prediction?

sgaur: <p>HI<br>
how weights are used for prediction the only thing I see is whenever we need to predict we used model <strong>ypred_valid =  torch.(model(valid_images))</strong> where model is architecture of my NN.</p>
<pre><code class="lang-auto">for index,(valid_images,valid_labels) in enumerate(validation_loader):
    valid_images = valid_images.to(device)
    valid_labels = valid_labels.to(device).float()
    ypred_valid =  torch.sigmoid(model(valid_images))
</code></pre>
<p>Usually I have multiple weights for same model So how I need to combine different models weights with same model architecture.</p>
<p>Thanks.!!</p>

Sudarshan_Babu: <p>Sorry, I am not able to understand your question, is your question on how to ensemble different models ?</p>

sgaur: <p>Sorry!! let me rephrase it.</p>
<p>So basically I capture weights of last three epochs (epoch =8,9, and 10) for same model.<br>
Now I want to predict my test data-set.</p>
<p>ypred_valid =  torch.sigmoid(model(valid_images)) ## where weights of epoch 8 is used.<br>
ypred_valid =  torch.sigmoid(model(valid_images)) ## where weights of epoch 9 is used.<br>
ypred_valid =  torch.sigmoid(model(valid_images)) ## where weights of epoch 10 is used.</p>
<p>In all three of them my model is same but model.state_dict() is different(which refers to weights.)</p>
<p>Now, How I load my model which takes weights(model.state_dict().) of epoch 8,9 and 10.</p>

Sudarshan_Babu: <p>Well why do something like  this:</p>
<pre><code class="lang-auto">
for epoch in range(total_epochs):
    current_model = train()
    ypred_valid = torch.sigmoid(model(valid_images))
    if epoch == 8 or epoch ==9 or epoch 10:
        print(ypred_valid)
</code></pre>
<p>Sorry, if this is not the solution you were looking for, but does this not solve your use case?</p>
Multiply conv layer weight by 2

Alpha: <p>For example, I want to explicitly multiply 2 on the convolution layers’ weight.</p>
<p>I motify the official <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py#L376-L377" rel="nofollow noopener">conv.py.py</a>  from :</p>
<pre><code class="lang-auto">    def forward(self, input):
        return self._conv_forward(input, self.weight)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">    def forward(self, input):
        new_weight = self.weight * torch.ones(self.weight.size())*2
        return self._conv_forward(input, new_weight)
</code></pre>
<p>I noticed that the type of <code>new_weight</code> becomes <code>torch.Tensor</code>  rather than <code>torch.nn.parameter.Parameter</code>.</p>
<p>So I wonder whether it’s OK to do this? And can the backward process works correctly?<br>
<a class="mention" href="/u/alband">@albanD</a><br>
Thanks in advance!</p>

albanD: <p>Hey,</p>
<p>No need to ping, we look at the posts <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>And yes it is expected that after doing any differentiable operation on it, you get a Tensor.<br>
Keep in mind that nn.Parameter are only the leafs stored in the nn.Module that nn recognize as being part of the parameters.</p>
<p>Also you can write the same thing as <code>new_weight = self.weight * 2</code> to make it easier to read <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Alpha: <p>Thank you very much.</p>
Placing measurement value into 2D tensor location by value of another tensor

sanjayk: <p>I have a series of measurements made over a 2D space. The measurements come as [x, y] and M where x,y are the coordinates of where the measurement was made and M is the result of the measurement. I want to use these measurements to update a 2D grid of values, let’s say dim [11,11], that represents my knowledge of the underlying function on a 1 meter grid. The 2D grid can be considered a discretized basis the underlying function.<br>
The x and y of each measurement are not discretized with each of x and y ranging from 0 to 10. I would like to convert each incoming measurement into a dim [11,11] tensor where the value of the tensor is zero everywhere except in the vicinity of x,y. For example:<br>
If x,y were 0,0 I want the [1,1] location of the tensor to be M.<br>
If x,y were 0.5,0.5, I want locations [1,1], [1,2], [2,1] and [2,2] to each be M/4 etc.<br>
If x,y were 10,10, I want location [11,11] to be M</p>

ptrblck: <p>Based on your description it seems your use case is a linear interpolation.<br>
Could you post a dummy input tensor with an example tensor containing the indices, please?</p>

sanjayk: <p>Yes, it is interpolation, but just over a few locations.<br>
Here’s an example for the case where x,y is 0,1 and the measurement is 3.4.<br>
I’ve showing it implemented using sparse tensors. How do I generalize it to the case where the x,y values are not integers?<br>
location = torch.LongTensor([[1],[2]])<br>
measurement = torch.FloatTensor([3.4])<br>
torch.sparse.FloatTensor(location,measurement, torch.Size([11,11])).to_dense()</p>
<p>tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 3.4000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000],<br>
[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,<br>
0.0000, 0.0000]])</p>
How to apply 3D convolution to stacked video frames

Erhan: <p>I am working on video classification for motion recognition.<br>
I selected 10 frame from video and applied optical flow these sequantial frames.<br>
After that, I found x and y  direction of each flows and stack them together.<br>
Finally, I have 3D matrix with shape H x W x 20.<br>
H = height<br>
W = weight<br>
20 = flows from 10 frame (2 times because of x and y dimension of optical flow).<br>
So, I want to apply 3D convolution layer this matrix above.<br>
When I look at pytorch documentation for 3D convolution, I saw 5 dimensional input like that (N,C,D,H,W).<br>
But my input is 4D dimensional like that (N,C,H,W) with N samples.<br>
So, How can I apply 3D convolutional to my matrix?</p>

ebarsoum: <p>N is the batch size, the number of sample. In your case, sample != single frame. Your sample is HxWx1x20., (c=1, you have only one channel) .</p>
<p>To map that to (N,C,D,H,W). C=1, D=20, H=H, and W=W. N depends on your batch size, and if it fits in memory or not.</p>

Erhan: <p>Thanks a lot for reply.</p>

Erhan: <p>I want to ask one more question.<br>
As <a class="mention" href="/u/ebarsoum">@ebarsoum</a> mentioned, I have HxWx1x20 numpy uint8 array.<br>
I want to implement a dataset class that inherit torch.utils.data.Dataset.</p>
<p>In <strong>getitem</strong> part of the function, I return this array by applying transforms like:</p>
<p>transforms.RandomCrop,<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ColorJitter(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]).</p>
<p>To be able apply the first three of them requires PiL image.<br>
How can handle this ?</p>
RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: train`

aayush: <pre><code class="lang-auto">RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: train`
</code></pre>

albanD: <p>Hi,</p>
<p>What were you doing when this happened? Can you give a full stack trace and the surrounding code?</p>

aayush: <pre><code class="lang-auto">`Traceback (most recent call last):
  File "pytorch_connectomics/scripts/main.py", line 72, in &lt;module&gt;
    main()
  File "pytorch_connectomics/scripts/main.py", line 65, in main
    trainer = Trainer(cfg, mode, args.checkpoint, device)
  File "/n/home11/kguliani/pytorch_connectomics/connectomics/engine/trainer.py", line 29, in _init_
    self.model = build_model(self.cfg, self.device)
  File "/n/home11/kguliani/pytorch_connectomics/connectomics/model/_init_.py", line 27, in build_model
    model = model.to(device)
  File "/n/home11/kguliani/.conda/envs/py3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 431, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: train`
</code></pre>

albanD: <p>Is your device “train” by any chance? This is not a device name.</p>
How to set retain_graph=true in C++?

cruzas: <p>Hi everyone,</p>
<p>Could someone tell me how to set <code>loss.backward(retain_graph=True)</code> in C++?</p>
<p>Thank you.</p>

ebarsoum: <p>Should be similar, here the signature:</p>
<pre><code class="lang-auto">void backward(const Tensor &amp;gradient = {}, c10::optional&lt;bool&gt; retain_graph = c10::nullopt, bool create_graph = false) const;
</code></pre>
Efficient way to produce batch specific tree-structured output

shivin7: <p>Hi,<br>
I’m trying to make kind of a recursive network that takes in sequential input and produces a binary-tree structured output.<br>
After processing the input and doing some computations, the model produces an intermediate <code>y_hat</code> of shape <code>(batch, )</code>, the values of which determine whether each specific sample in the batch needs to be split further or not (1 means split, 0 means don’t). Since different samples in the batch will produce different sized trees, I can’t figure out how to process this efficiently.</p>
<p>Currently I’m thinking of doing something like this :</p>
<pre><code class="lang-auto">def buildTree(self, inputs):
  y_hat =  """ Do some computations """
  completedTreeBatch = inputs[y_hat == 0]
  splitTreeBatch = inputs[y_hat == 1]
  if splitTreeBatch != None:  
    returnTreeBatch = buildTree(self, splitTreeBatch)
    completedTreeBatch = torch.cat((completedTreeBatch, returnTreeBatch))
  return completedTreeBatch

def forward(self, inputs):
  """ Do some computations """
  treeOutput = self.buildTree(inputs)
</code></pre>
<p>This would involve a lot of unnecessary padding and concatenating. Is there any better way to go about this?<br>
Also, is there a more efficient way to implement something like beam search to build the tree in PyTorch?</p>
<p>Any help would be really appreciated!</p>
How to store and resume visdom experiment window in environment after restart visdom.py

sekigh: <p>I created visdom experimental windows in environment " tutorial plot" by using sample codes, train.py and utils.py shown <a href="https://github.com/noagarcia/visdom-tutorial" rel="nofollow noopener">https://github.com/noagarcia/visdom-tutorial</a> and successfully obtained loss and accuracy curves on windows on browser. See these obtained figures below.  <a href="https://user-images.githubusercontent.com/26996180/83378691-722bdd80-a414-11ea-94ac-cb9612c9fd9a.png" rel="nofollow noopener">Screenshot at 2020-06-01 14-26-41</a></p>
<p>However, I have problem in how to store (or save) windows with created curves and resume them. Once I stop “$python visdom”, the windows are gone and never show up again even if I start “$ python visdom”. How should I do ? I reviewed <a href="https://github.com/facebookresearch/visdom#overview" rel="nofollow noopener">https://github.com/facebookresearch/visdom#overview</a> and followed description under “save” for window save under the name of “saved_data” but never restore them again.</p>
<p>Thank you in advance.</p>

ptrblck: <p>For completeness, the issue is also tracked <a href="https://github.com/facebookresearch/visdom/issues/737">here</a>.<br>
Unfortunately, I don’t know why it’s not working .</p>
<p>That being said, the <a href="https://github.com/facebookresearch/visdom/graphs/code-frequency">code frequency</a> doesn’t show much movement in the repository, so maybe you could try other libraries?</p>

sekigh: <p>Thank you for your reply. I am not sure what code frequency indicates on <a href="https://github.com/facebookresearch/" rel="nofollow noopener">https://github.com/facebookresearch/</a>. I suppose it means how frequently codes for visdom package are downloaded for use. or what frequency for code updates is. I did install visdom by “conda install visdom” and did not download and install visdom the source codes from <a href="https://github.com/facebookresearch/" rel="nofollow noopener">https://github.com/facebookresearch/</a>.  Then I downloaded the sample codes from <a href="https://github.com/noagarcia/visdom-tutorial" rel="nofollow noopener">https://github.com/noagarcia/visdom-tutorial</a> and tried them out for test case. Do you indicate that I may try another way to install visdom instead of conda ?<br>
Thank you in advance.</p>

ptrblck: <p>You could try to build from source in case the binaries are not up to date and rerun the script.</p>

sekigh: <p>I understood your suggestion.</p>
Understanding structure of a pytorch graph

Arun_Vishwanathan: <p>Hi,</p>
<p>For my use case, I require to be able to take a pytorch module and interpret the sequence of layers in the module so that I can create a “connection” between the layers in some file format. Now let’s say I have a simple module as below</p>
<pre><code class="lang-auto">class mymodel(nn.Module):
	def __init__(self, input_channels):
		super(mymodel, self).__init__()
		self.fc = nn.Linear(input_channels, input_channels)
	def forward(self, x):
		out = self.fc(x)
		out += x
		return out


if __name__ == "__main__":
	net = mymodel(5)

	for mod in net.modules():
		print(mod) 
</code></pre>
<p>Here the output yields:</p>
<pre><code class="lang-auto">mymodel(
  (fc): Linear(in_features=5, out_features=5, bias=True)
)
Linear(in_features=5, out_features=5, bias=True)
</code></pre>
<p>as you can see the information about the plus equals operation or plus operation is not captured as it is not a nnmodule in the forward function. My goal is to be able to create a graph connection from the pytorch module object to say something like this in json :</p>
<pre><code class="lang-auto">layers {
"fc": {
"inputTensor" : "t0",
"outputTensor": "t1"
}
"addOp" : {
"inputTensor" : "t1",
"outputTensor" : "t2"
}
}

</code></pre>
<p>The input tensor names are arbitrary but it captures the essence of the graph and the connections between layers. Is there a way to extract the information from a pytorch object? I was thinking to use the .modules() but then realized that hand written operations are not captured this way as a module. I guess if everything is an nn.module then the .modules() might give me the network layer arrangement. Looking for some help here. Thanks!</p>

dejanbatanjac: <p>Check out the  <code>torchviz</code> (<code>pip install torchviz</code>). This tool uses graphviz to print the graphs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21d3bee21a88798ba9f22b32671a646c513fcc21.png" data-download-href="https://discuss.pytorch.org/uploads/default/21d3bee21a88798ba9f22b32671a646c513fcc21" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21d3bee21a88798ba9f22b32671a646c513fcc21.png" alt="image" data-base62-sha1="4Pfq0pjdyypeOd1XHwXfdtIbz0d" width="357" height="499" data-dominant-color="F0F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">510×714 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Arun_Vishwanathan: <p>thanks! I was looking for something programmatic than visual to be able to store information in the format mentioned above. Any suggestions?</p>

Rami_Nasser: <p>try model.named_parameters()</p>
One padding for the first layer

GObriel: <p>Hello,</p>
<p>I would like to use 1- padding instead of 0.<br>
The default value is :  <em>padding_mode=‘zeros’</em>, but I would it to be ‘ones’, it doesn’t exist, how could i do it ?<br>
I think it’s not possible because mainly it is a bad idea because the value 1 has no meaning, but in my case, for the first layer, it does. Either i find a nice solution to do it like that or i change my first layer to : x=1-x</p>
<p>Thanks !</p>

Nikronic: <p>Hi,</p>
<p>You can use explicit padding before passing to conv layers (I assumed you are using conv2d). For this, you can use this snippet:</p>
<pre><code class="lang-auto">import torch.nn.functonal as F
import torch

x = torch.randn(1, 1, 3, 3)
x = F.pad(x, pad=(1,1,1,1), mode='constant', value=1)
x  # shape [1, 1, 5, 5]

</code></pre>
<p>Then you can pass it to a conv layer by not adding any paddings.</p>
<p>Bests</p>

GObriel: <p>Thanks Nik for taking the time to reply.<br>
So I use that only in the forward right  ? and i feed it to a conv2D</p>

Nikronic: <p>You are welcome, Actually, you have not provided much information about what you want to do. But if your problem is padding as you mentioned previously, you can do it using the snippet. But yes, you can now feed it in forward function to a conv2d.</p>

GObriel: <aside class="quote no-group" data-username="Nikronic" data-post="4" data-topic="84167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>n do it using the snippet.</p>
</blockquote>
</aside>
<p>I use this padding for the first conv layer of a Go IA.<br>
Thanks Nik, I’ll try next week (I first have to learn pybind11 to reuse C++ code) you’re solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Loss function not back propogating

aledjackson: <p>I made the following loss function to attempt to penalise the highest gradients out of the input. Everything runs fine to begin with however I notice that the extra loss function isn’t achieving anything. So I check to see if the gradient changes in the input after I call <code>grad_loss.backward()</code> and it doesn’t change at all.</p>
<p>Loss function:</p>
<pre><code class="lang-python">class LInfLoss:
    def __init__(self, weight=1, tolerance=0.001):
        self.weight = torch.tensor(weight).float()
        self.weight.requires_grad = True
        self.weight = self.weight.to('cuda:0')
        self.tolerance = tolerance

    def __call__(self, gradient: torch.Tensor):
        threshold = gradient.max() * (1 - self.tolerance)
        grads_over_threshold = gradient.ge(threshold).float() * self.weight
        return grads_over_threshold.sum()
</code></pre>
<p>Training loop</p>
<pre><code class="lang-python">    for (images, one_hot_labels) in tqdm(batched_train_data):
        # I collect batch size here because the last batch may have a smaller batch_size
        batch_size = images.shape[0]
        images.requires_grad = True
        
        optimizer.zero_grad()
        # as images is not a parameters optimizer.zero_grad() won't reset it's gradient
        if images.grad is not None:
            images.grad = None

        probabilities = model.forward(images)

        # I want to use .backward() twice rather than autograd because I want to accumulate the gradients
        loss = loss_func(probabilities, one_hot_labels)
        loss.backward(create_graph=True)
        grad_loss = grad_loss_func(images.grad)
        grad_loss.backward()

        optimizer.step()

        labels = one_hot_labels.detach().argmax(dim=1)
        predictions = probabilities.detach().argmax(dim=1)
        num_correct = int(predictions.eq(labels).sum())

        train_data_length += batch_size
        train_correct += num_correct
        train_loss += float(loss.detach()) * batch_size

        # To stop memory leaks
        images.grad = None
        del probabilities
        del loss
        del grad_loss
        del labels
        del predictions
        del num_correct
</code></pre>
<p>EDIT:<br>
just to clarify <code>grad_loss_func</code> = <code>LInfLoss</code></p>
Training on Cluster

Sentient07: <p>Hi,</p>
<p>I’m attempting to train my model over multiple nodes of a cluster, on 3GPUs. I am running the training script from Node 1, where GPUs 0, 1 are present while Node 2 has GPU 2. Following the distributed training example of <code>FasterRCNN</code> with this command, <code>CUDA_VISIBLE_DEVICES=0,1,2 python -m torch.distributed.launch --nproc_per_node=3 --use_env train.py</code> , I get the following error,</p>
<pre><code class="lang-auto">HCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=37 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "train.py", line 266, in &lt;module&gt;
    train_fpn()
  File "train.py", line 115, in train_fpn
    init_distributed_mode(args)
  File "/udd/rsundara/Code/head_detection/vision/utils.py", line 321, in init_distributed_mode
    torch.cuda.set_device(args.gpu)
  File "/udd/rsundara/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:37
</code></pre>
<p>Can someone please help me figure out how to train models across multiple nodes?</p>
<p>Regards,</p>

surya00060: <p>Since you are training on multiple nodes of a cluster, what does your output of <code>nvidia-smi</code> look like? I think you shouldn’t use <code>CUDA_VISIBLE_DEVICES=0,1,2</code> as not all 3 GPUs are bot connected and visible to the same system.</p>
<p><code>CUDA_VISIBLE_DEVICES</code> is used only when there are multiple GPUs in the system and we want to run on selected GPUs. Hope this solves some part of it.</p>
<p>Can you put the link for the tutorial for more help? Thanks.</p>

Sentient07: <blockquote>
<p>I think you shouldn’t use  <code>CUDA_VISIBLE_DEVICES=0,1,2</code>  as not all 3 GPUs are bot connected and visible to the same system.</p>
</blockquote>
<p>Yes, you’re right.</p>
<p>I’m following the training similar to the one mentioned here : <a href="https://github.com/pytorch/vision/tree/v0.4.0/references/detection" rel="noopener nofollow ugc">https://github.com/pytorch/vision/tree/v0.4.0/references/detection</a></p>
<p>Is there a way I can train across multiple nodes in pytorch?</p>

iffiX: <p>See <code>torch.distributed</code></p>

Sentient07: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a> could you please be more specific ? I am using distributed and please refer to the link in my previous post as to how I utilize it</p>

iffiX: <p>I see, then you will have to write your own launch script since <code>torch.distributed.launch</code> assumes you have the same configuration per node (i.e. launching the same number of processes per node, having same number of gpus per node).</p>

iffiX: <p>So basically you will have to set CUDA_VISIBLE_DEVICES to 0, 1 on node 1 and 0 on node two. and then think of some way to pass your commandline options to each of your processes.</p>
IndexError tensors used as indices must be long, byte or bool tensors

Xuyang_Bai: <p>When I train my model, I always randomly get the following error:</p>
<pre><code class="lang-auto">IndexError: tensors used as indices must be long, byte or bool tensors.
</code></pre>
<p>I know what this error message saying about but why this error comes out randomly? And should I add <code>.long()</code> whenever trying to index a tensor by another tensor?</p>

ptrblck: <p>If this error is thrown randomly, the index tensor is most likely changing the type “randomly” in your code base.<br>
I would recommend to narrow down, why the index tensor changes the type and make sure you are using valid indices.</p>

Xuyang_Bai: <p>Thanks, Since this error is thrown randomly, the program runs normally for some iterations until raise this error, it is really hard to find where does the index tensor get changed. And I think I haven’t change the type of index tensor explicitly, do you know which operations in pytorch might change the index tensor implicitly?</p>
<p>Thank you so much.</p>

ptrblck: <p>It’s hard to tell and maybe you could find some similar operations using the examples in the <a href="https://pytorch.org/docs/stable/tensor_attributes.html#torch-dtype">type promotion docs</a>.</p>

Xuyang_Bai: <p>Thanks ! I found that some times my index tensor has type torch.int32. BTW, why PyTorch doesn’t allow to use torch.int32 type as the indices tensor?</p>

someAdjectiveNoun: <p>May be it is not really random and some of the intermediate operations on data might return back different tensor dtypes.</p>
Installation pytorch upgrade the python version

Avi: <p>Hi everyone,<br>
I’m getting started even with python. I just faced a problem with installation of pytorch, which automatically upgrades the python version to 3.8.3.<br>
I tried installing using conda.<br>
Thanks<br>
Avi</p>

BlueTurtle: <p>What is the problem you are facing with PyTorch? What error does it give you?</p>

Avi: <p>The updating of python.<br>
I want to keep it 3.7.7.</p>

BlueTurtle: <p>This should update your Python version system-wise:<br>
<code>conda install python=3.7.7</code></p>
<p>Otherwise you can create a new environment in Anaconda itself and install 3.7.7 on there.</p>

Avi: <p>The updating happened just to the envs which I installed the pytorch.<br>
And the re-updating failed.</p>
RNN cuda out of memory

ramkikannan: <p>I am training a naive RNN for 1D time series data and have the following training loop. Since I cannot hold the total batch x seq_len x entire_time_steps data, I am transfering batch x seq_len for every time step data as the code below. But the memory keep growing for every time step as follows. I tried the following approaches.</p>
<p>Approach 1: allocating only once outside the time loop as inp and tried to copy the batch x seq_len to inp. But got an error saying cannot copy from torch.floattensor to torch.cuda.floattensor.<br>
Approach 2: Also tried cuda.empty_cache() for every 50 iterations and the memory still keeps growing.</p>
<p>For the 1D time series data, how to transfer to cuda for every timestep? What is a good way to execute this training loop if the entire batch x seq_len x entire_time_step is huge to fit in cuda memory.</p>
<p>Regards,<br>
Ramki</p>
<p>Memory growth:<br>
0 1048576<br>
1 43122688<br>
2 84017152<br>
3 124911616<br>
4 165806080<br>
5 207749120<br>
6 248643584<br>
7 289538048<br>
8 330432512<br>
9 371326976<br>
10 412221440<br>
11 454164480<br>
12 495058944<br>
13 535953408<br>
14 576847872<br>
15 617742336<br>
16 658636800<br>
17 699531264<br>
…<br>
393 16136667136 &lt;-- out of memory at this time step.</p>
<p>Code :</p>
<pre><code class="lang-auto">for epoch in range(1, NUM_EPOCHS + 1):
    rnn.train()
    rand_idxs = np.random.permutation(X_train.shape[0])
    batches = np.array_split(rand_idxs, NUM_BATCHES)
    print("Epoch: {:4}".format(epoch))    
    for j,batch in enumerate(batches,1): 
        current_batch_X = (X_train_torch[batch,:,:])                
        rnn.train()
        optimizer.zero_grad()
        hidden = torch.zeros((NUM_LAYERS, current_batch_X.shape[0],HIDDEN_SIZE)).cuda()
        # over all time steps
        for i in range(0, current_batch_X.shape[2]):
            inp = current_batch_X[:,:,i].reshape((current_batch_X.shape[0],
                                                  current_batch_X.shape[1],1)).cuda()            
            print(i, torch.cuda.max_memory_cached())
            _,hidden = rnn(inp, hidden)
            if i%50 == 0:
                torch.cuda.empty_cache()            
        current_batch_y = (y_train_torch[batch]).cuda()
        loss = criterion(output, current_batch_y)
        loss.backward()
        optimizer.step()
</code></pre>

ptrblck: <p><code>torch.cuda.empty_cache()</code> will not avoid the out of memory issue, but might instead just slow down your code, as PyTorch would need to reallocate the device memory.</p>
<p>I guess your memory usage grows, since you are storing the computation graphs for all time steps in memory before calling <code>backward</code> and thus freeing the intermediates.<br>
You could either lower the number of steps or alternatively call <code>backward()</code> after each (or a couple of iterations) and <code>detach()</code> the <code>hidden</code> state in the following operation.<br>
Let me know, if this would work for you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Training loss and validation loss decreasing upto 5 epochs and then increasing both

Sravani_Lekkala: <p>training loss and validation loss decreasing upto 5 epochs and then increasing both.what can we infer from it…accuracy also increasing upto 5 epochs and then decreasing…what changes will be better</p>

ptrblck: <p>You could try to e.g. lower the learning rate after 5 epochs and check, if the losses decrease any further.<br>
Alternatively, you could also try our different optimizers etc.</p>
Need Help in choosing pytorch normalization function for Normalizing utterence based time series data

Sasank_Kottapalli: <p>I am working in audio domain temporal convolutions.<br>
I have a 2d tensor N*K where N is features and K is time frames.</p>
<p>Which default pytorch normalization should I use for the below code?</p>
<pre><code>class ChannelwiseLayerNorm(nn.Module):
"""Channel-wise Layer Normalization (cLN)"""
def __init__(self, channel_size):
    super(ChannelwiseLayerNorm, self).__init__()
    self.gamma = nn.Parameter(torch.Tensor(1, channel_size, 1))  # [1, N, 1]
    self.beta = nn.Parameter(torch.Tensor(1, channel_size,1 ))  # [1, N, 1]
    self.reset_parameters()

def reset_parameters(self):
    self.gamma.data.fill_(1)
    self.beta.data.zero_()

def forward(self, y):
    """
    Args:
        y: [M, N, K], M is batch size, N is channel size, K is time-frames
    Returns:
        cLN_y: [M, N, K]
    """
    mean = torch.mean(y, dim=1, keepdim=True)  # [M, 1, K]
    var = torch.var(y, dim=1, keepdim=True, unbiased=False)  # [M, 1, K]
    cLN_y = self.gamma * (y - mean) / torch.pow(var + EPS, 0.5) + self.beta
    return cLN_y</code></pre>

ptrblck: <p>I haven’t tested your implementation, but wouldn’t the code be equal to <code>nn.BatchNorm1d</code> with <code>track_running_stats=False</code>?</p>

Sasank_Kottapalli: <p>Does it have N gamma and beta parameters as present in the code? Also dont need running mean and variance.</p>

ptrblck: <p>Yes, <code>gamma</code> and <code>beta</code> are the affine parameters, which are created by default as <code>weight</code> and <code>bias</code>.<br>
You can disable them by specifying <code>affine=False</code>, but they will be created by default.</p>