akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to initialize the conv layers with xavier weights initialization?

wuhoo: <p>Hi, I want to know how torch.nn.init.xavier_uniform is used. I’ve tried to apply it on net.parameters but got the following errors:</p>
<p>params = list(net.parameters())<br>
nn.init.xavier_uniform(params,gain=nn.init.calculate_gain(‘relu’))</p>
<p>Traceback (most recent call last):<br>
File “/Users/elaine/Desktop/YL-Net/train.py”, line 23, in <br>
nn.init.xavier_uniform(params,gain=nn.init.calculate_gain(‘relu’))<br>
File “/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/nn/init.py”, line 205, in xavier_uniform<br>
fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)<br>
File “/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/nn/init.py”, line 165, in _calculate_fan_in_and_fan_out<br>
dimensions = tensor.ndimension()<br>
AttributeError: ‘list’ object has no attribute ‘ndimension’</p>
<p>Thanks!</p>

QuantScientist: <p>This is how it is usually done:</p>
<pre><code class="lang-auto">linear1=torch.nn.Linear(N_FEATURES, hiddenLayerSize, bias=True) 
torch.nn.init.xavier_uniform(linear1.weight)
</code></pre>

wuhoo: <p>This is what I’m looking for. Thank you so much!</p>

MERAH_Samia: <p>What it used for Please</p>
Exact coordinates of bounding box for Mask R-CNN?

ZimoNitrome: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23febd94f361100fc8904249eb2411561ba0b189.png" data-download-href="https://discuss.pytorch.org/uploads/default/23febd94f361100fc8904249eb2411561ba0b189" title="img"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23febd94f361100fc8904249eb2411561ba0b189_2_690x214.png" alt="img" data-base62-sha1="58qupOiOQdWyA8WjNuu1TS4A5Zf" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23febd94f361100fc8904249eb2411561ba0b189_2_690x214.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23febd94f361100fc8904249eb2411561ba0b189_2_1035x321.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23febd94f361100fc8904249eb2411561ba0b189.png 2x" data-dominant-color="655352"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img</span><span class="informations">1324×411 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Given a 5x5 mask (see a.) (black = 0, red = 1) what are the coordinates for its bounding box?</p>
<p>Intuitively I would think of the image as a grid, and pixels being drawn from top left corner, giving the bounding box coords of (2, 2, 4, 4) (see b.).</p>
<p>Some image renderers like plt have me believed that pixels are drawn from the middle, resulting either in box coordinates of (2, 2, 3, 3) (see c.) or possibly (1.5, 1.5, 3.5, 3.5) (see d.).</p>
<p>Which one of these would be correct? Are any of them?</p>
<p>The regions are to be used in the training of a Mask R-CNN model but I guess that the way of calculating bounding boxes are similar for other models too.</p>

Usama_Hasan: <p>Hey <a class="mention" href="/u/zimonitrome">@ZimoNitrome</a>, the figure (b) makes sense according to implementations of popular R-CNN.</p>
GPU inference slows if requested consecutively

Fuad_Issa: <p>Pytorch gpu first prediction is really fast but then slows down if prediction is requested consecutively.</p>
<p>Please find notebook to run the whole thing here<br>
<a href="https://colab.research.google.com/drive/1gqSzQqFm8HL0OwmJzSRlcRFQ3FOpnvFh?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1gqSzQqFm8HL0OwmJzSRlcRFQ3FOpnvFh?usp=sharing</a></p>
<pre><code class="lang-auto">%%time
# Predict hidden states features for each layer
for each in range(10):
  with torch.no_grad():
      # See the models docstrings for the detail of the inputs
      outputs = model(tokens_tensor, token_type_ids=segments_tensors)
      # Transformers models always output tuples.
      # See the models docstrings for the detail of all the outputs
      # In our case, the first element is the hidden state of the last layer of the Bert model
      encoded_layers = outputs[0]
  # We have encoded our input sequence in a FloatTensor of shape (batch size, sequence length, model hidden dimension)
  # assert tuple(encoded_layers.shape) == (1, len(indexed_tokens), model.config.hidden_size)
</code></pre>
<pre><code class="lang-auto">CPU times: user 23.7 s, sys: 21.4 s, total: 45.1 s
Wall time: 45.2 s
</code></pre>
<pre><code class="lang-auto">%%time
# Predict hidden states features for each layer
for each in range(1):
  with torch.no_grad():
      # See the models docstrings for the detail of the inputs
      outputs = model(tokens_tensor, token_type_ids=segments_tensors)
      # Transformers models always output tuples.
      # See the models docstrings for the detail of all the outputs
      # In our case, the first element is the hidden state of the last layer of the Bert model
      encoded_layers = outputs[0]
  # We have encoded our input sequence in a FloatTensor of shape (batch size, sequence length, model hidden dimension)
  # assert tuple(encoded_layers.shape) == (1, len(indexed_tokens), model.config.hidden_size)
</code></pre>
<pre><code class="lang-auto">CPU times: user 20.4 ms, sys: 329 µs, total: 20.8 ms
Wall time: 22.9 ms
</code></pre>
<p>I have seen the same behaviour in tensorflow as well. Is this related to CUDA perhaps ?</p>

ptrblck: <p>CUDA operations are asynchronous so you would need to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timer.<br>
Otherwise the timing will be accumulated in the next blocking operation and will yield a wrong result.</p>
What is the best way to check if parameters already in optimizer?

marga: <p>Hi, I am freezing and un-freezing parameters of a model during training, so I want check if some parameters are already in my optimizer (so that I won’t put them again - when I do a get <code>ValueError: some parameters appear in more than one parameter group</code>).</p>
<p>I have tried the following:</p>
<pre><code class="lang-auto">... 
        for prm in self.model.parameters():
            prm.requires_grad = True
            if id(prm) not in list(self.optimizer.state_dict()['state'].keys()):
                new_trainable_parameters.append(prm)

        if new_trainable_parameters != []:
            my_dict = {'params': new_trainable_parameters}
            self.optimizer.add_param_group(my_dict)
</code></pre>
<p>but it is not working (I still get the same error).</p>
<p>Any ideas? Thanks in advance.</p>

NoobCoder: <p>Hey Katerina,</p>
<p>Wondering have you found an answer to your question?</p>

Nikronic: <p>Hi,</p>
<p>The taken approach is complex and not necessary really.<br>
The idea is that when you freeze a parameter using <code>requires_grad=False</code>, even you pass entire parameters to the optimizer, it will not consider them and literally just skip those parameters in the way that you have never provided them and this will be done by not accumulating gradients for that particular parameter which also leads to have <code>None</code> as the gradient.</p>
<aside class="quote quote-modified" data-post="2" data-topic="83358">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/passing-to-the-optimizers-frozen-parameters/83358/2">Passing to the optimizers frozen parameters</a> 
  </div>
  <blockquote>
    The parameters with requires_grad = False won’t accumulate a gradient during the backward pass and their .grad attribute will stay None by default (if no previous gradient was calculated). 
The optimizer will skip all parameters with a None gradient as seen <a href="https://github.com/pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96" rel="noopener nofollow ugc">here</a>. 


All parameters will accumulate gradients and the optimizer will only update the passed parameters. If you call optimizer.zero_grad() and don’t use model.zero_grad(), the “unused” parameters will continue to accumulate gradients. If…
  </blockquote>
</aside>

<p>Bests</p>
How to create a module from graph

Omer_B: <p>Hey,<br>
I plan to export a torchscript graph represantation by using <code>trace(...)</code> command, and then using <code>trace.graph()</code>. after that, I plan to apply some optimisations for the given graph and then I want to revert the result back to <code>ScriptModule</code> class.<br>
I am familiar with: <code>torch._C._create_function_from_graph('forward', graph)</code><br>
but it returns <code>ScriptFunction</code> class. I would like to use it as a module (ScriptModule class).</p>
<p>I need it as a module class because I need to be able to access its parameters by <code>parameters()</code> method.</p>
<p>thanks, Omer.</p>

Michael_Suo: <p>if you are mutating the graph in-place, the original ScriptModule should reflect your optimizations. Today we have no public way of creating a ScriptModule from a graph alone</p>

Omer_B: <p>Thanks for responding,<br>
what do you mean by “in-place” in this case ?<br>
lets assume that I have a traced module that I got by: <code>torch.jit.trace(model, sample_inputs)</code>.<br>
do you mean that I need to change its “graph” field or “inlined_graph” field to make the module to represent the new graph? (thats mean that all its parameters should be changed if needed)</p>

seliad: <p><a class="mention" href="/u/michael_suo">@Michael_Suo</a> In case <strong>we do not want to modify the graph in-place,</strong><br>
that is, we want an <code>nn.Module</code> that, given a graph, we will have the same <code>forward</code> function as the graph,<br>
and will have same parameters/buffers/modules as used in the graph,</p>
<p>Can we automatically create such <code>nn.Module</code> given a graph by inferring everything needed?</p>
How to apply a function for each image in the batch?

Mert_Cokelek: <p>Hello.<br>
My goal is to apply a user-defined function for each image in the batch, in the forward propagation.<br>
For example, batch size= 12 and I want to apply py360convert.e2c() function on each image in the batch.</p>
<p>In pandas, df.apply() does this job, however I could not find an example for PyTorch.<br>
Is there any approach?</p>
<p>Thanks in advance.</p>
How to see instances of incorrect images, labels and predictions in test data Pytorch - Fashion MNIST?

ScarletMcLearn: <p>Hi.<br>
In the notebook below I’ve created a Deep Learning Model in Pytorch and trained and tested it.<br>
I want to save the instances of the test image data and the test image labels which and predicted test labels which came incorrect running it on the test set.<br>
I have been trying to implement it using the for loops near the end of the Jupyter Notebook using :</p>
<pre><code class="lang-auto">correct = 0
total = 0

incorr_X = []
incorr_y = []
incorr_argmax = []

with torch.no_grad():
    for data in testset:
        X, y = data
        output = net(X.view(-1,784))
        #print(output)
        for idx, i in enumerate(output):
            #print(torch.argmax(i), y[idx])
            if torch.argmax(i) == y[idx]:
                correct += 1
            else:
                incorr_X.append(X)
                incorr_y.append(y)
                incorr_argmax.append(torch.argmax(i))
            total += 1
</code></pre>
<p>print("Accuracy: ", round(correct/total, 3))<br>
Is this the correct approach to get the incorrectly predicted image, label and predicted label?</p>
<p>How do I access the corresponding images, labels and predicted labels?</p>
<p>I have written some implementations in the Notebook, but they seem to not be working.</p>
<p>Any help will be greatly appreciated.</p>
<p>Thank you!!!</p>
<p>Link to Notebook:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/ScarletMcLearn/DataScience/blob/master/Deep%20Learning/Image%20Classification/PyTorch%20-%20Fashion%20-%20MNIST%20-%203.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/ScarletMcLearn/DataScience/blob/master/Deep%20Learning/Image%20Classification/PyTorch%20-%20Fashion%20-%20MNIST%20-%203.ipynb" target="_blank" rel="nofollow noopener">ScarletMcLearn/DataScience/blob/master/Deep Learning/Image Classification/PyTorch - Fashion - MNIST - 3.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "from torchvision import transforms, datasets\n",
    "\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
</code></pre>

  This file has been truncated. <a href="https://github.com/ScarletMcLearn/DataScience/blob/master/Deep%20Learning/Image%20Classification/PyTorch%20-%20Fashion%20-%20MNIST%20-%203.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Since you are iterating the <code>output</code> using <code>i</code>, you should append <code>i</code> in <code>incorr_X</code> and <code>y[idx]</code> in <code>incorr_y</code>.<br>
Let me know, if this isn’t working and what kind of error you are seeing.</p>
What does model.to(torch.Tensor) do?

sambaths: <p>I read in PyTorch docs that <code>.to(*args, **kwargs)</code> can also take Pytorch tensors as inputs.<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.to" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Module.to</a></p>
<pre><code class="lang-python">tensor = torch.tensor([[1, 2, 3], [4, 5, 6]], device='cuda:0', dtype=torch.float)
class MyModel(nn.Module):
   ...
model = MyModel()

model.to(tensor)
</code></pre>
<p>I just would like to know what does this do?<br>
Does this just moves the <code>model</code> to the <code>device</code> the <code>tensor</code> is on and set <code>dtype</code> to that of <code>tensor</code></p>

JuanFMontesinos: <p>It does what the docs says:</p>
<p>Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module</p>
<p>It allocates model in the given tensor’s device</p>

sambaths: <p>so, it is the same as:</p>
<pre><code class="lang-python">model.to(torch.device('cuda:0'))
model.float()
</code></pre>
<p>Thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>

JuanFMontesinos: <p>In that context it’s equivalent to<br>
<code>model.to(tensor.device())</code></p>
Difficulty in understanding position embedding

Mahsa_Afsharizade: <p>Hello friends. I want to have a representation for each sentence as <strong>sentence embedding vector + sentence position embedding vector</strong>.</p>
<p>I have no problem with sentence embedding vector, but I have trouble understanding sentence position embedding.</p>
<p>I read in <a href="https://ieeexplore.ieee.org/document/7756666" rel="nofollow noopener">an article</a> that first set a <strong>position value</strong> for each sentence that was 0.1, or 2.<br>
A value of zero for the first 3 sentences, a value of one for the last 3 sentences and a value of two for the other middle sentences of the document.</p>
<p>The authors’ idea was that they want the beginning and ending sentences of the document to be of greater importance than middle ones.</p>
<p>And they said the rest is like a normal embedding procedure …</p>
<p>Now I do not understand, for example, if I have 10 sentences in my document and the length of the embedding vectors is 4, then the sentence embedding matrix will be (10,4).<br>
But for the position embedding matrix, will the matrix size be (10,4)? Or (3,4)? (3 for the three values ​​0, 1, or 2 assigned to the position).</p>
<p>I do not understand, where should I use  those three values ​​of position 0,1 or 2 in the embedding layer?</p>
<p>Thanks in advance.</p>
Gathering elements along Z axis in 3D tensor following a mask

ucals: <p>I have a tensor like this example (x):</p>
<pre><code class="lang-auto">tensor([[[ 1,  2,  3,  4],
         [ 5,  6,  7,  8],
         [ 9, 10, 11, 12]],

        [[13, 14, 15, 16],
         [17, 18, 19, 20],
         [21, 22, 23, 24]]])
</code></pre>
<p>Using a tensor mask like this:</p>
<pre><code class="lang-auto">tensor([[0, 1, 0, 0],
        [0, 0, 1, 0],
        [1, 1, 0, 0]])
</code></pre>
<p>I want to gather the elements in the 3D tensor along the Z axis where the mask is 1, producing a result like this:</p>
<pre><code class="lang-auto">tensor([[ 9, 21],
        [ 2, 14],
        [10, 22],
        [ 7, 19]])
</code></pre>
<p>So far, I’ve managed to generate results in a very ugly way:</p>
<pre><code class="lang-auto">x = x.transpose(0, 2).reshape(-1, x.shape[0])
idx = mask.transpose(0, 1).flatten().nonzero()
result = x[idx].squeeze()
</code></pre>
<p>Is there a better way to do it?</p>
<p>Thanks!</p>

ptrblck: <p>You could create a bool mask and index the input tensor as:</p>
<pre><code class="lang-python">x = torch.tensor([[[ 1,  2,  3,  4],
                   [ 5,  6,  7,  8],
                   [ 9, 10, 11, 12]],

                  [[13, 14, 15, 16],
                   [17, 18, 19, 20],
                   [21, 22, 23, 24]]])

mask = torch.tensor([[0, 1, 0, 0],
                     [0, 0, 1, 0],
                     [1, 1, 0, 0]]).bool()

res = x[mask.unsqueeze(0).expand(2, -1, -1)]
print(res)
&gt; tensor([ 2,  7,  9, 10, 14, 19, 21, 22])
</code></pre>
<p>To get the posted output shape, you could use:</p>
<pre><code class="lang-python">res = res.view(2, -1).transpose(0, 1)
print(res)
&gt; tensor([[ 2, 14],
          [ 7, 19],
          [ 9, 21],
          [10, 22]])
</code></pre>
Is my model really performing well?

_joker: <p>Hi, I am wondering if there is anything wrong with my training as it achieves very high accuracy from the starting itself? The logs for training on CIFAR10 looks like this as below-</p>
<pre><code class="lang-auto">Files already downloaded and verified
Files already downloaded and verified

Epoch 000:
Training Loss: 64.2905438385933 | Training Accuracy: 91.71
Total Test Loss: 2.600952295586467 | Test Accuracy: 99.82
Saving...

Epoch 001:
Training Loss: 1.0865276760887355 | Training Accuracy: 99.81
Total Test Loss: 3.7665871270000935 | Test Accuracy: 99.92

Epoch 002:
Training Loss: 0.18417388796842715 | Training Accuracy: 99.986
Total Test Loss: 1.7079168632626534 | Test Accuracy: 99.96

Epoch 003:
Training Loss: 0.09753012470901012 | Training Accuracy: 99.994
Total Test Loss: 1.410150108858943 | Test Accuracy: 99.96
</code></pre>
<p>The good part is, after downloading the model and testing on examples it still gives the same accuracy. Still, I suspect if there is anything basic that I might be missing and all this accuracy is because of an error.</p>
<p>The model is stacked model of an autoencoder and a classifier and involves a custom loss for autoencoder including MSE loss and Cross-Entropy loss for Classifier.</p>
<p>I am looking for any suggestions or checks you may have to advise to understand the correctness of my pipeline.</p>
<p>Training Pipeline:</p>
<pre><code class="lang-auto">    for epoch in range(EPOCH+1):
        global best_train_acc
        running_loss, train_loss, correct, total = 0.0, 0, 0, 0
        autoencoder.train()
        for i, (inputs, labels) in enumerate(trainloader):
            inputs, labels = inputs.to(device), labels.to(device)
            encoded, decoded, _, _, _ = autoencoder(inputs, labels)
            loss_ae = custom_loss_fn()

            output_classifier = classifier(encoded)
            loss_classifier = ce(output_classifier, labels) # classifier loss
            
            optimizer.zero_grad()
            optimizer_classifier.zero_grad()
            
            loss_ae.backward(retain_graph=True)
            loss_classifier.backward()
            optimizer.step()
            optimizer_classifier.step()

            train_loss += loss_classifier.item()
            _, predicted = output_classifier.max(1)
            total += labels.size(0)
            correct += predicted.eq(labels).sum().item()
</code></pre>
<p>and testing in the same way just I add;<br>
<code>autoencoder.eval()</code> and <code>with torch.no_grad()</code></p>

ptrblck: <p>The code looks generally alright. I don’t know, how <code>custom_loss_fn</code> calculates the loss without any parameters, but you might use globals for them.<br>
You should probably also call <code>classifier.eval()</code> during the test case.</p>
<p>A minor concern might be the comparably high test loss and the high test accuracy, which don’t seem to really correlate well, so you could check the test loss calculation and accuracy again.</p>
How to deal with inf values when taking logs of tensors containing zeros

nowyouseeme: <p>In my model, in the forward function, I have one tensor on which I first apply sigmoid and then log. However, the output of sigmoid might be considerably low. The return value of the forward function is :</p>
<pre><code class="lang-auto">def forward(dots, path, probs)
        apply_paths = dots * path 
        apply_sigmoid = torch.sigmoid(apply_paths)
        apply_log = -1*torch.log(apply_sigmoid)

       return torch.sum(probs*apply_log)
</code></pre>
<p>I’m getting a lot of time <code>inf</code> as a return value. The return value is the loss itself, and so the gradients are calculated based on this return value. I’m not sure if this is okay or not. Should I do anything of these inf values after taking log? Or do something about very small values from the sigmoid function?</p>

ptrblck: <p>You could <code>clamp</code> these invalid values or just use <code>logsigmoid</code> as seen here:</p>
<pre><code class="lang-python">x = torch.tensor([-100., 100])
print(torch.log(torch.sigmoid(x)))
&gt; tensor([-inf, 0.])
print(torch.nn.functional.logsigmoid(x))
&gt; tensor([-100.,   -0.])
</code></pre>
Batched index_add

TermoPile: <p>Hi everyone,</p>
<p>I need to perform an index_add operation on 3D tensors, however, this operation is only implemented for 2D tensors and the indices needs to be a vector. I have been trying to think of a workaround which achieves the same but with no success. I would be glad if I could get some help!</p>
<p>My input tensor has the following shape (batch_size, max_length, emb_dim) and my index is of size (batch_size, max_length). I want to add vectors from the input tensor to itself along dim=1 based on the index values. For example, index[0, 1] = 3 will mean input[0, 1] and input[0, 3] are to be added.</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="TermoPile" data-post="1" data-topic="89474">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/e0b2c6/48.png" class="avatar"> TermoPile:</div>
<blockquote>
<p>however, this operation is only implemented for 2D tensors</p>
</blockquote>
</aside>
<p>That’s not the case, as you can use tensors with having multiple dimensions with <code>index_add_</code>.<br>
I don’t quite understand your use case, as it seems you would like to add the tensor to itself instead of another passed value tensor?<br>
Could you post an example using randomly initialized tensors?</p>
How to package project with pytorch CUDA/CPP extension through setuptools

Howie: <p>Hello. I want to package my project and upload to pypi.</p>
<p>My project contains some CUDA/CPP files, and I complie it as <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">the tutorial</a>. It works well.</p>
<p>In my project, I run setup.py for <code>roi_align</code> and then run another setup.py for <code>rod_align</code>. The files are listed as follows. Note that I used PyTorch to confirm whether GPU is available, then choose to build CPP extension or CUDA extension.</p>
<pre><code class="lang-auto">├── autocrop
│   ├── __init__.py
│   ├── cropper.py
│   ├── model
│   │   ├── __init__.py
│   │   ├── cropping_model.py
│   │   ├── moblienetv2.py
│   │   ├── rod_align
│   │   │   ├── __init__.py
│   │   │   ├── build
│   │   │   ├── functions
│   │   │   │   ├── __init__.py
│   │   │   │   └── rod_align.py
│   │   │   ├── modules
│   │   │   │   ├── __init__.py
│   │   │   │   └── rod_align.py
│   │   │   ├── setup.py # for rod_align
│   │   │   └── src
│   │   │       ├── rod_align.cpp
│   │   │       ├── rod_align.h
│   │   │       ├── rod_align_cuda.cpp
│   │   │       ├── rod_align_cuda.h
│   │   │       ├── rod_align_kernel.cu
│   │   │       └── rod_align_kernel.h
│   │   ├── roi_align
│   │   │   ├── __init__.py
│   │   │   ├── functions
│   │   │   │   ├── __init__.py
│   │   │   │   └── roi_align.py
│   │   │   ├── modules
│   │   │   │   ├── __init__.py
│   │   │   │   └── roi_align.py
│   │   │   ├── setup.py # for roi_align
│   │   │   └── src
│   │   │       ├── roi_align.cpp
│   │   │       ├── roi_align.h
│   │   │       ├── roi_align_cuda.cpp
│   │   │       ├── roi_align_cuda.h
│   │   │       ├── roi_align_kernel.cu
│   │   │       └── roi_align_kernel.h
│   │   └── shufflenetv2.py
│   └── utils.py
├── demo.py
├── imgs
│   └── demo.jpg
└── setup.py # This is the file I want to write, run once to package all the project
</code></pre>
<p>Now I want to package my whole project. I am confused about how to write the setup file. (just run once).</p>
<p>Can I copy the code <code>setup(xxxx)</code> from roi and rod setup.py to the final setup.py and write another <code>setup(xxxx)</code> after them, just like:</p>
<pre><code class="lang-python">from setuptools import setup
setup(roi)
setup(rod)
setuptools.setup(
    name="autocrop",
    python_requires='&gt;=3.6',
    install_requires=[
        "torch&gt;=1.1",
        "torchvision&gt;=0.3.0",
        "numpy",
    ]
xxxxxxx
)

</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you know who can answer packaging questions?</p>

ptrblck: <p><a class="mention" href="/u/seemethere">@seemethere</a> might be the right person.<br>
Also, Nikita Shulda, but I cannot find his user name (and unsure, if he’s registered here).</p>
How to make network recognize the important of position data

kolohe113: <p>So the input to my network is a depth image (64 x 64) and the robot position (x, y, z). I used torch.cat to combine them and feed them into fully connected layers. After I trained my model for 2000 episodes, it looks to me that my network is straight-up ignoring the position data. Maybe it is because the position data is only 3 x 1? How do I make sure my network recognizes the importantness of my position data?</p>
<p>Any recommendations will be appreciated, thank you!</p>

Valerio_Biscione: <p>Hi and welcome! What type of network are you using? Can you showing us how you actually combine the depth image with the robot position?</p>

kolohe113: <p>Yeah, so I have the depth image go through 3 layers of convolutional layers. Then I use<br>
the view function to flatten the output from the convolutional layers. Then I used torch.cat to concatenate the flattened output and position data. Here is my code:</p>
<pre><code class="lang-auto">class DualingDoubleDQN(nn.Module):
    def __init__(self, num_frames, pos_state_size, action_size, seed, fc4_units=128, fc5_units=128):
        self.action_size = action_size
        super(DualingDoubleDQN, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.conv1 = nn.Conv2d(1, 32, kernel_size=8, stride=3)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)
        self.fc4 = nn.Linear(64* 6* 6 + pos_state_size, fc4_units) 
        self.fc5 = nn.Linear(fc4_units, fc5_units)
        self.V = nn.Linear(fc5_units, 1)
        self.A = nn.Linear(fc5_units, action_size)

    def forward(self, pos_state, frame_state):
        c1 = F.relu(self.conv1(frame_state))       
        c2 = F.relu(self.conv2(c1))   
        c3 = F.relu(self.conv3(c2))
        c3 = c3.view(c3.size(0), -1)
        #print("c3 size after view: ", c3.size())
        concate = torch.cat((c3,pos_state),1) #concate flattened image and pos data
        f4 = F.relu(self.fc4(concate))
        f5 = F.relu(self.fc5(f4))
        Value = self.V(f5)
        Advantage = self.A(f5)
        return Value, Advantage
</code></pre>

kolohe113: <p>Also, I am doing a dueling double DQN for deep reinforcement learning.</p>

Valerio_Biscione: <p>I don’t see any problem in your code there. Maybe there is something wrong in the loss function? I am not super familiar with DQN but maybe show us your loss fun?</p>

kolohe113: <p>Interesting, for the loss function I just follow the dueling double dqn algorithm. And when I use only x,y,z,row,pitch,yaw as input everything was working properly, so I think my loss function is defined correctly. Maybe 2000 episodes is not enough for using images as input?</p>
What's the difference between module.train(False) and module.eval()?

Ryan_Burgert1: <p>What’s the difference between module.train(False) and module.eval()? They seem to have the same effect. On that note, why can’t I do module.eval(False) in place of module.train()?</p>

Valerio_Biscione: <p>EDIT: as mentioned here</p><aside class="quote" data-post="7" data-topic="4004">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-nn-modules-train-true-train-false-do/4004/7">What does nn.Module's train(True)/train(False) do</a> 
  </div>
  <blockquote>
    model.train(False) and model.eval() have the same effect, as the latter calls in fact the former function as seen in <a href="https://github.com/pytorch/pytorch/blob/2cdbb140e62d80906c4a6b1944e7e36650ec54b1/torch/nn/modules/module.py#L1002" rel="noopener nofollow ugc">this line of code</a>. 
If you see any differences, I guess this might be some random effects in your training.
  </blockquote>
</aside>
<p><code>eval()</code> and <code>train(False)</code> are equivalent</p>
Updating parameters with two combined network

lisyuan: <p>I have an image encoder and a question encoder branch combined together than connecting to 2 fc layers. How should I add the parameter together to update the network parameter?<br>
I have tried some code:</p>
<pre><code class="lang-auto">params = model.img_encoder.fc.parameters() + model.qu_encoder.parameters() + \
             model.fc1.parameters() + model.fc2.parameters()

optimizer = optim.Adam(params, lr=1e^-3)
</code></pre>
<p>but it will return</p>
<pre><code class="lang-auto">TypeError: unsupported operand type(s) for +: 'generator' and 'generator'
</code></pre>
<p>any idea or suggestion how to update the parameters of the combined network?</p>

Valerio_Biscione: <pre><code class="lang-auto">def concat_generators(*args):
      for gen in args:
          yield from gen
  params = concat_generators(model.img_encoder.fc.parameters(), model.qu_encoder.parameters() , model.fc1.parameters(),  model.fc2.parameters())

  optimizer = torch.optim.Adam(params, lr=learning_rate)
</code></pre>
<p>This may work.<br>
You got an error because you <code>parameters()</code> are not lists, but generators, and you can’t sum generators.</p>

lisyuan: <p>Yes, it work !!! But I think I should watch some tutorial about generator first <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

Valerio_Biscione: <p>Glad it worked! Read stuff about generators, it’s important as they are used everywhere in pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

lisyuan: <p>I found anothor solution. The following sample could work, too.</p>
<pre><code class="lang-auto">params = list(model.img_encoder.fc.parameters()) + list(model.qu_encoder.parameters()) \
             + list(model.fc1.parameters()) + list(model.fc2.parameters())

optimizer = optim.Adam(params, lr = LEARNING_RATE)
</code></pre>
How can I perform batch indexing?

lucam: <p>I have one batched tensor with matrices:</p>
<pre><code class="lang-auto">batch_size = 64 
num_elements = 500
channels = 2
a = torch.rand(batch_size, num_elements, channels)  # [64, 500, 2]
</code></pre>
<p>And one tensor with indices, that describe how to index each previous matrix:</p>
<pre><code class="lang-auto">indices = 80
b = (torch.rand(batch_size, indices) * num_elements).long()  # [64, 80]
</code></pre>
<p>I want to perform an indexing such that each matrix in the batch is indexed with the corresponding index in the other tensor. The resulting matrix has shape <code>[64, 80, 2]</code><br>
How can I do that?</p>

usami: <p>For two dimension indexing, try following:</p>
<pre><code class="lang-python">B, C1, C2 = 32, 1024, 3
input = torch.rand((B, C1, C2))  # [32, 1024, 3]
idx1 = torch.arange(B).view(-1, 1)  # [32, 1]
idx2 = torch.randint(low=0, high=C1, size=(B, 80))  # [32, 80]
output = input[idx1, idx2]  # [32, 80, 3]
</code></pre>
<p>Similarly, for three dimension indexing:</p>
<pre><code class="lang-python">B, C1, C2, C3 = 32, 1024, 16, 3
input = torch.rand((B, C1, C2, C3))  # [32, 1024, 16, 3]
idx1 = torch.arange(B).view(-1, 1, 1)  # [32, 1, 1]
idx2 = torch.arange(C1).view(1, -1, 1)  # [1, 1024, 1]
idx3 = torch.randint(low=0, high=C2, size=(B, C1, 8))  # [32, 1024, 8]
output = input[idx1, idx2, idx3]  # [32, 1024, 8, 3]
</code></pre>
Detecting arbitrary quadrilateral corners

l00p: <p>My problem is:</p>
<p>Given an image of some quadrilateral object, find the coordinates of the corners.</p>
<p>I have an idea about how I might start with this problem but I would love some feedback, suggestions, critiques from the community.</p>
<p>(image not real data, but just to illustrate the problem clearly)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d744215cf752a9ffaa960667f61ef0da6a5e5bd0.png" alt="image" data-base62-sha1="uIkCJk7R1B0lQqjQBwx0D1VN68w" width="564" height="357"></p>
<p>So I’m thinking to use an off the shelf vision classifier, resize the image to fit the input and modify the output to predict 8 values, which will be interpreted as 4 (x, y) coordinate pairs.</p>
<p>So, somewhat inspired by YOLO regression -  I’m thinking to replace the last layer of say, mnasnet, with an 8 output linear layer - pass the outputs of the linear layer through sigmoid functions to map them to the (-1, +1) range and interpret these values as normalised coordinates wrt. the centre.</p>
<p>eg. say the left-most corner in the attached image is 25% of the width of the image across from the left and 30% of the way down. mapping this to the range (-1, +1), taking the centre as the reference point, and taking the directions “up” and “left” to be positive, the x label would be 0.5 and the y label would be 0.4.</p>
<p>other corners would be labelled similarly, so for the top right corner, “up” and “right” would be taken to be positive.</p>
<p>So then the problem becomes a simple multiple regression problem and I can use  mean squared error for the loss function ( torch.nn.MSELoss )</p>
<p>Now these are arbitrary quadrilaterals, so there may be “corner cases” in which none or multiple corners will be in any given quadrant of the image. In order to deterministically label each datum, I’m thinking to first assign whichever corner is closest (euclidean distance) to the top-left corner of the image as the “top left” corner, then label whichever of the remaining three corners is closest to the top right of the image as the “top right” corner and so on until all four corners are assigned.</p>
<p>Does that sound like a reasonable approach?</p>
<p>It also occurred to me that I could use polar coordinates (angle, distance) labelling the data by first, second, third, fourth coordinate encountered in a clockwise rotation from the centre. this feels a little less arbitrary, somehow. thoughts?</p>
Strange CUDA Memory error

Ratan: <p>I am getting the following error. I am using batch size 1.<br>
I was able to train the very same network several times with batch size 6. the code does not show any error untill i use model.cuda(). Before i use model.cuda() the gpu memory fills up with 10gb.</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 10.24 GiB (GPU 3; 14.76 GiB total capacity; 10.31 GiB already allocated; 4.04 GiB free; 10.52 MiB cached)</p>
<p>When i first trained my model it used 10gb but now tries to use 10more gb.</p>

ptrblck: <p>Could you check, which part of the code fills up the GPU memory?<br>
Are you loading a <code>state_dict</code> directly to the device or any other data?</p>

Ratan: <p>Hi,<br>
This error occurs when i put the model for training.<br>
The model architecture is initialized to an object <code>model</code> . This is where the memory fills up. then i run model.cuda() -&gt; this point it tries to allocate that 10.24 GiB, results in cuda mem error.</p>
<p>Here is the story:<br>
Training for 1st time: works fine. used gpu 2,3 (takes 25min for one epoch) -&gt; process quit<br>
Training for 2nd time: works fine. switched to gpu 2 (takes 17min for one epoch) -&gt; process quit because i wanted to save best model. So made changes to check val loss and save.<br>
Training for 3rd time: above described cuda error.</p>
<p>I used the same code to run on a different system worked like a charm.</p>

ptrblck: <aside class="quote no-group" data-username="Ratan" data-post="4" data-topic="89176">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ratan/48/12876_2.png" class="avatar"> Ratan:</div>
<blockquote>
<p>The model architecture is initialized to an object <code>model</code> . This is where the memory fills up.</p>
</blockquote>
</aside>
<p>This would mean that you are already creating <code>CUDATensors</code> inside the <code>__init__</code> method of your model. Could you check for <code>to('cuda')</code> and <code>cuda()</code> calls inside it?<br>
The next <code>model.cuda()</code> call would then push all remaining parameters and buffers to the GPU and is apparently running out of memory.</p>

Ratan: <p>The issue was resolved. The was a small change in the FC layer as my initial code ran for 256x256 but in 512x512 the output tensors was very huge nearly 4times the previous model. I reduced the output features in nn.Linear by factor of 10000.</p>
More CUDA memory usage with features calculated on the fly

andersjing: <p>Hi there,</p>
<p>I tried two methods to generate the same features using a custom dataloader:</p>
<ol>
<li>
<p>Prepare the features beforehand and save it to disk as pickle file, and load the pickle file directly in the “_<em>getitem</em>_” function.</p>
</li>
<li>
<p>Generate the features on the fly in the “_<em>getitem</em>_” function, this process involves large matrix operation using NumPy.</p>
</li>
</ol>
<p>Both of the final returned data are the same shape and type, the only difference is the first one is loaded from disk while the second one is calculated on the fly with some matrix operation. However, the CUDA memory usage of the second one will be about twice the first one, without decreasing the minibatch_size, I will got “RuntimeError: CUDA out of memory”. Does it mean that the dataloader will take up GPU during the feature process?</p>
<p>I tried to fix this issue by setting the pin_memory to False, delete some intermediate variables in the “_<em>getitem</em>_” function, but none of them worked.</p>
<p>Any insight appreciated!</p>

ptrblck: <aside class="quote no-group" data-username="andersjing" data-post="1" data-topic="88738">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b19c9b/48.png" class="avatar"> andersjing:</div>
<blockquote>
<p>Does it mean that the dataloader will take up GPU during the feature process?</p>
</blockquote>
</aside>
<p>This shouldn’t be the case, if your <code>Dataset</code> performs all operations on <code>CPUTensors</code>.<br>
Could you run the data loading pipeline alone and check the GPU memory usage?<br>
If the <code>DataLoader</code> approach shows some GPU usage, could you post its code here, so that we could take a look?</p>

andersjing: <p>Thanks for the response. After further comparison, I finally found it was caused by a few dirty samples. It works fine after removing the dirty samples.</p>
How to merge labels and images in Pytorch

hectorist: <pre><code class="lang-auto">data = np.load("training_data.npy", allow_pickle=True)

X = torch.Tensor([i[0] for i in data]).view(-1, 50, 50) 
X = X / 255.0

print(X.size())
y = torch.Tensor([i[1] for i in data])
</code></pre>
<p>I have this and want to have:</p>
<pre><code class="lang-auto">t_Data = torch.from_numpy(data)
</code></pre>
<p>But it gives error as:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-23-ecb1b9da7431&gt; in &lt;module&gt;
----&gt; 1 t_Data = torch.from_numpy(data)

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.
</code></pre>
<p>Now I have two questions:</p>
<p>From what I get, when downloading dataset from torch vision and loading with dataloader it gives labels inside of it. However, I try to prepare my own dataset, so how could I merge labels and images into one dataset so that when loading with <code>shuffle=True</code> they won’t mismatch?</p>
<p>I have an another solution once that sampling batches sequentially with <code>torch.utils.data.SequentialSampler(data_source)</code> as it doesn’t require shuffle to be true. However, this doesn’t differ the batches(they are same in iteration), too, as if <code>shuffle=True</code> with none sampler! I don’t know 'sequential sampler’s point? Can anyone explain?</p>

Usama_Hasan: <p>Hy, I guess what you’re asking for is how to load your own dataset into dataloader. One way to do this is using TensorDataset.</p>
<blockquote>
<p>from torch.utils.data import DataLoader , TensorDataset<br>
dataset = TensorDataset(X , y)<br>
trainloader = DataLoader(dataset , batch_size = 16, shuffle=True)</p>
</blockquote>

hectorist: <p>that’s what I am looking for, thxx <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Does my for loop run parallely, if all the tensors involved in the loop are on the GPU?

melody_marks: <p>I have a list of tensors and all of them are present on the GPU. I obtained this list by splitting one tensor on the GPU using <code>torch.split</code>. I want to get list of sums of the list of tensors I have. So, in simple terms, I want to get a list in which, the first element is sum of first tensor in the list, and so on. If I run a for loop for this, does it get parallelised? If not, is there a way to make it run parallely? I want to parallelize it since the list is pretty long, and the sum operation can be done parallely, and independently on every tensor present on the list. If this operation can be performed on the GPU, the performance gain would be immense.</p>

melody_marks: <p>I have opened a similar post on SO as well. In this post, I’ve given my usecase with example. You may want to check that out. <a href="https://stackoverflow.com/questions/62966810/does-my-for-loop-run-parallely-if-all-the-tensors-involved-in-the-loop-are-on-t" rel="nofollow noopener">Link</a></p>

melody_marks: <p>A related topic was also created by me : <a href="https://discuss.pytorch.org/t/summing-a-tensor-according-to-a-different-tensor-provided/89619">Link</a></p>
How to optimize two models separately

ptnewbie: <p>I have two models defined as:</p>
<blockquote>
<p>model1 = Net1()<br>
model2 = Net2()<br>
opt1 = Adam(model1.parameters(),…)<br>
opt2 = Adam(model2.parameters(),…)</p>
</blockquote>
<p>When training, the model2 uses the output of model1 as input, and the two models have a different loss. I tried to train them:</p>
<blockquote>
<p>output1 = model1(data)<br>
output2 = model2(output1)<br>
loss1 = Loss(output1)<br>
loss2 = Loss(output2, labels)<br>
loss = loss1+loss2</p>
</blockquote>
<blockquote>
<p>model1.zero_grad()<br>
model2.zero_grad()<br>
loss.backward()<br>
opt1.step()<br>
opt2.step()</p>
</blockquote>
<p>In this way, I cannot train the model1 well. How can I train them separately? I only need loss1 to train model1. So I also tried:</p>
<blockquote>
<p>model1.zero_grad()<br>
loss1.backward()<br>
opt1.step()<br>
model1.eval()</p>
<p>model2.zero_grad()<br>
loss2.backward()<br>
opt2.step()<br>
model1.train()</p>
</blockquote>
<p>But I still cannot get the same results when only training model1, i.e., I removed all other codes related to model2.<br>
Thanks in advance.</p>

ptrblck: <p>The code looks generally alright.<br>
If you don’t want <code>loss2.backward()</code> to calculate gradients in <code>model1</code>, you could detach <code>output1</code> before passing it to <code>model2</code> via:</p>
<pre><code class="lang-python">output2 = model2(output1.detach())
</code></pre>
<p>This will make sure to cut the computation graph at this point.</p>

ptnewbie: <p>Yes, my code is correct. I changed the hyper parameters and it works as training a single network. Thank you for your help!</p>
Clone a model with copy.deepcopy(model)

silvester: <p>When I clone a model with copy.deepcoy, the cloned model is not training.</p>

ptrblck: <p>After cloning the model, you would probably need to recreate the optimizer, if not already done.<br>
Feel free to post a minimal executable code snippet, in case you get stuck, so that we could have a look at it.</p>
Difference between model_state_dict and generator_state_dict

Imran_Rashid1: <p>I have seen some code where they use both <code>model.model_state_dict()</code> and <code>model.generator_state_dict()</code>. I know that model_state_dict gives you the weights of the model neural network but I am unsure about the latter (generator_state_dict) as I cannot find any documentation on this. Can anyone shed some light on to this or at least provide a link with the relevant documentation for the generator_state_dict?</p>

ptrblck: <p>Neither of these two methods is a built-in method of <code>nn.Module</code>, so you would need to look into the custom model definition and search for these functions.</p>
<p><code>nn.Module</code> provides the <code>.state_dict()</code> method, which will return all registered parameters and buffers of this model.</p>
Older version support of SyncBatchNorm

Javix: <p>Hi,</p>
<p>I’m using torch 1.0 because of the gpu but I would like to include SyncBatchNorm in the code and it is not supported in the older version. In this way, I wonder how I can handle such issue. Any suggestions are appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I would recommend to update to the latest version and build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.<br>
This would not only include a lot of new features, but also bug fixes.</p>
<p>Let us know, if you get stuck or if this wouldn’t work for your setup.</p>