akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Check model performance

username: 111173
cooked: <p>Hi, I’m making a fully connected model.<br>
I’ve trained the model with same data set that is generated by ‘DataGenerator’ library and set same seed.<br>
The hyperparameters of the model were also fixed.<br>
However, the performance of the model was very different. some of training cases were trained well, and some of others were not at all. I don’t know how it is possible and how I interpret it. There I can’t go further steps like tuning the hyperparameters because I am not sure about my model works well.<br>
Please give me  advise. Thanks and have a nice day.</p>
Trainable parameter to select training frame in input (attention mechanism?)

username: rooks
cooked: <p>Hello</p>
<p>I am trying something which is new for me, and I don’t know if what I want to do makes sense or is possible to train with pytorch and autograd.</p>
<p>Basically, I would like to have a float scalar parameter used to define the start of the input data frame<br>
on which the forward function is applied (I guess this is kind of a attention mechanism?).</p>
<p>I want this scalar parameter to be trained together with the other parameters, so I define it as a <code>torch.nn.Parameter</code>.</p>
<p>To simplify, let’s say I want this parameter to select the start index of the frame between [0, 100]. And to simplify let’s say I manage to keep this parameter in the interval [0, 1]</p>
<p>So I just multiply 100 by the start parameter to select the beginning of my frame.</p>
<p>Then I simply apply the rest of my forward function on this particular frame of the input.</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):

    self.layer1 = nn.Linear(...)
    # ... normal MLP network

    self.start_param = torch.nn.Parameter(torch.tensor(0.1))

    def forward(self, x):
        start_index = torch.floor(self.start_param * 100).type(torch.int)       # seems start parameter not updated
        self.frame = range(0, start_index)
        x = x[:, self.frame]
        # continue and apply forward logic to the selected frame
        # ...
</code></pre>
<p>However, it seems my parameter <code>self.start_param</code> is not updated or trained. grad is always <code>None</code>.</p>
<p>Seems it is out of the graph scope.</p>
<ol>
<li>
<p>I am not sure how to check if a given parameter is part of the graph. How can I check is a given parameter (here my scalar parameter) is part of the parameters tracked during the forward pass?</p>
</li>
<li>
<p>I am not expert in autograd and pytorch, but I feel what I’m doing cannot be managed by autograd. Because it is based on frames, I guess the frame selection is not tracked by the graph and so the frame parameter won’t be updated during backward. Could you clarify this for me?</p>
</li>
<li>
<p>Is there another way to do what I want to do?</p>
</li>
</ol>
<p>Thanks.</p>
To require_grad or to not require_grad?

username: Gautam_Bhattacharya
cooked: <p>Hello,</p>
<p>Say I have a 6 layer network, that has been trained on some data. Now I want to re-train the lowest layer (layer closest to the data) only. I know that I can add only the specific layer (say self.fc1) to the optimizer,  so that only its parameters get updated.<br>
However, I am a little confused if I need to set requires_grad=False for the other layers. Do I need to? what happens if I do or do not?</p>
<p>Thanks in advance,<br>
Gautam</p>

username: pierre_bizeul
cooked: <aside class="quote no-group" data-username="Gautam_Bhattacharya" data-post="1" data-topic="5726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gautam_bhattacharya/48/21145_2.png" class="avatar"> Gautam_Bhattacharya:</div>
<blockquote>
<p>Say I have a 6 layer network, that has been trained on some data. Now I want to re-train the lowest layer (layer closest to the data) only. I know that I can add only the specific layer (say self.fc1) to the optimizer,  so that only its parameters get updated.</p>
</blockquote>
</aside>
<p>It does not matter, but it’s useless to compute the grad for those layers indeed.</p>

username: Small_Home
cooked: <p>Say you have a network composed of two parts: a feature extractor and a classifier. If I want to train only the feature extractor and not the classifier I should pass the feature extractor parameters to the optimizer and set require_grad to True for every parameter in the classifier right? Can I set require_grad to False in this scenario for every parameter in the classifier?</p>

username: suriname0
cooked: <p>I have no knowledge of the internals whatsoever, but from my simple testing it seems as though it doesn’t matter, i.e. you can set require_grad to false, but it doesn’t give you any performance speed-up.</p>
Regarding the usage of ASGD Optimizer in CNN

username: ina
cooked: <p>Hello everyone,</p>
<p>I am using pytorch to build a CNN model to classify DNA sequences. Regarding the optimizer, I tried both Adam and SGD but the accuracy was not improving no matter whether I increase the lr or decrease it. Then I found another optimizer called <a href="https://pytorch.org/docs/stable/_modules/torch/optim/asgd.html#ASGD" rel="nofollow noopener">ASGD</a> and using it instantly improved the accuracy of my model with the default parameters.</p>
<p><code>optimizer = torch.optim.ASGD(model.parameters(), lr=0.01, lambd=0.0001, alpha=0.75, t0=1000000.0, weight_decay=0)</code></p>
<p>So just wanted to know if ASGD is also commonly used because I could see that Adam and SGD are widely used but ASGD is not so common and I am a bit doubtful since I am new to Deep Learning. Does anyone have experience using it in CNN and let me know it’s advantages and disadvantages??</p>
<p>Thanks!</p>
Memory cost of nn.SyncBatchNorm

username: ginobilinie
cooked: <p>When I use nn.parallel.DistributedDataParallel for multi-gpu training in a single node, I use the nn.SyncBatchNorm to work as batch normalization across GPUs. However, I found the gpu memory cost increased a lot, at least 1gb for one gpu. When I use the SyncBatchNorm provided by apex (but I cannot successfully compile apex in this server), the gpu memory cost is normal. Can anyone help with it?</p>
How to use torchtext to build a vocabulary with binary file such as 'GoogleNews-vectors-negative300.bin'?

username: catqaq
cooked: <p>I can use the word vector model in txt format as follows:</p>
<pre><code class="lang-auto">if not os.path.exists(.vector_cache):
    os.mkdir(.vector_cache)
vectors = Vectors(name='myvector/glove/glove.6B.200d.txt')
TEXT.build_vocab(train, vectors=vectors)
</code></pre>
<p>However, when i turn to the binary format such as googlenews-vectors-negative300.bin, I got an error: could not convert string to float.<br>
The code is almost the same as above ：</p>
<pre><code class="lang-auto">if not os.path.exists(.vector_cache):
    os.mkdir(.vector_cache)
vectors = Vectors(name='GoogleNews-vectors-negative300.bin')
TEXT.build_vocab(train, vectors=vectors)
</code></pre>
<p>so, how to use the word vector model in binary format to build a vocab?<br>
In addition, should we use the vocabulary of the pre-trained model directly, or build a vocabulary from the training set, or build a vocabulary from the training set + test set? I am very confused about this.</p>
<p>Any help will be grateful！</p>

username: sai_m
cooked: <p>I have faced same issue with bin files. What I have done is, I have converted bin to text format and it started working.</p>
<p>When it comes to building custom embeddings, i think it depends on few factors like, how much data you have to generate good embeddings? Does available embeddings doesn’t serve the purpose?</p>
<p>Unless you data is very specific, like medical or related to something specific, which already available embeddings doesn’t have then you can train using gensim(which is very simple) to get your own embeddings.</p>
<p>However, I would suggest to play around glove or fasttext embedding using few custom words you have in your dataset and see the outputs before making any decision on using custom or available embeddings.</p>
Complex conjugate

username: blade
cooked: <p>I am trying to rewrite a process which was written for numpy arrays to perform using torch tensors. In one point, I need to compute <code>mat.conj().T</code> I was looking around and it seems that bringing complex values to pytorch <a href="https://github.com/pytorch/pytorch/issues/755" rel="nofollow noopener">is a hot topic</a>. I was wondering if there is an equivalent to <code>.conj()</code> in torch or I should simply go back and forth between numpy and torch to do this.</p>

username: nairbv
cooked: <p>It looks like a complex conjugate operator <code>torch.conj()</code> was added a couple of months ago on master, but isn’t available in v1.3.1:</p>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/26500" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/26500" target="_blank" rel="nofollow noopener">Vectorized complex unary and binary op support.</a>
    </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>dylanbespalko:unary_binary_ops</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-09-19" data-time="21:57:58" data-timezone="UTC">09:57PM - 19 Sep 19 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/dylanbespalko" target="_blank" rel="nofollow noopener">
          <img alt="dylanbespalko" src="https://avatars1.githubusercontent.com/u/11486236?v=4" class="onebox-avatar-inline" width="20" height="20">
          dylanbespalko
        </a>
      </div>

      <div class="lines" title="27 commits changed 25 files with 1462 additions and 82 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/26500/files" target="_blank" rel="nofollow noopener">
          <span class="added">+1462</span>
          <span class="removed">-82</span>
        </a>
      </div>
    </div>

  </div>
</div>
  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Can I train a CNN model for RL?

username: UnaiCorzo
cooked: <p>Hi there!<br>
I am working on a CNN model, which takes a chessboard as an input (an array containing the location of each piece type), and gives the value of the position (from 0 to 1); thus a regression problem. However, in order to do it using Reinforcement Learning, I would like the model to take as inputs the positions the chess engine is ‘seeing’ while it is playing training games against itself. Therefore, the model will ‘see’ single inputs (chessboard positions) at each time (the input size is 8x8x13), rather than a complete dataset. Furthermore, the network arquitecture will have to figure it out the value of each position on its own, without any help or target value. How can it be done?<br>
Thanks,</p>

username: nairbv
cooked: <blockquote>
<p>figure it out the value of each position on its own, without any help or target value.</p>
</blockquote>
<p>One way that’s been done before is that the “target” is winning the game – i.e. board positions from each game are labeled by whether or not they resulted in a win. There’s an explanation here, which used a database of 640,000 actual games to train their evaluation function:</p>
<p><a href="https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf</a></p>
1D convolution peak picking

username: Garnets
cooked: <p>Hello, I want to use 1D connection to pick a peak pick on every beat of music, but I want to know if I am doing right or missing anything.</p>
<p>Input uses a tensioner of Batch * 441 (SampleRate/100 of sound, that is, 0.01 second data) * 1 (Batch * 441 * 1) as input. After the 1D Convolution shown in the picture, the output of Batch * 2 is shown, which is the score for Class 1 and Class 2.</p>
<p>Target data is a tensor of (Batch * 2) size that displays every 0.01 second whether the two classes are true or false, respectively Because each of the two classes needs to be scored separately, the Loss function uses BCELoss with Multi Label Classification.</p>
<p>Assuming there’s no problem with the learning data, is there anything I’m missing? Or is there something wrong?</p>
<pre><code class="lang-auto">class convNet(nn.Module):

    def __init__(self):

        super(convNet, self).__init__()
        # model
        self.conv1 = nn.Conv1d(in_channels=441, out_channels=881, kernel_size=1, stride=2)
        self.conv2 = nn.Conv1d(in_channels=881, out_channels=1024, kernel_size=1, stride=2)
        self.fc1 = nn.Linear(1024, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 4)


    def forward(self, x, batch):

        x = F.max_pool1d(F.relu(self.conv1(x)), 1)
        x = F.max_pool1d(F.relu(self.conv2(x)), 1)
        x = F.dropout(x.view(batch, -1))
        x = F.dropout(F.relu(self.fc1(x)))
        x = F.dropout(F.relu(self.fc2(x)))

        return F.sigmoid(self.fc3(x))
</code></pre>
Do any of the recent transformer models work like this?

username: vainaijr
cooked: <pre><code class="lang-auto">'I ate __ apple' # task is to predict what word fills the blank

                      I ate a apple      # but correct phrase is 'I ate an apple', so we get a loss, and backprop
                    emb(I ate a apple)

               I ate __     (there is a blank, so we do softmax on all words)       apple
            emb(I ate __)   (, suppose 'a' got highest probability          )     emb(apple)
                                  
    I ate     softmax(   a     an     ...(all list of possible words))          a      p       p      l     e
  emb(I ate)         ( emb(a) emb(an) ...                            )      emb(a)  emb(p) emb(p) emb(l) emb(e)
                      
  I      ate              a        n 
emb(I) emb(ate)         emb(a)    emb(n)

        a      t      e
      emb(a) emb(t)  emb(e) 
</code></pre>
<p>use of (character, word, phrase, sentence level embeddings)</p>
<p>the way to combine two embeddings together could be using another weight matrix in middle like</p>
<pre><code class="lang-auto">emb(I ate) = emb(I) * weight_matrix * emb(ate)
</code></pre>
Is there a function to sort eigenvectors in by decreasing eigenvalues

username: alabijesujoba
cooked: <p>Having computed eigenvectors and eigenvalues of a matrix with pytorch,<br>
<code>w, lamb = torch.eig(matrix,eigenvectors=True)</code></p>
<p>Is there a function to sort the eigenvectors in the decreasing order of the eigenvalues?</p>

username: albanD
cooked: <p>You can use <a href="https://pytorch.org/docs/stable/torch.html#torch.sort"><code>.sort()</code></a> on the eigenvalues and use the indices to get the same sorting in the eigenvectors.</p>

username: alabijesujoba
cooked: <p><a class="mention" href="/u/alband">@albanD</a>, is the following right? Especially the dim?</p>
<pre><code class="lang-auto">sortedEig, indices=torch.sort(w, dim=0, descending=True, out=None)
lambSort = lamb.gather(dim=1, index=indices)


</code></pre>

username: albanD
cooked: <p>You don’t need to specify <code>out=None</code>.</p>
<p>This looks good according to the doc yes.</p>

username: alabijesujoba
cooked: <p>Thanks for your help, lambSort is a <strong>N x 2 tensor</strong> instead of <strong>N x N tensor</strong> which <strong>lamb</strong> actually is. How do I fix this?</p>

username: albanD
cooked: <p>Ho sorry I misread, I think you want <code>index_select()</code> instead of <code>gather</code> here.</p>

username: alabijesujoba
cooked: <p>OK! Since the first element/column of the eigenvalue tensor is the real part and the second element is the imaginary part, is it safe to use/take just the first element(real part) for sorting? Or do I need to combine them? If yes, is there a way to combine them?</p>

username: albanD
cooked: <blockquote>
<p>If yes, is there a way to combine them?</p>
</blockquote>
<p>That depends on your application !  Do you want to only consider the real part? Or the norm of the complex number?</p>

username: alabijesujoba
cooked: <p>I am actually implementing PCA so I guess I dabble ignore the complex elements.</p>
Indexing over a dimension

username: vadimkantorov
cooked: <p>Is there a loopless native PyTorch equivalent of this op?</p>
<pre><code class="lang-python">def index_dim(tensor, dim, index):
  return torch.stack([tensor[k].select(dim - 1, i) for k, i in enumerate(index.tolist())])

t = torch.rand(2, 4, 3)
i = torch.LongTensor([0, 1])
print(index_dim(t, 1, i).shape)
# torch.Size([2, 3])

</code></pre>
<p>In practice I need this to undo padding from a GRU output (I know I could use PackedSequence format, but for now I’d like to try padded format as it’s more compatible with ConvNet models) and get an embedding from the last meaningful timestep for every sequence in the batch.</p>

username: richard
cooked: <p>In the case you have you could do</p>
<pre><code class="lang-auto">t[torch.arange(t.size(0)), i]
</code></pre>
<p>but I think you’re trying to ask for something more general. How do you want to undo the padding from a GRU output?</p>

username: vadimkantorov
cooked: <p>For every batch element of output of GRU I’m taking a certain timestep slice that corresponds to the last timestep for that sequence. This discards hidden states corresponding to padding time steps - that’s what I mean by “undoing the padding”</p>
<p>Thanks for your suggestion! In my original post notation it should be the following, right?</p>
<pre><code class="lang-python">def index_dim(tensor, index):
  return tensor[torch.arange(len(tensor)), index]
</code></pre>

username: richard
cooked: <blockquote>
<p>For every batch element of output of GRU I’m taking a certain timestep slice that corresponds to the last timestep for that sequence. This discards hidden states corresponding to padding time steps - that’s what I mean by “undoing the padding”</p>
</blockquote>
<p>Thank you for that explanation, I understand your problem now.</p>
<p>Yes, that is my suggestion in the original post notation.</p>
Standarization with pytorch

username: CarlosHernandezP
cooked: <p>I am trying to solve a multiclass classification problem in which I have over 7000 pictures with labels. While preprocessing the data, I use a torchvision.Compose with Normalize() function included at the end. Said function performs the following operation for all the three channels:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a81ba73e07ef2e37416693f686b6fbe376d24063.png" alt="image" data-base62-sha1="nZ9sDGqlRiDbCgOIY7nq6uLtKIX" width="256" height="144"></p>
<p>Which is convenient for many cases but what I need is to <strong>standarize my data</strong>, that is to say, make it with 0 mean and variance 1. I cannot seem to find an easy way to do this without using other libraries.</p>
<p>I am not providing code as my problem is not related to a specific error but more towards a general lack of knowledge that could not be solved by searching on this forum/Stack Overflow/torchvision documentation.</p>

username: ptrblck
cooked: <p>The mentioned approach will standardize your data, as you will subtract the mean (so that the output will have zero mean) and divide by the standard deviation (so that the output will have unit variance).</p>

username: CarlosHernandezP
cooked: <p>But which values should I give to the Normalize function?</p>
<p>Or, if no arguments are given, the function automatically standardizes the data? I have seen multiple uses of this function, most of the times gave Normalize([0.5,0.5,0.5],[0.5,0.5,0.5]). As tensors have values between 0 and 1 this gives would mean that the values would vary from -1 to 1. I have seen also more wacky approaches with different numbers for each of the parameters. I feel like I’m at the grasp of finally realizing how to solve my problem but I’m not quite there yet.</p>

username: ptrblck
cooked: <p>If you are using “ImageNet-like” images, you could try to use the calculated stats from the ImageNet dataset as used in our tutorials:</p>
<pre><code class="lang-python">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
</code></pre>
<p>You could of course calculate the mean and std of your training dataset manually and use these stats.<br>
Note that the values are in the specified range, as you are usually calling <code>transforms.ToTensor()</code>, which returns a tensor in the range <code>[0, 1]</code>.</p>
<p>Even if your “real” stats are not exactly these numbers, the pretrained models will most likely work just fine.<br>
However, if your data comes from a completely different domain, e.g. CT/MRI images, you might need to use your calculated stats.</p>
NeurIPS 2019 PyTorch meetup on Thursday

username: dzhulgakov
cooked: <p>Hi everyone,</p>
<p>Let’s do an informal PyTorch meetup at NeurIPS on Thursday Dec 12 at 3:05pm-3:50pm (big coffee break).</p>
<p>We can meet at West Level 2 next to rooms 220-222 and hang out in the hallway.</p>
<p>Many of PyTorch core devs will attend.</p>

username: albanD
action_code: pinned_globally.enabled
unhandled cuda error

username: 1seck
cooked: <p>When trying out just 2 instances with 1 gpu each attached to test distributed training this error occurred:</p>
<blockquote>
<p>(base) ubuntu@ip-172-31-11-131:~/detectron2$ NCCL_SOCKET_IFNAME=ens3 NCCL_IB_DISABLE=1 python tools/train_net.py --num-gpus 1 --num-machines 2 --machine-rank 1 --dist-url tcp://32.273.122.180:3000 --config-file configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml<br>
Command Line Args: Namespace(config_file=‘configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml’, dist_url=‘tcp://34.253.142.180:3000’, eval_only=False, machine_rank=1, num_gpus=1, num_machines=2, opts=<span class="chcklst-box fa fa-square-o fa-fw"></span>, resume=False)<br>
Traceback (most recent call last):<br>
File “tools/train_net.py”, line 161, in <br>
args=(args,),<br>
File “/home/ubuntu/detectron2/detectron2/engine/launch.py”, line 49, in launch<br>
daemon=False,<br>
File “/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn<br>
while not spawn_context.join():<br>
File “/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 118, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/home/ubuntu/detectron2/detectron2/engine/launch.py”, line 70, in _distributed_worker<br>
comm.synchronize()<br>
File “/home/ubuntu/detectron2/detectron2/utils/comm.py”, line 79, in synchronize<br>
dist.barrier()<br>
File “/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 1424, in barrier<br>
work = _default_pg.barrier()<br>
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1573049306803/work/torch/lib/c10d/ProcessGroupNCCL.cpp:400, unhandled cuda error</p>
</blockquote>
<p>Exporting NCCL_SOCKET_IFNAME and NCCL_IB_DISABLE didn’t help, also other fixes as discussed in github issues and everything I found in the net on this topic.<br>
Maybe I forgot an argument. In AWS, “ens3” seems to be the ethernet connection, at least <code>ifconfig</code> does not reveal eth0 as usual.</p>
<p>What to do?</p>
Non-linearities in Denoising RNN Autoencoders

username: HenryPowell
cooked: <p>Hi there,</p>
<p>I’m employing a denoising RNN autoencoder for a project relating to motion capture data. This is my first time using auto encoder architectures and I was just wondering what non-linearities should be placed in these models and where they should go. This is my model as it stands:</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(EncoderRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.rnn_enc = nn.RNN(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
        self.relu_enc = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_enc(x, None)
        pred = self.relu_enc(pred)
        return pred


class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, num_layers):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers

        self.rnn_dec = nn.RNN(input_size=hidden_size, hidden_size=output_size, num_layers=num_layers, batch_first=True)
        self.relu_dec = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_dec(x, None)
        pred = self.relu_dec(pred)
        return pred


class RNNAE(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(RNNAE, self).__init__()
        self.encoder = EncoderRNN(input_size, hidden_size, num_layers)
        self.decoder = DecoderRNN(hidden_size, input_size, num_layers)

    def forward(self, x):
        encoded_input = self.encoder(x)
        decoded_output = self.decoder(encoded_input)
        return decoded_output
</code></pre>
<p>As you can see I have a ReLU non-linearity in each of the encoder and decoder networks but am not sure whether this is the correct implementation for these architectures.<br>
The model learns on the data OK but the MSE loss doesn’t really improve after the first few epochs and I have a suspicion it’s because of these ReLU functions.</p>
<p>Any advice on how to improve this/ whether this is basically correct would be very helpful.</p>
<p>Cheers!</p>
Writing a test_model function in pytorch

username: Home_Amino
cooked: <p>hi guys i wanna write a function  <code>test_model(model, criterion):</code><br>
It should  <strong>print</strong>  the  <strong>loss</strong>  and the test set  <strong>accuracy</strong> , and should  <strong>return  <code>y_true</code>  and  <code>y_pred</code></strong>  as NumPy arrays.</p>
<p>so i tried the following :</p>
<p>def test_model(model_ft, test_dataloaders,criterion,device):</p>
<pre><code># set model to evaluate model
model_ft.eval()
# Iterate over data.
for inputs, labels in dataloaders[phase]:
    inputs = inputs.to(device)
    labels = labels.to(device)
</code></pre>
<p>and now my question how do i get y_pred and y_true and the rest things?<br>
We are in the framework of transfer learning in pytorch so the function i wanna write is the same as the train_model see(<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a>) but somewhat edited for the prediction purpose. I also thought of concatenate the y_pred and y_true after each batch but i really cant put this into reality could you plz help me out ? Thanks</p>
How to implement the morphologyEx operation

username: Pengfei_Wang
cooked: <p><a href="https://docs.opencv.org/trunk/d9/d61/tutorial_py_morphological_ops.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.opencv.org/trunk/d9/d61/tutorial_py_morphological_ops.html</a><br>
if i can implement the morphologyEx  operation by built-in function such as maxpooling?</p>

username: v-prgmr
cooked: <p>Hi, did you implement this?</p>
Which line resulted in nan during the backprop?

username: Kong
cooked: <p>I am trying to debug my program that has nan gradients. How do I know which function caused the gradients to go nan when I use hooks? For example, in the snippet below, the assert is triggered at line 2. Does that mean that line 4, <code>x_skip = self.conv3dcaps4_nj_8</code> is the one that resulted in nan gradients during the backprop? Because the hook for x is registered at line 2 and the gradient of x obtained when backpropagating through line 4.</p>
<pre><code class="lang-auto">1 x = self.conv2dCaps4_nj_8_strd_2(x)  
2 x.register_hook(assert_enc5) # assert triggered here
3 assert not torch.isnan(x).any()
        
4 x_skip = self.conv3dCaps4_nj_8(x)  
5 x_skip.register_hook(assert_enc4)      
6 assert not torch.isnan(x_skip).any()
</code></pre>

username: Ceday
cooked: <p>I havent used it, but there is a context manager you might want to look at it.<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/autograd/anomaly_mode.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/autograd/anomaly_mode.html</a></p>
Bounding-Box Precision/Recall for Empty Images

username: maaft
cooked: <p>I have difficulties with calculating bounding-box recall and precision for a validation set that contains roughly 50/50 images with objects and images without objects.</p>
<p>As recall can’t be calculated for images without any objects, I use these images only for getting a better estimate of my precision:</p>
<p>precision = P_p * N_p / (N_p + N_n)<br>
recall = R_p</p>
<p>where P_p, R_p are the precision and recall values for non-empty images, N_p is the number of predicted bounding boxes for non-empty images and N_n is the number of predicted bounding-boxes for empty images.</p>
<p>Do you see any issues with this?<br>
Are there any papers available that address this issue?</p>
<p>This forum really needs LaTeX integration!</p>

username: Oli
cooked: <p>Hi! I don’t think that there are any issues with your approach. That’s generally how it’s done to the best of my knowledge. For papers, look at the <a href="http://host.robots.ox.ac.uk/pascal/VOC/pubs/everingham10.pdf" rel="nofollow noopener">Pascal VOC AP</a>, COCO mAP</p>

username: Oli
cooked: <p>Oh another thing I’ve been meaning to try is to calculate some other metrics for a greater model understanding.</p>
<ol>
<li>Average IoU for all predicted boxes (wrong predictions give a IoU of 0)</li>
<li>Average IoU for correct predicted boxes</li>
</ol>
<p>And if you haven’t seen it, there is a <a href="https://github.com/rafaelpadilla/Object-Detection-Metrics" rel="nofollow noopener">github repo</a> that can help with bounding box metrics.</p>
CUDA error: device-side assert triggered in optimizer.step()

username: a-parida12
cooked: <p>I am trying to train on the unet from <code>torch.hub</code>.  Some of the image-label pairs give me a <code>CUDA error: device-side assert triggered</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f276a6d2e74b08aa36b067e9723904447e2941.png" data-download-href="https://discuss.pytorch.org/uploads/default/39f276a6d2e74b08aa36b067e9723904447e2941" title="Screenshot%20from%202019-12-11%2000-52-03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39f276a6d2e74b08aa36b067e9723904447e2941_2_690x291.png" alt="Screenshot%20from%202019-12-11%2000-52-03" data-base62-sha1="8gCGRREnMxpGEpoufYwKUvR2c2R" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39f276a6d2e74b08aa36b067e9723904447e2941_2_690x291.png, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f276a6d2e74b08aa36b067e9723904447e2941.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39f276a6d2e74b08aa36b067e9723904447e2941.png 2x" data-dominant-color="F5F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-12-11%2000-52-03</span><span class="informations">806×341 40.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How Can I fix this?</p>

username: ptrblck
cooked: <p>Does this model work on the CPU without any errors?<br>
Usually you might get a better error message by running it on the CPU.</p>

username: a-parida12
cooked: <p>I tried running it on the cpu. I was able to fix the problem with the error message.</p>
Torch.nn.init.normal_()

username: Kenzd
cooked: <p>a pytorch beginner need your help! it seems a easy problem but i just can’t solve it <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><br>
i wana use torch.nn.init.nomal(model.parameters(),mean=0,std=1) to initialize my model, but i got this problem:<br>
<em>AttributeError: ‘generator’ object has no attribute ‘normal_’</em><br>
my pytorch version is 1.2.0</p>

username: Eta_C
cooked: <p><code>model.parameters()</code> return a generator.</p>
<p>An example:</p>
<pre><code class="lang-python">def init_function(module):
    if isinstance(module, nn.Conv2d):
        nn.init.normal(module.weight, mean=0, std=1)

net = YourNet()
net.apply(init_function)
</code></pre>

username: Kenzd
cooked: <p>It works! Thank u so much!</p>
Data Parallel for submodules not working properly

username: Sayan_Ghosh
cooked: <p>Hi, I have 4 separate nn.Modules which I instantiate inside my actual final model class (this is also nn.Module), so I have something like,</p>
<pre><code class="lang-auto">class M(nn.Module):
    def __init__():
          super(M, self).__init__()
          self.A = A()
          self.B = B()
          self.C = C()
          self.D = D()
</code></pre>
<p>for training I do the following :</p>
<pre><code class="lang-auto">model = M().to(device)
model = nn.DataParallel(model)
</code></pre>
<p>However I do not see the other 3 GPUs get utilised at all in a 4 GPU machine.<br>
I have done <code>export CUDA_VISIBLE_DEVICES=0,1,2,3</code> before starting training.</p>
<p>Can someone give an insight to what I am doing wrong ?</p>

username: nairbv
cooked: <p>One thought: Is your batch size larger than the number of GPUs? If smaller than it wouldn’t be able to divide your batch across the available GPUs.</p>

username: alex.veuthey
cooked: <p>Another thought: note that having 4 submodules and 4 GPUs won’t use one submodule per GPU, if that’s what you intended (unless perhaps only one module is selected and used in a forward pass).</p>
<p>What is your <code>nvidia-smi</code> output when running? What is your <code>device</code> variable?</p>
How to make the smaller the value(score), the higher the probability of being sampled

username: wwwangzhch
cooked: <p>If I have a tensor outputted by the neural network, representing the score of each category. I want to get two types of sampling results,：<br>
In the first case, the higher the value, the more likely it is to be sampled;<br>
In the second case, the lower the value, the more likely it is to be sampled.<br>
I can easily apply torch.multinomial() in the first case, but how can I get what I want in the second case?</p>
<p>For example, I have a tensor [0.1, 0.3, 0.4, 0.0, 0.7, 0.0, 0.2] and I want each case have 2 samples to be drawn, then:<br>
In the first case, [0.7, 0.4] are most likely to be drawn, and in the second case, [0.1, 0.2] are most likely to be drawn.<br>
Noticing that all the scores are from [0, 1] and if the score is 0, we want to mask it, that means this score shouldn’t be drawn in any case.</p>

username: KFrank
cooked: <p>Hi Wwwangzhch!</p>
<aside class="quote no-group" data-username="wwwangzhch" data-post="1" data-topic="63632" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwwangzhch/48/16678_2.png" class="avatar"> wwwangzhch:</div>
<blockquote>
<p>For example, I have a tensor [0.1, 0.3, 0.4, 0.0, 0.7, 0.0, 0.2] and I want each case have 2 samples to be drawn, then:<br>
In the first case, [0.7, 0.4] are most likely to be drawn, and in the second case, [0.1, 0.2] are most likely to be drawn.<br>
Noticing that all the scores are from [0, 1] and if the score is 0, we want to mask it, that means this score shouldn’t be drawn in any case.</p>
</blockquote>
</aside>
<p>In the first case your proposed use of <code>torch.multinomial()</code><br>
is fine.</p>
<p>Based on your statement that “all the scores are from [0, 1],”<br>
you can flip and mask your <code>scores</code> as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">scores = scores.sign() * (1.0 - scores)
</code></pre>
<p>If (before flipping) <code>scores</code> is <em>exactly</em> 0.0, <code>scores.sign()</code><br>
will return 0.0 (otherwise 1.0).  After flipping, 0.0 becomes 1.0,<br>
but gets set back to 0.0 when multiplied by the 0.0 from <code>sign()</code>.</p>
<p>You can now feed your flipped, masked scores to<br>
<code>torch.multinomial()</code><br>
(or <code>torch.distributions.categorical.Categorical</code>),<br>
and the masked entries will never get sampled because you’ve<br>
specified a probability of 0.0.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: wwwangzhch
cooked: <p>Hi Frank:<br>
Thank you for your reply, it’s perfect for my question!</p>
<p>I love Pytorch forever!!</p>
How to delete layer(not Sequential) in pretrained model?

username: wonchulSon
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae2548cd5beea95c48be8a158318e7d33f05676.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ae2548cd5beea95c48be8a158318e7d33f05676" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae2548cd5beea95c48be8a158318e7d33f05676.png" alt="image" data-base62-sha1="67mXM8eJ61frpCc1IgYIjOJ3Uma" width="690" height="357" data-dominant-color="F3F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1164×603 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Using pretrain model(ImageNet), I want to edit model to fit cifar dataset resnet architecture.</p>
<p>I can edit layer, but don’t know how to delete what I don’t want to use.<br>
Please let me know.<br>
Thank you.</p>

username: Eta_C
cooked: <p>See this topic.</p>
<aside class="quote" data-post="1" data-topic="63633">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ea5d25/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-remove-the-adaptive-average-pool-layer-from-vgg19/63633">How to remove the adaptive average pool layer from vgg19?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have loaded the pretrained model of vgg19. How to remove the adaptive average pool layer which is present before the classifier?
  </blockquote>
</aside>


username: wonchulSon
cooked: <p><a class="mention" href="/u/eta_c">@Eta_C</a></p>
<p>Oh, Thank you so much! ^^</p>
<p>Then can I ask you one more Q?</p>
<p>Do you know how to get hidden activation output?<br>
In pretrained resnet model, I want to get each blocks output !<br>
Thank you.</p>

username: Eta_C
cooked: <p>Search <code>register_hook</code> or <code>register_forward_hook</code> in PyTorch Forums.</p>
<p>One example is <a href="https://www.kaggle.com/sironghuang/understanding-pytorch-hooks" rel="nofollow noopener">Understanding-Pytorch-Hooks</a>.</p>
<p>Or you can also search <code>PyTorch Hook</code> in Google to learn more.</p>
transforms.RandomCrop()

username: yuyaya
cooked: <p>Hello,</p>
<p>I want to RandomCrop the numpy array but facing the error.<br>
Each input is shaped (channel=10, h=800, w=1200) and trying to transform in the custom dataset class.</p>
<p>the error message says,</p>
<blockquote>
<p>TypeError: cannot unpack non-iterable builtin_function_or_method object</p>
</blockquote>
<p>Does this happen because input wasn’t PIL image?<br>
If so, can someone share the way to transform numpy array.</p>
<p>Thank you</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/yuyaya">@yuyaya</a>,</p>
<p>A way to do would be to transform your numpy array to PIL Image in order to apply the transformation.</p>
<p>You can refer to this post <a href="https://discuss.pytorch.org/t/image-data-augmentation-for-numpy-data/34452/2" class="inline-onebox">Image data augmentation for numpy data</a></p>
Pytorch train network with gpu memory free problems

username: dtiny
cooked: <p>I use pytorch train network with same batch size 32. But network input size were different, such as 320x320, 416x416, 512x512.  I use command “nvidia-smi -l” to watch gpu memory occupation. Gpu memory was<br>
cost with max input size 512x512, and never reduced when input size was 320x320 or 416x416.  How to release the unused memory cache?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We use a special caching allocator that is much faster than the regular allocator. If you don’t need that memory, I would advise against clearing the cache. If you really need that memory, you can use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.empty_cache">cuda.empty_cache()</a> to release it.</p>
Multiprocess error

username: chunchun
cooked: <p>When I run the train.py in pysot which is an open sourece for trackers,it shows before any iteration which means it achieves no iteration:</p>
<p>DataLoader worker (pid 101735) is killed by signal: Killed. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.</p>
<p>But when I try to set  num_workers=0 ,it just does’t train.<br>
I really confused about this.Could you please tell why?</p>
<p>I  see some blog,it says memory is full when loading,is this right?if it isn’t GPU’s duty,how to modify my code?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="chunchun" data-post="1" data-topic="63560">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/97f17d/48.png" class="avatar"> chunchun:</div>
<blockquote>
<p>But when I try to set num_workers=0 ,it just does’t train.</p>
</blockquote>
</aside>
<p>What do you mean by that? Is the loss not decreasing in this case or is the code just stuck?</p>

username: chunchun
cooked: <p>The code just stuck.<br>
And now it stuck in model_load.<br>
It just stuck in different places witch makes me confused.<br>
Thanks for your attention and I would appreciate it  if you can give me some suggests.</p>

username: ptrblck
cooked: <p>Let’s continue the discussion on <a href="https://discuss.pytorch.org/t/the-training-wait-a-long-time-after-load-model/63604/3">this topic</a>.</p>
Proper autograd with split network

username: qwerty1024
cooked: <p>I’m building a network with a similar logic as <a href="https://discuss.pytorch.org/t/network-which-splits-and-joins-again/38357">here</a>, but this thread doesn’t quite solve my issue.</p>
<p>My network (simple, fully connected) in bad pseudocode:</p>
<pre><code class="lang-auto">common_layer = f(input)

branch1 = common_layer
branch1 = f(branch1)

branch2 = common_layer
branch2 = f(branch2)

output = a*branch1 + (1-a)*banch2
</code></pre>
<p>Is this construction guaranteed to backpropagate properly from output to branches to common layer? Do I need to make a deepcopy instead of just assigning <code>branch1 = common_layer</code> and <code>branch2 = common_layer</code>? Anything else that jumps out?</p>
<p>Network seems to be learning but if I add layers I see no real change in behavior which is making me think there is something funky with the branching and backprop.</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>That will work fine. You can directly do:</p>
<pre><code class="lang-auto">common_layer = f(input)

branch1 = f(common_layer)

branch2 = f(common_layer)

output = a*branch1 + (1-a)*banch2
</code></pre>
<p>As long as f is an nn.Module with no funky hooks. Or a simple python function.</p>
Grad is always None for leaf variable

username: dpstart
cooked: <p>I am trying to look at the activations for the feature in a CNN.<br>
However, when I look at the gradients of my <code>img_var</code> variable after calling <code>loss.backward()</code>, the gradient is always <code>None</code>. Note that the variable is always a leaf node.</p>
<pre><code class="lang-auto">class SaveFeatures():
    def __init__(self, module):
        self.hook = module.register_forward_hook(self.hook_fn)
    def hook_fn(self, module, input, output):
        self.features = torch.tensor(output,requires_grad=True).cuda()
    def close(self):
        self.hook.remove()

class FilterVisualizer():
    def __init__(self, size=224):
        self.size = size
        self.model = vgg16(pretrained=True).to(DEVICE)
        self.model.eval()

        for p in self.model.parameters(): p.requires_grad=False

    def visualize(self, layer, filter_n, lr=0.1, opt_steps=30):
        sz = self.size
        img = np.uint8(np.random.uniform(100, 200, (3, sz, sz)))/255  # generate random image

        activations = SaveFeatures(self.model.features[28])  # register hook

        img_var = torch.autograd.Variable(torch.Tensor(img).to(DEVICE).unsqueeze(0), requires_grad=True)  

        print(img_var.cpu())

        optimizer = torch.optim.Adam([img_var], lr=lr, weight_decay=1e-6)


        for n in range(opt_steps):  # optimize pixel values for opt_steps times
            optimizer.zero_grad()
            self.model(img_var)
            loss = -activations.features[0, filter_n].mean()
            loss.backward()

            print(img_var.grad)  ### This is always None
            optimizer.step()

        img = img_var.detach().cpu().numpy()[0].transpose(1,2,0)
        activations.close()


</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You don’t need to use <code>Variable</code> anymore. You can call <code>.requires_grad_()</code> on your Tensor after building it.</p>
<p>The problem here I think is your “SaveFeatures”. You rewrap the output in a brand new Tensor. That means that you break the link with the existing graph.<br>
Why not do <code>self.features = output</code> ?<br>
Also this is not recommended to do this in general as states can be tricky to handle properly.<br>
You can override the forward functino of the vgg16 net to remove what you don’t want maybe?</p>

username: dpstart
cooked: <p>Thank you, the problem was in fact in <code>SaveFeatures</code>.</p>