akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model accuracy with contrastive loss and triplet loss

username: dengio
cooked: <p>Is there a way to calculate some sort of accuracy when I have to deal with loss functions which calculates distance beetwen pairs? Siamese networks outputs embedded representations of input data and then these are fed to the contrastive loss which calculates their dissimilarity. How can I say that these representations belongs to the same class according to the margin I choose? For triplet networks I think a possible metric to calculate accuracy would be to check if the distance between anchor and positive has to be less than the distance between anchor and negative for each input of my data. <a href="https://gombru.github.io/2019/04/03/ranking_loss/" rel="nofollow noopener">Here</a> is an article which talks about these types of loss functions. Would you share your opinion about this? Thanks.</p>

username: Raul_Gombru
cooked: <p>I usually monitor the percentange of correct triplets in each batch. With that I mean the triplets where the distance between the anchor and the negative is bigger than the distance between the anchor and the positive by the margin.</p>
<p>Pytorch triplet loss does not provide tools to monitor that, but you can code it easily as I do in <a href="https://github.com/gombru/face_attributes_retrieval/blob/master/model/model.py" rel="nofollow noopener">here</a>.</p>
<p>I’m the author of the blog post you link <a href="https://gombru.github.io/2019/04/03/ranking_loss/" rel="nofollow noopener">Understanding Ranking Loss, Contrastive Loss, Margin Loss, Triplet Loss, Hinge Loss and all those confusing names</a></p>
[Android] Issue with pytorch android

username: vishalagarwal
cooked: <p>I am using a SSD object detection model to run on android using jit trace, developed in PyTorch 1.3.</p>
<p>2 days back, I was using the nightly <code>pytorch_android:1.5.0-SNAPSHOT</code> it was working fine and I was getting the expected output. But today the app is crashing and giving the following error <code>Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 17354</code>.</p>
<p>After this I tried other versions also.</p>
<p><code>pytorch_android:1.4.0</code> and  <code>pytorch_android:1.4.0-SNAPSHOT</code> : This is giving different output. For the same input Tensor, output tensors in python and android are coming out to be different.</p>
<p><code>pytorch_android:1.3.0</code> : <code>com.facebook.jni.CppException: false CHECK FAILED at ../c10/core/Backend.h (tensorTypeIdToBackend at ../c10/core/Backend.h:106)</code></p>
<p>Very confused with what’s going on! Can I revert back to the nightly build which was 2 days old?</p>
Static_quantization_tutorial of pytorch

username: Majid_Shirazi
cooked: <p>Hi everyone,</p>
<p>I am trying to use <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a> tutorial to do model quantization but I get error when I am trying do run the line of</p>
<pre><code class="lang-auto"># Convert to quantized model
torch.quantization.convert(myModel, inplace=True)

</code></pre>
<p>there is error of : untimeError: Didn’t find engine for operation quantized::conv_prepack NoQEngine (operator() at /opt/conda/conda-bld/pytorch_1573049306851/work/aten/src/ATen/native/quantized/cpu/qconv_prepack.cpp:264)</p>
<p>BTW, I am using pytorch 3.6 env. Would be appreciated if you help me. <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:"></p>

username: remya
cooked: <p>I also following the same tutorial.<br>
Can we  quantize the standard pytorch models with the same approach?  How to fuse the layers if they are not given as a separate class?<br>
Any one tried the same tools for  models like VGG/ResNet ?</p>

username: Deepali
cooked: <p>check what is set for - <code>torch.backends.quantized.engine</code></p>
In-place operations and mypy

username: tetratrio
cooked: <p>In-place operations on tensors is reported by mypy as an error.</p>
<p>Example:</p>
<pre><code class="lang-python">import torch

t = torch.zeros(3)
t.add_(1)
</code></pre>
<pre><code class="lang-bash">test.py:5: error: "Tensor" has no attribute "add_"; maybe "addr_", "addmv_", or "addmm_"?
</code></pre>
<ul>
<li>PyTorch Version: 1.3.1</li>
<li>OS: Ubuntu 16.04</li>
<li>How you installed PyTorch: conda</li>
<li>Python version: 3.7.4</li>
</ul>
<p>Is this an error or intended?</p>

username: richard
cooked: <p>Definitely unintended, please file an issue at <a href="https://github.com/pytorch/pytorch/issues">https://github.com/pytorch/pytorch/issues</a> if there isn’t one already</p>

username: tetratrio
cooked: <p>I opened an issue and also added all the other errors I found with typing in pytorch. Cheers!</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/32824" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/32824" target="_blank" rel="nofollow noopener">Incorrect typing errors</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-01-30" data-time="12:46:35" data-timezone="UTC">12:46PM - 30 Jan 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/Tetratrio" target="_blank" rel="nofollow noopener">
          <img alt="Tetratrio" src="https://avatars0.githubusercontent.com/u/7781148?v=4" class="onebox-avatar-inline" width="20" height="20">
          Tetratrio
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Mypy will report typing errors for some operations that should be allowed. I have encountered four different type check errors...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Indexing hints for torch.utils.data.Dataset

username: deeplearner20
cooked: <p>I have a three dimensional (x,y,z) NumPy array of audio clips. X is the number of data points, length of y is basically the length of utterances, and z is the number of features which remain constant. Each utterance represents a group of frames, and for the sake of this problem I have to randomly choose a frame with its corresponding labels. While choosing each frame it is important to choose it with some context, so basically that means I will be indexing my dataset to select (context+frame) x z  subset of data randomly from any utterance to train the model. I need to pad zeros at the start and beginning of each data point x to ensure that frames at the edges have a context on both sides.</p>
<p>Now, y (length of utterance is different for different data points) and I thus cannot directly convert it to a tensor directly. I need to use np.vstack to stack the data points. Now the problem I am facing is while writing the getitem function within torch.utils.data.Dataset, where I only have to index the original elements of y, ignoring the padded elements in the new stacked matrix. Are there any hints that you can provide for this?</p>
Tracing custom autograd.Function

username: G.M
cooked: <p>I have a custom autograd.Function:</p>
<pre><code class="lang-auto">class raw(autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        ctx.a = (inp * inp + 1).reciprocal()
        ctx.b = ctx.a.sqrt()
        return inp * ctx.b

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output * ctx.a * ctx.b
</code></pre>
<p>and I want to trace this function, but when I do:</p>
<pre><code class="lang-auto">jit.trace(raw.apply, example_inputs=tc.randn(1))
</code></pre>
<p>, I get the error from this line:</p>
<pre><code class="lang-auto">......................
    jit.trace(raw.apply, example_inputs=tc.randn(1))
  File "...\Python37\lib\site-packages\torch\jit\__init__.py", line 903, in trace
    name = _qualified_name(func)
  File "...\Python37\lib\site-packages\torch\_jit_internal.py", line 696, in _qualified_name
    "__module__ can't be None.".format(name))
RuntimeError: Could not get qualified name for class 'apply': __module__ can't be None.
</code></pre>
<p>This code used to work for pytorch 1.1.0, but after I updated it to 1.4.0 recently, I got this error. I’m using Python3.7.3 on windows10.</p>

username: peterjc123
cooked: <p>Reproduced. Could you please open a new issue at <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>?</p>

username: G.M
cooked: <p>Opened it: <a href="https://github.com/pytorch/pytorch/issues/32822" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32822</a></p>
Load and save a model

username: Hille
cooked: <p>I got a problem when I want to load my trained models Therefore I created me a simple example to find out what the problem of my save and load method is.</p>
<p>Here you can see the file where I save my model:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable


class MeinNetz(nn.Module):
    def __init__(self):
        super(MeinNetz, self).__init__()
        self.lin1 = nn.Linear(10, 10)
        self.lin2 = nn.Linear(10, 10)

    def forward(self, x):
        x = F.relu(self.lin1(x))
        x = (self.lin2(x))
        return x

netz = MeinNetz()

input = Variable(torch.ones(1, 10))
torch.save(netz.state_dict(), './net.pth')
output = netz(input)
print(output)
</code></pre>
<p>And here is the file where I want to load my model:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable


class MeinNetz(nn.Module):
    def __init__(self):
        super(MeinNetz, self).__init__()
        self.lin1 = nn.Linear(10, 10)
        self.lin2 = nn.Linear(10, 10)

    def forward(self, x):
        x = F.relu(self.lin1(x))
        x = (self.lin2(x))
        return x

netz2 = MeinNetz

netz2.load_state_dict(torch.load('./net.pth'))
input = Variable(torch.ones(1,10))
output = netz2(input)
print(output)

</code></pre>
<p>When I run the load file I get the following error :</p>
<pre><code class="lang-auto">load_state_dict() missing 1 required positional argument: 'state_dict'
</code></pre>
<p><strong>What is the problem here?</strong></p>
<p><strong>Sry im stupid I missed brackets. Problem of the simple example is solved</strong></p>

username: Subash_Gandyer
cooked: <p>How to define our model class if we are using Fastai’s ConvLearner object from a pretrained model? Can you provide me an example for this just as you did there for your question?<br>
Thanks in advance.</p>
<p>Subash</p>

username: angelinaG
cooked: <p>Hi Hille!<br>
I am unable to understand the bracket error.<br>
I am facing the same problem when executing the following line of code:</p>
<pre><code class="lang-auto">model1= nn.Module
model1.load_state_dict(torch.load("model1_weights.pt"))
</code></pre>
<p>There is no mismatch as far as I understand in the number of brackets.<br>
Could you please help me understand this?</p>
<p>Thank you.</p>
.cuda.is_available() returns false after updating

username: kaiseryet
cooked: <p>Hey guys, so I am using Ubuntu 18.04 and CUDA 10.2. Installed pytorch and torchvision by conda. Initially.cuda.is_available() returns True. Now after updating it returns false… Any good solution besides reinstalling?</p>
<p>ps I am a big fan of conda, so please suggest a solution under conda environment, not something like compiling binaries. And idk when pytorch for cuda 10.2 is gonna be out, so won’t count on that…</p>

username: ptrblck
cooked: <p>Could you post the log from the conda installation?<br>
Maybe the wrong version was picked (cpu instead of e.g. <code>cudatoolkit=10.1</code>).</p>

username: kaiseryet
cooked: <p>I picked cudatoolkit=10.2 at initial installation, but updating didn’t ask me to pick this option. I’ll post log later.</p>

username: ptrblck
cooked: <p>The binaries are shipping with <code>cudatoolkit=9.2</code> or <code>10.1</code>.<br>
If you specify <code>10.2</code>, you might get the package from another source and PyTorch might fall back to an older release.<br>
I just tested this command and got the following suggestion:</p>
<pre><code class="lang-auto">## Package Plan ##

  environment location: /home/pbialecki/anaconda3/envs/tmp

  added / updated specs:
    - cudatoolkit=10.2
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-10.2.89        |       hfd86e86_0       364.5 MB
    pytorch-1.0.0              |py3.7_cuda9.0.176_cudnn7.4.1_1       498.7 MB  pytorch
    torchvision-0.2.2          |             py_3          44 KB  pytorch
    ------------------------------------------------------------
                                           Total:       863.2 MB

The following NEW packages will be INSTALLED:

  ninja              pkgs/main/linux-64::ninja-1.9.0-py37hfd86e86_0
  pytorch            pytorch/linux-64::pytorch-1.0.0-py3.7_cuda9.0.176_cudnn7.4.1_1
  torchvision        pytorch/noarch::torchvision-0.2.2-py_3
</code></pre>
<p>As you can see, PyTorch would be downgraded to <code>1.0.0</code> with CUDA9.0.</p>
Location of aten files in conda installed pytorch

username: ash95
cooked: <p>I would like to modify the Normalization.cpp file in the aten submodule of pytorch (which I had installed via conda). However, I can’t seem to find it. So how do I modify it and rebuild the dependencies.</p>

username: ptrblck
cooked: <p>If you’ve installed the PyTorch binaries via conda, you should fine some files in your conda directory, e.g.:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.6/site-packages/torch/include/...
</code></pre>
<p>However, modifying them won’t have any effect.<br>
Instead, you should build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.<br>
Also, the <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">Contributing guidelines</a> might be helpful.<br>
If you change some files in the source folder and rebuild, you will be able to see the changes.</p>
Find indices with value (zeros)

username: cbaziotis
cooked: <p>I have a 1D Variable (LongTensor) and i want to find the indices of the elements with a given value (zero). Is there a way to do this efficiently in PyTorch?</p>
<p>For instance, in order to get the indices of non-zero elements, i do this:</p>
<blockquote>
<p>non_zeros = torch.nonzero(lengths_c.view(-1).data).squeeze()</p>
</blockquote>
<p>I need the opposite of this (indices of zero elements).</p>

username: SimonW
cooked: <p><code>(x != 0).nonzero()</code></p>

username: Nicolas_Beaudoin-Gag
cooked: <p>I think it would be (x == 0).nonzero() instead of (x != 0).nonzero(). The last one is the same as x.nonzero().</p>

username: imsrgadich
cooked: <p><a class="mention" href="/u/nicolas_beaudoin-gag">@Nicolas_Beaudoin-Gag</a>  How can we use output of <code>nonzero()</code> function as index for accessing data from a tensor <code>min_dist_sort</code> (<code>torch.DoubleTensor</code>) which is <code>torch.Size([9567, 4])</code>. It has sorted distance in each column. It has zeros, I’m looking for minimum non-zero distance in each column. Hence, I’m doing the following (where <code>ind</code> some column number).</p>
<p><code>min_dist_sort[(torch.nonzero(min_dist_sort[:,ind]&gt;0))[0],ind]</code></p>
<p>When I do this, I get the below error message.</p>
<p><code>TypeError: Performing basic indexing on a tensor and encountered an error indexing dim 0 with an object of type torch.LongTensor. The only supported types are integers, slices, numpy scalars, or if indexing with a torch.LongTensor or torch.ByteTensor only a single Tensor may be passed.</code></p>
<p>I’m not able to understand what the problem might be. Any suggestions here. Thank you.</p>

username: saba
cooked: <p>hi,<br>
I want to find the location of specific value in 3 dimention in pytorch. I used in2sub function in Matlab. Do you know what is the best option in Pytorch?</p>

username: ptrblck
cooked: <p>Would <a href="https://discuss.pytorch.org/t/equivalent-function-like-numpy-diff-in-pytorch/35327/6">this answer</a> from the other post work?</p>

username: saba
cooked: <p>many thanks for your reply.<br>
i should use torch.nonzero . I think it is the same as you recommend.</p>
Torch.cuda.is_available() returns false

username: Nishant_Hegde
cooked: <p>Hi folks!</p>
<p>I’m trying to use a pytorch code and torch.cuda.is_available() returns False. The cuda drivers are installed, nvidia drivers are upto date.</p>
<ol>
<li>nvidia-smi works</li>
<li>nvcc --version works as well</li>
</ol>
<p>Any ideas on how I can resolve this issue?</p>
<p>System config;<br>
Ubuntu 18.04<br>
NVIDIA driver version 410.48<br>
Cuda 10.0</p>

username: ptrblck
cooked: <p>How did you install PyTorch?<br>
If you are using the binaries, they will use the specified <code>cudatoolkit</code> not your local CUDA installation.<br>
Based on <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">this table</a> you would need the NVIDIA driver <code>&gt;= 418.39</code> for CUDA10.1, while your current version should work for CUDA9.2.</p>

username: Nishant_Hegde
cooked: <p>Thanks man!</p>
<p>Turns out there was a problem with the PyTorch version. Anyway, the issue is now resolved. Thanks for your help.</p>
How to make Tensor pass by reference?

username: Nick_8229
cooked: <p>In-place function could achieve it, but many operators has no in-place method, such as followings. How should I do pass by reference for such cases?</p>
<pre><code class="lang-auto">def foo(a, b):
   a = torch.max(a, b)

def foo2(a, b, c):
   a = torch.where(c, a, b)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>All objects are passed by reference in python.<br>
But doing <code>a = </code> does not try to change <code>a</code> inplace. It only give the name “a” to the object returned by the right hand side.<br>
You can change <code>a</code> inplace by doing explicit inplace change: <code>a.copy_(torch.max(a, b))</code> for example.</p>
Gradient of (test) Loss wrt (training) Input

username: vmk
cooked: <p>I’m trying to implement <a href="https://github.com/SsnL/dataset-distillation" rel="noopener nofollow ugc">Dataset Distillation</a> using basic optimizers and autograd. A crucial part of the algorithm is training the network on a small dataset, evaluating the loss of this network on <em>another</em> dataset, and calculating the gradient of this final loss wrt the training data</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad7c7aa36ca4bb45bc1355860e32def86818ca5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad7c7aa36ca4bb45bc1355860e32def86818ca5e" title="algo_snippet"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad7c7aa36ca4bb45bc1355860e32def86818ca5e_2_690x149.png" alt="algo_snippet" data-base62-sha1="oKJirJ2OWFrSHop1JmEwCGFyUUK" width="690" height="149" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad7c7aa36ca4bb45bc1355860e32def86818ca5e_2_690x149.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad7c7aa36ca4bb45bc1355860e32def86818ca5e_2_1035x223.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad7c7aa36ca4bb45bc1355860e32def86818ca5e_2_1380x298.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">algo_snippet</span><span class="informations">1908×414 39.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I try to directly implement this, by using an optimizer to update weights and <code>torch.nn.loss</code> for loss, autograd gives the following error:</p>
<pre><code class="lang-auto">One of the differentiated Tensors appears to not have been used in the graph
</code></pre>
<p>How can this be implemented using basic autograd and optimisers?</p>

username: albanD
cooked: <p>Hi,</p>
<p>All the optimizers in torch.optim do <strong>not</strong> perform their updates in a differentiable manner.<br>
Similarly, all the layers in <code>nn</code> contain <code>nn.Parameter()</code> that are leaf Tensors. Meaning they cannot have gradient history and so you cannot propagate gradients through their update.</p>
<p>I would recommend checking the <a href="https://github.com/facebookresearch/higher">higher</a> library from FAIR that does this though.</p>
Using `backward()` of custom autograd function with loss.backward()

username: Vigneswaran
cooked: <p>Hi community, I wanted to create a custom activation function which requires custom autograd function (something similar to <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html#sphx-glr-beginner-examples-autograd-two-layer-net-custom-function-py" rel="nofollow noopener">MyRelu example here</a>. When I apply it,  the <code>forward()</code> is called and its definition is used to compute activation but when I use <code>loss.backward()</code>, the <code>backward()</code> defined is not called. Even in the <code>MyRelu</code> example (attached above) doesn’t seem to utilize the <code>backward()</code> function.<br>
I’m not sure which part I’m understanding wrongly. Any insights and corrections would be much appreciated. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>How did you check, if the <code>backward</code> was called or not?<br>
If I add a simple print statement into the <code>backward</code> method, it gets printed once <code>loss.backward()</code> is called.</p>

username: albanD
cooked: <p>Note that if your forward is not doing anything (returns the input or a view of the input), this might happen if you do inplace changes on the output.</p>
<p>Otherwise, I would follow <a class="mention" href="/u/ptrblck">@ptrblck</a> advice and double check how you test this.</p>
Folding 2D-tensor back to 1D

username: ygabuev
cooked: <p>I have a 1D signal, which I subject to the following operations:</p>
<ol>
<li>Split into overlapping frames. I can do that with <code>torch.Tensor.unfold</code> function.</li>
<li>Work with each frame individually.</li>
<li>Combine the result back into 1D. The originally overlapping regions need to be summed up.</li>
</ol>
<p>How can i perform the 3rd step efficiently? There is a class <code>torch.nn.Fold</code>, but it only works for image-like tensors with a channel dimension.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think we support this indeed. You can maybe create dummy dimensions of size 1. To make it into a image-like Tensor?</p>
Select column based on binary 2D tensor

username: ElleryL
cooked: <p>consider I have a matrix <code>x = torch.randn(2,5)</code></p>
<p>I have a binary matrix <code>b = torch.Tensor([[1,0,1,0,1],[0,1,1,1,0]])</code> # assume that <code>b.sum(-1)</code> are equal across <code>dim=0</code>; namely, each row has same number of one’s</p>
<p>I’m wondering if there is an efficient way such that it returns  a matrix with shape <code>NxD</code> where <code>N</code> is number of rows in <code>b</code> and <code>D</code> is number of ones in each row of <code>b</code>; since we assume that each row of b has same number of ones so this may not raise error.</p>
<p>The return matrix, should have its first row<code>x[0,x[b[0,0]]],x[0,x[b[0,2]]],x[0,x[b[0,4]]]</code> and second row `x[1,x[b[1,2]]],x[1,x[b[1,2]]],x[1,x[b[1,3]]]’</p>
<p>Note that <code>x</code> is output of a neural network but the returned new matrix will be in the loss function so I still need to backprogate. Any ideas ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you are sure they contain the same number of non zero elements D, you can do <code>x[b].view(D, N)</code>. If you want an <code>[N, D]</code> result, you can transpose that with <code>.t()</code>.</p>
Segmentation Fault on CPU

username: mohsenkiskani
cooked: <p>I am using PyTorch version 1.4.0 on Amazon EC2 and I am running the following simple command:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; num = 2133; import torch; A = torch.ones(10, num);  layer1 = torch.nn.Linear(num, 6); B = layer1(A); B.size()
torch.Size([10, 6])
</code></pre>
<p>which goes through successfully. However, if I increase the parameter <code>num</code> by one, I get a segmentation fault error and python terminates.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; num = 2134; import torch; A = torch.ones(10, num);  layer1 = torch.nn.Linear(num, 6); B = layer1(A); B.size()
Segmentation fault
[ec2-user@my_instance my_dir]$ 
</code></pre>
<p>I think that this happens because of a memory issue but the tensors I am using are not super large. These are memory specs of my system:</p>
<pre><code class="lang-auto">[ec2-user@my_instance my_dir]$ ipcs -l

------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767
</code></pre>
<p>Any idea how to avoid this error?</p>

username: crowsonkb
cooked: <p>Running the line starting with <code>num = 2134</code> results in a Python process that uses 63.7 MB of memory on my system. I think it is unlikely to be an out of memory error. I don’t know what the problem could be though.</p>

username: Yaroslav_Bulatov
cooked: <p>backtrace can help isolate the problem,</p>
<pre><code class="lang-auto">sudo service apport start
ulimit -Sc unlimited
&lt;run the crashing script which dumps "core" into current directory&gt;
gdb python
core core
bt
</code></pre>
How to avoid rounding in pytorch

username: Saharkakavand
cooked: <p>I want to implement L2 norm in pytorch, and compare the result with np.linalg.norm. they have different result, how can I fix it?</p>
<pre><code class="lang-auto">num = np.linalg.norm(
                    (extracted_feat[str(key)].cpu().detach().numpy() - extracted_feats_aug[str(key)].cpu().detach().numpy()) / np.mean(extracted_feat[str(key)].cpu().detach().numpy()))```
 num = 13982.017 (float32)

`torch.sqrt(torch.sum(((extracted_feat[str(key)] - extracted_feats_aug[str(key)]) / torch.mean(extracted_feat[str(key)]))**2))`
valu: tensor(13989.3135, device='cuda:0')

</code></pre>
<pre><code class="lang-auto">(((extracted_feat[str(key)] - extracted_feats_aug[str(key)]) / torch.mean((extracted_feat[str(key)])).norm(p=2)
value:  tensor(13989.3135, device='cuda:0')
</code></pre>
<pre><code class="lang-auto"></code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>How large are these Tensors?<br>
Could you give a small code sample that we can run locally that reproduces the issue?</p>

username: Saharkakavand
cooked: <p>Thank you for your response.<br>
extracted_feat[str(key)]: torch.Size([1, 64, 600, 1200])<br>
extracted_feat_aug[str(key)]: torch.Size([1, 64, 600, 1200])<br>
what I am trying to do is subtract the activations of relu layers for clean image and augmented image then  divide it by the mean value of clean activations of the same layer, then compute the L2 norm of it.</p>

username: albanD
cooked: <p>Is the difference between <code>13982.017</code> and <code>13989.3135</code> ?<br>
For an accumulation of 46 million entries, I guess this is expected due to floating point imprecisions.<br>
If you want more precision, you can use <code>.double()</code> to get double precision numbers for which the accuracy is greater (from 6 significant digits to 12).</p>
Check if model is wrapped in nn.DataParallel

username: Ilkin
cooked: <p>Hi,</p>
<p>I wondered if there is an efficient way to check if a model is wrapped in nn.DataParallel. I get a lot errors due to DataParallel objects being wrapped in module object and wondered if there is a more natural way of checking data parallelism than len(list(Model.named_children())) == 1.</p>

username: alex.veuthey
cooked: <p>I am using something like this for getting the model’s state dict (for saving):</p>
<pre><code class="lang-python">try:
    model_state_dict = model.module.state_dict()
except AttributeError:
    model_state_dict = model.state_dict()
</code></pre>
<p>It essentially checks if the model has an attribute <code>module</code>, which is created by <code>nn.DataParallel</code>. I don’t know if there is a cleaner way introduced in more recent versions.</p>
<p>P.S: you have written <code>DataLoader</code> in your post title, you might attract more readers / helpers by correcting that <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: Ilkin
cooked: <p>I thought of doing something similar as well, but in that case I will still have to repeat all lines of code following <code>try</code> in <code>except</code> as well which I thought was a bad practice to follow in general. But still thanks for your insight.</p>

username: SimonW
cooked: <p><code>isinstance(m, nn.DataParallel)</code></p>

username: Ilkin
cooked: <p>This is ingenious, thank you!</p>
Torch.sparse to find inverse of a block-diagonal matrix?

username: kaiseryet
cooked: <p>Hey guys, so I am trying to find the inverse of a block diagonal matrix, namely, torch.inv</p>
<p>The issue is that, invert a large block diagonal matrix is not memory and computational efficient if inverting as an ordinary matrix…</p>
<p>So the right idea is to invert each diagonal block and put it back. Can torch.sparse do it? Does it have a future plan to implement this?</p>
<p>p.s. my block diagonal has each block of different size, so reshape and invert is not the right way to do it</p>
Are torch.stack and cat inplace operations?

username: veeresh_d
cooked: <p>Hi all,<br>
just wondering, are <code>torch.stack, torch.cat</code> inplace operations?  If we use the same variable name on both sides of the statement.</p>
<p>example .</p>
<p><code>torch_list = [torch.randn(2, 2), torch.randn(2, 2), torch.randn(2, 2)] torch_list = torch.stack(torch_list, dim=0)</code></p>
<blockquote>
<p>print(torch_list.shape) #  torch.Size([3, 2, 2])</p>
</blockquote>
<p>Thanks in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>They are not inplace as we do not support a Tensor backed by multiple small memory storage.</p>
<p>When you do <code>some_name = </code> in python. It just assigns the returned object to that name.</p>
Validation loss increasing in pytorch but not in keras

username: Sharut_Gupta
cooked: <p>I have been working on pretrained resnet50 in Keras. When I wrote the same code in PyTorch (same hyperparameters and layers), the validation loss keeps increasing or decreasing by a very small amount. The training loss curve decreases pretty well but something fishy is happening in Keras.<br>
Any help would be appreciated.<br>
Here is the link to both of my codes</p>
<p><a href="https://drive.google.com/drive/folders/1LpUpdhlU6RLsuIgzSYxFVf0g2pv6USZc?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/drive/folders/1LpUpdhlU6RLsuIgzSYxFVf0g2pv6USZc?usp=sharing</a></p>
Use FastText Embedding model on PyTorch

username: malioboro
cooked: <p>As explained in <a href="https://pytorch.org/docs/stable/nn.html#embedding" rel="noopener nofollow ugc">the documentation of Embedding Layer</a>:</p>
<blockquote>
<p>Embedding Layer is “a simple lookup table”.</p>
</blockquote>
<p>I think it means I can’t use it to handle OOV. But as I have a FastText model for my language, what’s the best way to use my FastText model together with PyTorch?</p>

username: Julio_Marco_A_Silva
cooked: <p>I know this answer comes “a bit” late, but…</p>
<p>An embedding layer turns known words into embedding vectors, and yes: it is just a lookup table that maps words to their vectors (although those vectors are trained, not just assigned).</p>
<p>IMHO, the best way, if you are worried about OOV words, is to train your embedding layer with texts from your domain, using a corpus as large and varied as you can get. In this way you will minimize the impact of OOV words, as the most relevant words that give meaning to the sentences in your domain will be well covered.</p>

username: sai_m
cooked: <p>As suggested by <a class="mention" href="/u/julio_marco_a_silva">@Julio_Marco_A_Silva</a>, best way would be to train on custom data set. If we still face OOV, one way to initialize OOV is using unk_init = torch.Tensor.normal_, while loading pre trained vectors. so that, pytorch will initialize unknown words via Gaussian distribution and this can be applied to train and test sets</p>
Implementing dropconnect for conv2d and transposeconv2d

username: Swaroop_Bhandary
cooked: <p>I am trying to code dropconnect for Conv2D and transposeconv2D layer. Followed the tutorial in <a href="https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/nn/weight_drop.html" rel="nofollow noopener">https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/nn/weight_drop.html</a> to create it.</p>
<pre><code class="lang-auto">
def _weight_drop(module, weights, dropout):
    for name_w in weights:
        w = getattr(module, name_w)
        del module._parameters[name_w]
        module.register_parameter(name_w + '_raw', Parameter(w))
    original_module_forward = module.forward

    def forward(*args, **kwargs):
        for name_w in weights:
            raw_w = getattr(module, name_w + '_raw')
            w = torch.nn.functional.dropout(raw_w, p=dropout, training=module.training)
            setattr(module, name_w, w)
        return original_module_forward(*args, **kwargs)
    setattr(module, 'forward', forward)
        
class WeightDropConv2d(torch.nn.Conv2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)
        
class WeightDropConvTranspose2d(torch.nn.ConvTranspose2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)
</code></pre>
<p>torch.version.cuda: 1.1.0<br>
torch.<strong>version</strong>: 9.0.176</p>
<p>I get the following error in the 2nd epoch:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "dropconnect.py", line 110, in &lt;module&gt;
    out = model(image)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 73, in forward
    out = self.c1(x)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 34, in forward
    setattr(module, name_w, w)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 558, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>
<p>This error occurs in the second epoch when I switch from .eval() to .train(). This error does not occur if I don’t call .eval()</p>
<p>Any suggestions on why this error is occurring or how to implement dropconnect in a better manner?</p>
New Convolutional Layer

username: lazlazari
cooked: <p>Hi</p>
<p>Considering a linear convolutional layer we compute a result given by y = w^{T}x +b .</p>
<p>Considering a quadratic convolutional layer we can compute the value using <img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8e7d4d0b0345451adbdcdcdd168579e6f26537f.gif" alt="CodeCogsEqn (1)" data-base62-sha1="zvV6KQqo6HqXGaRiCgbquIvdE9F" width="130" height="20"> . More specifically, given an input image X and a convolutional layer with kernel of size k x k, we decompose X as a sequense of k x k patches. Then, we vectorize each patch as a vector of length K^2. For each patch x in input image we would like to compute the aforementioned quadratic equation value where w and b are the same as those appearing in a linear convolution and Q is a k^2 x k^2 matrix which contributes a quadratic non-linearity.</p>
<p>How can I extend pytorch in order to create the quadratic convolutional layer? Any ideas?</p>
Profile Layerwise Inference Latency

username: surya00060
cooked: <p>Hello All,</p>
<p>I want to profile the layerwise time taken during inference on CPU. Is there any way to measure the time between the starting and ending point. Let us consider MobileNet, which has many Inverted Residual Blocks. I need to get the inference latency for each Block.</p>
<p>Thanks in Advance for any pointers/help.</p>
Understanding pytorch

username: samra-irshad
cooked: <p>Hi, I am new to pytorch and trying to understand the pytorch tutorial. <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py</a></p>
<p>I dont understand how an object’s variable can accept arguments or act as a function as given in x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) where ‘self.conv1(x)’ is acting as a function?</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    # 1 input image channel, 6 output channels, 5x5 square convolution
    # kernel
    self.conv1 = nn.Conv2d(1, 6, 5)
    self.conv2 = nn.Conv2d(6, 16, 5)
    # an affine operation: y = Wx + b
    self.fc1 = nn.Linear(16 * 5 * 5, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    # Max pooling over a (2, 2) window
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    # If the size is a square you can only specify a single number
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

def num_flat_features(self, x):
    size = x.size()[1:]  # all dimensions except the batch dimension
    num_features = 1
    for s in size:
        num_features *= s
    return num_features
</code></pre>
<p>net = Net()<br>
print(net)</p>

username: ptrblck
cooked: <p>Each module derives from <code>nn.Module</code>, while implements the <a href="https://github.com/pytorch/pytorch/blob/5e2311033ebc4bb6009f70db68e4d63ba73a1633/torch/nn/modules/module.py#L530"><code>__call__</code></a> function. This function is called, when you can directly an object e.g. as <code>model(x)</code>.<br>
You can see in the linked source code that some maintenance will be performed (e.g. hooks will be called) and finally <code>forward</code> will be executed, which defines the actual forward pass of this module.</p>
Cuda Memory Full during Feature extraction

username: srinivasan.vasu89
cooked: <p>I am working on video and getting frames and using the pretrained model, I tried predict features. I am trying to use GPU and end up having OOM error. Algo looks like</p>
<p>for loop --&gt; for each video<br>
—with torch.no_grad()<br>
-----process video n fetch frames<br>
-----batch frames<br>
-----ft = model(frames)<br>
-----x = torch.cat(ft)</p>
<p>However, memory is full after processing few videos. Should I use<br>
-----x = torch.cat(ft).cpu()<br>
-----del ft, model</p>
<p>I really don’t understand how things are working with tensors in Pytorch. Please let me understand the concept. Thanks</p>
How to declare and train RNN/LSTM in skorch?

username: Nidhal_Baccouri
cooked: <p>the skorch library is amazing. it’s really very easy to use but I’m having troubles implementing an LSTM model with it. Since it abstract the training loop and all, I don’t know what should be returned in the forward function of the nn model. My task is regression, the problem is I found many results on the internet for word embeddings and something like that, I don’t need that, my task is simpler than that I only want to do regression like predicting house prices for example. here is my LSTM declaration</p>
<pre><code class="lang-auto"> class LstmNetwork(nn.Module):

    def __init__(self, n_features, n_lstm_hidden=16, n_fc_hidden=64, n_out=2, num_layers=1, bi=False, dropout=0.0):
        super().__init__()

        self.n_features = n_features
        self.n_lstm_hidden = n_lstm_hidden
        self.n_fc_hidden = n_fc_hidden
        self.n_out = n_out
        self.num_layers = num_layers
        self.bi = bi
        self.dropout = dropout
        self.hidden_cell = self.init_hidden()

        self.lstm = nn.LSTM(
            input_size=self.n_features,
            hidden_size=self.n_lstm_hidden,
            num_layers=self.num_layers,
            bidirectional=self.bi,
            dropout=self.dropout)

        self.fc = nn.Linear(
            in_features=n_fc_hidden,
            out_features=n_out
        )

    def init_hidden(self):

        return (torch.zeros(1, 1, self.n_lstm_hidden),
                torch.zeros(1, 1, self.n_lstm_hidden))

    def forward(self, x):
        # I found this example on internet, but there is no initialisation for hidden states here, I tried to 
       # initialize hidden states as stated above but it throws an error when I run it on GPU
     # also I thought that we should also return the hidden states in the forward function in pytorch but 
    # if I do that here I get an Error.
    # I'm not sure if my declaration is correct, can someone help?
        out, hn = self.lstm(x)
        out = self.fc(out)
        return out
</code></pre>
<p>please help me out with this, I’m struggling for two weeks now</p>
Different speedups with multi GPU training on two machines with same code

username: deepak242424
cooked: <p>Hi All,</p>
<p>I am trying to run one code on multi-GPU setting using DataParalllel. I have access to two machines having 2 GPUs each.</p>
<p>Machine 1: Two 1080Ti<br>
Machine 2: Two 2080Ti</p>
<p>I am running the same code with same dataset in multi GPU setting on each machine separately. Code is working fine on Machine 2 and giving the expected speedup. However on Machine 1 code starts fine but after few training steps its speed drops drastically. On both the machines I am working with Pytorch Version 1.1.0.</p>

username: ptrblck
cooked: <p>Are these machines identical besides using different GPUs?<br>
I.e. are they using approx. the same SSDs, CPUs etc?</p>
<p>Also, are you using the same CUDA and cudnn versions?</p>
<p>The difference might not directly come from the GPUs but possibly other bottlenecks, e.g. data loading and processing.</p>
Maskrcnn_benchmark train

username: lyh
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b14ccaf059c6e962f7726bddd50fa66b8037cfd2.png" alt="1580213391(1)" data-base62-sha1="pit2tvxv18WhMg18KekwuKB7I9s" width="422" height="140"></p>
<p>When I train " python setup.py build develop ".The machine returns :</p>
<p>error: command ‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe’ failed with exit status 2</p>
<p>What should i do for this problem?Thanks a lot.</p>
<p>The machine :<br>
Window10-Home<br>
python-3.6<br>
torch-1.4<br>
pycharm</p>

username: ptrblck
cooked: <p>Could you post the complete stack trace please?<br>
Also, make sure to wrap the log in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Build speed_benchmark_torch for x86 instead of android?

username: solvingPuzzles
cooked: <p>In <a href="https://discuss.pytorch.org/t/speed-benchmarking-on-android/67340/5">this thread</a>, I learned how to compile for Android the file <code>pytorch/binaries/speed_benchmark_torch.cc</code>, which lets you benchmark the speed of torchscript models. However, for debugging purposes, I’d like to build this file for my desktop computer’s Intel CPU, so that I can better debug some failures I am getting.</p>
<p>Does anyone know how to compile  <code>speed_benchmark_torch.cc</code> for desktop CPU?</p>

username: solvingPuzzles
cooked: <p>I figured out a way to do it. I do a fresh build of pytorch as follows:</p>
<pre><code class="lang-auto">cd pytorch
export BUILD_BINARY=1
python setup.py build_ext 
</code></pre>
<p>An x86 CPU compatible (rather than Android or iOS compatible) binary appears here:<br>
<code>pytorch/build/bin/speed_benchmark_torch</code></p>
<p>And, I was able to run it successfully on my desktop computer’s CPU.</p>
How to assign to specific columns in each row?

username: f3ba
cooked: <p>Suppose I have a list of indices: <code>ind = torch.tensor([0, 2, 1, 3])</code>, and a 4x4 tensor <code>A</code>.</p>
<p>I need to assign some values to the items <code>A[i, ind[i]</code>, for <code>i=1,4</code>. For example:</p>
<pre><code class="lang-auto">for i in range(4):
    A[i, ind[i]] = r[i]
</code></pre>
<p>where <code>r</code> contains the target values in this example.</p>
<p>Given <code>A</code>, <code>ind</code>, <code>r</code>, Is there a way to vectorize this operation?</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python"># Setup
ind = torch.tensor([0, 2, 1, 3])
A = torch.zeros(4, 4)
r = torch.randn(4)

# Your code
for i in range(4):
    A[i, ind[i]] = r[i]

# Alternative
B = torch.zeros(4, 4)
B[torch.arange(4), ind] = r
print((B == A).all())
&gt; tensor(True)
</code></pre>
Questions about image classification based on feature vector

username: E4CE
cooked: <p>Hello, I am trying to implement image classification based on feature vector. I followed this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">tutorial </a> and added</p>
<p>model_conv = models.resnet18(pretrained=True)<br>
feature_extractor = torch.nn.Sequential(*list(model_conv.children())[:-1])<br>
x = torch.randn([1,3,224,224])<br>
output1 = feature_extractor(x)<br>
print(output1)</p>
<p>and got a result of 512 output. My questions are</p>
<p>“Are there outputs are features of the trained model?”<br>
if I want to extract features of one image should I follow the same procedure?<br>
Thank you</p>

username: G.M
cooked: <p>The output of the model should be feature extracted by the model.</p>

username: E4CE
cooked: <p>Thank you for the reply. If I understood correctly, I trained my model with dataset “A” and extracted feature of this model which means feature of dataset “A”. Right?</p>

username: G.M
cooked: <p>Yes, you are correct.</p>