akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom torch.autograd.Function Inherited Class

username: hashbrown
cooked: <p>I want to implement my own quantized and clipped ReLU. This is how I implemented it:</p>
<pre><code class="lang-auto">class _quantAct(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, clip_low=0., clip=6., bits=8, inplace=False):
        if inplace:
            ctx.mark_dirty(input)
            output = input
        else:
            output = input.clone()
        output[output&lt;clip_low]=clip_low
        output[output&gt;clip]=clip
        output = output.div(clip).mul((2**bits)-1).round().div((2**bits)-1).mul(clip)

        ctx.save_for_backward(output.eq(clip_low)+output.eq(clip))
        return output

    @staticmethod
    def backward(ctx, grad_output):
        # saved tensors - tuple of tensors with one element
        mask, = ctx.saved_tensors
        grad_input = grad_output.masked_fill(mask,0)
        return grad_input, None, None, None, None

class quantReLU(nn.ReLU):
    def __init__(self, clip=6., bits=8, inplace=False):
        super(quantReLU, self).__init__()
        self.clip = clip
        self.bits = bits
        self.inplace = inplace

    def forward(self, inputs):
        return _quantAct().apply(inputs, 0, self.clip, self.bits, self.inplace)
</code></pre>
<p>How many grads do I have to return from the static backward method of my torch.autograd.Function inherited class? Why does it expect me to return 5 of them?</p>
<p>Appreciate your inputs, thanks!</p>

username: TomB
cooked: <p>You need to return as many values from backwards as were passed to to forward, this includes any non-tensor arguments (like<code>clip_low</code> etc). For non-Tensor arguments that don’t have an input gradient you can return <code>None</code> but still need to return a value. So, as there were 5 inputs to forward, you need 5 outputs from backward. Technically, I gather if the user only passed <code>input</code> and left the others as default you could just return one gradient, but then you’d have to track that, instead you can also return extra <code>None</code>s which will be ignored.<br>
Explained in the docs on <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">extending autograd</a>.</p>

username: hashbrown
cooked: <p>Thanks <a class="mention" href="/u/tomb">@TomB</a>, this answers my question. Appreciate it.</p>
Converting from pytorch to coreml

username: Stev3
cooked: <p>Hi All,</p>
<p>I’m struggling to convert an image classifier trained with fast.ai to coreml for iphone and hoping to find someone who can help. Full detail is listed in my stack overflow question here: <a href="https://stackoverflow.com/questions/58276161/problem-when-converting-pytorch-image-classifier-to-mlmodel-returns-same-softma" rel="nofollow noopener">https://stackoverflow.com/questions/58276161/problem-when-converting-pytorch-image-classifier-to-mlmodel-returns-same-softma</a></p>
<p>Any suggestions or ideas would be awesome!</p>
<p>Thanks,<br>
Steve</p>
Pixel wise binary classification - which loss function to use?

username: WeiQin_Chuah
cooked: <p>I would like my model to predict say K number of planes from an RGB image. After outputting a tensor (final layer is softmax) with size [B K+1 H W], with the +1 being the non-planar mask, I summed up all the planar masks. Therefore, after the summing operation, I will have [B 2 H W]. Also, I have ground-truth labels with 1 indicating planar regions and 0 indicating the nonplanar region.</p>
<p>Just wondering which loss function will be the best for this application?</p>
<p>Also, I’ve tried using cross entropy however the loss becomes nan after 2 iterations.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I’m not sure to fully understand your use case, however it seems you are dealing with a two-class classification (2 output channels and two class indices).<br>
If your output has the shape <code>[B, 2, H, W]</code>, your target should have the shape <code>[B, H, W]</code> and contain the class indices in the range <code>[0, 1]</code> to be able to use <code>nn.CrossEntropyLoss</code>.<br>
Also, make sure to pass raw logits to this criterion, as internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied. If you are using a <code>softmax</code> activation as the last non-linearity, could you please remove it?</p>
<p>Could you check your inputs for invalid values (Inf or NaN)?<br>
If you can’t find any invalid values, could you lower the learning rate a bit and check, if you get a NaN loss again?</p>

username: WeiQin_Chuah
cooked: <p>After some digging in PyTorch documentation, I found BCEloss which is cross entropy loss for binary classification. Shouldn’t I use that instead? I can repeat my target from <code>[B, H, W]</code> to <code>[B, 2, H, W]</code> so that it matches the shape of my output.<br>
And by using BCEloss, I will not have to remove the last layer of cross entropy loss.</p>
<p>I tried using cross entropy loss as you suggested but I am getting another error:</p>
<p><code>RuntimeError: copy_if failed to synchronize: device-side assert triggered</code></p>

username: ptrblck
cooked: <p><code>nn.BCE(WithLogits)Loss</code> would be the alternative and your output and target should then both have the shape of <code>[B, 1, H, W]</code>.<br>
I would recommend to use <code>nn.BCEWithLogitsLoss</code> for numerical stability and still remove the last activation layer (<code>sigmoid</code> for <code>nn.BCELoss</code>).</p>
Getting dimension out of range while normalizing weight

username: Surekha_Gaikwad
cooked: <p>I am trying to normalize the weight that I get from embedding layer using F.normalize function but getting error as “IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)”.<br>
Not sure why.</p>
<pre><code class="lang-auto">def forward(self, imgs):

        ### complete the forward path --------------------

        cls_scores = []
        ## YOUR CODE HERE
        for i in range(len(self.classes)):
            v = self.backbone(imgs)
            
            class_out = random.choice(self.classes)
            class_out = self.classes.index(class_out)
            class_out = torch.tensor(class_out)
            
            wt = self.embeddings(class_out) # not sure what should be the input
        
            wt_normalized = F.normalize(wt, p=2, dim=1)        
            self.dc.weight = nn.Parameter(wt_normalized)            
            v_out = self.dc(v)
            out = nn.Upsample(size=(8, 8), mode='bilinear')(v_out)
            
            out = nn.Upsample(size=(64, 64), mode='bilinear')(v_out)
            score = self.mlp(out)
            cls_scores.append(score)
        
        ### ----------------------------------------------

        return cls_scores # Dim: [batch_size, 10]
</code></pre>

username: ptrblck
cooked: <p>Based on the error message, if seems <code>wt</code> has only a single dimension.<br>
If you want to normalize using <code>dim=1</code>, make sure your tensor has at least two valid dims:</p>
<pre><code class="lang-python">F.normalize(torch.randn(10, 10), p=2, dim=1) # works
F.normalize(torch.randn(10), p=2, dim=1) # throws your error
</code></pre>
How to freeze a single weight of a conv2d layer

username: XW324
cooked: <p>For example, we have a model only containing a single conv2d layer(1 feature to 1 feature and kernel size of 3). How do I fixed the center weight of this conv layer?</p>
<p>Last time I tried, I used model.conv2d.weight.grad[index for center weight]=0. It does set the weight.grad to zero, but after calling  optimizer.step(), the center weight is still updated. I called loss.backward() before mannually setting center weight grad to zero.</p>
<p>So how do I do things like this in pytorch?</p>

username: ptrblck
cooked: <p>Zeroing out the gradients should work for optimizers without running estimates.<br>
If you are using an optimizer with internal states (e.g. <code>Adam</code>), this approach will still work, if you zero out the gradients from the beginning:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 1 ,3)
weight_reference = conv.weight.clone()
optimizer = torch.optim.Adam(conv.parameters(), lr=1.)

x = torch.randn(1, 1, 10, 10)
out = conv(x)
out.mean().backward()

conv.weight.grad[:, :, 1, 1] = 0.
optimizer.step()

print(weight_reference == conv.weight)
&gt; tensor([[[[False, False, False],
          [False,  True, False],
          [False, False, False]]]])
</code></pre>
<p>However, after a valid update was performed, the running averages might update the parameter further.</p>
<pre><code class="lang-auto"></code></pre>

username: XW324
cooked: <p>Very clean answer. Thanks.</p>

username: Alpha
cooked: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>thank you very much for your answer.</p>
<p>In my case, I have more than one conv layers, and for almost all of these layers, I have to fix part of the parameters of the layer. Is there a convenient way?</p>
<p>If I use <code>conv.weight.grad[:, :, 1, 1] = 0.</code>  I have to do this for every layers, which is not elegant and convenient.</p>

username: ptrblck
cooked: <p>Would using a for loop and the layer names be a solution?<br>
You would have to somehow access each <code>weight</code> parameter of all layers you would like to set to zero.</p>
Multiclass Semantic Segmentation (UNet++)

username: munna
cooked: <p>Hi!<br>
I am trying to use Unet++ for semantic segmentation.<br>
input shape is:  [N, H, W] i.e: [16, 256, 256]<br>
target shape is: [N, 1, H, W] i.e:  [16, 1, 256, 256]<br>
and output shape is: [N, 1, H, W] i.e:  [16, 1, 256, 256]<br>
loss function is: BCEWithLogitsLoss<br>
I have trained my model using <a href="https://github.com/4uiiurz1/pytorch-nested-unet/blob/87ac6c553d4824cdae6fa2944b28c9c2fef75d9f/archs.py#L73" rel="nofollow noopener">this model architecture</a>.  And it’s working fine for 2 classes.</p>
<p>Now I am trying to use this <strong>trained</strong> model for <strong>3 classes</strong>. I have converted my mask pixel value to corresponding class values already. Now I don’t understand how could I use this same model for 3 classes. As in the model architecture, the out_channels = 1 which is fixed.</p>
<p>I have tried with<br>
class_number = 3<br>
model.final1.out_channels = class_number<br>
model.final2.out_channels = class_number<br>
model.final3.out_channels = class_number<br>
model.final4.out_channels = class_number<br>
But seems like it  stills generate the output to [16, 1, 256, 256] which I hoped to get [16, 3, 256, 256]</p>
<p>I am new in PyTorch. So your suggestions would highly be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Since the last convolution layer has only a single channel, you won’t be able to use it for 3 classes.<br>
Instead you would have to assign a new <code>nn.Conv2d</code> layer with <code>out_channels=3</code> to <code>model.final</code> and retrain this layer for your new use case.</p>

username: munna
cooked: <p>Thank you so much. It worked <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>What I have done is:</p>
<pre><code class="lang-auto">class AddnewConvLayer(nn.Module):
    def __init__(self):
        super().__init__()
        self.convolution_layer = nn.Conv2d(32, 3, kernel_size=1)

    def forward(self, x):
        return self.convolution_layer(x)

model.final1 = AddnewConvLayer()
model.final2 = AddnewConvLayer()
model.final3 = AddnewConvLayer()
model.final4 = AddnewConvLayer()
</code></pre>
a.sub_(lr * a.grad) ??

username: Luis_Anaya
cooked: <p>as I understand it, this subtracts the coefficients by (learning rate * gradient)…</p>
<p>But why is it necessary to subtract?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: akshayk07
cooked: <p>If anyone has the same question, it’s answered <a href="https://stackoverflow.com/a/58266479/7615877" rel="nofollow noopener">here</a></p>
No module named hypothesis error while running test

username: PyExtreme
cooked: <p>I am  getting</p>
<blockquote>
<p>no module named hypothesis</p>
</blockquote>
<p>error when I ran the command</p>
<blockquote>
<p>python test/ran_test.py</p>
</blockquote>
<blockquote>
<p>Running test_expecttest … [2019-10-07 05:34:41.949386]<br>
Traceback (most recent call last):<br>
File “test_expecttest.py”, line 8, in <br>
import hypothesis<br>
ModuleNotFoundError: No module named ‘hypothesis’<br>
Traceback (most recent call last):<br>
File “test/run_test.py”, line 458, in <br>
main()<br>
File “test/run_test.py”, line 450, in main<br>
raise RuntimeError(message)<br>
RuntimeError: test_expecttest failed!</p>
</blockquote>

username: pbelevich
cooked: <p>If you use conda, run <code>conda install hypothesis</code>,for pip I believe it should be <code>pip install hypothesis</code></p>
ONNX export failed:Couldn't export Python operator Im2Col

username: saurav_agarwal
cooked: <p>Hi,<br>
I am trying to convert a pytorch model to Onnx format. I am using F.unfold in between.<br>
But while converting I am getting below error.</p>
<p>import torch.nn as nn<br>
import torch.nn.init as init<br>
import torch.nn.functional as F<br>
import torch</p>
<p>class Test_model(torch.nn.Module):<br>
def forward(self, x, y):<br>
x = torch.rand(1, 64, 157, 317)<br>
#x=F.unfold(x, kernel_size=3, padding=1)<br>
unfold_p0=nn.Unfold(kernel_size=3,padding=0, stride=4)<br>
x=unfold_p0(x)<br>
return x</p>
<p>model = Test_model()<br>
dummy_input = torch.ones(2, 3, dtype=torch.long)<br>
loop_count = torch.tensor(5, dtype=torch.long)</p>
<p>torch.onnx.export(model, (dummy_input, loop_count), ‘loop.onnx’, verbose=True)</p>
<p>proto, export_map = graph._export_onnx(params, _onnx_opset_version, defer_weight_export, operator_export_type)<br>
RuntimeError: ONNX export failed: Couldn’t export Python operator Im2Col</p>
<p>I am using torch version: 1.2.0</p>

username: spt
cooked: <p>torch.unfold is not supported in the PyTorch to ONNX exporter yet.  There’s an open PR for initial export support.<br>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/24970" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/neginraoof" target="_blank" rel="nofollow noopener">
    <img alt="neginraoof" src="https://avatars1.githubusercontent.com/u/16928382?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/pull/24970" target="_blank" rel="nofollow noopener">[ONNX] Unfold export</a>
</h4>

<div class="date">
  by <a href="https://github.com/neginraoof" target="_blank" rel="nofollow noopener">neginraoof</a>
  on <a href="https://github.com/pytorch/pytorch/pull/24970" target="_blank" rel="nofollow noopener">07:01PM - 21 Aug 19 UTC</a>
</div>

<div class="github-commit-stats">
  <strong>17 commits</strong>
  changed <strong>4 files</strong>
  with <strong>148 additions</strong>
  and <strong>1 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Just curious - which model is this needed for?</p>

username: neginraoof
cooked: <p>The PR mentioned above provides export for torch.Tensor.Unfold op. However, I think your model uses torch.nn.Unfold export (this one internally uses Im2Col). nn.Unfold is currently in out backlog of unsupported ops. I will updated the thread once we have started a PR for this one.</p>
Selecting from tensor

username: Shahar_Hoory
cooked: <p>Hi,<br>
I have a 3d tensor sized (batch_size, S, 5).<br>
I also have a list containing a list for each batch, that contains indices.<br>
For example, if the batch size is 2, the list looks like this:<br>
[[idx0_0, idx0_1, …], [idx1_0, idx1_1, …]] (where each idx is between 0 to S).</p>
<p>I want to get a tensor that contains, for each batch, the 3rd dimension’s stuff from the original tensor, only from the indices that appear in the list of the same batch, <strong>while keeping the gradient of the original tensor.</strong></p>
<p>For example, if the original tensor is</p>
<pre><code class="lang-auto">[[[A],
  [B], 
  [C]]]
</code></pre>
<p>(batch_size = 1, S = 3, A, B, C are sized 5)</p>
<p>and the list is [[0, 2]]<br>
The result will be:</p>
<pre><code class="lang-auto">[[[A], 
  [C]]]
</code></pre>
<p>and that it’ll be possible to backpropagate through this tensor.</p>
<p>I heard about the gather function but i’m not sure if it fits here and how to use it in this case.</p>
<p>Thank you!</p>

username: el_samou_samou
cooked: <p>Hi, can you specify what you mean by “keeping the gradient of the original tensor”?</p>
<p>If it means that you just want to propagate the gradient incoming to:</p>
<pre><code class="lang-auto">[[[A], 
  [C]]]
</code></pre>
<p>we can note:</p>
<pre><code class="lang-auto">[[[grad(A)], 
  [grad(C)]]]
</code></pre>
<p>to the input of the operation in the most straight forward way you would obtain:</p>
<pre><code class="lang-auto">[[[grad(A)], 
  [None],
  [grad(C)]]]
</code></pre>
<p>Which is what would torch.gather do.</p>
<p>In case you want something else, please specify but I assume you will have to extend torch.autograd as in:<br>
<a href="https://pytorch.org/docs/stable/notes/extending.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/extending.html</a></p>
<p>Best,</p>
<p>Samuel</p>

username: Shahar_Hoory
cooked: <p>Hi and thank you for the reply.<br>
I meant that i want to know the operation needed to get the tensor with the appropriate values from the original tensor &amp; the list (as i explained), in such way that it’ll be possible to backpropagate through the result. (This computation is done while computing the loss so it has to be differential).<br>
Thank you!</p>

username: el_samou_samou
cooked: <p>Ok in that case, torch.gather will do the job.</p>
<p>I think you could also do:</p>
<pre><code class="lang-auto">results = [[[A],
  [B], 
  [C]]]
l = [0, 2]
updatd_results = results[:, l, :]
</code></pre>
How I can build my own optimizer?

username: isalirezag
cooked: <p>Is there a way to write my custom optimizer from scratch ?</p>

username: Konpat_Ta_Preechakul
cooked: <p>The code for SGD is very readable given that you know how it works: <a href="https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html</a></p>
<p>You might ignore in-place operations they tend to use, and also ignore the “param_groups” if you don’t care.</p>
How to feed multiple images in per record

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>Could anyone point me to information explaining how to feed multiple images per record?  For example, each record I have represents a property.  I have 5 - 8 images/per property and a tabular data set where each observation is one distinct property.   The target variable is a binary variable.  How can I feed in the images where multiple images are attributed to the same property?</p>

username: ptrblck
cooked: <p>Could you post a sample of your <code>records</code>?<br>
How are you reading them and in which format are they stored?</p>

username: Jordan_Howell
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>.  I don’t have the data ingested yet as I don’t know exactly how to do it (admittedly I’m learning pytorch).  The tabular record consists of the following:<br>
ID<br>
Age of Home<br>
Zip Code<br>
Home Value<br>
Number of families living in home<br>
Target</p>
<p>For each record that consists of the above variables, I have multiple photos.  I have another data frame conisting of the photo name, the ID of the observation in the tabular data.  The location on my hard drive and what I call the meta-data label (side of home, front of home, roof, etc).  Does that help?</p>

username: ptrblck
cooked: <p>I think the easiest way would be to load the data using <code>pandas</code> in your custom <code>Dataset</code> and return the features with all images in its <code>__getitem__</code>.</p>
<p>Do you want to return all images for the corresponding ID and are you always dealing with the same number of images?</p>

username: Jordan_Howell
cooked: <p>I do want to return all images but the number of images will be different every time.</p>

username: Jordan_Howell
cooked: <p>Now that I’m back at my work desk, maybe this record would help.  The following is one actual record.</p>
<p>ID: 5000686<br>
Age of Home: 34<br>
Zip Code: 49508<br>
Home Value: 200000<br>
Number of families living in home: 1<br>
Target: 1</p>
<p>Photos:<br>
5000686-136279724.jpg / meta-label = dwelling<br>
5000686-136279725.jpg / meta-label = dwelling<br>
5000686-136279726.jpg / meta-label = roof<br>
5000686-136279727.jpg / meta-label = roof<br>
5000686-136279728.jpg / meta-label = roof<br>
5000686-136279729.jpg / meta-label = Other</p>
<p>So for reach photo, the title is the ‘id-photo number.’ I have a csv with three columns that shows the location on my hard drive, full photo ID, and meta label.  I have 8 to 28 photos per record.</p>
Why the alignment score function (in seq2seq attention model) in the tutorial seems different from thoes in papers?

username: brz
cooked: <p>I am learning attention mechanism.<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a><br>
In the tutorial, the alignment score is calculated based on the decoder’s input and hidden state.<br>
However, I read several papers about attention. They used current target hidden<br>
state ht with each source hidden state hs to compute the alignment score, such as this one, entitled  Effective Approaches to Attention-based Neural Machine Translation. I do not know why.</p>
Equivalent data preprocessing methods get entirely different results

username: Hongkai_Zheng
cooked: <p>Hi. I changed the transforms code part of DCGAN tutorial(<a href="https://github.com/pytorch/tutorials/blob/master/beginner_source/dcgan_faces_tutorial.py" rel="nofollow noopener">https://github.com/pytorch/tutorials/blob/master/beginner_source/dcgan_faces_tutorial.py</a>) from</p>
<pre><code class="lang-auto">dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
</code></pre>
<p>to</p>
<pre><code class="lang-auto">dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
</code></pre>
<p>by removing the resize and Centercrop.</p>
<p>Then the results becomes entirely different. After removing the resize and centercrop, the DCGAN just cannot get good results regardless of different hyperparameters.  The discriminator’s loss drops dramatically to 0.</p>
<p>But to my knowledge, since the celeA image size is 64 by 64, removing resize(64) and Centercrop() won’t make any difference. Could anyone explain it? I really appreciate it.</p>

username: ptrblck
cooked: <p>Did you make sure all image tensors have the same width and height after removing the two mentioned transformations?<br>
Also, is this effect reproducible, i.e. how many times did you run the code?</p>

username: Hongkai_Zheng
cooked: <p>I thought celeA’s images are 64 by 64. That is the reason. Thanks</p>
Habana backend compile error

username: leejaymin
cooked: <p>Hello,</p>
<p>I was trying to build Glow with Habana backend option like below:</p>
<ul>
<li><code>cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DLLVM_DIR=/usr/local/opt/llvm@7/lib/cmake/llvm -DGLOW_WITH_HABANA=ON ..</code></li>
</ul>
<p>However, the error occurred during cmake sync.<br>
The error message is the following:</p>
<pre><code class="lang-bash">CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/Users/jeminlee/development/glow/lib/Backends/Habana/SYNAPSE_INCLUDE_DIR
   used as include directory in directory /Users/jeminlee/development/glow/lib/Backends/Habana
/Users/jeminlee/development/glow/tests/unittests/SYNAPSE_INCLUDE_DIR
   used as include directory in directory /Users/jeminlee/development/glow/tests/unittests
</code></pre>
<p>I would like to know whether there is the right way to compile Glow on Habana backend.</p>
<p>Thanks</p>
Input to NLLLoss: label out of range?

username: serenayj
cooked: <p>Hi,<br>
I have a softmax output over a list: [10, 11, 12, 13, 14, 15, 16, 17, 18], where each number represents the word token index. The softmax predicts the probability distribution as [1,9] tensor. And the ground truth label is 13.<br>
I want to compute the NLL loss between ground truth label: 13, and the prediction. Since the length of list is 9, it gave me error as:<br>
<code>RuntimeError: Assertion</code>cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes’ failed.  at …/aten/src/THNN/generic/ClassNLLCriterion.c:94`</p>
<p>I thought the NLLLoss could take ground truth label as it is, is that true?</p>

username: ptrblck
cooked: <p><code>nn.NLLLoss</code> expects the class labels to be in the range <code>[0, nb_classes-1]</code>.<br>
If you are dealing with class indices starting at e.g. 10, you should subtract the offset from your target tensor.</p>
Truly Random Sampling

username: shivam2298
cooked: <p>I am currently dealing with a large image dataset. For each epoch I am using 10000 samples and 128 batch size. in <strong>getitem</strong> I am randomly sampling images from my dataset. What can I do so that after every epoch the sampling is different from the previous epoch and not repeated?</p>

username: ptrblck
cooked: <p>If you create a <code>DataLoader</code> using the default sampler (<code>RandomSampler</code>) and specify <code>shuffle=True</code>, each batch will be randomly sampled in each epoch.<br>
Here is a small example:</p>
<pre><code class="lang-python">data = torch.randn(10, 3, 224, 224)
target = torch.arange(10)

dataset = TensorDataset(data, target)

loader = DataLoader(
    dataset, 
    batch_size=2,
    shuffle=True
)

for data, target in loader:
    print(target)

for data, target in loader:
    print(target)
</code></pre>
<p>As you can see, the two loops (representing an epoch) return different batches in each run.</p>

username: shivam2298
cooked: <p>Actually I don’t want to have a fixed dataset and then sample random batches. After every epoch, I want to construct a random dataset. Essentially what I want is that regardless of epochs, every batch randomly picks images from a given folder having millions of images.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shivam2298" data-post="3" data-topic="57574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivam2298/48/9316_2.png" class="avatar"> shivam2298:</div>
<blockquote>
<p>I want to construct a random dataset.</p>
</blockquote>
</aside>
<p>Based on your description, if sounds rather as if you would like to just load images randomly.<br>
If that’s the case, just swap by <code>Dataset</code> for <code>ImageFolder</code> and the code should work nevertheless.</p>
Differences in MSELoss calculations

username: Pawel
cooked: <p>Hello,<br>
I’m trying to understand the calculations and manually calculate this simple network.<br>
All calculations are good except for the weight.<br>
What am I doing wrong?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

weight_0 = 0.25
bias_0 = 0.68
l_rate = 0.01

input_data = torch.Tensor([[2.2], [4.0]])
target_data = torch.Tensor([[4.1], [5.1]])

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(1, 1)
        self.fc1.weight.data = torch.tensor([[weight_0]])
        self.fc1.bias.data = torch.tensor([bias_0])

    def forward(self, x):
        x = self.fc1(x)
        return x

net = Net()

loss_f = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr = l_rate)

#net.train() #one epoch
net_out = net(input_data)
loss = loss_f(net_out, target_data)
optimizer.zero_grad()
loss.backward()
optimizer.step()
net.eval()

print("loss:  ", loss.data) # 9.9666
print("weight:", net.fc1.weight.data, net.fc1.weight.grad.data) # 0.4499, -19.9940
print("bias:  ", net.fc1.bias.data, net.fc1.bias.grad.data) # 0.7429, -6.2900

print("calculations:")

in_mean = (input_data[0] + input_data[1]) / 2

out_1 = (input_data[0] * weight_0 + bias_0)
out_2 = (input_data[1] * weight_0 + bias_0)

loss_1 = (out_1 - target_data[0]) ** 2
loss_2 = (out_2 - target_data[1]) ** 2
loss_out = (loss_1 + loss_2) / 2

loss_d_1 = (out_1 - target_data[0]) * 2
loss_d_2 = (out_2 - target_data[1]) * 2
loss_d_out = (loss_d_1 + loss_d_2) / 2

weight = weight_0 - loss_d_out * l_rate * in_mean
bias = bias_0 - loss_d_out * l_rate

print("loss:", loss_out) # 9.9666
print("loss_d:", loss_d_out) # -6.2900
print("weight:", weight) # 0.4450 (should be 0.4499)
print("bias:", bias) # 0.7429

print(net.fc1.weight.data, "-", weight, "=", net.fc1.weight.data-weight.data) # 0.4499 - 0.4450 = 0.0050

</code></pre>
<p>Thanks!</p>

username: phan_phan
cooked: <p>It’s because you should have separated the computations for each element of you batch!<br>
Here, instead of:</p>
<pre><code class="lang-auto">loss_d_out = (loss_d_1 + loss_d_2) / 2
weight = weight_0 - l_rate * loss_d_out * in_mean
bias = bias_0 - l_rate * loss_d_out
</code></pre>
<p>it should rather be:</p>
<pre><code class="lang-auto">weight = weight_0 - l_rate * (loss_d_1 * in_1 + loss_d_2 * in_2) / 2
bias = bias_0 - l_rate * (loss_d_1 + loss_d_2) / 2
</code></pre>
<p>(Though it doesn’t change anything for the bias)</p>

username: Pawel
cooked: <p>It works. Many thanks!</p>
How can I achieve this efficiently?

username: linyu
cooked: <p>Hello everyone, now, I have a 2-D tensor, A = [1, 2 ; 3, 4], and a 1-D vector, B = [5, 6]. I want to achieve C which is C = [1<em>5, 2</em>5; 3<em>6, 4</em>6]. Can pytorch0.4.1 calculate this efficiently? Sure, I know that this can be achieved by A*diag(B), but diag(B) need too much memory. Looking forward to your reply.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use <code>expand</code> and element-wise multiplication:<br>
<code>C = A * B.unsqueeze(1).expand_as(A)</code>.</p>
How to evaluate a multi linear regression model with pytorch

username: Nidhal_Baccouri
cooked: <p>I started using Pytorch. I find it very Nice, I tried to implement a basic linear regression model using a simple Feed forward neural network, I trained the model and I get to observe the loss decreasing over time but I didnt figure out how to get the accuracy of such a model with pytorch! in Keras that’s easy using  <code>metrics = ["accuracy"] inside the compile function</code> same in scikit learn it is also very easy using r2_score or other score functions. why such things didn’t exist in Pytorch? and if I have to implement an Accuracy function myself for a Linear regression model, how can I do that ?</p>
<p>ps: I know how to implement such a function for a Classification Problem because it have discrete values. In linear Regression it is different because we have continuous values that’s why I couldn’t find a way to implement an Accuracy function for it. I could just see the value of the loss and based on it say whether my model is good or not but it would be better if I achieve to implement an Accuracy function like in keras or sklearn to print the accuracy of the Model for every epoch. that would be great</p>
Compilation C++ extension using recentpytorch version and cuda (1.20 and 10.1)

username: lelouedec
cooked: <p>Good evening,</p>
<p>I needed to upgrade pytorch to use a new function implemented since 1.0. However my C++ extension for pointnet++ which was perfectly compiling and working for previous pytorch version is not working anymore and return the following error when importing :</p>
<p>ImportError: /home/lelouedec/anaconda3/envs/test1/lib/python3.7/site-packages/point-0.0.0-py3.7-linux-x86_64.egg/point.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E</p>
<p>The implementation can be found here :<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/lelouedec/3DNetworksPytorch/tree/master/cppattempt" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/16105109?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/lelouedec/3DNetworksPytorch/tree/master/cppattempt" target="_blank" rel="nofollow noopener">lelouedec/3DNetworksPytorch</a></h3>

<p>Implementation of Popular Deep Learning Network using pytorch - lelouedec/3DNetworksPytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>EDIT :<br>
Installing pytorch but specifying cudatoolkit=10.1 seems to remove the error, but havent tested fully yet.<br>
Also the toolkit version dowloading when specifying cudatoolkit is the one for cuda 10.0 on the CUDNN website</p>

username: tom
cooked: <p>Maybe it’s something about having the version it is compiled with and the one it is run with go out of sync?<br>
My impression is that unless you use some internal API, that is the most common source of the error.</p>
<p>Best regards</p>
<p>Thomas</p>

username: lelouedec
cooked: <p>Good morning,</p>
<p>Sorry for the late answer, it was indeed a problem of versions. Somehow it compiled using the system cuda and not the one installed with pytorch (problem with path or something) and it yielded this error.<br>
The way to avoid these symbols error is to always check the cudatoolkit version used at compilation, the system one and pytorch version.</p>
Custom Sampler for Omniglot

username: raghuvanshraj
cooked: <p>I’m trying to create a Siamese network for the Omniglot dataset, for which I need to implement a custom sampler that will yield batches of pairs of images on which the neural net will be run. I need a little help with building this custom sampler. A little guidance in the right direction should be enough I believe.</p>

username: ptrblck
cooked: <p>The usual approach would be to write a custom <code>Dataset</code> and load and return the image pairs in the <code>__geitem__</code> method of it.<br>
How would you like to create these pairs, i.e. do you have any specific logic in mind?<br>
Also, have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading tutorial</a> for more information on how to create a <code>Dataset</code>.</p>

username: raghuvanshraj
cooked: <p>Hey!</p>
<p>Yeah so basically I have to just generate random pairs of images and if they are same I label the output as 1, otherwise I label it as 0. My only concern is I want each batch to have the same number of different pairs as there are same pairs of images, because of which I was hoping to create a <code>Sampler</code> that can generate these batches for me, which I can then use to initialize the <code>DataLoader</code> class.</p>
Batch of diagonal matrix

username: theevann
cooked: <p>Hi !<br>
I have a matrix n*m of n different vectors of dimensions m.<br>
I would like to get n matrices of size m*m with each matrix being a diagonal of a vector.</p>
<p>I guess I could do:</p>
<pre><code class="lang-python">d = []
for vec in mat:
   d.append(torch.diag(vec))
torch.stack(d)
</code></pre>
<p>But isn’t there any better ‘pytorchic’ way ?</p>

username: SimonW
cooked: <p>Yes! It’s kind of hidden, but here you go:</p>
<pre><code class="lang-python">&gt;&gt;&gt; mat = Variable(torch.randn(3, 4))
&gt;&gt;&gt; res = Variable(torch.zeros(3, 4, 4))
&gt;&gt;&gt; res.as_strided(mat.size(), [res.stride(0), res.size(2) + 1]).copy_(mat)
Variable containing:
 0.0422  0.0896  1.4919 -0.7167
-2.4854  0.3412 -1.4421 -0.5081
-0.6238  0.2446 -0.2848 -0.4184
[torch.FloatTensor of size (3,4)]

&gt;&gt;&gt; res

(0 ,.,.) =
  0.0422  0.0000  0.0000  0.0000
  0.0000  0.0896  0.0000  0.0000
  0.0000  0.0000  1.4919  0.0000
  0.0000  0.0000  0.0000 -0.7167

(1 ,.,.) =
 -2.4854  0.0000  0.0000  0.0000
  0.0000  0.3412  0.0000  0.0000
  0.0000  0.0000 -1.4421  0.0000
  0.0000  0.0000  0.0000 -0.5081

(2 ,.,.) =
 -0.6238  0.0000  0.0000  0.0000
  0.0000  0.2446  0.0000  0.0000
  0.0000  0.0000 -0.2848  0.0000
  0.0000  0.0000  0.0000 -0.4184
[torch.FloatTensor of size 3x4x4]
</code></pre>
<p>I’ve created an issue for this at <a href="https://github.com/pytorch/pytorch/issues/5198">https://github.com/pytorch/pytorch/issues/5198</a></p>

username: varunagrawal
cooked: <p>For the sake of completeness and to save future readers trouble, the function you are looking for is <code>torch.diag_embed</code>.</p>

username: linyu
cooked: <p>However, pytorch0.4.1 has not this function. How can I do this on pytorch0.4.1?</p>
Multilayer Neural Network doesn't work

username: Woo
cooked: <p>HI guys!<br>
I’m college student who is studying ML.</p>
<p>I made Binary Classifier for classifying which one is horse picture or human picture.<br>
It worked in one layer NN but it did’t work in multi layers like two or three.</p>
<p>This is my code for train part.</p>
<p>if someone know the problem on my code , Let me know what is wrong</p>
<p>Thanks!!</p>
<p><span class="hashtag">#This</span> is code</p>
<p>for epoch in range(NUM_EPOCH+1):<br>
<span class="hashtag">#forward</span> propagation(train)<br>
trZ1=np.dot(u,trainX)+a#Layer 1<br>
trZ2=np.dot(v,trA1)+b#Layer 2<br>
trA2=sigmoid(trZ2)</p>
<pre><code>trZ3=np.dot(w,trA2)+c#Layer 3
trA3=sigmoid(trZ3)

#get train loss
trloss=-(np.multiply(trainY,np.log(trA3))+np.multiply((1-trainY),np.log(1-trA3)))
trloss=1/trDataNum*np.sum(trloss)
trLossArray[epoch]=trloss

#forward propagation(test)
tZ1=np.dot(u,testX)+a#Layer 1
tA1=sigmoid(tZ1)

tZ2=np.dot(v,tA1)+b#Layer 2
tA2=sigmoid(tZ2)

tZ3=np.dot(w,tA2)+c#Layer 3
tA3=sigmoid(tZ3)

#get test loss
tloss=-(np.multiply(testY,np.log(tA3))+np.multiply((1-testY),np.log(1-tA3)))
tloss=1/tDataNum*np.sum(tloss)
tLossArray[epoch]=tloss

#get Accuracy
trainPY=np.where(trA3&gt;=0.5,1.,0.)
trAccuracy=((trainPY == trainY).sum())/trDataNum
trAcArray[epoch]=trAccuracy

testPY=np.where(tA3&gt;=0.5,1.,0.)
tAccuracy=((testPY == testY).sum())/tDataNum
tAcArray[epoch]=tAccuracy

#backward propagation
dz3=trA3-trainY
dw=1/trDataNum*np.dot(dz3,trA2.T)
dc=1/tDataNum*np.sum(dz3,axis=1,keepdims=True)

dz2=np.multiply(np.dot(w.T,dz3),sigmoid(trA2,True))
dv=1/trDataNum*np.dot(dz2,trA1.T)
db=1/tDataNum*np.sum(dz2,axis=1,keepdims=True)

dz1=np.multiply(np.dot(v.T,dz2),sigmoid(trA1,True))
du=1/trDataNum*np.dot(dz1,trainX.T)
da=1/tDataNum*np.sum(dz1,axis=1,keepdims=True)

#update weight and bias
u=u-lr*du
a=a-lr*da    

v=v-lr*dv
b=b-lr*db

w=w-lr*dw
c=c-lr*dc

#check the data per epoch 50
if epoch%50==0:
    
    print("epoch :" + str(epoch+1))
    print("train loss :  " +np.array2string(trloss))
    print("test loss :  " +np.array2string(tloss))
    print("train accuracy :  " +np.array2string(trAccuracy))
    print("test accuracy :  " +np.array2string(tAccuracy))</code></pre>
How to calculate the layer-level inference time of a pytorch network?

username: yunfeng
cooked: <p>Hi friends in Pytorch community,<br>
I want to find out the inference time of each (conv, bn, relu) layer in a pytorch model (like resnet101 implemented in torchvision package), is there any way to do this? I guess maybe <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module.register_buffer" rel="nofollow noopener">register_buffer</a>  is suitable for this task, but I don’t know how to figure it out. Does anyone has any idea? Thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>register_buffer</code> is used to save buffers that you then use during the forward pass.<br>
I would recommend using the <a href="https://pytorch.org/docs/stable/autograd.html#profiler" rel="nofollow noopener">autograd profiler</a> to measure the runtime.</p>

username: yunfeng
cooked: <p>Hi,<br>
Thanks for you reply, I will check docs of autograd profiler.</p>
Torch.cuda.is_available() returns False with cuda10.0

username: whisht
cooked: <p>torch.cuda.is_available() returns False with <strong>cuda10.0</strong>, but True with <strong>cuda8.0</strong>. So I am confused with this problem.</p>
<p>I have installed cuda with different version(8.0 as well as 10.0) and the corresponding cudnn, but the returned value is opposite. Is this relative to the NVIDIA  gpu driver version(which is <strong>375.26</strong>)?  Does this makes the cuda10.0 not working?</p>

username: kshitij
cooked: <p>I strongly believe that you are correct. The issue has to do with the CUDA and driver incompatibility.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6c334e39577a721541396f3d991174296026293b.png" alt="cuda_compatibility" data-base62-sha1="frbuhFRVEW1XQGuF6kKSbXmpNLt" width="422" height="314"></p>
<p>Reference : <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html" class="inline-onebox" rel="noopener nofollow ugc">CUDA Compatibility :: NVIDIA Data Center GPU Driver Documentation</a></p>
<p>So you may have to upgrade drivers for CUDA 10 support</p>

username: joellerena
cooked: <p>Also with torch.cuda.is_available () had false.<br>
But when installing the Nvidia driver to the most updated version 436.48, True is displayed. I previously updated Pytorch to 1.2.0.<br>
I have windows 10 and anaconda.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a" title="py"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a_2_690x272.jpeg" alt="py" data-base62-sha1="bkJmh9ArNw6cgwfbIwAYO930eCK" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a_2_690x272.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f6fd9cb09198cb9e9adba3b73d1f10092c5f12a.jpeg 2x" data-dominant-color="F6F7FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">py</span><span class="informations">923×364 80.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Model Initialized but Parameters are Empty

username: Rafael_R
cooked: <p>Hi,</p>
<p>I am creating a simple network:</p>
<pre><code class="lang-auto">class RandomNet(nn.Module):
    def __init__(self, vocab_size):
        super(RandomNet, self).__init__()
        self.vocab_size = vocab_size
        self.linear = nn.Linear(1, 2)

    def save_checkpoint(self, path):
        th.save(self.state_dict(), path)

    def load_checkpoint(self, path, cpu=False):
        if cpu:
            self.load_state_dict(th.load(path,
                map_location=lambda storage, loc: storage))
        else:
            self.load_state_dict(th.load(path))

    def forward(self, y):
        h = self.linear(y)
        return [h[0], h[1]]
</code></pre>
<p>When I wrapped this model inside a wrapper model, and just print the parameters:</p>
<pre><code class="lang-auto">self.net = RandomNet(src_vocab_size)
print(model.parameters())

AttributeError: 'RandomModel' object has no attribute 'parameters'
</code></pre>
<p>But I do have the linear parameter. How could I fix the error?</p>
<p>Am I making a mistake in using the linear layer?</p>

username: dejanbatanjac
cooked: <p>I am not sure where you used <code>self.net</code> but I create a working example for you:</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn


class RandomModel(nn.Module):
    def __init__(self):
        super().__init__();
        self.net = None
        self.linear = None
    def forward(self, x):            
        return x
    

class RandomNet(nn.Module):
    def __init__(self, vocab_size):
        super(RandomNet, self).__init__()
        self.vocab_size = vocab_size
        self.linear = nn.Linear(1, 2)

    def save_checkpoint(self, path):
        th.save(self.state_dict(), path)

    def load_checkpoint(self, path, cpu=False):
        if cpu:
            self.load_state_dict(th.load(path,
                map_location=lambda storage, loc: storage))
        else:
            self.load_state_dict(th.load(path))

    def forward(self, y):
        h = self.linear(y)
        return [h[0], h[1]]
    
src_vocab_size=20
net = RandomNet(src_vocab_size)
print(list(net.parameters()))

model = RandomModel()
model.net = RandomNet(src_vocab_size)
print(list(model.parameters()))
</code></pre>
<p>Out of the the Random Model will be:</p>
<pre><code class="lang-auto">[Parameter containing:
tensor([[0.1979],
        [0.7473]], requires_grad=True), Parameter containing:
tensor([-0.4504,  0.0435], requires_grad=True)]
</code></pre>

username: Rafael_R
cooked: <p>thanks! yes, so the <a href="http://model.net" rel="nofollow noopener">model.net</a> params have to be passed to the optimizer, silly mistake since my model inherited from object</p>
DataParallel: why only parallize the `model.feature` part of the vgg architecture?

username: zeakey
cooked: <p>I found that the imagenet classification example <a href="https://github.com/pytorch/examples/blob/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/imagenet/main.py#L162-L164" rel="nofollow noopener">here</a>  has a special branch to handle vgg and alexnet which have dense <strong>fully connected layers</strong>.</p>
<p>After a simple benchmark on VGGNet I found that ONLY parallelizing the <code>model.feature</code> part is obviously faster than parallelizing the whole model.</p>
<p>So what’s the theory behind this behaviour?</p>
<p>Are fully connected layers not friendly to parallelism?</p>

username: albanD
cooked: <p>I think that the computation required for the fully connected layers is so small, that the overhead of the parallelization makes it slower if you parallelize.<br>
Parallelization is only usefull if you have heavily waiting for computations to be performed.</p>

username: zeakey
cooked: <p>Okay I got it.</p>
<p>So the computational overhead in FC layers is not so heavy, however the number of parameters is<br>
extremely large. The will be much time on param/gradient sync.</p>