akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is weight_norm different in different python versions?

username: BierOne
cooked: <p>Hi! guys! I have a question about weight_norm.</p>
<p>My PyTorch version is 1.0.1 in all environments.  By the way, I have set the same seeds. But when I use weight_norm in python2.7 and python3.7 respectively, I got different weight_g but the same weight_v.</p>
<p>Moreover, I got the same weight_g when I init the linear layer with a lower dimension. I am wondering what happened when I changed the initial dimension.</p>
<p>Hope someone could give a detailed analysis! Thanks!</p>
<p>For example:</p>
<p>Init with lower dimension</p>
<pre><code class="lang-python"># python 3.7
weight_norm(nn.Linear(20, 20), dim=None).weight_g
"Parameter containing: tensor(2.6137, requires_grad=True)"

# python 2.7
weight_norm(nn.Linear(20, 20), dim=None).weight_g
"Parameter containing: tensor(2.6137, requires_grad=True)"
</code></pre>
<p>Init with higer dimension</p>
<pre><code class="lang-python"># python 3.7
weight_norm(nn.Linear(2048, 2048), dim=None).weight_g
"tensor(295.5971, requires_grad=True)"

# python 2.7
weight_norm(nn.Linear(2048, 2048), dim=None).weight_g
"tensor(26.1274, requires_grad=True)"
</code></pre>

username: ptrblck
cooked: <p>I just tried to reproduce this issue, but get similar values for Python2.7 and 3.7 in both use cases (<code>~2.6</code> and <code>~26.1</code>).</p>
<p>Could you update to the latest stable release (<code>1.4</code>) in your Python3.7 environment and rerun the code?</p>
<p>That being said, note that Python2.7 support is dropped and you shouldn’t use it anymore. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: BierOne
cooked: <p>Hi, ptrblck. I am so glad you give me a reply. Thanks!<br>
I have been updated my Pytorch to v1.4 and rerun the code in Python2.7 and 3.7 respectively. As you have been said, I also get similar values in two Python environment. It seems like Pytorch have been fixed this problem as they update the version. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> But I still don’t know why this issue happens. Maybe some computations in  C++ are different? <img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"></p>

username: ptrblck
cooked: <p>It seems to be weird, that the Python version somehow interacts with the results.<br>
However, we should let Python2.7 rest in peace and stick to 3.x. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Batch normalization layer after nn.ConvTranspose2d layers

username: veeresh_d
cooked: <p>Hi all,</p>
<p>just wondering can we use the batch normalization layer after the <code> nn.ConvTranspose2d</code> layers ?</p>

username: ptrblck
cooked: <p>Sure, that’s possible. Are you seeing any issues or errors using it?</p>
How is the efficiency when doing computation with Tensor of [w,h] or [1, w*h]?

username: songyuc
cooked: <p>Hi, guys,<br>
I am implementing the computation of mIoU in semantic segmentation,<br>
and I want to know how is the efficiency when doing computation with Tensor of [w,h] or [1, w*h]?<br>
Because this would help me decide whether I need to tranform the Tensor into a 1-dimension Tensor before I do the mIoU computation.</p>
<p>Any idea or answer will be appreciated!<br>
Thanks in advance!</p>

username: ptrblck
cooked: <p>I would recommend to run a quick profiling code and just get the runtimes for your current workload and setup. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Also, if you are using the GPU, don’t forget to synchronize the code via <code>torch.cuda.synchronize()</code>.</p>
Write tensor to .t7 file

username: Littisha_Lawrance
cooked: <p>I am using a simple image classification example for my code.I would like to extract the tensor values in the net.state_dict(). So far, I am able to extract the keys and the values separately using .keys and .values.I would like to write these values to a .t7 file. Is it possible to do the same with torch .save, such that it can be loaded later on to get the key value.</p>

username: ptrblck
cooked: <p>You could store the <code>state_dict</code> directly or alternatively wrap the <code>odict_keys</code> and <code>odict_values</code> into lists and store them:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
sd = model.state_dict()

torch.save(list(sd.keys()), 'tmp1.pth') # use list, otherwise: TypeError: can't pickle odict_keys objects
torch.save(list(sd.values()), 'tmp2.pth')
</code></pre>
<p>Let me know, if I misunderstood your use case.</p>

username: Littisha_Lawrance
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for the reply.<br>
I saved the state dictionary values into files with their names as the respective keys.<br>
I would like to know if there is any way to know the structure of a respective file to get the values of the weights and biases in binary;</p>

username: ptrblck
cooked: <p><code>torch.save</code> uses <code>pickle</code> by default as seen <a href="https://github.com/pytorch/pytorch/blob/c18cb1eb52605d7600de711d8f6a843099cef3a6/torch/serialization.py#L334">here</a>, so you could probably use <a href="https://docs.python.org/3/library/io.html#io.BytesIO">io.BytesIO</a> to read the file, but I haven’t tested it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
[SOLVED] Index based element update for pytorch tensor

username: mho2
cooked: <p>Hi, I have a simple snippet of code, which im wondering if there is a way to make it faster:</p>
<pre><code class="lang-auto">        cls_score_shifted = torch.zeros((cls_score.shape[0],cls_score.shape[1]),device='cuda')
        for i in range(cls_score.shape[0]):
            for j in range(cls_score.shape[1]):
                if(targets[i] == j):
                    cls_score_shifted[i,j] = cls_score[i,j] + 1
                else:
                    cls_score_shifted[i,j] = cls_score[i,j]
</code></pre>
<p>I want to conditionally update (shift) my class score if it is the correct (target) logit, and leave the other element alone. is there a quicker way to do this?</p>
<p>Thanks!</p>
<p>Edit:</p>
<p>This seems a lot faster</p>
<pre><code class="lang-auto">        cls_score_mask = torch.zeros((cls_score.shape[0],cls_score.shape[1]),device='cuda')
        for i,mask_row in enumerate(cls_score_mask):
            mask_row[targets[i]] = 1
        cls_score_shifted = cls_score + cls_score_mask
</code></pre>
<p>Probably still a better way to do this.</p>
<p>Edit2:<br>
Okay, got it.</p>
<pre><code class="lang-auto">        cls_score_mask = torch.zeros_like(cls_score).scatter_(1, targets.unsqueeze(1), 1)
        cls_score_shifted = cls_score + cls_score_mask
</code></pre>
How to implement (nueral net inside neural net) in Pytorch?

username: John_Deterious
cooked: <p>I have input data (images) that come with obscene number of channels (700+). To address this, I intend to have conv filter with (1x1) size so that it acts on a single pixel. <strong>what I want is</strong>: this filter acts only on this pixel (no spatial scanning!).</p>
<p>PS: I read about the “groups trick”, I think it is useful too and I used it in the first layer. But I completely forgot that I still have the problem of 1x1 filter acting on all pixels.</p>
<blockquote>
<pre><code>    self.torso = nn.Sequential(OrderedDict([
        ('nn_in_nn', nn.Conv2d(in_channels=ch_in, out_channels=64 * ch_out_multip, kernel_size=(1, 1), stride=1, padding=(0, 0),
                               groups=8 )),
</code></pre>
</blockquote>

username: John_Deterious
cooked: <p>I think I got it, make filter size = size of the image, thus, there’s no scanning, each set of weights from the filter will act only on corresponding pixel, <strong>only</strong>.</p>
<p>But now I think this is too much parameteres, and I really want to do depth-wise convolution to save some parameters. Groups is not helping at all, it is just separating the filters, no real depth-wise conv is taking place. How can I do that?</p>

username: John_Deterious
cooked: <p>I got it, reshape the data to be PIXELS X CHANNELS (i.e. image has one axis only). Next, use 1D convolutions that operate on channels axis as required. To inforce single pixel requirement, use the group trick in 1DConv. (now pixels are thought of as channnels).</p>
How to use shared weights in different layers of a model

username: Dongyue_Oliver
cooked: <p>I am trying to share the weights in different layers in one model. Please take a look at this example code:</p>
<p>this code tries to share weights in fc1 and some parts of fc2</p>
<pre><code>import torch
import torch.nn as nn
import torch.optim as optim

class testModule(nn.Module):

    def __init__(self):
        super(testModule, self).__init__()
        self.fc1 = nn.Linear(5, 10, bias=True)
        self.fc2 = nn.Linear(10, 10, bias=False)

        self.shared_weights = nn.Parameter(torch.randn(10, 5), requires_grad=True)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x

    def share_weight(self):
        index = [1, 3, 5, 7, 9]
        self.fc1.weight.data = self.shared_weights
        self.fc2.weight.data[:, index] = self.shared_weights
</code></pre>
<p>After calling the .share_weight() method and training, the weight in fc1.weight and fc2.weight[:, index] become different.</p>
<p>Why would this happen and what is the behavior behind assigning weight.data as another tensor. How could I reach the goal of sharing fc1.weight and fc2.weight[:, index] in training.</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>.data</code> is in the process of being removed and should not be used. As you have experienced, it only does very confusing things <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>You will need to have only <code>nn.Parameter</code>s to be the true parameters and you will have to recompute other things at each forward:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.optim as optim

class testModule(nn.Module):

    def __init__(self):
        super(testModule, self).__init__()
        self.fc1 = nn.Linear(5, 10, bias=True)
        self.fc2 = nn.Linear(10, 10, bias=False)

        # Remove the weights as we override them in the forward
        # so that they don't show up when calling .parameters()
        del self.fc1.weight
        del self.fc2.weight

        self.fc2_base_weights = nn.Parameter(torch.randn(10, 10))
        self.shared_weights = nn.Parameter(torch.randn(10, 5))

    def forward(self, x):
        # Update the weights
        index = [1, 3, 5, 7, 9]
        self.fc1.weight = self.shared_weights
        self.fc2.weight = self.fc2_base_weights.clone()
        self.fc2.weight[:, index] = self.shared_weights

        x = self.fc1(x)
        x = self.fc2(x)
        return x

    def _weight(self):
</code></pre>

username: Dongyue_Oliver
cooked: <p>Thank you! This way works!</p>
Is it possible to calculate the second-order derivative of a custom CUDA function?

username: sh8
cooked: <p>Is possible to calculate the second-order derivative of a custom CUDA function?<br>
When I try to calculate it with a twice backward technique, I got the error as follows.</p>
<pre><code class="lang-auto">RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
<p>I guess the reason is the computation graph is not created while backward pass in the custom CUDA function…</p>

username: albanD
cooked: <p>Hi,</p>
<p>How do you define your custom CUDA function? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If you use <code>autograd.Function</code> in python to create new autograd Functions, you need to make sure that your backward can be auto-diffed. If it cannot, then your backward should call the apply of another Function that corresponds to your backward and double-backward.</p>

username: sh8
cooked: <p>Hi,</p>
<p>Thank you for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
<pre><code class="lang-auto">If it cannot, then your backward should call the apply of another Function that corresponds to your backward and double-backward.
</code></pre>
<p>I see! That’s what I want to know!<br>
I’m gonna try to call the apply inside the backward function.</p>

username: sh8
cooked: <p><a class="mention" href="/u/alband">@albanD</a><br>
After I implemented a double backward function following what you said, it finally started working!!<br>
Thank you for your wonderful advise!</p>
Torch.save FileNotFoundError

username: hurazh
cooked: <p>when i try to call torch.save to save my model in a tmp_file, it rises a FileNotFoundError. the trace back is as follow:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 234, in <br>
test_hdr_continuous()<br>
File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 195, in test_hdr_continuous<br>
model = fit_mdn(X[:split], y[:split], nepochs=20)<br>
File “C:\Users\Haoran\Documents\GitHub\dose-response\python\simulations\continuous.py”, line 192, in fit_mdn<br>
torch.save(model, tmp_file)<br>
File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 161, in save<br>
return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))<br>
File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 116, in _with_file_like<br>
f = open(f, mode)<br>
FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp_file_4358f298-a1d9-4c81-9e44-db4d8f1b4319’</p>
</blockquote>
<p>it is weird that everything works perfectly on my mac, but I got this error on my Windows desktop</p>

username: albanD
cooked: <p>Hi,</p>
<p>Does the <code>/tmp</code> folder exists on windows? How do you generate the temporary file name? Is it a valid windows filename?</p>

username: dhinesh
cooked: <p>Found any solution?<br>
Even I face the same problem.</p>

username: albanD
cooked: <p>The problem here was that the path does not exist on windows machines. So if you’re using windows, you should set a path that matches a windows machines, not Unix machine.</p>

username: dhinesh
cooked: <p>Actually, the path existed. But, I had a weird file name. I think, that’s the reason why it wasn’t behaving as expected. Thanks <a class="mention" href="/u/alband">@albanD</a></p>
BCELoss with positive weights

username: hkitano
cooked: <p>Hi, I’m doing multi-label binary classification (8 classes), and have a model whose outputs have already passed through the sigmoid function (thus, I don’t want to use BCELossWithLogits). How do I incorporate positive label weighting for each of the 8 classes I have? Thanks!</p>

username: ptrblck
cooked: <p><code>pos_weight</code> is currently implemented in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss">nn.BCEWithLogitsLoss</a>, so you would have to either reimplement <code>BCELoss</code> with these weight arguments or switch to <code>BCEWithLogitsLoss</code>.<br>
I would recommend to second approach, as it will also have more numerical stability.</p>

username: hkitano
cooked: <p>To change to BCEWithLogitsLoss, would I simply get rid of my final Sigmoid layer, and during training, pass those outputs to the loss function? In my validation, I’ll have to call the sigmoid function on my outputs before i compare predicted labels and ground truths, right? Am I missing anything?</p>

username: G.M
cooked: <p>Yes, just remove the sigmoid and  pass those outputs directly to BCEWithLogitsLoss. Well, u don’t really have to call sigmoid again in Validation because for sigmoid, when x=0, y=0.5, u can use 0 as an indicator.</p>
Torch Distributed Class Definitions

username: Vibhatha_Abeykoon
cooked: <p>Hi,</p>
<p>I am trying to locate ReduceOp definitions, referred in the following comment in the source.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/243af17d65fb147901bf9efef21b0cb3e4f25fee/torch/distributed/distributed_c10d.py#L92" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/243af17d65fb147901bf9efef21b0cb3e4f25fee/torch/distributed/distributed_c10d.py#L92" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/243af17d65fb147901bf9efef21b0cb3e4f25fee/torch/distributed/distributed_c10d.py#L92</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="82" style="counter-reset: li-counter 81 ;">
<li># TODO: remove them when users are ready to take a hard dependency on PyTorch 1.</li>
<li>_backend = Backend.UNDEFINED</li>
<li>dist_backend = Backend</li>
<li>
</li>
<li>
</li>
<li>class reduce_op(object):</li>
<li>    r"""</li>
<li>    Deprecated enum-like class for reduction operations: ``SUM``, ``PRODUCT``,</li>
<li>    ``MIN``, and ``MAX``.</li>
<li>
</li>
<li class="selected">    :class:`~torch.distributed.ReduceOp` is recommended to use instead.</li>
<li>    """</li>
<li>
</li>
<li>    def __init__(self):</li>
<li>        # __members__ is a dict storing key-value pairs for enum classes</li>
<li>        for k, v in ReduceOp.__members__.items():</li>
<li>            setattr(self, k, v)</li>
<li>        self.__members__ = ReduceOp.__members__</li>
<li>
</li>
<li>    def __getattribute__(self, key):</li>
<li>        warnings.warn("torch.distributed.reduce_op is deprecated, please use "</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>(I did search the git source and local installation both)</p>
<p>I couldn’t find the class definition. Am I missing something here?</p>

username: osalpekar
cooked: <p>ReduceOp is a C++ enum, and is exposed to the python interface using pybind (<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/init.cpp#L145" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/init.cpp#L145</a>). That enum is defined here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/Types.hpp#L8" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/Types.hpp#L8</a></p>
Loading Different size tensors

username: adithya1
cooked: <p>Hi,</p>
<p>I am trying to load multiple tensors where each of them are either (3,64,64), (3,128,128) and (3,256,256) I want to use them without resizing, as an input to the <strong>segmentation</strong> network but for that, I will have to make my custom collate_fn. The issue with that is, I am unable to use the data in the form of a mini-batch as they are of different sizes. Is there any workaround for this ?.</p>
<p>I thought of making a data loader to load all 32x32 images in one iteration and 256x256 in the next iteration and so on. I’m not sure how to achieve this, please can some help me out with this? Thank you</p>

username: adithya1
cooked: <p>Please, can someone help me out with this ? Thank you</p>