akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to reverse shuffle process by pytorch

username: yzleroy
cooked: <p>If I have a tensor <strong>A</strong> [1,5,3], then I perform shuffle operation, like torch.randperm(len(<strong>A</strong>)) and get shuffled index [2,1,0]. Then I shuffle <strong>A</strong> according to shuffled index, the resulted tensor is [3, 5, 1]. but how can I get original tensor ([1,5,3])? Is there any reverse shuffle function of pytorch? I know I can assign elements according to shuffled index, but it’s very slow. Can anybody help me? Thank you very much!!!</p>

username: ptrblck
cooked: <p>If I understand your use case correctly, this should work:</p>
<pre><code class="lang-python">a = torch.randn(7)
print(a)
&gt; tensor([ 1.1022, -0.3638, -0.4358,  0.2048,  1.3897, -2.2341, -0.3528])

idx = torch.randperm(a.size(0))
print(idx)
&gt; tensor([5, 6, 0, 2, 4, 3, 1])

b = a[idx]
print(b)
&gt; tensor([-2.2341, -0.3528,  1.1022, -0.4358,  1.3897,  0.2048, -0.3638])

c = torch.zeros_like(a)
c[idx] = b
print(c)
&gt; tensor([ 1.1022, -0.3638, -0.4358,  0.2048,  1.3897, -2.2341, -0.3528])

print((c == a).all())
&gt; tensor(True)
</code></pre>

username: yzleroy
cooked: <p>Yes, thank you very much!!! <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=9" title=":heart_eyes:" class="emoji" alt=":heart_eyes:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=9" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:"></p>
Where to find implementation of Cross Entropy loss function

username: thoger
cooked: <p>Hi</p>
<p>I love the amazing Pytorch docs, but one issue we ran into is that they seem a bit sparse on the topic of how loss functions, specifically Cross Entropy, is <strong>implemented</strong> (i.e. mathematically and/or the raw code).</p>
<p>Does someone have an insight into where I would go about finding that?</p>
<p>Context: We are implementing a custom loss function that can incorporate class weights in “distribution-based classes” (not quite sure what the specific term is).</p>
<p>(Sorry if this is a out-of-scope question – feel free to refer me to where a question like this would be more appropriate)</p>

username: ptrblck
cooked: <p>A manual (and slow) implementation can be found <a href="https://discuss.pytorch.org/t/cross-entropy-loss-clarification/103830/2">here</a> and the CPU implementation calls into <a href="https://github.com/pytorch/pytorch/blob/99b641169b4086225479ddad7a8c027c1faf8f19/aten/src/ATen/native/LossNLL.cpp#L458-L471"><code>cross_entropy_loss</code></a> and then into <a href="https://github.com/pytorch/pytorch/blob/99b641169b4086225479ddad7a8c027c1faf8f19/aten/src/ATen/native/LossNLL.cpp#L27-L28"><code>nll_loss</code></a>.</p>

username: thoger
cooked: <p>Thank you, very helpful! Highly appreciated! If I had been a bit more lucky in my search, I should of course have found your reference on my own, but alas.</p>
<p>(Out of topic-context: And just now that I have the chance, also thank you so much for not just this answer but the tons of other answers you provide in here that prove hyper-helpful on a daily basis)</p>
Help with versions?

username: mtarun02
cooked: <p>Hello, I am new to pytorch and CUDA, and have been struggling to make all the versions work together successfully. Here is my system information:</p>
<p>These GPUs:<br>
GeForce GTX TITAN X<br>
TITAN X (Pascal)<br>
TITAN X (Pascal)<br>
GeForce GTX TITAN X</p>
<p>I have figured out that TITAN x is compute capability 5.2, but cannot find which version of CUDA or pytorch should go along with this, as some threads online say 5.2 may be too old, etc. It seems that the version of pytorch also determines which versions of CUDA i can install, as only certain ones with with each version of pytorch? Thank you for the help.</p>

username: ptrblck
cooked: <p>Compute capability 5.2 is compatible with the current release (<code>1.9.0</code>) and you should be able to install the binaries or build from source using CUDA&gt;=9.2 (I believe the min. CUDA toolkit version will be bumped to 10.0).<br>
Note that you do not need a local CUDA toolkit installation, as the binaries (pip wheels and conda binaries) will ship with their own CUDA runtime (cudnn, NCCL, etc.), so your workstation only needs a valid NVIDIA driver for the selected CUDA version.<br>
However, you would need a local CUDA toolkit in case you want to build PyTorch from source or build any custom CUDA extension.</p>
Convolution slowness on Turing architectures

username: luisa1
cooked: <p>Hi, I’ve found that two downsampling 1x1 convolutions result in very slow backward passes on a Quadro RTX 5000 with deterministic enabled. The timing result is 0.3ms forward, and 115ms backward.  On other GPUs, the backward timing is 3ms. The input image has shape 256x512.</p>
<p>The module code:</p>
<pre><code class="lang-auto">class CustomModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        inplanes = [64, 64]
        planes = [16, 64]
        layers = []
        for i in range(len(inplanes)):
            layers.append(nn.Conv2d(inplanes[i], planes[i] * 4, stride=2, kernel_size=1, bias=False))
        self.layers = nn.Sequential(*layers)
        self.inplanes = inplanes

    def forward(self, x):
        out = []
        for layer in self.layers:
            x = layer(x)
            out.append(x)
        return out[-1]
</code></pre>
<p>Pytorch version: <code>1.9.0+cu111</code></p>
<p>It seems that cudnn chooses a slow op, <code>dgrad2d_alg1_1</code> on this particular GPU architecture, but not on other GPU architectures. I believe a similar timing occurs on all Turing architectures.</p>
<p>Profiling code:</p>
<pre><code class="lang-auto">class Timer(object):
    def __init__(self, name: str, verbose: bool) -&gt; None:
        super().__init__()
        self.name = name
        self.verbose = verbose

    def __enter__(self):
        torch.cuda.synchronize()
        self.start = time.time()

    def __exit__(self, type, value, traceback):
        torch.cuda.synchronize()
        if self.verbose:
            print(self.name, "{:.3f}ms".format((time.time() - self.start) * 1e3))


def custom_step(custom_model, data, verbose=True):
    with Timer("forward", verbose):
        output = custom_model(data)
    loss = torch.ones_like(output)
    with Timer("backward", verbose):
        output.backward(loss)


def profile(model, data, model_step, warmup_steps):
    for _ in range(warmup_steps):
        model_step(model, data, verbose=False)
    model_step(model, data)


if __name__ == "__main__":
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True
    custom_module = CustomModule().cuda()
    model_input = torch.randn((1, custom_module.inplanes[0], 256, 512), device="cuda")
    profile(custom_module, model_input, custom_step, warmup_steps=10)
</code></pre>
<p>These results reproduce with benchmark = True and False.</p>

username: ptrblck
cooked: <p>Deterministic algorithms are known to be potentially slow and apparently <code>dgrad2d_alg1_1</code> was the one suitable for your device.<br>
Using <code>cudnn.benchmark=True</code> wouldn’t change it, as the kernel selection is in itself non-deterministic and should thus be disabled.</p>
How to resolve the below error

username: SHASHANK_KUMAR_MISHR
cooked: <p>on running the below line</p>
<p>from models import *<br>
Following error occurs<br>
2 from utils import *<br>
3 import os, sys, time, datetime, random<br>
4 import torch<br>
5 from torch.utils.data import DataLoader</p>
<p>~/Downloads/models/<strong>init</strong>.py in <br>
23 from base import *<br>
24 from props import *<br>
—&gt; 25 from references import *<br>
26 from utils import *</p>
<p>ModuleNotFoundError: No module named ‘references’</p>

username: ptrblck
cooked: <p>The <code>references</code> module cannot be found in your current environment.<br>
Make sure the files are located in your working directory or in some subdirectory they should be loaded from.<br>
Have a look at the <a href="https://docs.python.org/3/reference/import.html">Python Import System</a> for more information.</p>

username: jsievers77
cooked: <p>I am having trouble resolving the same error.  pip install has worked pretty well at resolving missing modules, but I am unable to figure out what module is named ‘references’.</p>

username: ptrblck
cooked: <p>I was assuming that the original post used a custom module named <code>references</code>, which wasn’t found in the current working directory, so you could check for the same issue.</p>
Pytorch model taking too much space

username: kountaydwivedi
cooked: <p>Hi,<br>
While saving a model, I came to realize that they are taking huge space. Is there any way to store Pytorch’s models in a compressed format?<br>
Thanks.<br>
:-}</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.save.html"><code>torch.save</code></a> already compresses the file via the <code>_use_new_zipfile_serialization=True</code> argument. I don’t know, if another <code>pickle_module</code> or <code>pickle_protocol</code> could potentially yield more space savings.</p>
Help manually getting gradients from network

username: anon1
cooked: <p>I have created the following MWE.</p>
<pre><code class="lang-auto">import torch


class Net(nn.Module):
    def __init__(self, input_size, h1, output_size):
        super().__init__()
        self.fc1 = nn.Linear(input_size, h1)
        self.ac1 = nn.ReLU()
        self.fc2 = nn.Linear(h1, output_size)
        
    def forward(self, x):
        x = self.fc1(x)
        x = self.ac1(x)
        x = self.fc2(x)
        return x


net = Net(1, 16, 11)

x = torch.rand(1, requires_grad=True)
output = net(x)

#I'm trying to get the gradient dx/doutput[i]
for out in output:
    print(out.backward(retain_graph=True))


</code></pre>
<p>This prints None a bunch of times. Why? These should be the derivatives dx/doutput[I]</p>

username: tom
cooked: <p>tensor.backward() doesn’t return anything, it sets/updates the <code>.grad</code> attribute on the autograd leaves.<br>
If you want gradients to be returned instead of accumulated in <code>.grad</code>, you would want to use <code>torch.autograd.grad</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: anon1
cooked: <pre><code class="lang-auto">torch.autograd.grad(output, x)
torch.autograd.grad(x, output)
</code></pre>
<p>both do not work either. Am I making a silly mistake? I expect this to return a tensor of all the d_output/dx (I just realized I typed it wrong in my initial problem description). I want the partial derivatives of all the outputs with respect to the one input.</p>
The hyperparameters settings when apply the ddp

username: CBGxd
cooked: <p>Hi guys,</p>
<p>I don’t know someone asked it before or not, but I really wanna to make sure everything I did is correct. Says we have learning rate <em>lr</em>, epoch <em>e</em>, and batch size <em>b</em> <strong>as normal setting</strong>. And now, we apply it to the ddp based on <strong>2 gpus</strong>:<br>
1). If we wanna the batch size keep unchanged by compared with the single card, we don’t change the lr and b = b/2<br>
2). If we wanna apply double batch size, given we have 2 gpus, we don’t need to modify the b, but lr = lr * 2 because of the larger batch size.<br>
3). Should we modify the epoch number?<br>
4). I’m doing a semi-supervised learning,  which including two losses (i.e., supervised loss and semi loss). During training, there is a weight that apply for the semi-supervised loss, and we don’t need to change it right?</p>
<p>Cheers,</p>

username: Yanli_Zhao
cooked: <ol>
<li>and 2) are correct.</li>
<li>if you have 1) setting, I guess you do not need to modify the epoch number; if you have 2) setting, I guess you may need to adjust the number a little bit as it trains faster?</li>
</ol>

username: CBGxd
cooked: <p>That’s wired. When I double up the batch size, the convergence speed of the algorithm turns down… I’m testing the b=4 and b=8 (per gpu) for two cards. After the <strong>first epoch</strong>, the smaller batch size always archives better accuracy. Is that normal?</p>

username: CBGxd
cooked: <p>Thanks so much for you help.</p>
<blockquote>
<p>After the <strong>first epoch</strong>, the smaller batch size always archives better accuracy.</p>
</blockquote>
<p>Do you have any suggestion with this?</p>

username: Yanli_Zhao
cooked: <p>batch size will impact accuracy, this is normal</p>

username: CBGxd
cooked: <p>Yes, the final result get improved a lot.</p>
Train and val loss learning curve

username: Alok1
cooked: <p>Hey guys.</p>
<p>I am trying to implement a DCNN for the image reconstruction task. I am facing a weird issue with my end results on the test set vs training and validation loss curve.</p>
<p>My end results look great, however, there’s a huge difference between training loss and validation loss, which is a clear sign of overfitting. Could anyone suggest the reason behind this?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/eda3a7efff6b4fbc239b45101a87e67a990c215a.png" alt="exp_07" data-base62-sha1="xUfN5XdWlKL2FR7tncCwGK4JEh4" width="640" height="480"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1e6382844ea84da6487775e564ad165f1cabfca.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1e6382844ea84da6487775e564ad165f1cabfca" title="mnist_SE"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1e6382844ea84da6487775e564ad165f1cabfca_2_455x500.png" alt="mnist_SE" data-base62-sha1="yvWjpwx78e1qYoVwkBI2Fae4IP0" width="455" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1e6382844ea84da6487775e564ad165f1cabfca_2_455x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1e6382844ea84da6487775e564ad165f1cabfca.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1e6382844ea84da6487775e564ad165f1cabfca.png 2x" data-dominant-color="918AA5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mnist_SE</span><span class="informations">501×550 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>P(X) is a predicted image and R(X) is the target image.</p>
<p>Hypermeter settings<br>
epochs = 50<br>
batch size = 64<br>
LR  = 0.001<br>
Shuffle training data = True</p>
Face recognize with Depthmap

username: Huynh_Viet_Trung
cooked: <p>Hi everybody.<br>
Sorry my English not good. Im from Viet Nam<br>
I have question, My Project about recognize identify. I using camera Intel D435 to collect and get depthmap of face, I folllow this paper with my dataset student. link<br>
After I train  Model include one pictures(.png) and one depthmap(.npy) of face, I train my model ResNet50 with pretrain Imagenet, using Pytorch, my dataset have 500 class with 70 picture/class. I got model 99.7% But i have problem my model only veryfi  data trained in my dataset, and not verify real data.<br>
My data with ratio 8:1:1 8 train, 1 eval, 1 test<br>
I think my model is overfit or something wrong??. Please help me.<br>
How i can fine tune or do something to improve accuracy in real data.??</p>

username: Huynh_Viet_Trung
cooked: <p>Link paper: <a href="https://github.com/XingwXiong/Face3D-Pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - XingwXiong/Face3D-Pytorch: 3D Face Recognition, implemented with PyTorch</a></p>
How to change cudnn version in pytorch

username: c.justin
cooked: <p>Hi there,</p>
<p>I’m really new to pytorch and when I do validation in training I got this error in using batchnorm1d:<br>
cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.<br>
By taking a took at <a href="https://github.com/ttumiel/pytorch/commit/36d310827c73bfed6aae6f4d68725d86f447d428" rel="nofollow noopener">https://github.com/ttumiel/pytorch/commit/36d310827c73bfed6aae6f4d68725d86f447d428</a>, I though it might be fixed if I update my cudnn version.  Previously I have torch.backend.cudnn.version() to be 7603. I installed a 7.6.5 version of cudnn in Conda, but I don’t know how to get pytorch to use that one.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You could try to install the binaries with <code>cudatoolkit=10.2</code>, which should ship with a newer cudnn version, use the <a href="https://ngc.nvidia.com/catalog/containers/nvidia:pytorch">NGC container</a>, or build PyTorch from source with your local CUDA and cudnn installation.</p>

username: jia_lee
cooked: <p>Do you mean Pytorch installed by pip never uses the local cudnn in our computer?</p>

username: ptrblck
cooked: <p>Yes, that is correct. The install log shows which CUDA runtime and cudnn is used (in the file name).<br>
If you want to use your local libraries, you would have to build from source.</p>

username: Kk_L
cooked: <p>Could you please write a detailed tutorial document to show the steps for setting up this container? Thanks a lot!</p>
Showing image from data loader

username: Mutairu_onaido
cooked: <p>Somebody please help me on this i keep having this error!<br>
<a href="/uploads/short-url/phRNc92pEy3QKF6e6u9Q0mTw9Yh.png">image|690x331</a></p>

username: Monne
cooked: <p>what is <code>helper</code>? can you provide more code?</p>
About change of way to store tensors in GPU?

username: vuhoangminh
cooked: <p>Dear All,</p>
<p>At the end of an intermediate layer in the forward pass, I want to store a modified version of the output of this layer instead of the original one. Then in the backward pass, I would like to use the modified version to compute the gradient for the next layer. Let’s assume that the operation of that intermediate layer (ReLU, conv. …) is unchanged.</p>
<p>My questions are</p>
<ul>
<li>How to ask PyTorch to save the modified tensor and detach the original one from the graph?</li>
<li>If PyTorch allows to do so, how will the GPU allocation be affected? Will it be different by the amount of the difference between the original and modified tensors?</li>
</ul>
<p>Help is much appreciated</p>

username: ptrblck
cooked: <p>I think you could create custom <code>autograd.Function</code>s as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> and store the tensors via <code>ctx.save_for_backward</code> and use them afterwards in the <code>backward</code> operation.<br>
Since this new tensor would be stored the memory would depend on its size.</p>

username: vuhoangminh
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you so much for your reply.</p>
<p>Assume that in the forward pass I use the following codes to compute output and modify the output:</p>
<blockquote>
<p>output = run_function(input)<br>
output = modify(output)<br>
ctx.save_for_backward(output)</p>
</blockquote>
<p>where <code>run_function</code> is ReLU, conv., etc. Then, in the backward pass I retrieve the modified output by:</p>
<blockquote>
<p>(output, ) = ctx.saved_tensors</p>
</blockquote>
<p>Together with the <code>grad_output</code>, will the following code is sufficient to compute the gradient for the input?</p>
<blockquote>
<p>torch.autograd.backward(output, grad_output)</p>
</blockquote>
<p>If not, what should I do?</p>
<p>Thank you so much for your time.</p>

username: ptrblck
cooked: <p>No, you would have to implement the backward method manually as shown in the linked example.<br>
E.g. for <code>ReLU</code>:</p>
<pre><code class="lang-python">    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input &lt; 0] = 0
        return grad_input
</code></pre>
<p>where you could now change this computation and add your custom methods to it.</p>

username: vuhoangminh
cooked: <p>I got it. Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
Installing torch

username: Khabbab_Zakaria
cooked: <p>I am trying to install torch in linux with cuda version 11.1<br>
I checked this: <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a><br>
It says that the code is</p>
<pre><code class="lang-auto">pip3 install --user torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f
</code></pre>
<p>However, this line gives this error</p>
<pre><code class="lang-auto">-f option requires 1 argument
</code></pre>
<p>Can someone help?</p>

username: ptrblck
cooked: <p>You are missing the URL after the <code>-f</code> argument:</p>
<pre><code class="lang-auto">pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
Result = torch._C._nn.leaky_relu(input, negative_slope) RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 754974720 bytes

username: anoud_abdalziz
cooked: <p>I have this problem althoght I reduced my batch size of dataset</p>
<p>RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 754974720 bytes.</p>

username: ptrblck
cooked: <p>How much host RAM does your setup have? The posted number would correspond to 720MB, so it seems your RAM size is quite limited or you’ve already filled out the majority of it.</p>

username: anoud_abdalziz
cooked: <p>Because I run this model <a href="https://github.com/Deep-Imaging-Group/RED-WGAN/blob/master/preprocessing.py" class="inline-onebox" rel="noopener nofollow ugc">RED-WGAN/preprocessing.py at master · Deep-Imaging-Group/RED-WGAN · GitHub</a> on CPU, I had 16gram , and because I had a huge data volume, and even though I had Nvidia GPU , I always have an error with data volume.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a160319c05b68d34da4d439eeae27b3cf001f596.png" data-download-href="https://discuss.pytorch.org/uploads/default/a160319c05b68d34da4d439eeae27b3cf001f596" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a160319c05b68d34da4d439eeae27b3cf001f596.png" alt="image" data-base62-sha1="n1AXX4uNwbwoeod2I9WOoDPytoi" width="690" height="275" data-dominant-color="E8E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">763×305 7.62 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4a2372cea800b5e889cd2a740e4ea18a9445af0.png" alt="image" data-base62-sha1="wCApk0svNVgreCRy1keLIO2z9PG" width="395" height="230"></p>

username: ptrblck
cooked: <p>You could try to increase the swap, which would result in a significant slowdown, but you might be able to at least allocate the needed memory.<br>
The better approach would most likely be to reduce the memory usage on this system, e.g. by lowering the batch size.<br>
I’m unsure, if you are preloading the entire dataset, but you might want to lazily load each sample in this case.</p>
Retrieve names of namedtuple output

username: dfalbel
cooked: <p>I wonder if it’s possible to retrieve the names of the resulting output when calling a traced function, for example:</p>
<pre><code class="lang-auto">import torch
from collections import namedtuple
output = namedtuple("output", ["hello", "bye"])

def f (x):
  return output(x, 2*x)

traced_fn = torch.jit.trace(f, torch.tensor(1))
traced_fn(torch.tensor(2))
</code></pre>
<p>Returns an unamed tuple:</p>
<pre><code class="lang-auto">(tensor(2), tensor(4))
</code></pre>
<p>I know I can return a dictionary and disable <code>strict</code>, to get named outputs but would like to know if this this possible with named tuples.</p>
Failed to install pytorch without root permission

username: jiapei100
cooked: <p>Hmmm. Finally, I had <strong>PyTorch</strong> successfully built from source, using <strong>CMake</strong>.<br>
However, in order to have <strong>python torch</strong> enabled, it seems I had to use the following commands: <code>python setup.py build</code> and <code>python setup.py install</code>, but failed with <strong>Permission denied</strong>. If I install with <strong>sudo</strong>, everything will be installed under <code>/root/.local</code>, instead of <code>/home/username/.local</code> .</p>
<pre><code class="lang-auto">➜  pytorch git:(master) ✗ python setup.py install --user
Building wheel torch-1.8.0a0
-- Building version 1.8.0a0
cmake --build . --target install --config Release -- -j 48
[0/1] Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:119 (file):
  file failed to open for writing (Permission denied):

    ........./pytorch/build/install_manifest.txt


FAILED: CMakeFiles/install.util 
cd ........./pytorch/build &amp;&amp; /usr/local/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 762, in &lt;module&gt;
    build_deps()
  File "setup.py", line 312, in build_deps
    build_caffe2(version=version,
  File "........./pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "........./pytorch/tools/setup_helpers/cmake.py", line 346, in build
    self.run(build_args, my_env)
  File "........./pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '48']' returned non-zero exit status 1.

</code></pre>
<p>Any anybody help please?</p>
<p>Cheers</p>

username: ptrblck
cooked: <p>Could you check the permissions of the <code>pytorch/build</code> folder and if your user doesn’t have the proper permissions, remove it and retry the installation?</p>

username: calvinmccarter
cooked: <p>For what it’s worth, I had a similar error when installing via <code>python setup.py develop</code>. But my “Permission denied” error was coming from the build-script running Ninja, and for some reason it was attempting to run <code>/usr/local/bin/ninja --version</code> rather than <code>/home/username/conda-env/bin/ninja --version</code>.</p>
<p>I couldn’t get the build-script to use the correct Ninja, but the solution that worked for me was to run  <code>python setup.py develop --cmake</code> instead.</p>
How to generalize softmax to allow more than one true class

username: zhenqin
cooked: <p>My input is a list of 5 embeddings and I want to predict whether the 5 entities are selected. If I only need to output one entity, I can use softmax activation. Then what if two of them are selected?</p>
<p>Basically I want a generalized softmax and the sum of the output vector is 2 instead of 1. Also each element of the output is less than or equal to 1.</p>

username: eqy
cooked: <p>It sounds like you are describing a multilabel objective. One possible loss function for something like this would be multilabel softmarginloss: <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelSoftMarginLoss.html?highlight=multilabel#torch.nn.MultiLabelSoftMarginLoss" rel="noopener nofollow ugc">MultiLabelSoftMarginLoss — PyTorch 1.9.0 documentation</a></p>

username: zhenqin
cooked: <p>Yes, it is a multi-class objective but I am not sure it that is also multilabel. From my understanding multi-lable means different fields, a valid output can have the elements being all ones or all zeros, but in my problem, the sum of sample output vector y must be 2, i.e. only two of the labels are 1.</p>

username: eqy
cooked: <p>In this case I wonder if simply computing the product of the labels (2**5 = 32 classes) would work.</p>

username: zhenqin
cooked: <p>I can use 5C2 = 10 classes, but it is quite hard to scale up using this method. Thanks anyways!</p>
Model optimization

username: Rajat_Sharma1
cooked: <p>I am training a network on 3D array (voxels) which have a range of [1, -1]. The distribution is such that most of values are 1, very few are between 1 and -1. During training the network does optimize but only for the location where there are 1’s as they dominate the loss. I am using adam optimizer. What can be done such that the network also learns for the location where there are no 1’s. I have tried assigning high weights to the location where there values less than 1 but it does not help.</p>

username: Epoching
cooked: <p>What’s your loss function? And how much of a higher weight did you assign to the location where there are values less than 1?</p>

username: Rajat_Sharma1
cooked: <p>I use L1 loss function. Initially I calculated loss as mean of L1 loss but which it is dominated  by 1’s as 70-80% values are equal to 1. Later, I tried to mask out values equal to 1 and less than 1 and take their seperate mean. Later I set weight equal to 10 for values less than 1. But some how the network is not converging.</p>

username: Epoching
cooked: <p>Oh interesting, when you weighted the smaller class by a factor of 10, it just struggled to converge in general right? (while in the original case without weights, it converges to a point where it always predicts the more populated class).</p>
<p>Is data augmentation or self-supervised pretraining an option, or perhaps just need more data instead of overweighting the smaller class? <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=9" title=":open_mouth:" class="emoji" alt=":open_mouth:"></p>
Torch.stack 10x slower than numpy.stack in default_collate

username: PaulS
cooked: <p>Hi,<br>
I have Dataset class which holds its entire data in a Numpy array in memory and returns a tuple of two Numpy arrays via <strong>getitem</strong>.<br>
The data item returned by <strong>getitem</strong> has shape (3, 244, 244) and the target item is a 1 dimensional array.<br>
I’m using the default Dataloader with batch size of 64 which in turn uses default_collate to arrange the batches. When using numpy.stack instead of torch.stack in the default_collate function I see a speedup of over 10 times (from ~1s to ~0.1s per batch) in the collate function.</p>
<p>Specifically, when changing line 68 in utils/data/_utils/collate.py from</p>
<pre><code class="lang-auto">return default_collate([torch.as_tensor(b) for b in batch])
</code></pre>
<p>to</p>
<pre><code class="lang-auto">return torch.as_tensor(numpy.stack(batch))
</code></pre>
<p>How is that?</p>

username: eqy
cooked: <p>I think a performance difference with these two implementations is expected, as converting to tensors before stacking introduces a second iteration step in Python (which will be slower than doing everything in a single native function like <code>numpy.stack</code>). However, I can’t reproduce the 10x difference (it might be because my environment or timing) is somewhat different.</p>
<p>If you have a small code snippet that shows the 10x difference it might be easier to see if there is another issue, but as is I would expect converting to tensors after stacking in <code>numpy</code> to be faster.</p>
PyTorch Geometric -- behaviour when dealing with node labels no in range (0, ..., N-1)

username: David_Ireland
cooked: <p>Suppose we have a graph with N nodes, but our node labels are not 0, 1, 2, …, N-1.  What is the behaviour of PyTorch methods here?</p>
<p>Suppose I create a graph using <code>graph = Data(edge_index=torch.LongTensor([[0, 1, 1, 2, 5, 6],                                           [1, 0, 2, 1, 6, 5]]))</code>, and I want to pass this information to a GCN layer, does the node feature matrix I define have to have node features for 7 nodes or 5?</p>
<p>If it is the former, then I suppose I can just just pad these with 0’s. If it is the latter, would the output of a GCN layer still be a 5xF tensor? (assuming it is a standard GCN layer where the first dimension of the output tensor is the same as the input). If so, then would the rows still be in numerical order, i.e. the output array would be in order (0, 1, 2, 5, 6) in terms of which node the row relates to?</p>
<p>Finally, for whichever behaviour it is, then is the behaviour consistent across the convolutional and dense convolution layers?</p>
Create diagonal matrices from batch

username: Samue1
cooked: <p>I have a batch</p>
<p><code>x = torch.rand(size=(M, N))</code></p>
<p>and want to create for each of the <code>M</code> inputs a diagonal matrix with dimensions <code>N x N</code> such that the output has dimensions <code>M x N x N</code>. How can I do that? If I pass <code>x</code> to <code>torch.diag</code> I get a one-dimensional output.</p>
<p>Any idea what I do wrong?</p>

username: KFrank
cooked: <p>Hi Samuel!</p>
<aside class="quote no-group" data-username="Samue1" data-post="1" data-topic="125380" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f04885/48.png" class="avatar"> Samue1:</div>
<blockquote>
<p><code>x = torch.rand(size=(M, N))</code></p>
<p>and want to create for each of the <code>M</code> inputs a diagonal matrix</p>
</blockquote>
</aside>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.diag_embed (torch.rand (size = (M, N)))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Error with gradient backprop when not doing batch

username: Fredrik_Lundberg
cooked: <p>Hi!<br>
I have implemented a form of option critic network with batch update for the critic but iterative for the actor. The first update is done after a batch has accumulated on both and then iterative on the actor but intermittently on the critic (when enough new samples have accumulated). This first update works fine but then when I only update the actor  I get the following where the trackback reveals this to be the error:<br>
nonspatial_latent_output = torch.unsqueeze(self.latent_nonspatial(nonspatial_input),0)<br>
…<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [262176, 256]], which is output 0 of TBackward, is at version 2; expected version 1 instead</p>
<p>The only difference between the batch update is that that variable is then squeezed, but from what I gater neither squeeze nor unsqueeze should be inplace operations.</p>
<p>Any help would be greatly appreciated!!</p>

username: Ayman_Al_Jabri
cooked: <p>Try this:</p>
<pre><code class="lang-auto">nonspatial_latent_output = self.latent_nonspatial(nonspatial_input)
nonspatial_latent_output = nonspatial_latent_output.unsqueeze(0)
</code></pre>

username: Fredrik_Lundberg
cooked: <p>Thank you for your reply, I think I tried it but it did not work. I have however found a solution, I was also using that feature network to produce the state for the next observation before computing gradients. When I changed that so that I only had put one observation through the feature network before computing the gradients it did not raise the error anymore. Not quite sure why since I did not use that state for any loss or anything, but maybe it confused its histories since there were no two observation having been put through the feature network.</p>
Can I set values in a model's state_dict/ a layers dropout rate after it has been loaded from a checkpoint?

username: jpc
cooked: <p>Summary: I’d like to load a model from a .pt file that was created with a dropout of x, and set a new value y for the dropout and continue training the model with the new dropout rate.</p>
<p>I have a trained model I save in a .pt file. I load the model and optimizer like this:</p>
<pre><code class="lang-auto">checkpoint = torch.load('./models/foo.pt')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
</code></pre>
<p>In my model class, I create an LSTM layer like this:</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.n_layers, dropout=dropout_rate)
</code></pre>
<p>When initially created, the model had a dropout value of 0 passed. In the code to load and retrain, I get the value for dropout_rate from the cmd line arguments, and set a global variable that is referenced in the class definition (not sure if this could cause the issue, and dropout_rate needs to be passed as an argument to the class instead). If I call the code to retrain and pass a value of 0.5 as dropout rate, it seems that loading the model from the .pt file overwrites the parameters passed when creating the model object.</p>
<p><strong>It is possible to set the dropout rate of the LSTM layer to a new value after the model has been loaded from a .pt file?</strong></p>
<p>Running this code nets me the following output:</p>
<pre><code class="lang-auto">for param_tensor in model.state_dict():
    print(param_tensor, "\t", model.state_dict()[param_tensor].size())

Model's state_dict:
lstm.weight_ih_l0 	 torch.Size([300, 177])
lstm.weight_hh_l0 	 torch.Size([300, 75])
lstm.bias_ih_l0 	 torch.Size([300])
lstm.bias_hh_l0 	 torch.Size([300])
linear.weight 	 torch.Size([1, 75])
linear.bias 	 torch.Size([1])
</code></pre>
<p>Printing model.parameters() provides no useful info, as it is a list of nameless parameters mapped to large arrays of tensors. I don’t see where I could find the value of the dropout, as it must be stored somewhere if it is overwriting the initial value.</p>
<p>Is this at all possible?</p>

username: ptrblck
cooked: <p>Setting the <code>dropout</code> after initializing the model should work, but you would have to make sure, the passed value makes sense for the current setup, as <a href="https://github.com/pytorch/pytorch/blob/9f7708eecb25268702ac107471fd13972fb4f5d2/torch/nn/modules/rnn.py#L41-L50">this check</a> will be missing.</p>
<p>I’m not too familiar with loading a model directly from a file, as this might have side effects, but I assume as long as you can successfully create an instance of your model, the workflow should still work.</p>

username: muammar
cooked: <p>Did this work for you?</p>
GPU is not fully used

username: tash
cooked: <p>My network cannot utilize the full GPU. Only 10% of the GPU was utilized when the batch size was 8.<br>
I increased the batch size from 8 to 16 and in that case it was giving me a Out of memory issue. Even for batch size 12, the network used only 10/12 % of the GPU.<br>
Any suggestions would be appreciated.</p>
<p>Thanks</p>

username: eqy
cooked: <p>It might be useful to check what the relative time spent processing data is, and if there are any overheads or preprocessing that can be done to this up.</p>

username: Epoching
cooked: <p>Yea as <a class="mention" href="/u/eqy">@eqy</a> mentioned, there could be an instance where you really are using your gpu for a decent amount of it’s capacity, but you don’t “see it” being used because most of the time there’s other things running (non-gpu).</p>
<p>In addition to checking for that, I’d also double check that things aren’t staying on the gpu longer then they should be! (maybe the batch size 12 case said the gpu was too much because as things keep running, more gpu space was getting utilized and not freed? not sure if you’re saying batch size 12 <em>instantly</em> errored out of space, but just an open-end thought</p>

username: tash
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9582709f1fb2656518c8081e6033879bee1f7474.png" data-download-href="https://discuss.pytorch.org/uploads/default/9582709f1fb2656518c8081e6033879bee1f7474" title="gpu usage"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9582709f1fb2656518c8081e6033879bee1f7474.png" alt="gpu usage" data-base62-sha1="lkCBMm57VbyfhQw3BE9aPVZR6vi" width="690" height="382" data-dominant-color="3B172F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu usage</span><span class="informations">716×397 48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I should have mentioned that my GPU has 8GB RAM and 6.2GB has been used. But on the right side, it is showing 12%is being used. Does that mean the network is computationally expensive but it has been using the GPU fully?</p>

username: Epoching
cooked: <p>I believe <code>Volatile GPU-Util</code> shows how <em>active</em> the GPU is while the 6.2 GB/8 GB is showing <em>how much</em> is on the GPU, according to <a href="https://medium.com/analytics-vidhya/explained-output-of-nvidia-smi-utility-fc4fbee3b124" rel="noopener nofollow ugc">Explained Output of nvidia-smi Utility | by Shachi Kaul | Analytics Vidhya | Medium</a></p>
<blockquote>
<p><em><strong>GPU-Util:</strong></em> It indicates the percent of GPU utilization i.e. percent of time when kernels were using GPU. <em>For instance</em> , output in table above shown 13% of the time. In case of low percent, GPU was under-utilised when if code spends time in reading data from disk (mini-batches).</p>
</blockquote>
<p>I’m guessing there’s a lot going on in the code besides stuff processing on the GPU!</p>
List of Embedding objects for Transformer

username: cinna
cooked: <p>I have a model where each input from a list of inputs gets processed by its own Embedding and the results are concatenated. The underlying Embedding objects don’t have to be consistent with each other in any way.<br>
I subclass my model from Transformer and store Embeddings in a list. But this list does not show up in transformer.named_parameters(), it doesn’t get transferred to CUDA when I transfer the transformer object to CUDA, it doesn’t really get used. Is there an alternative way to store these Embedding objects so that Transformer treats them correctly?</p>
<p>If I had a couple of them I could make each one a separate member of by subclass but in my case I need to store them in something like a list.</p>

username: ptrblck
cooked: <p>I guess you might have been using plain Python <code>list</code>s or <code>dict</code>s to store the embedding layers.<br>
If that case, use <code>nn.ModuleList/Dict</code> instead, which will make sure to properly register these modules and push them to the desired devices via the <code>to()</code> operation on the parent model.</p>

username: cinna
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="125330">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>nn.ModuleList</p>
</blockquote>
</aside>
<p>Thank you for the timely response. nn.ModuleList works perfectly here. If you have come across examples of nlp models with concatenated embeddings please let me know.</p>
Expanding part of a tensor

username: ohayonguy
cooked: <p>Hi<br>
I have a 4D tensor of shape NxCxHxW allocated on GPU memory.<br>
I want to take some of the elements (along the N dimension), and expand them without allocating new memory (i.e., repeating each element n times). I can do this for all elements as follows:<br>
<code>batch.unsqueeze(0).expand(n, -1, -1, -1, -1)</code><br>
Is there a way to do this but only to selected elements from the batch? (for instance, only for the first element of the batch, and still without allocating new memory).</p>
<p>Example for how to do this with memory allocation:<br>
<code>torch.cat((batch.unsqueeze(0), batch[:num_elems_to_expand].unsqueeze(0).expand(n, -1, -1, -1, -1)), dim=0)</code></p>
<p>Thanks</p>
Add support for `optimized` mode

username: luisasanmartin
cooked: <p>When debugging a model, it’s useful to add asserts and other similar logic to catch bugs. However, these asserts can be slow, so for production use cases, it’s useful to remove them all. Running with python -O will do that. However, when exporting a model, the asserts remain in the compiled graph even with the optimized flag turned on.</p>
<pre><code class="lang-auto">class DummyNet(torch.nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.conv = torch.nn.Conv2d(20, 20, 3)

    def forward(self, x):
        assert x.shape[1] == 20
        return self.conv(x)


def export():
    net = DummyNet()
    jit_net = torch.jit.script(net)
    print(jit_net.graph)
</code></pre>
<p>Produces this graph</p>
<pre><code class="lang-auto">graph(%self : __torch__.DummyNet,
      %x.1 : Tensor):
  %25 : str = prim::Constant[value="AssertionError: "]()
  %4 : int = prim::Constant[value=1]()
  %6 : int = prim::Constant[value=20]()
  %3 : int[] = aten::size(%x.1) # &lt;string&gt;:7:9
  %5 : int = aten::__getitem__(%3, %4)
  %7 : bool = aten::eq(%5, %6) 
   = prim::If(%7)
    block0():
      -&gt; ()
    block1():
       = prim::RaiseException(%25)
      -&gt; ()
  %13 : __torch__.torch.nn.modules.conv.Conv2d = prim::GetAttr[name="conv"](%self)
  %15 : Tensor = prim::CallMethod[name="forward"](%13, %x.1)
  return (%15)
</code></pre>

username: ptrblck
cooked: <p>That sounds like a good feature request.<br>
Would you mind creating the request on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> as well so that the code owners could take a look at it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: luisasanmartin
cooked: <p>Added the ticket here <a href="https://github.com/pytorch/pytorch/issues/60953" class="inline-onebox" rel="noopener nofollow ugc">Add support for Python's optimized mode to torchscript · Issue #60953 · pytorch/pytorch · GitHub</a></p>
Fork: function not implemented

username: Philip_Solomon_See
cooked: <p>Hi ,<br>
I code a distributed data parallel in pytorch (windows 10) , everything goes fine without error . When I load it in Linux , it give me and error “runtime error: fork: function not implemented”.<br>
This is how I code it</p>
<p>I used</p>
<p>for machine_rank in range(world_size):<br>
mp.spawn( train,args=(world_size,backend,c_param),nprocs=world_size,join=True)</p>
<p>then follow by:</p>
<p>def train(*args):<br>
if (args[3].training_type==1):<br>
torch.distributed.init_process_group(<br>
backend=args[2],<br>
init_method=r"file:///" +args[3].classifier+".log",<br>
world_size=args[1],<br>
rank=args[0]   )</p>
<p>I suspect the problem is in  join=True    in mp.spawn</p>
<p>any help will be much appreciated</p>
Tracing back the features that contribute most in an Autoencoder

username: kountaydwivedi
cooked: <p>Hi,<br>
I have created a Stacked Autoencoder to reduce dimensions of my dataset from ~20k to 100.<br>
Now, how could I track those original features that contribute most to these new space of 100 features?<br>
Thanks.<br>
:-}</p>