akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to speed up torch.load

username: vince62s
cooked: <p>At inference, when it comes to load / unload models very frequently, model loading time becomes crucial.</p>
<p>Right now, a decent model of 700MB takes up to 5 seconds to load from a SSD on a fast machine.</p>
<p>I was wondering if there could be some kind of optimization. (use of HDF5 for the model itself instead of pickle ?)</p>
<p>I would like to take it down to 1 second (as it is for Lua Torch …)</p>
<p>Cheers,<br>
Vincent</p>

username: suke
cooked: <p>do you resolve this problem?</p>
Deep residual nets traning

username: Ashish_Gupta1
cooked: <p>Hello everyone I am working on deep residual nets and I have to train it . Over shape net model. Now I don’t have much idea about traning these big nets.<br>
I have use batch norm,skip connection. In my nets.<br>
But my training loss started decreasing in start and stops after 3-4epochs.<br>
So what may be the reasons. I am using cross entropy with Adam as optimiser. learning rate=.001.</p>
Libtorch cmake error on Centos7

username: XiaXuehai
cooked: <p>Download libtorch from <a href="https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-latest.zip" rel="nofollow noopener">official website</a> without CUDA. Cmake version:2.8.12. Then <code>find_package(Torch REQUIRED)</code> and got an error:</p>
<pre><code class="lang-auto">[root@xxx build]# cmake .. -DCMAKE_PREFIX_PATH=/home/xxx/cnn/libtorch/
-- /home/xxx/cnn/libtorch
-- torch;caffe2_library;/home/for_faiss/crnn/libtorch/lib/libc10.so
-- /home/for_faiss/crnn/libtorch/include;/home/for_faiss/crnn/libtorch/include/torch/csrc/api/include
-- Configuring done
CMake Error at libtorch/share/cmake/Caffe2/public/threads.cmake:5 (add_library):
  Cannot find source file:

    INTERFACE

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx
Call Stack (most recent call first):
  libtorch/share/cmake/Caffe2/Caffe2Config.cmake:17 (include)
  libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:4 (find_package)


CMake Error in CMakeLists.txt:
  Cannot find source file:

    IMPORTED

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx


-- Build files have been written to: /home/for_faiss/crnn/build

</code></pre>

username: XiaXuehai
cooked: <p>Solved. Cmake 3.10 is fine.</p>
DQN with bitmap as input

username: Milky
cooked: <p>I am trying to implement the DQN algorithm from the example, however, the environment that I use has as states bitmaps. When I put my state in the training loop, I get error in:<br>
return policy_net(state).max(1)[1].view(1, 1), for choosing the next action that isn’t exploration (the random action). The error is:<br>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [16, 3, 5, 5], but got 2-dimensional input of size [300, 300] instead.<br>
My input is indeed [300,300], but it expects 4-dimensional weight [16.3.5.5]. I believe this is because of the get_screen function from the cart game using the BCHW format tensor, after transposing and unsqueezing the rgb_array from the screen. Since I don’t have rgb values but a bitmap I can’t have the BCHW format tensor. So I wanted to ask, how to best go about implementing a DQN algorithm. Is there perhaps another “skeleton” algorithm that I can use?</p>

username: ptrblck
cooked: <p>It looks like you would like to pass a grayscale image into the model.<br>
Based on the error message it also sounds like the first layer is an <code>nn.Conv2d</code> layer with <code>in_channels=3</code> and <code>out_channels=16</code>.<br>
If that’s the case, you should unsqueeze dim0 and dim0 for your input (conv layers expect an input of <code>[batch_size, channels, height, width]</code>), and change the number of input channels of the first conv layer to 1.</p>
<pre><code class="lang-python">input = input[None, None, :, :]
</code></pre>
<p>Let me know, if that helps.</p>
Can't optimize variable with Inception v3: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

username: Roman_Koshkin
cooked: <p>I am trying to create a generative network based on the pre-trained Inception_v3.</p>
<ul>
<li>I fix all the weights in the model</li>
<li>create a Variable whose size is (2, 3, 299, 299)</li>
<li>create targets of size (2, 1000) that I want my final layer activations to become as close as possible to by optimizing the Variable.</li>
</ul>
<p>And it gives me the error: «RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation».</p>
<p>This is strange, because if I do the same thing with VGG16, everything works:</p>
<pre><code class="lang-auto"># minimalist code with Inception_v3 that should work but throws the error:

import torch
from torch.autograd import Variable
import torch.optim as optim
import torch.nn as nn
import torchvision

torch.set_default_tensor_type('torch.FloatTensor')
Iv3 = torchvision.models.inception_v3(pretrained=True)
for i in Iv3.parameters():
    i.requires_grad = False

criterion = nn.CrossEntropyLoss()
x = Variable(torch.randn(2, 3, 299, 299), requires_grad=True)
target = torch.empty(2, dtype=torch.long).random_(1000)
output = Iv3(x)
loss = criterion(output[0], target)
loss.backward()
print(x.grad)

# ########the same code but with VGG16 works ##########################

import torch
from torch.autograd import Variable
import torch.optim as optim
import torch.nn as nn
import torchvision

torch.set_default_tensor_type('torch.FloatTensor')
vgg16 = torchvision.models.vgg16(pretrained=True)
for i in vgg16.parameters():
    i.requires_grad = False

criterion = nn.CrossEntropyLoss()
x = Variable(torch.randn(2, 3, 229, 229), requires_grad=True)
target = torch.empty(2, dtype=torch.long).random_(1000)
output = vgg16(x)
loss = criterion(output, target)
loss.backward()
print(x.grad)
</code></pre>
<p>Please help.</p>

username: xml94
cooked: <p>It seems that we are facing same questions.</p>
<p>There are really strange.</p>
<p>My question is : <a href="https://discuss.pytorch.org/t/why-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/45298">here</a></p>

username: xml94
cooked: <p>I managed to solve this question.</p>
<p>I guess that the models, Inception_v3 and VGG16, are a little bit different.</p>
<p>Please check whether there is any nn.ReLU(True) or nn.LeakyReLU(True). And you need change it from ‘Ture’ to ‘False’</p>
<p>good luck</p>

username: Roman_Koshkin
cooked: <p>Ok, when I create a model (it is called <code>Iv3</code>), I look inside by <code>list(Iv3.children())</code>. Surprisingly, I don’t see any ReLU or LeakyReLU whatsoever. Again, when I run <code>list(vgg16.children())</code>, I see the ReLU</p>

username: Roman_Koshkin
cooked: <p>Turns out the problem was due to Pytorch 1.0.0. No problem with Pytorch 0.4.1. though.</p>
Illegal memory access caused by clamp?

username: Wei_Wang
cooked: <p>I have created my own function with forward and backward. The msg says the error occurs in the backward pass with clamp operation. Is <strong>tensor.clamp(min=1.e-5)</strong> operation forbidden in the backward pass?</p>
<p>THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/generated/…/THCReduceAll.cuh line=317 error=77 : an illegal memory access was encountered<br>
Traceback (most recent call last):<br>
File “main.py”, line 347, in <br>
train(epoch)<br>
File “main.py”, line 209, in train<br>
loss.backward()<br>
File “/home/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 102, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
File “/home/anaconda3/lib/python3.7/site-packages/torch/autograd/function.py”, line 76, in apply<br>
return self._forward_cls.backward(self, *args)<br>
File “/home/pycharm/pytorch-cifar/torch_utils.py”, line 96, in backward<br>
denominator = torch.norm(M.mm(v_k)).clamp(min=1.e-5)<br>
File “/home/anaconda3/lib/python3.7/site-packages/torch/functional.py”, line 713, in norm<br>
return torch._C._VariableFunctions.frobenius_norm(input)<br>
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/generated/…/THCReduceAll.cuh:317</p>

username: tom
cooked: <p>Most likely, you’re seeing an earlier error. Use <a href="https://lernapparat.de/debug-device-assert/" rel="nofollow noopener">blocking launches</a> to get the location of the error.</p>
<p>Best regards</p>
<p>Thomas</p>
How to add additional cost to create_supervised_trainer?

username: JesseLivezey
cooked: <p>I’m using <code>ignite.engine.create_supervised_trainer()</code> and I would like to train a network using a crossentropy loss plus a function of the model parameters. I want something like</p>
<pre><code class="lang-auto">def create_supervised_trainer(model, optimizer, loss_fn,
                              model_loss=None,
                              output_transform=lambda x, y, y_pred, loss: loss.item()):

       def _update(engine, batch):
        model.train()
        optimizer.zero_grad()
        x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        if model_loss is not None:
            loss = loss + model_loss(model)
        loss.backward()
        optimizer.step()
        return output_transform(x, y, y_pred, loss)

    return Engine(_update)
</code></pre>
<p>Is there some builtin way to get something similar?</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/jesselivezey">@JesseLivezey</a> this can be done in a multiple way:</p>
<ol>
<li>You would like to reuse <code>create_supervised_trainer</code>, you can create a custom function with a signature (y_pred, y)</li>
</ol>
<pre><code class="lang-python">model = ...
loss_1 = nn.CrossEntropy()
model_loss = ...

def my_custom_loss(y_pred, y):
    return loss_1(y_pred, y) + model_loss(model)

trainer = create_supervised_trainer(model, optimizer, my_custom_loss)
</code></pre>
<ol start="2">
<li>You would like to reuse <code>create_supervised_trainer</code>, you can subclass a loss function as</li>
</ol>
<pre><code class="lang-python">class ComposedLoss(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.l1 = nn.CrossEntropy()
        self.model_loss = ... 
        self.model = model

    def forward(self, y_pred, y):
         return self.l1(y_pred, y) + self.model_loss(self.model)
        
</code></pre>
<ol start="3">
<li>You can directly create a trainer with you own update function</li>
</ol>
<pre><code class="lang-python">def update_fn(engine, batch):
        model.train()
        optimizer.zero_grad()
        x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        if model_loss is not None:
            loss = loss + model_loss(model)
        loss.backward()
        optimizer.step()
        return output_transform(x, y, y_pred, loss)

trainer = Engine(update_fn)
</code></pre>
<p>HTH</p>

username: JesseLivezey
cooked: <p>Thanks! That’s helpful.</p>
Distribution backward grad inspection not matching numerical reproduction

username: jfsantos.ds
cooked: <p>Hello,</p>
<p>I have implemented a loss based on the Negative Binomial Distribution probability mass function.<br>
I tried to validate this implementation by reproducing the gradients wrt. the distribution’s parameters.<br>
Unfortunately the obtained values do not match. I am unsure if it is my loss function implementation that is at fault since I believe that the numerical reproduction is correct according to sources:<br>
[<a href="http://vixra.org/pdf/1211.0113v1.pdf" rel="nofollow noopener">http://vixra.org/pdf/1211.0113v1.pdf</a>](source 1 - (Negative Binomial distribution PMF and likelihood gradient))<br>
or<br>
[<a href="https://en.wikipedia.org/wiki/Negative_binomial_distribution" rel="nofollow noopener">https://en.wikipedia.org/wiki/Negative_binomial_distribution</a>](source 2 - (Negative Binomial distribution PMF and likelihood gradient))</p>
<p>Here is the code snippet with an example and prints:</p>
<pre><code>import torch
from scipy.special import digamma
import numpy as np

output_soft = torch.tensor([[5],[0.1]],requires_grad=True) # model output 

def loss(input,target):
    total_count = input[0]
    probability = input[1]
    target_p_tc_gamma = torch.tensor([target + total_count],dtype = torch.float,requires_grad=True).lgamma()
    r_gamma = total_count.lgamma()
    target_factorial = torch.tensor([target + 1], dtype = torch.float).lgamma()
    combinatorial_term = torch.tensor([target_p_tc_gamma-r_gamma-target_factorial],dtype = torch.float,requires_grad = True).exp()
    prob_term = probability.pow(target)
    comp_prob_term = torch.tensor([1-probability],dtype = torch.float,requires_grad = True).pow(total_count)
    likelihood_target = combinatorial_term*prob_term*comp_prob_term
    return - likelihood_target.log()

target = torch.tensor([5.])
loss = loss(output_soft,target)
loss.backward()
print("Backward gradient inspection",output_soft.grad.detach().numpy())

def neg_PMF_gradient_check(input,target):
    target = target.detach().numpy()
    total_count = input[0].detach().numpy()
    probability = input[1].detach().numpy()
    dg = digamma
    comp_prob = 1-probability
    grad_tc = dg(target + total_count) - dg(total_count) + np.log((total_count)/(total_count + target))
    grad_prob = target*(1/probability) - total_count/(comp_prob)
    return [-grad_tc[0],-grad_prob[0]]

print("Numerical reproduction",neg_PMF_gradient_check(output_soft,target))
</code></pre>
<p>Any clues are very much appreciated!</p>
ImportError: ImportError: No module named torch.nn and No module named torchvision.transforms -

username: duygusar
cooked: <p>I have an old piece of code that I run on a virtual environment,</p>
<p>when I import torchvision.transforms in a python environment, it works fine, but when I run a python script that imports the same, from terminal, it gives this error:</p>
<p>ImportError: No module named torchvision.transforms</p>
<p>it was working fine before I switched to CUDA10.1 and then back to CUDA9.0 if that matters, but I don’t understand how it works just fine when I simply import in a python environment, but won’t work when I call a python script.</p>
<p>Same with torch.</p>

username: ptrblck
cooked: <p>Could you print the <code>sys.executable</code> in your Python environment and inside the script?<br>
If you can run it in a REPL session, but not as a script, maybe you are not using the current Python install/environment.</p>

username: duygusar
cooked: <p>Thank you, you are right, for some reason the script doesn’t run on the venv I run it on. It prints</p>
<blockquote>
<p>/usr/bin/python</p>
</blockquote>
<p>while in a session it prints the venv</p>
<blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>import sys<br>
print sys.executable<br>
/media/d/DATA_2/action-recognition/venv/bin/python</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p>It seems to be related to user/root settings</p>
Combine matrix multiplication and indexing to reduce memory usage

username: pimdh
cooked: <p>Hi!</p>
<p>Consider the following example:</p>
<pre><code class="lang-auto">n = 100
a = torch.randn(n, 1, 100)
b = torch.randn(3, 100, 101)
i = torch.randint(0, 3, size=100)
c = b[i]  # shape (n, 100, 101)
d = torch.bmm(a, c)  # shape (n, 1, 101)
</code></pre>
<p>Here, the <code>c</code> matrix is allocated an memory, which becomes prohibitively expensive if <code>n</code> becomes large. However, I could imagine that a CUDA kernel could be written that merges the indexing operation and the batched matrix multiplication, so that the <code>c</code> matrix is never allocated.</p>
<p>Does such a method already exist? Or will I have to write this CUDA kernel myself?</p>
<p>Thanks!</p>
Joblib with pytorch to parallelize sampling process

username: gcnn
cooked: <p>I tried to use Pytorch(v=0.4.1) together with Joblib in order to parallelize the sampling process in a RL task. Setting the seed for Pytorch, Numpy, and python wasn’t enough to get reproducible experiments. In other words, Pytorch yielded me different predictions for different runs of the same algorithm in the same task with the same seeds. I am wondering if there is any known issue between Pytorch and Joblib. With Multiprocessing the results are just fine.</p>
TorchSeg -- Fast, modular reference implementation and easy training of Semantic Segmentation algorithms in PyTorch

username: ycszen
cooked: <p>Hi all,<br>
We have released the <a href="https://github.com/ycszen/TorchSeg" rel="nofollow noopener">TorchSeg</a>.<br>
In this repo, we will release the fast, modular reference implementation of semantic segmentation algorithms in PyTorch.<br>
Now, this repo has implemented some state-of-art semantic segmentation algorithms: <a href="https://arxiv.org/abs/1612.01105" rel="nofollow noopener">PSPNet</a>, <a href="https://arxiv.org/abs/1804.09337" rel="nofollow noopener">DFN</a>, <a href="https://arxiv.org/abs/1808.00897" rel="nofollow noopener">BiSeNet</a>, <a href="https://link.springer.com/chapter/10.1007/978-3-030-01240-3_17" rel="nofollow noopener">PSANet</a>.<br>
More algorithms will be supported.</p>
<p><strong>Highlights</strong></p>
<ul>
<li>
<strong>Modular Design:</strong>  easily construct customized semantic segmentation models by combining different components.</li>
<li>
<strong>Distributed Training:</strong>   <strong>&gt;60%</strong>  faster than the multi-thread parallel method(<a href="https://pytorch.org/docs/stable/nn.html#dataparallel" rel="nofollow noopener">nn.DataParallel</a>), we use the multi-processing parallel method.</li>
<li>
<strong>Multi-GPU training and inference:</strong>  support different manners of inference.</li>
<li>Provides pre-trained models and implement different semantic segmentation models.</li>
</ul>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/ycszen/TorchSeg" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/6136494?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/ycszen/TorchSeg" target="_blank" rel="nofollow noopener">ycszen/TorchSeg</a></h3>

<p>Fast, modular reference implementation and easy training of Semantic Segmentation algorithms in PyTorch. - ycszen/TorchSeg</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Simultaneous Training and Validation on SWA

username: mitulraj
cooked: <p>Hi, I am using <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/" rel="nofollow noopener">Stochastic Weight Averaging from torch contrib</a> for the first time. I noticed that model weights are updated at the end of training by calling swap_swa_sgd method and we need to update Batch Normalization also at the end of training.</p>
<p>So I believe while training, I can not do validation simultaneously as weights are not updated. Is my understanding correct?<br>
If yes, is there any workaround to do simultaneous training and validation with SWA?</p>

username: ptrblck
cooked: <p><code>swap_swa_sgd</code> just swaps the current parameters for the swa ones in each call as you can see in the <a href="https://github.com/pytorch/contrib/blob/c8319c6f6b4c6da8f01246b2b19b1345fcba37ef/torchcontrib/optim/swa.py#L177" rel="nofollow noopener">source code</a>.<br>
If you would like to validate your model, you could call it once before the actual validation, and again afterwards to get your original parameters back.</p>
<p>I’m not sure if you can revert the batchnorm update.</p>
Complete distributions package

username: solsol
cooked: <p>Hi,<br>
How can I install distributions package for pytorch?</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch.distributions<br>
import torch.distributions.uniform<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
ModuleNotFoundError: No module named ‘torch.distributions.uniform’; ‘torch.distributions’ is not a package</p>
</blockquote>
</blockquote>
</blockquote>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using (<code>print(torch.__version__)</code>)?<br>
If you are using an older one, you could update it using the instructions on the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">website</a>.</p>

username: solsol
cooked: <p>I need to use version 0.3.1</p>

username: ptrblck
cooked: <p>A lot was added into PyTorch since <code>0.3.1</code>.<br>
Why do you have to use this old version?<br>
If you need help porting your code, feel free to post it here or create a new topic.</p>

username: solsol
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>By updating torch version to 0.4, I would face some errors in some parts of the code. Such:</p>
<blockquote>
<blockquote>
<blockquote>
<p>return x.cuda(self.primary_gpu, async=True)<br>
SyntaxError: invalid syntax</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>I could recommend to update to <code>1.1.0</code> and use the <code>.to(device, non_blocking=True)</code> method.<br>
Have a look at the <a href="https://github.com/pytorch/pytorch/releases/tag/v0.4.0" rel="nofollow noopener">release notes for 0.4</a> to see some necessary changes. Also the most recent release notes will be useful.</p>

username: solsol
cooked: <p>Thanks, anyway there is no way to use “distributions package” in pytorch 0.3?</p>

username: ptrblck
cooked: <p>There are some distributions in <code>0.3.1</code> as seen <a href="https://pytorch.org/docs/0.3.1/distributions.html?highlight=distributions#module-torch.distributions" rel="nofollow noopener">here</a>. However a lot of distributions were added after the <code>0.3.1</code> release.</p>
SGD has no attribute CyclicLR

username: Niklas_Hildebrandt
cooked: <p>Hi, I’m having issues with implementing the cyclical learning rate policy. When I my code, I get the error message that ‘SGD’ object has no attribute ‘CyclicLR’. I have checked to ensure that I have the nightly version. I have followed the example given in the docs,</p>
<p>optim = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)<br>
scheduler = torch.optim.CyclicLR(optim)</p>
<p>my import is as follows<br>
from torch.optim.lr_scheduler import StepLR, ExponentialLR, CyclicLR</p>
<p>Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Niklas_Hildebrandt
cooked: <p>Solved this myself, realized that i just needed to write CyclicLR(optim) without the torch.optim</p>
Using C++ frontend vs. Loading python model in C++

username: RuFAI
cooked: <p>Hello World,</p>
<p>could anyone explain me the advantage of loading a PyTorch model written in python in C++ instead of writing the model, data handling etc. directly in the PyTorch C++ frontend? At this moment the latter option makes much mor sense to me since it is less effort. What do you think??</p>
Loss from multiple GPUs

username: haowxu
cooked: <p>Hi guys I am trying to build a large classification model with a huge number of classes like training a word embedding. At the end of the model there is a linear layer with like 1 million outputs, followed by a softmax layer.</p>
<p>What I do is splitting the large linear layer into 8 independent layers with their weights on 8 GPUs respectively. The softmax scores can computed on the same GPUs through gathering sums. Then, what is the best way to do the backward?</p>
<p>My current solution is to implement a CrossEntropyLoss method manually by picking the softmax probabilities from different GPUs and do backward on this loss. This seems quite inefficient. Since computing the gradients can be quite simple, is it possible to feed the gradients directly rather than executing a ‘loss.backward()’?</p>
GPU memory debug

username: Dhorka
cooked: <p>Hi,<br>
I would like to ask some little advices for beginners debugging gpu usage. For instance, my network has this behaviour:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9833cde4703b412e2e95e3db6cbdc065043c606b.png" data-download-href="https://discuss.pytorch.org/uploads/default/9833cde4703b412e2e95e3db6cbdc065043c606b" title="gpu_memory"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9833cde4703b412e2e95e3db6cbdc065043c606b_2_690x190.png" alt="gpu_memory" data-base62-sha1="lIrz4jrASY3Clepkh9ufUXSHPir" width="690" height="190" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9833cde4703b412e2e95e3db6cbdc065043c606b_2_690x190.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9833cde4703b412e2e95e3db6cbdc065043c606b_2_1035x285.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9833cde4703b412e2e95e3db6cbdc065043c606b_2_1380x380.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu_memory</span><span class="informations">2008×554 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Overtime the gpu memory usage is increased a little bit, the memory usage was optained using the torch method torch.cuda.max_memory_allocated at the end of each epoch. I would like to know which are the recommended procedure to find memory leaks and so on in pytorch. I was thinking to use the garbage collector, but I am not pretty sure if it is the best way to tackle this problem. There is any tool that helps to solve this task?</p>
<p>Thanks!!</p>
How to convert tensor entry to a float or double?

username: arslansadiq
cooked: <p>Hi all,<br>
In C++ when we print a tensor like this:</p>
<pre><code class="lang-auto">torch::Tensor tensor = torch::zeros({10,1,7}, torch::dtype(torch::kFloat32));
std::cout&lt;&lt;output[i]&lt;&lt;'\n';
</code></pre>
<p>we get an output like this:</p>
<pre><code class="lang-auto">-0.3716 -0.6210 -0.4650 -0.2274  0.8657 -0.6912 -0.9256
[ Variable[CPUFloatType]{1,7} ]
</code></pre>
<p>here I am interested in only float values not the whole output along its datatype. I tried with accessing it with via array style access like <code>output_real[j][0][0]</code> even then it outputs datatype like this: <code>-0.371585 [ Variable[CPUFloatType]{} ]</code>. I just need to access the value, is there any way for that?</p>
<p>cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: EsdeathYZH
cooked: <p>Like this:</p>
<pre><code class="lang-auto">torch::Tensor tensor = torch::randn({3,4});
std::cout &lt;&lt; tensor[1][2].item&lt;double&gt;() &lt;&lt; std::endl;
</code></pre>

username: arslansadiq
cooked: <p>Thank you so much, it works <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
How do you a set a seed for weight initialization in the nn module?

username: pietromarchesi
cooked: <p>How do you set a seed for the random initialization of weights provided by the nn module?</p>

username: dgriff
cooked: <p>torch.manual_seed(seed)<br>
torch.manual_seed_all(seed)</p>
<p>or for cuda</p>
<p>torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)</p>

username: x0s
cooked: <p>Thanks for your answer.</p>
<p>There is no <code>torch.manual_seed_all(seed)</code> when dealing with CPU (check <a href="https://github.com/pytorch/pytorch/blob/master/torch/random.py" rel="nofollow noopener">source</a>)</p>
<p>If you want to seed <em>CPU</em> and <em>every GPU</em>, you can use <code>torch.manual_seed(seed)</code> (see <a href="https://github.com/pytorch/pytorch/pull/1762" rel="nofollow noopener">Merged Request</a>)</p>

username: sdv4
cooked: <p>This doesn’t seem to work. I did this and initialized two LSTMs. When I printed out the parameters for each, they were different.</p>
<p>Have you inspected parameters when using this method to verify that two different initializations result in the same outcome?</p>

username: ptrblck
cooked: <p>Did you set the seed right before instantiating both models?</p>
<pre><code class="lang-python">torch.manual_seed(seed)
modelA = MyModel()

torch.manual_seed(seed)
modelB = MyModel()
</code></pre>

username: sdv4
cooked: <p>Nope, but doing so worked. Thanks!</p>
Extra parenthesis at loss function

username: Ali_Mirzaeyan
cooked: <p>HI,<br>
can anyone explain what is the difference between these two pieces of code:</p>
<pre><code class="lang-auto">b = nn.MSELoss(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b
loss.backward()
</code></pre>
<pre><code class="lang-auto">b = nn.MSELoss()(output_x, x_labels)
a = nn.CrossEntropyLoss()(output_y, y_labels)
loss = a + b
loss.backward()
</code></pre>

username: JuanFMontesinos
cooked: <p>In the first case you are instantiating the class passing those parameters as arguments.<br>
In the 2nd case you are instantiating the class, calling the instance with two inputs and assigning output to b and a.</p>
<p>Does the 1st case even run?</p>

username: Ali_Mirzaeyan
cooked: <p>No the first one is not working. I’m a little bit confused with the “constructor” which will be used by <strong>init</strong> and the “forward” function in the pytorch classes. whenever we instantiate a class the constructor will run automatically. and when we call an instance of a class the “forward” function will run automatically. am I right ?</p>

username: JuanFMontesinos
cooked: <p>It’s a pythonic thing.</p>
<p>A class is a scheme of an object. When you instantiate a class, you are creating an object (an instance) which has specific properties. Constructor is a function called automatically during that process in which you can set up some variables, paramters etcetera…</p>
<p>Once you have that object, which has been defined in the <strong>init</strong> method, you can call it. By calling, I mean to use it as a function itself.</p>
<p>You could rewrite second funtion as</p>
<pre><code class="lang-auto">mse_loss_function = nn.MSELoss()
cross_entropy_loss_function=nn.CrossEntropyLoss()
b=  mse_loss_function(output_x, x_labels)
a= cross_entropy_loss_function(output_y, y_labels)
loss = a + b
loss.backward()
</code></pre>
<p>This does exactly the same than you snippet but following the logical way of coding. You instantiate the  class “MSE” and you use the object called mse_loss_function as a function.</p>
<p>But yeh, you are right, init is the constructor which instantiates and when you call the instance it runs forward</p>
How to make a cuda available using CUDA_VISIBLE_DEVICES?

username: chois_good
cooked: <p>What I’d like to do is</p>
<pre><code class="lang-auto">import os, torch

print(torch.cuda.is_available())  # True

os.environ["CUDA_VISIBLE_DEVICES"]="-1"
print(torch.cuda.is_available())  # False

os.environ["CUDA_VISIBLE_DEVICES"]=""
print(torch.cuda.is_available())  # True
</code></pre>
<p>But <code>os.environ["CUDA_VISIBLE_DEVICES"]=""</code> does not make cuda available again. How can I do that?</p>

username: Anna_Araslanova
cooked: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I use:</p>
<ul>
<li>os.environ[“CUDA_VISIBLE_DEVICES”]="" to switch to cpu mode</li>
<li>os.environ[“CUDA_VISIBLE_DEVICES”]=“0” to cuda mode, if I have only 1 GPU</li>
<li>os.environ[“CUDA_VISIBLE_DEVICES”]=“0,2,5” to use only special devices (note, that in this case, pytorch will count all available devices as 0,1,2 )</li>
</ul>

username: ptrblck
cooked: <p>Setting these environment variables inside a script might be a bit dangerous and I would also recommend to set them before importing anything CUDA related (e.g. PyTorch).</p>
<p>It looks as if you would like to mask the GPU dynamically inside your script.<br>
If that’s the case, I would rather use the <code>device</code> variable and write device-agnostic code than rely on <code>CUDA_VISIBLE_DEVICES</code>.</p>

username: Anna_Araslanova
cooked: <p>Thank you for the great response!</p>
<p>You are right, I hide devices, for example, to show tensorboard, then all GPUs are used (via <code>export CUDA_VISIBLE_DEVICES=""</code> in a console)</p>
<p>…I never fought about consequences of dynamic masking in the code, because didn’t have problems. Thanks</p>
Bert Embedings are same for all layers

username: Zoran_Peric
cooked: <p>I am using pretrained Bert model by Huggingface  from <a href="https://github.com/huggingface/pytorch-pretrained-BERT" rel="nofollow noopener">https://github.com/huggingface/pytorch-pretrained-BERT</a>  to get word emebedings by getting hiden states.</p>
<p>However it seems that the emebeding  vectors that I get are the same no matter which layer I choose and also it seems that for example Flair implementation gives different results, when doing cosine similarity between different words.</p>
<p>What I am doing wrong? Is there any processing I have to do with the hiddent states to convert them to vectors. Maybe normalising or something like this.</p>
<pre><code class="lang-auto">   tokens_tensor_1 = torch.tensor([indexed_tokens_1])
    segments_tensors_1 = torch.tensor([segments_ids_1])

    
    tokens_tensor_1 = tokens_tensor_1.to('cuda')
    segments_tensors_1 = segments_tensors_1.to('cuda')

    with torch.no_grad():
        hidden_states_1, _  = model(tokens_tensor_1)#, segments_tensors_1)

    
    print ("tokenized_text_1:",tokenized_text_1)   
    vectors = []
    for index in range(0, len(tokenized_text_1)):
        torch_vector = hidden_states_1[0][0][index]
        torch_vector = torch_vector.to('cpu')
        numpy_vector = torch_vector.numpy()
        vectors.append(numpy_vector)`
</code></pre>

username: Zoran_Peric
cooked: <p>I was lucky and found that normalising was the problem. So I did vector/np.linalg.norm(vector) and now all layers gives different cosine similarity between words.</p>
<p>And then I get the last layer by using:<br>
torch_vector = hidden_states_1[-1][0][index]</p>
Cosine similarity between 2 vectors

username: Zoran_Peric
cooked: <p>I am really suprised that pytorch function nn.CosineSimilarity is not able to calculate simple cosine similarity between 2 vectors. How do I fix that?</p>
<p>vector: tensor([ 6.3014e-03, -2.3874e-04,  8.8004e-03,  …, -9.2866e-09,<br>
-3.9112e-05,  2.2280e-03])<br>
vector1: tensor([ 6.3014e-03, -2.3874e-04,  8.8004e-03,  …, -9.2866e-09,<br>
-3.9112e-05,  2.2280e-03])</p>
<p>cos = nn.CosineSimilarity()<br>
print (cos(vector,vector1))</p>
<p>I get error:<br>
Traceback (most recent call last):<br>
File “I:\software1\SpellChecker\Bert_embeding.py”, line 188, in <br>
print (cos(vector,vector1))<br>
File “C:\Users\pera\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\pera\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\distance.py”, line 80, in forward<br>
return F.cosine_similarity(x1, x2, self.dim, self.eps)<br>
RuntimeError: The size of tensor a (1024) must match the size of tensor b (0) at non-singleton dimension 0</p>

username: Zoran_Peric
cooked: <p>It  works when i put cos = nn.CosineSimilarity(dim=0)</p>

username: vimalthilak
cooked: <aside class="quote no-group" data-username="Zoran_Peric" data-post="1" data-topic="45167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zoran_peric/48/10136_2.png" class="avatar"> Zoran_Peric:</div>
<blockquote>
<p>CosineSimilarity</p>
</blockquote>
</aside>
<p>I think you figured out the answer yourself.</p>
<p>The documentation ( <a href="https://pytorch.org/docs/master/nn.html#cosinesimilarity" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch master documentation</a> ) makes it clear that the default dim is 1.</p>
How to debug: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time?

username: LeanderK
cooked: <p>I was changing my model and encountered a  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time, but the error message is not very useful.</p>
<p>I have a normal training-loop and don’t use LSTMs or something, so this should not happen.</p>
<p>is there any way to find out which part of the model was responsible?</p>

username: MariosOreo
cooked: <p>There might be something calculate globally, and when you <code>.backward()</code> first time, it breaks the computation graph, and the second time you call <code>.backward()</code>, it still need the ‘global’ variable to compute gradient, but it has beed freed (as the error raised). So checking your code, whether there are some variables in computation graph globally.</p>

username: LeanderK
cooked: <p>Ok. My code is quite complicated, so this is easier said than done <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: LeanderK
cooked: <p>Is there really no way to get the offending calculation?</p>
<p>I’ve stripped down my network, but it’s still occurring…Like I said, my architecture is quite complicated and there are quite a few moving parts. So there are still a lot of places that could be responsible.</p>
<p>I bet the error is just a stupid mistake somewhere deep in my code</p>

username: LeanderK
cooked: <p>ok, I found the perpetrator. But I am not sure why it happens…</p>
<p>This should close the question. Althought it does not really help anyone.</p>
<p>I just uncommented code everywhere.</p>
A small tensor occupy 600MiB on GPU?

username: kangkang_li
cooked: <p>I just put a small <code>3*4 tensor</code> to <code>GPU0</code>, why the memory on <code>GPU0</code> increases<br>
from <code>0</code> to <code>600</code> so rapidly? It’s too high.How to compute it?(just forget the GPU1 , I am using it.) Is there anything wrong with my server?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0da36f7ad2174128c02177bd50381865754d4250.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0da36f7ad2174128c02177bd50381865754d4250" title="8C14444C-4ED2-4F1B-9498-F16937D4AB81"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0da36f7ad2174128c02177bd50381865754d4250_2_690x241.jpeg" alt="8C14444C-4ED2-4F1B-9498-F16937D4AB81" data-base62-sha1="1WEmGRQJgXs104oW2eNjraGfbRC" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0da36f7ad2174128c02177bd50381865754d4250_2_690x241.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0da36f7ad2174128c02177bd50381865754d4250_2_1035x361.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0da36f7ad2174128c02177bd50381865754d4250_2_1380x482.jpeg 2x" data-dominant-color="333130"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">8C14444C-4ED2-4F1B-9498-F16937D4AB81</span><span class="informations">2284×800 336 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Geeks_Sid
cooked: <p>That’s probably the initialization of CUDA and CuDNN.</p>

username: kangkang_li
cooked: <aside class="quote no-group" data-username="Geeks_Sid" data-post="2" data-topic="45128">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geeks_sid/48/10013_2.png" class="avatar"> Geeks_Sid:</div>
<blockquote>
<p>the initialization of CUDA and CuDNN</p>
</blockquote>
</aside>
<p>what should I do? modify my code or do something on my server?</p>

username: Geeks_Sid
cooked: <p>No, it is always going to consume that bare minimum. Atleast that’s what I have noticed.</p>
Slightly changing model causes 'modified by inplace operation' error

username: whoab
cooked: <p>Previous code:<br>
return n_vals</p>
<p>I added one line:<br>
n_vals[:,1,:] = n_vals[:,1,:] * torch.tensor(2)</p>
<p>And now get:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
<p>I don’t understand this error because the line I added is not in place.</p>
<p>Another potential cause of the error is because I am preloading the model’s weights from disk and changing the model (with the above line) after preloading the weights. Still, doesnt seem like it should cause this error…</p>

username: whoab
cooked: <p>The answer is that assigning the variable with slice notation is an inplace operation. The solution is to add a clone call:<br>
n_vals[:,1,:] = n_vals[:,1,:].clone() * torch.tensor(2)</p>
Try to build cpp extension on Win10 but got "fatal error C1021: invalid preprocessor command 'warning'"

username: JiangPQ
cooked: <p>I followed the instruction of <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this</a> to build a custom cpp extension.</p>
<p>The extension code works fine on Ubuntu but fails on Win10.<br>
When I run <code>python setup.py install</code> it gives bunches of warnings and an error like below:</p>
<blockquote>
<p>C:\Users\jiang\Miniconda3\envs\pytorch\lib\site-packages\torch\lib\include\torch\csrc\api\include\torch/torch.h(7):<br>
<strong>fatal error C1021: invalid preprocessor command ‘warning’</strong><br>
error: command ‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe’ failed with exit status 2</p>
</blockquote>
<p>I didn’t paste those warnings because there are too many.</p>
<p>Anyone knows how to fix this?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I think your Visual Studio might be too old.<br>
<a class="mention" href="/u/peterjc123">@peterjc123</a> suggested in <a href="https://discuss.pytorch.org/t/xstddef-372-error-c-2338-the-c-standard-doesnt-provide-a-hash-for-this-type/31484/3?u=ptrblck">another thread</a> to update VS to 15.3.</p>

username: dashesy
cooked: <p>pragma warning with VS2017</p>
<blockquote>
<p>C:\Users\tobai\AppData\Local\conda\conda\envs\IRIS30\lib\site-packages\torch\lib\include\torch\csrc\api\include\torch/torch.h(7): fatal error C1021: invalid preprocessor command 'warning’error: command ‘C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\HostX86\x64\cl.exe’ failed with exit status 2</p>
</blockquote>

username: peterjc123
cooked: <p>Looks like <code>#warning</code> preprocessor command is invalid in MSVC. I’m find a portable way to do this.</p>

username: peterjc123
cooked: <aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/20484" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/peterjc123" target="_blank" rel="nofollow noopener">
    <img alt="peterjc123" src="https://avatars3.githubusercontent.com/u/9998726?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/pull/20484" target="_blank" rel="nofollow noopener">Portable way of the warning clause</a>
</h4>

<div class="date">
  by <a href="https://github.com/peterjc123" target="_blank" rel="nofollow noopener">peterjc123</a>
  on <a href="https://github.com/pytorch/pytorch/pull/20484" target="_blank" rel="nofollow noopener">10:26AM - 14 May 19 UTC</a>
</div>

<div class="github-commit-stats">
  <strong>1 commits</strong>
  changed <strong>1 files</strong>
  with <strong>9 additions</strong>
  and <strong>1 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to exclude a model parameter from Model.parameters() without setting its required_grad to False

username: klywa
cooked: <p>I want to do this since I need two optimizers to take care of different model parameters.</p>
<p>for example, my model has many layers and an architecture parameter model.arch:</p>
<pre><code class="lang-auto">class model(nn.Module):
  def __init__(self):
    ...
    self.linear_layers = nn.ModuleList()
    for i in range(4):
      self.linear_layers.append(nn.Linear(...))
    # many other layers
    self.arch = torch.zeros([...], requires_grad=True)
  
  def get_arch_parameters(self):
    return [self.arch]

  def get_other_parameters(self):
     # I want to return all parameters other than self.arch
</code></pre>
<p>Now I want to use two optimizers to update different parameters:</p>
<pre><code class="lang-auto">opt1 = torch.optim.SGD(model.get_other_parameters(), ...)
opt2 = torch.optim.SGD(model.get_arch_parameters(), ...)
</code></pre>
<p>However, I do not know how to implement model.get_other_parameters(), which returns all model parameters but model.arch.<br>
I cannot simply set model.arch.requries_grad=False and use model.parameters() instead, since opt2 needs to update it.<br>
Hence I’d like to know if there is any method to exclude a parameter from model.parameters() without setting its requires_grad to False.</p>
<p>Thanks in advance.</p>

username: acobobby
cooked: <p>Can you simply return the <code>ModuleList</code>? Like <code>return self.linear_layers</code>. If I understood correctly these layers are not in the <code>arch</code> variable so it should work.</p>

username: klywa
cooked: <p>Hi Andrea,</p>
<p>Thanks. You inspired me.</p>
<p>Though it will not work to just return a list of layers since the <code>parameter</code> that an optimizer take should be an iterator of tensors, you remind me that each layer is actually a <code>Module</code> so it has its own <code>parameters</code> method. And I found that a <code>ModuleList</code> also has a <code>parameter</code> method.</p>
<p>So I just wrote:</p>
<pre><code class="lang-auto">def get_other_parameters(self):
  return itertools.chain(self.linear_layers.parameters(), other.parameters(), ...)
</code></pre>
<p>And seemingly it works.</p>
<p>However, it’s still a little clumsy since I need to enumerate all the parameters I want to optimize. In cases where the number of parameters is huge while I only want to exclude several of them, it is neither efficient nor elegant.</p>
<p>Hence I still want to know if there is a better solution.</p>
Spacial Transformer Network and Pre-Trained Model

username: rhyang6
cooked: <p>Hi,</p>
<p>Is there any way to add a spacial transformer network (seen in the tutorial: <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html</a>) to a pre-trained model? or do we have to copy the architecture of the pretrained model in order to add it?</p>
<p>Thanks</p>