akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Net in DataParallel make training aware quantization convert model acc error

username: dingyongchao
cooked: <p>I run net train and test in single GPU is good, and I try to train model in multi-gpus, and I found that the process of training is good, however, when I convert model into quantization, and the model acc is 1% in CIFAR100 which means random.<br>
My convert code is as follows:</p>
<pre><code class="lang-auto">126     if gpus == 1:
127         quantized_model = torch.quantization.convert(net.eval().to(device), inplace=False)
128     else:
129         print("expert model from torch DataParallel")
130         single_net = deepcopy(net.module)
131         #single_net = net
132         quantized_model = torch.quantization.convert(single_net.eval().to(device), inplace=False)
133         #quantized_model = quantized_model.module
134     quantized_model.eval()
</code></pre>
<p>if gpu number is more than one, I use net = torch.nn.DataParallel(net), so when I need to convert it, I first do single_net = deepcopy(net.module), but the acc of quantized_model is 1% , well the training acc is about 77%. Meanwhile,  it has warning:</p>
<blockquote>
<p>/home/autolab/anaconda3/lib/python3.7/site-packages/torch/quantization/observer.py:208: UserWarning: Must run observer before calling calculate_qparams.                           Returning default scale and zero point.<br>
Returning default scale and zero point.")</p>
</blockquote>
<p>So if anyone has any ideas?</p>

username: supriyar
cooked: <p>There are currently some issues with nn.DataParallel and Quantization Aware Training. There is a WIP PR to fix it - <a href="https://github.com/pytorch/pytorch/pull/37032" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/37032</a><br>
You can follow the toy example here to make sure you’re following the steps for QAT correctly <a href="https://gist.github.com/vkuzo/78b06c01f23f98ee2aaaeb37e55f8d40" rel="nofollow noopener">https://gist.github.com/vkuzo/78b06c01f23f98ee2aaaeb37e55f8d40</a></p>

username: dingyongchao
cooked: <p>Thanks， I think I has solve this problem by follows:</p>
<pre><code class="lang-auto">178     test(net.module, epoch, device, args.gpus)
179     qat_model = deepcopy(net)
180     qat_model.eval().to('cpu')
181     torch.quantization.convert(qat_model, inplace=True)
182     test(qat_model.module, epoch, 'cpu', args.gpus)
183     scheduler.step()
</code></pre>
<p>I need to run net.module first, then the qat_model in cpu is correct</p>
How to train a model that similar to another model

username: J.H_Y
cooked: <p>Hi, I have a question about training a model that similar to another model.<br>
For example, I have a classification model A. which output is</p>
<pre><code class="lang-auto">outputA = modelA (input), the dimension is [batchsize, nclass]
</code></pre>
<p>I also have another classification model B, the output dimension is the same as model A.<br>
How can I train the model B that make its output similar to modelA?</p>
<p>I have tried the L1Loss to calculate the loss of two outputs and make it similar. But it does not works.<br>
Any ideas about this area?  Thank you so much. Appreciate it.</p>
No kernel image is available for execution on the device pytorch1.1 compiled from source

username: Dhorka
cooked: <p>Hi,</p>
<p>I have the following problem:</p>
<blockquote>
<p>THCudaCheck FAIL file=/home/dhhorka/libs/pytorch/aten/src/THC/THCTensorMathCompareT.cuh line=69 error=48 : no kernel image is available for execution on the device.</p>
</blockquote>
<p>I compiled from source pytorch 1.1 (using the code tagged as v1.1.0). The thing is, I am running this code in a remote server where there are two different nodes with different gpus architectures. I compiled the code in the node that contains old gpus . WhenI execute the pytorch script on these GPU’s the code code is working properly but… when I execute this code on the new gpus (2080ti) I get the previous error.</p>
<p>I tried to compile pytorch in the new gpus and then the code It is working on the new gpus but it does not work in the old gpus… I am using cuda 10. Is there anyway to make it work in both GPUs?</p>
<p>P.S: I do not put a category to this topic because I am not pretty sure witch one fits better in this case.</p>

username: agrija9
cooked: <p>Hi <a class="mention" href="/u/dhorka">@Dhorka</a>, I am having this issue too. Were you able to solve it or find help somewhere else?</p>

username: ptrblck
cooked: <p>This error is raised, if your PyTorch binary or source build wasn’t compiled with the right compute capability for your GPU.</p>
<p>Which GPU are you using and how did you install / build PyTorch?</p>

username: Dhorka
cooked: <p>Is there to build pytorch from source for more than one gpu capability? The thing is that I have two different gpus with different gpu capabilities…</p>

username: hartbx
cooked: <p>You can build for multiple GPU architectures at once by setting <code>TORCH_CUDA_ARCH_LIST</code>, for example we build with:</p>
<pre><code class="lang-auto">export TORCH_CUDA_ARCH_LIST="3.7;6.0;7.0;7.5"
</code></pre>
<p>Wikipedia has a good map of adapters to compute versions: <a href="https://en.wikipedia.org/wiki/CUDA" rel="nofollow noopener">https://en.wikipedia.org/wiki/CUDA</a></p>

username: ptrblck
cooked: <p>The binaries should are built with the same architecture list as posted by <a class="mention" href="/u/hartbx">@hartbx</a>, if I’m not mistaken.</p>

username: agrija9
cooked: <p>I am using an Nvidia Tesla v100.</p>
<p>Thanks for pointing out the error. This is what happened to me indeed.</p>
<p>More specifically, I was running a script on a remote cluster using GPUs and retrieving some GPU tensors. While trying to read this tensors in my local computer (which only supports CPU tensors) I encountered this error.</p>
<p>I’m now converting these tensors while on the cluster.</p>
<p>Thanks a lot!</p>
Inplace Fill Starting From a Specific Index

username: Faria_Huq
cooked: <p>I have a 1D tensor of a fixed dimension 11. I want to fill each position with a certain value starting from an index. For example,</p>
<blockquote>
<p>x = Tensor([1,2,3,4,5,6,7,8,9,10,11])</p>
</blockquote>
<p>I want to replace each value with -1 starting from index 7 to get a result,</p>
<blockquote>
<p>x = Tensor([1,2,3,4,5,6,7,-1,-1,-1,-1])</p>
</blockquote>
<p>How can I attain this easily?</p>

username: LeviViana
cooked: <p>This way:</p>
<pre><code class="lang-python">import torch
x = torch.Tensor([1,2,3,4,5,6,7,8,9,10,11])
x[7:].fill_(-1)
print(x)
</code></pre>

username: Faria_Huq
cooked: <p>Thank you so much for the help!</p>
How to add 3 Fully Connected layers to a ResNet50 pretrained on Imagenet?

username: pulpaul
cooked: <p>Let me explain myself. I have to train a preTrained (on Imaganet) ResNet50 network, but instead of having one last Fc layer of 1000 I need to add 3 more FC layers (1000-&gt;128,128-&gt;64 and 64-&gt;10). I do this because i want train with a custom dataset (10 classes of my making). My code is this:</p>
<pre><code class="lang-auto">resnet50 = models.resnet50(pretrained=True)
resnet50.fc = nn.Linnear(
    nn.Linear(2048, 1000),
    nn.Linear(1000, 128),
    nn.Linear(128, 64),
    nn.Linear(64, 10)
)
</code></pre>
<p>But i dont know if I have to use a ReLU() after the last FC (nn.Linear(64, 10)).</p>
<p>Thanks for your help.</p>

username: LeviViana
cooked: <p>Successive linear layers may not be very effective: <em><strong>(i)</strong></em> everything still just a linear operation, and <em><strong>(ii)</strong></em> it adds a lot of parameters. Generally we add differentiable non-linear operations between linear layers (sigmoid, tanh, relu -&gt; which is differentiable almost everywhere, etc).</p>
<p>Regarding the last layer, since you are dealing with a classification problem, you want to get a distribution as output. So, generally the last layer will output the logits, therefore no ReLU would be applied at the end.</p>
Source install GPU mac CUDA

username: pyroll
cooked: <p>Hello all,<br>
im following this:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://medium.com/@enactdev/compiling-nvidia-cuda-enabled-pytorch-on-a-2012-macbook-pro-385fb5963079" target="_blank" title="11:39AM - 19 August 2019" rel="nofollow noopener">Medium – 19 Aug 19</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://medium.com/@enactdev/compiling-nvidia-cuda-enabled-pytorch-on-a-2012-macbook-pro-385fb5963079" target="_blank" rel="nofollow noopener">Compiling NVIDIA CUDA-enabled pytorch on a 2012 Macbook Pro</a></h3>

<p>Getting pytorch to compile with CUDA support requires having the right libraries and packages installed. These steps should work on any…</p>

  <p><span class="label1">Reading time: 5 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I get all the way to the end however, once I try<br>
“torch.cuda.is_available()”</p>
<p>i get a painful False…</p>
<p>Id like to confirm, how should my bash_profile look like? any other suggestions so I can get a True?</p>
<p>Please see the link to log file, as its quite verbose</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/images/favicon-vflUeLeeY.ico" class="site-icon" width="16" height="16">
      <a href="https://www.dropbox.com/s/24a591habk72dpk/pytorch.txt?dl=0" target="_blank" rel="nofollow noopener">Dropbox</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-txt-large.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://www.dropbox.com/s/24a591habk72dpk/pytorch.txt?dl=0" target="_blank" rel="nofollow noopener">pytorch.txt</a></h3>

<p>Shared with Dropbox</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Cheers</p>
DataParallel - How to collect data from each gpu?

username: HaziqRazali
cooked: <p>Is there a function that I can use to manually collect tensors from all the gpu into one common gpu ? Like if my batch size is 16 and I am using 4 GPUs then the output of the network has a batch size of 4. How can I <strong>manually</strong> collect the output from each thread so that I now have a tensor that has a batch size of 16 ?</p>
<p>And how do I make sure the results are in the same order that they are split ?</p>
<p>I looked through <a href="https://pytorch.org/docs/stable/nn.html?highlight=nn%20dataparallel#torch.nn.DataParallel" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=nn%20dataparallel#torch.nn.DataParallel</a> but could not find anything.</p>
<p>For example</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def forward(data):
        datalist = []
        for x in data:
            datalist.append(x)
    return torch.stack(datalist)

net = torch.nn.DataParallel(Model())
datalist = net(data)

</code></pre>
<p>but datalist does not have the same shape as data and even if I reshape datalist to have the same shape as data the values are no longer equal because the <code>append</code> does not come in order i.e. gpu0, gpu1, gpu2, gpu3, etc. How can I fix this ?</p>

username: MachT
cooked: <p>I have the same problem. Could someone provide any pointers?</p>
Why is PyTorch called so?

username: debvrat
cooked: <p>I see that PyTorch is based on the Torch library. What is the etymology of the word ‘Torch’ here? Why was that library called so?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think I ever saw an explanation from Ronan about that.<br>
You can check the old iterations on Ronan’s website though: <a href="http://torch.ch/torch3/">http://torch.ch/torch3/</a></p>
Plot the Decision Boundary of a Neural Network in PyTorch

username: Lightning_Blade
cooked: <p>I’ve been trying to plot the decision boundary of my neural network which I used for binary classification with the sigmoid function in the output layer but gor one error after another, I found many posts discussing the plotting of the decision boundary of a scikit-learn classifier but not a neural network built in PyTorch. Below is my neural network:</p>
<pre><code>class NeuralNetwork(torch.nn.Module):
  def __init__(self):
  super(NeuralNetwork, self).__init__()
  self.fc1 = torch.nn.Linear(23, 16)
  self.fc2 = torch.nn.Linear(16, 14)
  self.fc3 = torch.nn.Linear(14, 10)
  self.fc4 = torch.nn.Linear(10, 5)
  self.fc5 = torch.nn.Linear(5, 1)

def forward(self, x):
  x = torch.relu(self.fc1(x))
  x = torch.relu(self.fc2(x))
  x = torch.relu(self.fc3(x))
  x = torch.relu(self.fc4(x))
  x = torch.sigmoid(self.fc5(x))
  return x

model = NeuralNetwork().double()

CUDA = torch.cuda.is_available()
if CUDA:
 model.cuda()

criterion = torch.nn.BCELoss(reduction='mean')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)

model_1.train()

Precision = []
Cost = []

for epoch in range(10001):

  if CUDA:
    inputs = X_train.cuda()
    label = Y_train.cuda()
else:
  inputs = X_train
  label = Y_train

prediction = model_1(inputs)
loss = criterion(prediction, label)
accuracy = ((prediction &gt; 0.5) == label).float().mean().item()

Cost.append(loss.item())
Precision.append(accuracy)

if epoch % 1000 == 0 or epoch == 30000:
  print("Epoch:", epoch, ",", "Loss:", loss.item(), ",", "Accuracy:", accuracy)

# Backpropagation process
optimizer.zero_grad()
loss.backward()
optimizer.step()

model_1.eval()

X_test = torch.from_numpy(X[27000:,:])
Y_test = torch.from_numpy(y[27000:,:]).double()

with torch.no_grad():

 y_pred = model_1(X_test)
print("Accuracy: ", ((y_pred &gt; 0.5) == Y_test).float().mean().item())
</code></pre>
<p>As you can see my neural network has 23 features and I plan to choose only two features in my plot: X_test[:, 0] and X_test[:, 5].<br>
I would greatly appreciate your help, thanks in advance.</p>

username: Miguel_Chaparro
cooked: <p>hello, did you had any advances on implementing decision boundary?, I’m interested in the same topic</p>
Does PyTorch require an nVidia GPU card?

username: a.vill
cooked: <p>Good day, I’m currently doing R&amp;D on image processing, and I stumbled upon an example that uses PyTorch. I’m currently in the process of installing PyTorch, and I’m wondering does PyTorch need an nVidia GPU? I’ve seen other image processing code that require CUDA, but CUDA requires an nVidia card to work. Right now, I’m on a MacBook pro and I have no access to a desktop with an nVidia card. I couldn’t exactly find a “System Requirements” page. So any help is very much appreciated. Thank you.</p>

username: SimonW
cooked: <p>Some Mackbook Pro comes with nvidia GPUs. To check, see “About this mac” from the top left apple button. Even if you don’t have an nvidia GPU, you can still run pytorch in cpu only mode.</p>

username: a.vill
cooked: <p>I understand, my MBP uses an Intel GPU. Does that mean I should select “None” in the CUDA version when picking the command to download PyTorch?</p>

username: mratsim
cooked: <p>PyTorch can work in CPU mode as well, it will be (much) slower though.</p>
<p>In general there is very low support for OpenCL in any deep learning frameworks (Tensorflow, Theano, Chainer, Dynet, Mxnet, Caffe…), because Nvidia provided very efficient deep learning primitives. Researchers and library developers built on top of that. AMD/Intel are still playing catchup.</p>
<p>You can find some port of popular frameworks on github, especially by <a href="https://github.com/hughperkins" rel="nofollow noopener">Hugh Perkins</a> but you will probably spend a lot of time compiling from source, patching in upstream updates, etc. And then you will realize that you only have basic layers/loss functions available.</p>

username: richard
cooked: <p><a class="mention" href="/u/a.vill">@a.vill</a> To answer your question, if your computer says you have an Intel GPU then yes you should select “None” for the CUDA version.</p>

username: manish_singla
cooked: <p><a class="mention" href="/u/richard">@richard</a> my system has Intel GPU and after selecting “None”, it still throws below error:<br>
File “/home/****/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 101, in _check_driver<br>
<a href="http://www.nvidia.com/Download/index.aspx" rel="nofollow noopener">http://www.nvidia.com/Download/index.aspx</a>""")<br>
AssertionError:<br>
Found no NVIDIA driver on your system. Please check that you<br>
have an NVIDIA GPU and installed a driver from</p>

username: richard
cooked: <p>What command did you use to install pytorch?</p>
JIT Compiling Extensions

username: MauroPfister
cooked: <p>Hi everyone,</p>
<p>I’m trying to use the deformable convolutions cpp extensions from the <a href="https://github.com/open-mmlab/mmdetection" rel="nofollow noopener">mmdetection repo</a> without a <code>setup.py</code> but by compiling them just in time with <code>torch.utils.cpp_extension.load()</code> as suggested <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">here</a>. However, I’m having some trouble giving the <code>load()</code> function the correct path.</p>
<p>My folder structure is as follows:</p>
<pre><code class="lang-auto">├── dcn
│   ├── deform_conv.py
│   ├── deform_pool.py
│   ├── __init__.py
│   └── src
│       ├── deform_conv_cuda.cpp
│       ├── deform_conv_cuda_kernel.cu
│       ├── deform_pool_cuda.cpp
│       └── deform_pool_cuda_kernel.cu
└── test_dcn.py
</code></pre>
<p>In <code>test_dcn.py</code> I import the deformable convolutions with <code>from dcn import DeformConvPack</code> and in the file <code>deform_conv.py</code> I inserted the following at the top:</p>
<pre><code class="lang-auto"># deform_conv.py

from torch.utils.cpp_extension import load
deform_conv_cuda = load(name='deform_conv_cuda', sources=['src/deform_conv_cuda.cpp', 'src/deform_conv_cuda_kernel.cu'])

# Rest of the code
# ...
</code></pre>
<p>I expected this to work but the compilation fails because the <code>.cpp</code> and <code>.cu</code> files cannot be found. If I instead specify the sources as <code>'dcn/src/deform_conv_cuda.cpp'</code> and <code>'dcn/src/deform_conv_cuda_kernel.cu'</code> it works.</p>
<p>Could someone explain me the logic behind this? Thank you very much =)</p>

username: googlebot
cooked: <p>Basically, a working directory of process and python’s path used with module import are distinct things; as compilation is about creating child processes, and module with load() lives in entirely different directory, the former directory is used as base.</p>

username: MauroPfister
cooked: <p>Thanks for the answer! In the mean time I came to the same conclusion and used the following snippet as a workaround. That way I can import the <code>nn.Module</code> defined in <code>deform_conv.py</code> from anywhere I want without having to worry about getting the path right. I’m just a bit surprised, that the <code>torch.utils.cpp_extension.load()</code> function doesn’t take care of that.</p>
<pre><code class="lang-auto"># deform_conv.py

import os
from torch.utils.cpp_extension import load
parent_dir = os.path.dirname(os.path.abspath(__file__))
sources = ['src/deform_conv_cuda.cpp', 'src/deform_conv_cuda_kernel.cu']  # Paths of sources relative to this file
abs_sources = [os.path.join(parent_dir, source) for source in sources]  # Absolute paths of sources
deform_conv_cuda = load(name='deform_conv_cuda', sources=abs_sources)  # JIT compilation of extensions
</code></pre>
Custom Image Dataset good practice? (Working Memory Issues)

username: mayool
cooked: <p>Hello everyone!<br>
I am creating my own custom image dataset using torchs Dataset class.<br>
So far, I iterate through all .jpg files in a given folder and store them as a list by appending.<br>
This costs a lot of working memory + it takes ages to load the dataset.<br>
I was wondering what a smart way is to load the images? What is considered good practice when working with a lot of images?</p>
<p>I was thinking about storing the paths where the images are (as .csv / .json) and only load the images in the def <strong>getitem</strong>(self, idx): method, when they are actually needed.</p>
<p>What’s the most efficient way?<br>
Thanks for any suggestions!</p>

username: simaiden
cooked: <p>The second one is actually the way that -for example- ImageFolder dataset class load the images. So, the best practice is load the path directory where your images are ( for example with os.listdir() ) , and load the image in <code>__getitem__</code> method</p>
Saving output of a layer

username: pedram1
cooked: <p>Hi everyone</p>
<p>I have a kind of funny question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Who knows that how can I save (store) output of a layer ?<br>
Codes below represents my NN. Basically I would like  to save last layer of my encoder to a tensor</p>
<p><span class="hashtag">#Autoencoder</span> and Autodecoder conv layer…<br>
class Autoencoder(nn.Module):</p>
<p>def <strong>init</strong>(self):<br>
super(Autoencoder,self).<strong>init</strong>()</p>
<pre><code>    self.encoder = nn.Sequential(
        nn.Conv1d(1,3,kernel_size=4, stride=2),
        nn.MaxPool1d(3, stride=2),
        nn.Tanh(),
                
        
        nn.Conv1d(3,6,kernel_size=4, stride = 2),
        nn.MaxPool1d(3,stride=1),
        nn.Tanh(),
                 
        
        nn.Conv1d(6,9,kernel_size=3, stride=1),
        nn.MaxPool1d(2,stride = 1),
        nn.Tanh(),

        nn.Conv1d(9,12,kernel_size = 3, stride = 1),
        nn.MaxPool1d(2,stride = 1),
        nn.Tanh())

        

        

                 
                
                   
                     
                          
    self.decoder = nn.Sequential(          
        nn.ConvTranspose1d(12,9,kernel_size = 3, stride = 2),
        nn.Tanh(),
        

        nn.ConvTranspose1d(9,6,kernel_size = 4,stride = 1),
        nn.Tanh(),
                    

        nn.ConvTranspose1d(6,3,kernel_size= 8,stride = 2),
        nn.Tanh(),

        nn.ConvTranspose1d(3,1,kernel_size = 8, stride = 4),
        nn.Tanh())
</code></pre>
<p>def forward(self,x):<br>
x = self.encoder(x)<br>
x = self.decoder(x)<br>
return x</p>
<p>Thanks all</p>

username: simaiden
cooked: <p>You can do</p>
<pre><code class="lang-auto">net = Autoencoder()
encoded = net.encoder(input)
</code></pre>

username: pedram1
cooked: <p>Worked properly  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thank you.</p>

username: pedram1
cooked: <p>The output is what I was expecting. it is a tensor of size (65536, 12, 2) which 65536 means number of my sample, 12 is number of my channel and 2 is length of each sample. Actually I wanna apply a PCA on this tensor to first figure out which channel is important, (basically just looking for the one channel , the most important one) and then apply a second PCA for plotting that channel.<br>
Do u have any idea about this ?</p>
<p>I Appreciate this</p>

username: tomt
cooked: <aside class="quote no-group" data-username="pedram1" data-post="4" data-topic="71206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>2 is length of each sample. Actually I wanna apply a PCA on this</p>
</blockquote>
</aside>
<p>I do wonder the same. Did you figure it out or have any links?</p>
Conv2d_weight in nn.grad for transposed convolution

username: ttoosi
cooked: <p>Hi,</p>
<p>I am trying to write a customized backward function for <strong>ConvTranspose2d</strong>. I am using <strong>nn.grad</strong> as a reference, where it computes the gradients w.r.t weight in <strong>conv2d_weight</strong>. I thought if I only change <strong>torch.conv2d</strong> with <strong>nn.functional.conv_trasnpose2d</strong> it would do the job, but it didn’t.<br>
So my question is how I can change <strong>conv2d_weight</strong> function (copied below from <strong>nn.grad</strong> for convenience) to be able to compute the gradient of <strong>ConvTranspose2d</strong> w.r.t its weights?</p>
<p>Thanks,<br>
Tahereh</p>
<pre><code class="lang-auto">def conv2d_weight(input, weight_size, grad_output, stride=1, padding=0, dilation=1, groups=1):
    r"""
    Computes the gradient of conv2d with respect to the weight of the convolution.
    Args:
        input: input tensor of shape (minibatch x in_channels x iH x iW)
        weight_size : Shape of the weight gradient tensor
        grad_output : output gradient tensor (minibatch x out_channels x oH x oW)
        stride (int or tuple, optional): Stride of the convolution. Default: 1
        padding (int or tuple, optional): Zero-padding added to both sides of the input. Default: 0
        dilation (int or tuple, optional): Spacing between kernel elements. Default: 1
        groups (int, optional): Number of blocked connections from input channels to output channels. Default: 1

    """
    stride = _pair(stride)
    padding = _pair(padding)
    dilation = _pair(dilation)
    in_channels = input.shape[1]
    out_channels = grad_output.shape[1]
    min_batch = input.shape[0]

    grad_output = grad_output.contiguous().repeat(1, in_channels // groups, 1,
                                                  1)
    grad_output = grad_output.contiguous().view(
        grad_output.shape[0] * grad_output.shape[1], 1, grad_output.shape[2],
        grad_output.shape[3])

    input = input.contiguous().view(1, input.shape[0] * input.shape[1],
                                    input.shape[2], input.shape[3])

    grad_weight = torch.conv2d(input, grad_output, None, dilation, padding,
                               stride, in_channels * min_batch)

    grad_weight = grad_weight.contiguous().view(
        min_batch, grad_weight.shape[1] // min_batch, grad_weight.shape[2],
        grad_weight.shape[3])

    return grad_weight.sum(dim=0).view(
        in_channels // groups, out_channels,
        grad_weight.shape[2], grad_weight.shape[3]).transpose(0, 1).narrow(
            2, 0, weight_size[2]).narrow(3, 0, weight_size[3])

</code></pre>
`__getitem__()` does not support returning tensors with different sizes

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">class TrainDataset(torch.utils.data.Dataset):
...
def __getitem__(self, index):
...
# suppose we return this image
img = torch.randn(3, 240, 240)

channel_tmp = 10
other_info = torch.randn(channel_tmp , 240, 240)
return img, other_info
</code></pre>
<p>However, here, in my case, I need something like this:</p>
<pre><code class="lang-python">class TrainDataset(torch.utils.data.Dataset):
...
def __getitem__(self, index):
...
# suppose we return this image
img = torch.randn(3, 240, 240)

channel_tmp = np.random.randint(5, 10)
other_info = torch.randn(channel_tmp , 240, 240)
return img, other_info
</code></pre>
<p>Of course, this snippet does not work, as   this line of code <code>return torch.stack(batch, 0, out=out)</code> in <code>torch/utils/data/_utils/collate.py</code>.</p>
<p>Edit:</p>
<ol>
<li>
<p>The <code>channel_tmp</code> varies a lot (from very small to a very large number), so if always set the <code>channel_tmp</code> the largest possible number and when the batchsize is 4 or 8, it will be OOM.</p>
</li>
<li>
<p>In fact, I want something like this, for this specific variable <code>other_info</code>, I need data from each thread can be stacked on the <code>channel_tmp</code> dimension.`</p>
</li>
</ol>
<p>Is there any workaround to this?</p>

username: ptrblck
cooked: <p>You could use a custom <code>collate_fn</code> as described <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3">here</a> which would work for variable shapes.</p>

username: Zhaoyi-Yan
cooked: <p>Thank you, ptrblck, I will take a look!</p>

username: Zhaoyi-Yan
cooked: <pre><code class="lang-auto"> def my_collfn(batch):
     img = [item[0] for item in batch]
     other_info= [item[1] for item in batch]

     img = torch.stack(img, 0)
     other_info= torch.cat(other_info, dim=0)

     return img, other_info

</code></pre>
CUDA memory issue

username: holoholger
cooked: <p>I just bought a PC with an NVidia GTX1660 Super with 6GB of VRAM for CUDA.<br>
Never used CUDA before</p>
<p>When I run my code, I get the error message:</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 38.00 MiB (GPU 0; 5.80 GiB total capacity; 1.41 GiB already allocated; 14.06 MiB free; **1.54 GiB reserved in total by PyTorch**)</code></p>
<p>I don’t get it, I’ve got 6GB of VRAM, but PyTorch has only reserved 1.54 GB out of it.<br>
What am I doing wrong here?</p>
<p>I’ve tried torch.cuda.empty_cache() - no effect.<br>
I run Ubuntu Linux on an AMD Ryzen 3400G. Are the other 4.5GB of GPU VRAM reserved for graphics output? I killed the xserver, no improvement. I do have another Ryzen built-in GPU but there doesn’t seem to be any driver, so as of now, I’m still outputting video signal through my GTX1660 Super.</p>
<p>Would appreciate some help here, I’m new to CUDA.</p>

username: ptrblck
cooked: <p>Could you check the memory usage via <code>nvidia-smi</code>?<br>
Also, what kind of model are you using and how large is the input?<br>
Does this out of memory error occur in the first iteration or after a while?</p>

username: ranger
cooked: <p>You can run watch -n 2 nvidia-smi in side-tab.<br>
If the model has a bigger size than available gpu memory , memory consumption will rise uptil a point and go back to zero.</p>

username: holoholger
cooked: <p>Thanks for the nvidia-smi hint, guys! That did the job.<br>
There were some other apps running in the background, sucking all the GPU VRAM away.</p>
Dataloader loading color distorted image

username: haeminjung
cooked: <p>Quick show and tell will be the easiest way to tell you the problem.</p>
<pre><code class="lang-auto"># Define the dataset and the loader as usual
dataset = dset.ImageFolder(root=datapath,
                           transform=transforms.Compose([
                                transforms.Resize(image_size),
                                #transforms.CenterCrop(image_size),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5414, 0.5333, 0.5338),(0.1897, 0.1911, 0.1932))]))

dataloader = torch.utils.data.DataLoader(dataset, batch_size=64,
                                         shuffle=False, num_workers=workers)

#load the image
data = next(iter(dataloader))

#save the image
dlX = data[0].to(device)
vutils.save_image(dlX[0], 'test.jpg')

</code></pre>
<p>Result of save_image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/0118edbe82b4b8058f7d0802a29eedb70e858864.jpeg" alt="test" data-base62-sha1="9HSZoj2keptbmGpjZUVWJDEaYA" width="128" height="128"></p>
<p>Am I missing something? or doing something wrong?<br>
I have trained a DCGAN with this dataset with results look like this but can only post a single image)</p>
<p>I am attaching a link to my dropbox where you can see the result, original, and the training results<br>
<a href="https://www.dropbox.com/sh/apbfcuezzlj1kbf/AAB_-fzU1_8oLRorZ9kWF0fAa?dl=0" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.dropbox.com/sh/apbfcuezzlj1kbf/AAB_-fzU1_8oLRorZ9kWF0fAa?dl=0</a></p>

username: boto
cooked: <p>Take a look at <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Normalize" rel="nofollow noopener">torchvision.transforms.Normalize</a>. Your picture is being normalized, which results in this weird picture. If you remove <code>transforms.Normalize((0.5414, 0.5333, 0.5338),(0.1897, 0.1911, 0.1932))]</code>, the image should look normal.</p>

username: haeminjung
cooked: <p>Thank you for the reply.</p>
<p>As far I as I understand normalizing, I thought it was crucial that I normalize the dataset accordlingly to it’s avg and std, thus those are the calculated value.</p>
<p>Then, to get results more resembling the original, is it okay to skip normalization?</p>

username: boto
cooked: <p>Yes, normalization should be done. I wouldn’t skip it, as normalizing data improves performance.<br>
If you want the original image back and you know the mean and std, you could simply reverse the normalization:</p>
<pre><code class="lang-auto">normalized_image = (original_image - mean) / std
original_image = (normalized_image * std) + mean
</code></pre>

username: haeminjung
cooked: <p>Thank you for the reply, I had not been able to come back to this issue for a bit while.<br>
I will check within next week to see if it work, which I highly think would.</p>
<p>Thanks for your time</p>
Tensor creation (Python bindings / C++)

username: kl_divergence
cooked: <p>Hi,</p>
<p>I am trying a fix an issue. Some methods are broken. I wanted to know the source of where tensor is created in Pytorch. Like when we pass arguments like <code>requires_grad</code>, how are they taken care of in C++.<br>
What does <code>wrap</code> and <code>THPVariable_wrap</code> exactly do ?<br>
I’m looking into <code>/csrc/autograd/generated/</code>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Is it this issue: <a href="https://github.com/pytorch/pytorch/issues/36455">https://github.com/pytorch/pytorch/issues/36455</a> ?<br>
I think this particular comment on the issue answers your question <a href="https://github.com/pytorch/pytorch/issues/36455#issuecomment-614097391">https://github.com/pytorch/pytorch/issues/36455#issuecomment-614097391</a>?</p>

username: kl_divergence
cooked: <p>Yes, thanks. Still not sure how to go about it. Waiting for someone more experienced to pitch in as suggested by ezyang.</p>
Codebase for publish on torch hub

username: nicolalandro
cooked: <p>I use this <a href="https://github.com/pytorch/hub" rel="nofollow noopener">guide</a> to create a project for torch hub. I create the repo <a href="https://github.com/nicolalandro/ntsnet_cub200" rel="nofollow noopener">here</a>. I write my template.md file. I read thet I push it in a codebase, but I do not find the github link of this codebase. Someone know the link?</p>

username: nicolalandro
cooked: <p>I try with the same of instruction: <a href="https://github.com/pytorch/hub" rel="nofollow noopener">https://github.com/pytorch/hub</a> maybe it works.</p>
Loading 3D data

username: Raul-Ronald_Galea
cooked: <p>Hello,</p>
<p>I have a dataset composed of 3D volumes, small enough to be loaded in memory.</p>
<p>I need to process these volumes a batch of slices at a time (GPU memory restrictions)</p>
<p>What would be a clean way to go about this?<br>
My current solution is subclassing torch.utils.data.Dataset to return one volume at a time, then getting slices of the returned volume manually. But this just feels like a somewhat hacky way to create batches…</p>
<p>Code snipped for the Dataset:</p>
<pre><code class="lang-auto">class My_Dataset(Dataset):

    def __init__(self, paths):
        """
        Args:
        paths: list of paths to the volumes
        """
        self.paths = paths
        self.load_everything_in_memory()

    def __len__(self):
        return len(self.images)

    def __getitem__(self, idx):
        image, mask = self.images[idx], self.masks[idx]
        return image, mask

    def load_everything_in_memory(self):
        """
        self.images = list of volumes
        self.masks = labels for those volumes
        """
        images, masks = [], []
        for path in self.paths:
            # do some loading
            # get image, mask
            images.append(image)
            masks.append(mask)
            
        self.images = images
        self.masks = masks
</code></pre>
<p>It feels like it should be some better way of doing this.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>This sounds like a valid approach.<br>
Alternatively, you could of course create a tensor or list with all slices and use the <code>index</code> to grab a slice and create the desired batch from it, but I guess <code>load_everything_in_memory</code> might then become a bit more complicated.</p>
<p>You could try to use <code>torch.utils.checkpoint</code> to trade compute for memory, but I’m not sure how large your memory requirement is for a single batch.</p>

username: Raul-Ronald_Galea
cooked: <p>Thanks, for your reply! I suppose its good enough for now as it is, I can always look to optimise later if I really need to.</p>
Right way to place tensors on devices?

username: sharvil
cooked: <p>Suppose I have a (fake) module that looks like this:</p>
<pre><code class="lang-python">class MyLinearModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.mask = torch.ones(1, 100)
    self.mask[:, :50] = 0
    self.kernel = nn.Parameter(torch.rand(1, 100))

  def forward(self, x):
    return x @ (self.kernel * self.mask)
</code></pre>
<p>This code is fine for CPU but doesn’t work on GPU because <code>self.mask</code> is a CPU tensor and <code>.cuda()</code> won’t move it since it’s not a buffer or parameter. Fine. So we add the following method to <code>MyLinearModule</code>:</p>
<pre><code class="lang-python">[...]
  def cuda(self, device=None):
    self.mask = self.mask.cuda(device)
    return super().cuda(device)
</code></pre>
<p>Which gets the job done. Then we decide to train on multiple GPUs by wrapping our module with <code>DataParallel</code> and we get this failure:</p>
<pre><code class="lang-auto">  File "test.py", line 13, in forward
    return x @ (self.kernel * self.mask)
RuntimeError: expected device cuda:1 but got device cuda:0
</code></pre>
<p>I could write <code>return x @ (self.kernel * self.mask.to(self.kernel.device))</code> but it seems silly to copy a constant from CPU to each GPU on every forward call. I feel like I’m missing something fundamental here.</p>
<p>What’s the right way to make sure <code>self.mask</code> is on the appropriate device?</p>

username: ptrblck
cooked: <p>Since <code>mask</code> is not trainable, you should register it as a buffer via:</p>
<pre><code class="lang-python">mask = torch.ones(1, 100)
mask[:, :50] = 0.
self.register_buffer('mask', mask)
</code></pre>
<p>This will make sure to push it to the right device as is done with parameters.</p>

username: sharvil
cooked: <p>I considered that, but treating the mask as a buffer will also make it a part of the module’s <code>state_dict</code> which is unnecessary since the mask is just a constant. Is there a way to specify that the buffer doesn’t need to be written out to disk?</p>

username: sharvil
cooked: <p>Ah, never mind, I see that there’s an open feature request for what I described (<a href="https://github.com/pytorch/pytorch/issues/18056" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/18056</a>). Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
How to load pre-trained model weight (.pth) by using keras?

username: thida
cooked: <p>I have exported a trained model’s weight by using<br>
<code> torch.save(model, '/path/to/weight/entire_model.pth')</code><br>
I am wondering is there any way I could load the weight by using keras and modify the pre trained model’s layers of my own?</p>

username: anilg
cooked: <p>Hi,<br>
You can use <a href="https://github.com/nerox8664/pytorch2keras" rel="nofollow noopener">Pytorch2Keras</a> library.</p>
<pre><code class="lang-auto">k_model = pytorch_to_keras(model, input_var, [(10, input_size, input_size,)], verbose=True, names='short')  
</code></pre>

username: thida
cooked: <p>Thank you so much for the suggestion.<br>
I was running through the documentation, found that the model converted by pytorch2Keras seems not include CNN. Is that so?<br>
Is there any method to load pre trained CNN weights using keras as well?</p>

username: anilg
cooked: <aside class="quote no-group" data-username="thida" data-post="3" data-topic="77438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8baadc/48.png" class="avatar"> thida:</div>
<blockquote>
<p>ough the documentation, found that the model converted by pytorch2Keras seems not include CNN. Is that so?</p>
</blockquote>
</aside>
<p>What do you mean by CNN layer? I assume that you are pointing CONV2D or CONV. They are both supported. You can easily restore you weights using Keras.</p>
Same parameters in different models

username: circa
cooked: <pre><code class="lang-auto">import concurrent.futures
import torch
import torch.nn as nn
import torch.optim as optimizer
from torch.distributions import Categorical

class mymodel1(nn.Module):
    def __init__(self):
        super(mymodel1,self).__init__()
        self.weight = nn.Linear(3,2)
        
    def forward(self, X):
        out = self.weight(X)
        out = nn.Softmax(dim = 0)(out)
        return out

class mymodel2(nn.Module):
    def __init__(self):
        super(mymodel2,self).__init__()
        self.weight = nn.Linear(3,2)
        
    def forward(self, X):
        out = self.weight(X)
        out = nn.Softmax(dim = 0)(out)
        return out


class mymodel3(nn.Module):
    def __init__(self):
        super(mymodel3,self).__init__()
        self.weight = nn.Linear(3,2)
        
    def forward(self, X):
        out = self.weight(X)
        out = nn.Softmax(dim = 0)(out)
        return out


        
def doTrain(model, X):     
    a1 = model()
    return list(a1.parameters())

X = torch.randn(12,3)


updatedParams = []
results = []


with concurrent.futures.ProcessPoolExecutor() as executor:
    f1 = executor.submit(doTrain, mymodel1, X[0*4:(0+1)*4])
    f2 = executor.submit(doTrain, mymodel2, X[1*4:(1+1)*4])
    f3 = executor.submit(doTrain, mymodel3, X[2*4:(2+1)*4])
    

print(f1.result())
print(f2.result())
print(f3.result())
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">[Parameter containing:
tensor([[-0.3413, -0.4291,  0.0850],
        [-0.4270, -0.4523, -0.3700]], requires_grad=True), Parameter containing:
tensor([0.5327, 0.2588], requires_grad=True)]
[Parameter containing:
tensor([[-0.3413, -0.4291,  0.0850],
        [-0.4270, -0.4523, -0.3700]], requires_grad=True), Parameter containing:
tensor([0.5327, 0.2588], requires_grad=True)]
[Parameter containing:
tensor([[-0.3413, -0.4291,  0.0850],
        [-0.4270, -0.4523, -0.3700]], requires_grad=True), Parameter containing:
tensor([0.5327, 0.2588], requires_grad=True)]
</code></pre>
<p>Can somebody tell me why I am getting same parameters returned from the different processes although the models are different ?</p>

username: mrshenli
cooked: <p>This might be because the per-process RNG are initialized with the same seed by default? Can you try if manually changing seed using <a href="https://pytorch.org/docs/stable/random.html#torch.random.manual_seed" rel="nofollow noopener"><code>manual_seed</code></a> solves the problem?</p>

username: circa
cooked: <p>Thanks,<br>
I used<br>
<code>torch.manual_seed(num)</code><br>
with a different value for num in the <strong>init</strong> method of each of the  models and that did it.</p>
RuntimeError: "log_cuda" not implemented for 'Long'

username: Kiran_Ikram
cooked: <p>Hi Patrick,</p>
<p>To train my MAML, I am experimenting with loss functions. In switching from CrossEntropyLoss to KLDivLoss I got: RuntimeError: “log_cuda” not implemented for ‘Long’</p>
<p>My loss function takes in:</p>
<p><span class="hashtag">#k_way</span> = 5<br>
<span class="hashtag">#data_shape</span> = x.shape[2:] , where x are my input samples<br>
<span class="hashtag">#create_nshot_task_label</span> , where y is : Label vector for n-shot task of shape [q * k, ]<br>
and y = torch.arange(0, k, 1 / q).long()</p>
<p>logits = model(torch.zeros((k_way, ) + data_shape).to(device, dtype=torch.double))<br>
loss = loss_fn(logits, create_nshot_task_label(k_way, 1).to(device))</p>
<p>I was hoping you could help me overcome this! Thank you!</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.KLDivLoss">nn.KLDivLoss</a> expects the model output and target to have the same shape and as <code>FloatTensors</code>.<br>
I guess you are passing the same targets from the <code>nn.CrossEntropyLoss</code> to the new criterion, which will yield this error.</p>
How do I get numpy related functions to run on a GPU?

username: JasonChenhx
cooked: <p>How do I get numpy related functions to run on a GPU</p>

username: ptrblck
cooked: <p>PyTorch implements a lot of numpy-equivalent functions, which can be executed on a GPU, if the data is already pushed to the GPU.<br>
Are you missing some numpy methods or which functions would you like to use?</p>

username: JasonChenhx
cooked: <p>I want to use np.dot.</p>

username: ptrblck
cooked: <p>You could use <code>torch.dot</code>. You can find a lot of functions in <a href="https://pytorch.org/docs/stable/torch.html">the docs</a>.</p>

username: JasonChenhx
cooked: <p>But the dot function only applies to 1 dimension.</p>

username: ptrblck
cooked: <p>As stated in the <a href="https://numpy.org/doc/stable/reference/generated/numpy.dot.html?highlight=dot#numpy.dot">numpy docs</a> for multi-dimensional arrays other methods (e.g. <code>matmul</code>) should be preferred, which are also implemented in PyTorch.</p>
Is it possible to use pytorch c++ frontend API in CUDA kernel

username: yessyw
cooked: <p>Like this,<br>
<strong>global</strong> void function(torch::Tensor A, torch::Tensor B, torch::Tensor C, int n){<br>
int tid = blockIdx.x * blockDim.x + threadIdx.x;<br>
if(tid &lt; n) {<br>
C[tid]  =  A[tid]  +  B[tid];<br>
}<br>
}</p>

username: ptrblck
cooked: <p>I don’t think that’s possible and you would usually pass the data pointers to the method.</p>

username: yessyw
cooked: <p>Thanks.</p>
<p>You mean that example code dosen’t works or libtorch api can’t use in nvcc compiler(.cu file)?</p>
<p>I just want to use libtorch api in .cu file<br>
My actual problem is when i try #include&lt;torch/torch.h&gt; in .cu file(nvcc) there is error  E1866<br>
on “C10_DEFINE_DEPRECATED_USING(IntList, ArrayRef&lt;int64_t&gt;)”.<br>
Am i wrong or Is it impossible?</p>

username: ptrblck
cooked: <p>You can use <code>Tensor</code>s in <code>.cu</code> files as seen <a href="https://github.com/pytorch/pytorch/blob/40eff454ce5638fbff638a7f4502e29ffb9a2f0d/aten/src/ATen/native/cuda/Unique.cu#L56">here</a>. I just don’t think that the actual kernel will accept these.</p>
<p>Regarding the error message: could you use <code>ArrayRef</code> instead of <code>IntList</code>?</p>

username: tom
cooked: <p>You probably want PackedTensorAccessors for this.<br>
See e.g. the <a href="https://github.com/pytorch/pytorch/blob/60a3e82c4e0272a748b402b31e9f3843a61445a6/aten/src/ATen/native/cuda/Normalization.cuh#L159" rel="nofollow noopener">BatchNorm</a> kernels for details (this currently is called <code>GenericPackedTensorAccessor</code>, if you know your required index size, you can use <code>PackedTensorAccessor32</code> or <code>64</code>.<br>
They offer array-like elementwise access in kernels and can be passed (by value(!)) to the kernel instead of pointer + dimensions + strides.</p>
<p>Best regards</p>
<p>Thomas</p>

username: 21fa417e3fb06e56040c
cooked: <p>Did you solve the problem?<br>
are you PytorchKorea community Son?</p>

username: yessyw
cooked: <p>Thanks for your reply.</p>
<p>sorry i am late.<br>
I can make .cu file include &lt;Aten.h&gt;.<br>
but, still there are error if i include &lt;torch.h&gt;</p>
<p>at::Tensor can not save gradient right?<br>
so, i need torch::Tensor. Is it possible to use torch::Tensor in cuda(.cu file)</p>