akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unsupervised Image Similarity model

berkayalan: <p>Hi,</p>
<p>I have product images more than 40k and would like to find similar images from the pool. I have looked some example such as:</p>
<ul>
<li><a href="https://medium.com/pytorch/image-similarity-search-in-pytorch-1a744cf3469" class="inline-onebox" rel="noopener nofollow ugc">Image Similarity Search in PyTorch | by Aditya Oke | PyTorch | Medium</a></li>
</ul>
<p>Do you have any suggestion for that? I really need this and all your suggestions are appreciated.</p>
<p>Thanks in advance.</p>
Saving GAN generated images

HassanAli: <p>Hi,<br>
I am following the DC GAN tutorial of pytorch for generating synthetic images. I want to store generated after the last epoch individually.<br>
<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a><br>
I am new to this. I don’t know how to save these images individually in a folder on local drive. Please help me.</p>

Matias_Vasquez: <p>Torchvision has a function called <code>save_image()</code>. <a href="https://pytorch.org/vision/stable/generated/torchvision.utils.save_image.html#torchvision.utils.save_image" rel="noopener nofollow ugc">Here</a> is the documentation.</p>
<p>You might want to take a look at <a href="https://discuss.pytorch.org/t/how-to-save-image/48808/7">this old discussion</a> to see some examples.</p>
<p>Here is also a minimal example.</p>
<pre><code class="lang-python">from torchvision.utils import save_image

# Batch x Channel x Height x Width = BxCxHxW = 64x3x28x28
rand_tensor= torch.rand(64, 3,28,28) 

# Taking the first image from the batch (B)
# Img size CxHxW
img1 = rand_tensor[0]           
save_image(img1, 'img1.png')
</code></pre>

HassanAli: <p>Thank you very much for your reply. This code will save only one image. Do I have to use loop to save all images individuallly?</p>

Matias_Vasquez: <p>Depending on what you want to save, if you want to save every image individually then yes, you can make a loop and save every one of them.</p>
<p>Or you can use the following code to save a grid of all of the images generated.</p>
<pre><code class="lang-python">import torch
from torchvision.utils import save_image

t = torch.rand((64, 3, 50, 50))
save_image(t, "img_grid.png", nrow=8)
</code></pre>
<p>In the link provided in the previous post is the documentation for <code>save_image</code> and it says you can use other <code>**kwargs</code> specified in the <code>make_grid</code> function.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

HassanAli: <p>Thank you. It will solve my issue.</p>

HassanAli: <p>I was able to save the images individually like this:</p>
<p>import os</p>
<p>sample_dir = ‘samples’<br>
if not os.path.exists(sample_dir):<br>
os.makedirs(sample_dir)</p>
<p>if(epoch == num_epochs-1 ):<br>
fake_fname = ‘fake-{0:0=4d}.png’.format(i)<br>
<span class="hashtag">#save_image</span>(fake, os.path.join(sample_dir, fake_fname), nrow=8)<br>
save_image(fake[i], os.path.join(sample_dir, fake_fname), nrow=1)</p>
Torch.fx.wrap will not work, when encapsulate the code

MartinZhang: <p>a.py</p>
<pre><code class="lang-auto">import torch
import torch.fx
from math import sqrt

def normalize(x):
    return x / sqrt(len(x))

torch.fx.wrap('len')
torch.fx.wrap('sqrt')
</code></pre>
<p>setup.py</p>
<pre><code class="lang-auto">from distutils.core import setup
from Cython.Build import cythonize

setup(name='a', ext_modules=cythonize(['a.py']))
</code></pre>
<blockquote>
<p>python setup.py bulid_ext --inplace</p>
</blockquote>
<p>b.py</p>
<pre><code class="lang-auto">from a import *
normalize(torch.randn(3, 4))
traced = torch.fx.symbolic_trace(normalize)
</code></pre>
<blockquote>
<p>python b.py</p>
</blockquote>
<p>error from <code>_symbolic_trace.py</code></p>
<pre><code class="lang-auto">AssertionError
</code></pre>
<p>How to deal it?</p>
Is there anyway to load data into GPU directly?

zezhong_zhang: <p>In every training loop, I use DataLoader to load a batch of image into CPU, and move it to GPU like this:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader

batchsize = 64
trainset = datasets.CIFAR10(blahblah…)
train_loader = DataLoader(train_dataset, batch_size=batchsize, shuffle=True, num_workers=2)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def train(epoch):
    for batch_index, data in enumerate(train_loader, 0):
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)
</code></pre>
<p>I know that I can wrap dataloader and calling <code>.to(device)</code> in advance instead of using it in every training batch. But <code>.to(device)</code> itself is time-consuming, I mean, transfer a tensor from CPU to GPU is much slower than creating a tensor directly on GPU, isn’t it?</p>
<p>for example:<br>
(randomTensorA is created on CPU and using .to() function to transfer it on GPU,<br>
randomTensorB is created on GPU)</p>
<pre><code class="lang-auto">import time
import torch

shape = [300, 300, 300]

a = time.time()
for _ in range(100):
    randomTensorA = torch.randn(shape).to(torch.device('cuda'))
b = time.time()
print('Elapsed Time: %f' % (b-a))

a = time.time()
for _ in range(100):
    randomTensorB = torch.randn(shape, device='cuda')
b = time.time()
print('Elapsed Time: %f' % (b-a))
</code></pre>
<p>Terminal output:</p>
<pre><code class="lang-auto">Elapsed Time: 24.316857
Elapsed Time: 1.658716
</code></pre>
<p>So, is there anyway to let dataloader load dataset directly on GPU? Please let me know, thanks.</p>

ptrblck: <p>The <code>CIFAR10</code> dataset is stored as binary data on your SSD so you would need to move it to the GPU at one point and cannot “create” it on the device.</p>

Forceless: <p>My solution is save these data in gpu use torch.save() in advance<br>
And load them like</p>
<pre><code class="lang-python">torch.load('./data.pt',map_location='cuda:0')
</code></pre>
Go from jit::ClassDef to python code?

DBraun: <p>I’m looking at this example</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/test/cpp/jit/test_class_parser.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/jit/test_class_parser.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/jit/test_class_parser.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/test/cpp/jit/test_class_parser.cpp</a></h4>


      <pre><code class="lang-cpp">#include &lt;gtest/gtest.h&gt;

#include &lt;torch/csrc/jit/frontend/parser.h&gt;
#include &lt;torch/csrc/jit/frontend/resolver.h&gt;

namespace torch {
namespace jit {
constexpr c10::string_view testSource = R"JIT(
  class FooTest:
    def __init__(self, x):
      self.x = x

    def get_x(self):
      return self.x

    an_attribute : Tensor
)JIT";

TEST(ClassParserTest, Basic) {
  Parser p(std::make_shared&lt;Source&gt;(testSource));
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/jit/test_class_parser.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is it possible to get Python code from the <code>classDef</code>, to loosely recover the <code>testSource</code>? I read <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/docs/serialization.md" class="inline-onebox" rel="noopener nofollow ugc">pytorch/serialization.md at master · pytorch/pytorch · GitHub</a> but don’t know where to go for C++ code. This is something close <a href="https://github.com/pytorch/pytorch/blob/51e50a2ddb9844e0abaf6028fe5e4246448dffff/torch/csrc/jit/python/script_init.cpp#L1670-L1728" class="inline-onebox" rel="noopener nofollow ugc">pytorch/script_init.cpp at 51e50a2ddb9844e0abaf6028fe5e4246448dffff · pytorch/pytorch · GitHub</a></p>
<p>Should I copy this code into a function to achieve my goal? Right now it’s just in a python binding that I can’t use.</p>
<p>Suppose I do that, then I will have an instance of a <code>ClassDef</code>.</p>
<p>Then I could try PythonPrint to print the Python code: <code>https://github.com/pytorch/pytorch/blob/91ef3c82615d6ede05d5b86f1bd5571ea95e4ef1/torch/csrc/jit/python/script_init.cpp#L1235-L1243</code> I noticed that <code>PythonPrint</code> is missing <code>printClass</code> whereas <code>PythonPrintImpl</code> does have <code>printClass</code>. So I would have to use PythonPrintImpl or duplicate its code, both seem wrong to do.</p>
My model still doesn't learn, what horribly stupid mistake did I make this time?

hananelroe: <p>is it the network structure or the code itself?<br>
the code takes a random image from a random directory, determines a target and does the train thing</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import transforms
from PIL import Image
import matplotlib.pyplot as plt
from tqdm import tqdm
import random
import os

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"device: {device}")

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.L1 = nn.Linear(28*28, 1000)
        self.L2 = nn.Linear(1000, 1500)
        self.L3 = nn.Linear(1500, 1000)
        self.L4 = nn.Linear(1000, 500)
        self.L5 = nn.Linear(500, 250)
        self.L6 = nn.Linear(250, 100)
        self.L7 = nn.Linear(100, 50)
        self.L8 = nn.Linear(50, 2)

    def forward(self, x):
        x = torch.sigmoid(self.L1(x))
        x = torch.sigmoid(self.L2(x))
        x = torch.sigmoid(self.L3(x))
        x = torch.sigmoid(self.L4(x))
        x = torch.sigmoid(self.L5(x))
        x = torch.sigmoid(self.L6(x))
        x = torch.sigmoid(self.L7(x))
        x = self.L8(x)
        return x


net = Network().to(device)
optimizer = optim.SGD(net.parameters(), lr=0.01)
criterion = nn.MSELoss()
epochs = 3000

loss_hist = []
net.train()  # puts the network in "training mode" (idk)

for epoch in tqdm(range(0, epochs)):
    directory = os.path.join("data", random.choice(os.listdir("data")))
    imagePath = os.path.join(directory, random.choice(os.listdir(directory)))  # get a random image file from a random directory
    target = torch.tensor([0., 1.]) if imagePath[5:6] == "0" else torch.tensor([1., 0.])  # determine target using the file's parent directory
    img = Image.open(imagePath)
    img = transforms.functional.to_grayscale(img, 1)  # convert image to grayscale
    '''
    plt.imshow(img)
    plt.title(imagePath[5:6])
    plt.show()
    '''
    imageTensor = transforms.ToTensor()(img)  # convert the image to tensor
    imageTensor = imageTensor.squeeze()  # from size(1, 28, 28) to size(28, 28)
    imageTensor = torch.flatten(imageTensor)  # from size(28, 28) to size(784)
    input = imageTensor
    optimizer.zero_grad()
    out = net(input)
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
    loss_hist.append(loss.item())

plt.plot(loss_hist)
plt.show()
</code></pre>

ptrblck: <p>Your model is quite deep and the <code>sigmoid</code> nonlinearities could kill the gradient.<br>
Try to overfit a small dataset first by playing around with some hyperparameters (e.g. lower the learning rate, use an adaptive optimizer, replace the sigmoids with relus, remove layers from the model etc.) and make sure your model is able to overfit this small data sample before scaling up the use case again.</p>
Trained ASR model on the GPU and trying to evaluate but getting the errorRuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same - how do I move the data to the GPU

user12233: <p>I’m trying to evaluate an ASR model that I trained on a GPU.  From what I’ve read to evaluate I also need to move the data to the GPU but I’m struggling to do that.  I’m trying to evaluate the ASR model by decoding the test data using <code>pyctcdecode</code>.  The code I’m trying to implement is <a href="https://github.com/kensho-technologies/pyctcdecode/blob/main/tutorials/02_pipeline_huggingface.ipynb" rel="noopener nofollow ugc">here</a>.  Specifically the code below:</p>
<pre><code class="lang-auto">import soundfile as sf
arr, _ = sf.read('1919-142785-0028.wav')
input_values = asr_processor(arr, return_tensors="pt", sampling_rate=16000).input_values  # Batch size 1
logits = asr_model(input_values).logits.cpu().detach().numpy()[0]
</code></pre>
<p>When I run the last line of code I get the following error:</p>
<p><code>errorRuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</code></p>
<p>Doing some research I think I have to move the data to the GPU as well but that’s where I’m struggling.  I’ve tried to the following:</p>
<p><code>logits = model(input_values.to("cuda")).logits.cpu().detach().numpy()[0]</code><br>
<code>logits = model(input_values.to("cuda")).logits</code></p>
<p>That will run without producing the error but all the values of the logits are nans.  So I feel like I’m still not doing it correctly.</p>
<pre><code class="lang-auto">array([[nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       ...,
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan]], dtype=float32)
</code></pre>

suraj.pt: <aside class="quote no-group" data-username="user12233" data-post="1" data-topic="147694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/ac91a4/48.png" class="avatar"> user12233:</div>
<blockquote>
<p>errorRuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</p>
</blockquote>
</aside>
<p>From the error message, it looks like your input is already on GPU, but your model is on CPU? What happens if you do <code>model = model.to('cuda')</code> before running inference?</p>
How to optimizer weights in the same layer with different weight_decay values

yuchiq: <p>I have a fully connected layer in torch.</p>
<pre><code class="lang-auto">self.fc = nn.ModuleList([nn.Sequential(nn.Linear(10, 1),self.activation)])
</code></pre>
<p>In the optimizer, I want to assign different weight_decay for the parameters in this layer. Some thing like (not correct code here)</p>
<pre><code class="lang-auto">optimizer = optim.Adam([{'params':self.fc.parameters()[0:5],'weight_decay':0.01},
                                        {'params':self.fc.parameters()[5:10],'weight_decay':0.01},])
</code></pre>

KFrank: <p>Hi Paul!</p>
<aside class="quote no-group" data-username="yuchiq" data-post="1" data-topic="147583" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ecb155/48.png" class="avatar"> yuchiq:</div>
<blockquote>
<p>I want to assign different weight_decay for the parameters in this layer.</p>
</blockquote>
</aside>
<p>If you want to use different values of <code>weight_decay</code> for different<br>
parameters, use the <em>parameter group</em> facility of <code>Optimizer</code>.</p>
<p>However, if you want to use different weight decays for different<br>
elements of the <em>same</em> parameter, things become more complicated.</p>
<p>The issue is that an entire tensor gets updated “all at once.”  That is, you<br>
can’t update some elements of a tensor one way and other elements of<br>
the same tensor some other way (without indexing into the tensor “by<br>
hand”).</p>
<p>One approach is to split the tensor in question up into multiple tensors<br>
(and then put them into separate parameter groups that have different<br>
<code>weight_decay</code> values).  While splitting up tensors like this is certainly<br>
doable, it tends to be a hassle.</p>
<p>Instead, you can recognize that weight decay is, in essence, the same<br>
as applying a quadratic (<em>L2</em>) penalty to the weights.  (Note, an optimizer<br>
may treat a quadratic penalty and a <code>weight_decay</code> parameter somewhat<br>
differently in detail.)</p>
<p>It’s then easy to give different quadratic penalties – and hence different<br>
weight decays – to parts of the same tensor, say, <code>some_parameter</code>:</p>
<p>Let the tensor <code>penalty_mask</code> have the same shape as <code>some_parameter</code><br>
and consist, for example, of <code>1</code>s in the locations of the elements of<br>
<code>some_parameter</code> for which you want the weaker weight decay and <code>2</code>s<br>
for those elements for which you want the weight decay to be stronger<br>
(in this example, twice as strong).</p>
<p>Then:</p>
<pre data-code-wrap="python"><code class="lang-python">quadratic_penalty = some_scale_factor * penalty_mask * some_parameter**2
loss_plus_penalty = loss + quadratic_penalty
loss_plus_penalty.backward()
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
What's the difference between tensor.register_hook and grad_fn.register_hook?

111568: <p>I noticed that both tensor and grad_fn have a register_hook method, and they all seem to work after the grad is computed, are they the same?</p>
<pre><code class="lang-auto">a = torch.tensor([2.], requires_grad=True)
b = a * 3
b.backward()
grad_acc = b.grad_fn.next_functions[0][0]


# what's the difference between these two operations?
a.register_hook(...)
grad_acc.register_hook(...)
</code></pre>

hjm-aws: <p>I think the hooks on <code>Tensor</code> run <strong>before</strong> the backward pass, while the hooks on <code>grad_fn</code> run <strong>after</strong> the gradients are generated.</p>
How can I force python build to use existing successfully configured CMakeCache?

jiapei100: <p>How can I force python build to use existing successfully configured CMakeCache ?</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  pytorch git:(master) ✗ TORCH_CUDA_ARCH_LIST="7.5" NO_TEST=1 USE_MKLDNN=0 BUILD_CAFFE2=1 python setup.py build -cmake
Building wheel torch-1.12.0a0+git1c5a812
-- Building version 1.12.0a0+git1c5a812
cmake -GNinja -DBUILD_CAFFE2=1 -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=......pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python3.8/site-packages -DCUDA_NVCC_EXECUTABLE=/usr/local/cuda/bin/nvcc -DCUDNN_INCLUDE_DIR=/usr/local/cuda/include -DCUDNN_LIBRARY=/usr/local/cuda/lib64/libcudnn.so -DJAVA_HOME=/usr/lib/jvm/default-java -DNUMPY_INCLUDE_DIR=~/.local/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/usr/bin/python -DPYTHON_INCLUDE_DIR=/usr/include/python3.8 -DPYTHON_LIBRARY=/usr/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.12.0a0+git1c5a812 -DUSE_MKLDNN=0 -DUSE_NUMPY=True ......pytorch
CMake Error: Error: generator : Ninja
Does not match the generator used previously: Unix Makefiles
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.
</code></pre>
.cuda() and .cpu() insanely slow on A4000 with Cuda 11.1

soulslicer: <p>Ubuntu 20.04</p>
<p>I have ‘1.9.0+cu111’</p>
<p>cuda ‘11.1’</p>
<p>CUDNN 8005</p>
<p>I have a server with A4000’s, and they are all operating at 16x PICE express.</p>
<p>Any first call to .cuda() takes 7 mins to complete. And transfer’s between GPU and CPU via the .cpu() or .to(device) take forever. Is there</p>

ptrblck: <p>Could you update to the latest stable release and rerun your code, please?</p>

soulslicer: <p>I did and I get the same issue</p>

soulslicer: <p>Okay it turns out i needed to install the server version of the base nvidia driver and not the regular version. That fixed it</p>
<p>nvidia-driver-510-server instead of nvidia-driver-510</p>
How to scale/warmup the learning rate for large batch size?

caesar025: <p>I am trying to run ImageNet training on a large number of GPUs (&lt;64) with the help of PyTorch DDP and a batch size of 64 per GPU. I am unsure how to scale and warm-up the learning rate:</p>
<ul>
<li>the original <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py#L429" rel="noopener nofollow ugc">PyTorch DDP ImageNet example</a> does not scale the learning rate at all and only decays it every 30 epochs</li>
<li>the DALI dataloader with PyTorch DDP <a href="https://github.com/NVIDIA/DALI/blob/2b25d340983ad2b6ee8297d718c39a725b0f7453/docs/examples/use_cases/pytorch/resnet50/main.py#L536" rel="noopener nofollow ugc">implementation</a> scales the learning rate with the number of workers (in relation to a base batch size 256 and also uses 5 epochs of warm-up)</li>
</ul>
<p>However, both cases fail to reach a validation accuracy &lt; 70% when trained with a global batch size larger than 4096 in my case. As a comparison, Horovod reaches ~74% validation accuracy out of the box up to a global batch size 32k using the exact same lr schedule as in the DALI example. How do I need to tweak the LR for PyTorch to work in this case?</p>

wanchaol: <p><a class="mention" href="/u/caesar025">@caesar025</a> thanks for posting!</p>
<p>There’s some previous discussions about how to adjust learning rate when scaling up batch size, did you try it already? <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769/19" class="inline-onebox">Should we split batch_size according to ngpu_per_node when DistributedDataparallel - #19 by junb</a></p>

caesar025: <p>I was already scaling the learning rate with the number of workers, so that was not the issue. My mistake was in the warm-up of the learning rate. As I figured the correct way to do this is:</p>
<pre><code class="lang-auto">    if epoch &lt; args.warmup_epochs:
        lr = lr*float(1 + step + epoch*len_epoch)/(args.warmup_epochs*len_epoch)
</code></pre>
<p>where <code>len(epoch) = len(train_loader)</code>. With this fix I get ~74 validation accuracy for a batch size 32k, so everything good now!</p>
Reading webdataset with TorchData

austinmw: <p>Hi, I’ve been unable to figure out how to read webdataset tars with TorchData. I have images and annotations generated in webdataset format with jpg and cls keys. How can I pair up the jpg and cls items in the tar that are located next to each other?</p>

nivek: <p>It depends on how the files are stored, but here are the DataPipes that will be helpful:</p>
<ol>
<li>
<a href="https://pytorch.org/data/beta/generated/torchdata.datapipes.iter.FileLister.html#torchdata.datapipes.iter.FileLister" rel="noopener nofollow ugc"><code>FileLister</code></a> lists out all your files in your directories in order.</li>
<li>
<a href="https://pytorch.org/data/beta/generated/torchdata.datapipes.iter.TarArchiveLoader.html#torchdata.datapipes.iter.TarArchiveLoader" rel="noopener nofollow ugc"><code>TarArchiveLoader</code></a> (<code>.load_from_tar</code>) loads/decompresses your TAR archives.</li>
<li>
<a href="https://pytorch.org/data/beta/generated/torchdata.datapipes.iter.Grouper.html#torchdata.datapipes.iter.Grouper" rel="noopener nofollow ugc"><code>Grouper</code></a> (<code>.groupby</code>) can group files of the same names together, after that you can open the files, transform them, and return a sample at a time as you see fit.</li>
</ol>
IndexError: select(): index #num out of range for tensor of size #num at dimension 0 in backward for rnn based model

Prerna_Dhareshwar: <p>Hello,</p>
<p>I have a model that has an LSTM layer and during the backward pass after almost 70 iterations of training without error, I suddenly get this error-<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3d62c0011e997a508612a77fb6b5d4eeecce6378.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d62c0011e997a508612a77fb6b5d4eeecce6378" title="52%20AM"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3d62c0011e997a508612a77fb6b5d4eeecce6378.png" alt="52%20AM" data-base62-sha1="8L2HbRcMptfhL5fhhcYjRgnmH5m" width="690" height="129" data-dominant-color="0B0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">52%20AM</span><span class="informations">1132×212 3.57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have no idea how to go about debugging this.</p>
<p>Thanks</p>
<p>EDIT: (SOLVED) I was making an error in the inputs to pack_padded_sequence. The size of inputs was 3290800 but the argument lengths was 3290801, I wasn’t slicing my tensor correctly.</p>

Nikos_Spanos: <p>I have a similar error. But my input size is correct with the sequence length. [32,1,100] and argument length [32]. Do you have any clue what might the problem be? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <a href="https://discuss.pytorch.org/t/loss-backward-indexerror-select-index-1-out-of-range-for-tensor-of-size-1-32-100-at-dimension-0/139530">Here</a> is my question on this forum</p>

Ziyu_Huang: <p>I think this relates to cpp code, and simply because of index error. I meet this error and solve it.<br>
I am writing cuda code using cpp version’s torch, and I create a tensor of size 4*2, initially I guess in cpp it will be of 1D, but I got: IndexError: select(): index 4 out of range for tensor of size [4, 2] at dimension 0<br>
Then I use xxx[i][j] and restrict i (0, 3), j (0, 1), and it works!<br>
So check the index, and maybe you could use a higher dimension expression.</p>
Need for un-normalizing DCGAN results?

fyp_fighter: <p>I have been working on a DCGAN to generate images, based on this <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.<br>
I performed transformation on the training dataset.</p>
<pre><code class="lang-auto">dataset_mean = [0.05941391, 0.05687975, 0.05648475]
dataset_std = [0.05552825, 0.05573225, 0.05731734]

dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.RandomHorizontalFlip(p=0.1),
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               # SCALES DATA TO RANGE [0,1]
                               transforms.ToTensor(),
                               # NORMALIZES DATA TO CERTAIN RANGE,
                               # WITH DATASET-SPECIFIC MEAN AND STD
                               transforms.Normalize((dataset_mean[0], dataset_mean[1], dataset_mean[2]),
                                                    (dataset_std[0], dataset_std[1], dataset_std[2])),
</code></pre>
<p>After normalization, the training data should look very weird and noisy.<br>
I am just wondering <strong>Q1) if an un-normalization operation a must on the output of DCGAN after training?</strong></p>
<p>I have seen people discussing and even <a href="https://discuss.pytorch.org/t/simple-way-to-inverse-transform-normalization/4821/13">implementing it</a> (This is the method I used to calculate <em>dataset_mean</em> and <em>dataset_std</em>) . But it is at the same time hardly seen on DCGAN/GAN tutorials.</p>
<p>Also, I tried following the tutorial and setting the mean and std all as 0.5. The output already looks pretty well. After using the specific mean and std, the results are not even close to expected. <strong>Q2) Why is it so?</strong> Although I have not tried un-normalizing it.</p>
<p>Thanks in advance.</p>
Tensor.cuda() works but tensor.to("cuda") does not work

JanoschMenke: <p>Hi guys, I have this issue where <code>to("cuda")</code> does nothing to some of my tensors. Does not throw an erro but when I check afterwards, the tensor still throws <code>tensor.is_cuda &gt;&gt;&gt; False</code> Only when I am using <code>.cuda()</code> The loss calculation can be performed. I cannot find a minimal working example that reproduces the error, I assume it has something to do with the data preparation.  Below is an lose example of what I am talking about. Maybe someone has experienced a similar problem, and knows what my mistake is.</p>
<pre><code class="lang-auto">device = "cuda"
model.to(device)
clf_targets.to(device)
input.to(device)
output = model(input)
loss = criterion(output, clf_targets) 
</code></pre>
<p>does not work as <code>clf_targets</code> remains on the cpu.</p>
<pre><code class="lang-auto">device = "cuda"
model.to(device)
clf_targets.to(device)
input.to(device)
output = model(input)
loss = criterion(output, clf_targets.cuda()) 
</code></pre>
<p>does work.</p>

Forceless: <p>you need</p>
<pre><code class="lang-python">device = "cuda"
model.to(device)
clf_targets.to(device)
input = input.to(device)
output = model(input)
loss = criterion(output, clf_targets.cuda()) 
</code></pre>
<p>Model don’t need assign after move, but tensor need</p>

JanoschMenke: <p>Yes that is the right solution. Thanks. I had a  Brainfart</p>
Weight Gradients Not Same for Same Input

mirazul_haque: <p>I am trying calculate weight gradients of a layer w.r.t CE loss function given an input. However, if I do the inference for multiple times, for same input I am getting different weight  gradients. I am using ResNet-18 model for this. What can be the cause for this?</p>

ptrblck: <p>I cannot reproduce the issue by checking the weight gradient of a random layer:</p>
<pre><code class="lang-python">model = models.resnet18()
x = torch.randn(1, 3, 224, 224)

out = model(x)
out.mean().backward()
print(model.layer4[0].conv1.weight.grad.abs().sum())

model.zero_grad()
out = model(x)
out = model(x)
out = model(x)
out.mean().backward()
print(model.layer4[0].conv1.weight.grad.abs().sum())
</code></pre>
<p>and get the same grad stats.<br>
Could you post a minimal, executable code snippet showing the issue?</p>

mirazul_haque: <p>I found the issue by looking at your code. I missed zero_grad() and after adding that code works fine. Thanks a lot!</p>
How to convert tensor image from Generator to base64 for sending?

prismspecs: <p>I have a flask server that I want to listen to JSON requests for Z input (working), generate an image using that vector (working), and then send the image using localhost to another app. It should be base64 encoded, as I understand it. How do I transform the tensor into this format?</p>
<pre><code class="lang-auto">
@app.route("/json", methods=['GET', 'POST', 'PUT'])
def getjsondata():


    if request.method=='POST':
        print("received POST")

        data = request.get_json()

        #print(format(data['z']))
        jzf = [float(i) for i in data['z']]
        jzft = torch.FloatTensor(jzf)
        jzftr = jzft.reshape([1, 512])

        z = jzftr.cuda()
        c = None                   # class labels (not used in this example)
        trunc = 1
        img = G(z, c, trunc)
        img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)

        # this is how I was saving image to disk, but I would rather not save before sending
        # PIL.Image.fromarray(img[0].cpu().numpy(), 'RGB').save('newtest.png')

        # this isn't working
        pil_image = PIL.Image.fromarray(img[0].cpu().numpy(), 'RGB')
        encoded_string = base64.b64encode(pil_image)

        return encoded_string</code></pre>

prismspecs: <p>I managed a solution</p>
<pre><code class="lang-auto">@app.route("/json", methods=['GET', 'POST', 'PUT'])
def getjsondata():

    if request.method=='POST':
        # print("received POST")

        data = request.get_json()

        #print(format(data['z']))
        jzf = [float(i) for i in data['z']]
        jzft = torch.FloatTensor(jzf)
        jzftr = jzft.reshape([1, 512])

        z = jzftr.cuda()
        c = None                   # class labels (not used in this example)
        trunc = 1
        img = G(z, c, trunc)

        #img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)
        img = (img * 127.5 + 128).clamp(0, 255).to(torch.uint8)

        # turn into PIL image
        pil_img = transforms.ToPILImage()(img[0]).convert("RGB")
        #pil_img = PIL.Image.fromarray(img[0].cpu().numpy(), 'RGB')
        pil_img.save('newtest.png')

        response = serve_pil_image64(pil_img)
        response.headers.add('Access-Control-Allow-Origin', '*')
        # response.headers.add('Content-Transfer-Encoding', 'base64')
        return response


    return 'OK'

def serve_pil_image64(pil_img):
    img_io = BytesIO()
    pil_img.save(img_io, 'JPEG', quality=70)
    img_str = base64.b64encode(img_io.getvalue()).decode("utf-8")
    return jsonify({'status': True, 'image': img_str})

</code></pre>
How to change cyfunction to function

MartinZhang: <p>Hello EveryOne.</p>
<p>there is a cyfunction , as follows:</p>
<pre><code class="lang-auto">&lt;cyfunction Tanh.forward at ...&gt;
</code></pre>
<p>how to change it  to function?</p>
<pre><code class="lang-auto">&lt;function Tanh.forward at ...&gt;
</code></pre>
<p>Thanks.</p>
Creating a Dataloader Buffer

DerekGloudemans: <p>As I see it, the standard use of the DataLoader class is a series of operations:</p>
<ol>
<li>Call the dataloader.</li>
<li>It performs some loading operations and returns the result.</li>
<li>Then, the result of the dataloader is used for some operations by the main code.</li>
</ol>
<p>My issue with this is that the loading operations are blocking and take sometimes significant portions of time. Conceivably, though, the loading operation could be performed ahead of time such that the result was stored in memory associated with the dataloader. Then, when the dataloader was called, the result could simply be returned with no processing (already in memory). Then, the next batch of data could be processed and loaded into memory WHILE THE MAIN CODE WAS EXECUTING, IN PARALLEL.</p>
<ol>
<li>Call the dataloader, and it immediately returns result stored in buffer</li>
<li>Result of dataloader is used by main code, and simultaneously a separate process loads next data into the buffer</li>
</ol>
<p>This should be possible using multiprocessing, essentially creating a buffer. However, using the multiprocessing functionality of the DataLoader I don’t think it is.</p>
<p>Does anyone know of a way to do this cleanly using existing Pytorch libraries? I’m confident I can do it with python’s multiprocessing module but I’d prefer to avoid that since working with tensors across mutiple processes can get a bit hairy.</p>

ptrblck: <aside class="quote no-group" data-username="DerekGloudemans" data-post="1" data-topic="80902">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/derekgloudemans/48/24109_2.png" class="avatar"> DerekGloudemans:</div>
<blockquote>
<p>However, using the multiprocessing functionality of the DataLoader I don’t think it is.</p>
</blockquote>
</aside>
<p>Why do you think it is not working in the current <code>DataLoader</code> implementation?<br>
The <code>num_workers</code> argument will specify the number of processes, which will load and process each batch in the background, while the main training loop is busy.</p>

mythrex: <p>So basically, there is a buffer implementation when using Dataloaders, right?</p>

ptrblck: <p>Each process will load an entire batch and add it to a queue. Assuming you are seeing this as a buffer, then yes.</p>

Matias_Vasquez: <p>Maybe what you want is something like the <code>@functools.lru_cache()</code> decorator.</p>
<p>You could try putting it in your dataset, right before the <code>__get_item___(self, index)</code> method.</p>
<p><a href="https://docs.python.org/3/library/functools.html" rel="noopener nofollow ugc">Here</a> is the documentation for the decorator.</p>
<p><a href="https://www.manning.com/books/deep-learning-with-pytorch#toc" rel="noopener nofollow ugc">Here</a> is a book with an example of how to use it. (Chapter 10.2 - Listing 10.2 - Book: Deep Learning with PyTorch)</p>
Gradients through multiple forwards

Guillaume_Jeanneret: <p>Hi all,</p>
<p>I’m trying to compute the loss gradient using a model multiple times. Unfortunately, my GPU is small, so I cannot calculate the backprop directly from the loss function.<br>
ATM, I tried to compute it like this:</p>
<pre><code class="lang-auto">x_in  # input image
x_in.requires_grad = True
x_t = torch.clone(x_in)
for i in range(50):
    x_t = model(x_t)
    model.zero_grad()
loss = criterion(x_t)
grads = torch.autograd.grad(loss, x_in)[0]
</code></pre>
<p>Knowing the problem, I tested it using the chain rule iteratively, i.e., I computed the derivatives of the output tensor wrt the input tensor. Something like this:</p>
<pre><code class="lang-auto">x_in  # is an input image
x_t = x_in.detach()
grads = torch.ones_like(x_t)

for i in range(50):
    x_in = x_t.detach()
    x_in.requires_grad = True
    x_t = model(x_in)

    # compute the derivatives wrt the output
    prev_grads = torch.autograd.grad(x_t.sum(), x_in)[0]

    # chain rule
    with torch.no_grad():
        grads *= prev_grads

x_in = x_t.detach()
x_in.requires_grad = True
loss = criterion(x_in)

return grads * torch.autograd.grad(loss, x_in)[0]
</code></pre>
<p>If I do it this way, it works. Nonetheless, I am unsure if what I’ve done is correct. Can you help me by giving me some insights into correctly doing it?</p>
<p>Thank you very much</p>

suraj.pt: <p>Hi,</p>
<p>I’m not sure about your approach. If your GPU is small, you can try using gradient checkpointing (<a href="https://github.com/cybertronai/gradient-checkpointing#how-it-works">nice visual explanation</a>) with <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a> in your model’s forward.</p>

Guillaume_Jeanneret: <p>Thank you very much! I didn’t know that function existed.</p>
Question about inference pipeline for object detection

Lu_Bin_Liu: <p>I’m new to PyTorch and I’m wondering what is the most efficient way to load data (from a folder of .jpgs or downsampling an mp4) and running batch inference using a trained Detectron2 model, to minimize the bottleneck from loading the data.</p>

Matias_Vasquez: <p>If your data is subdivided in folders like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfafa1b37b0f01c2ffa933a37a67d96d6a57c835.png" alt="image" data-base62-sha1="vUOLqyHMWEawMnHc1gycLjLT5Dn" width="245" height="173"></p>
<p>You might want to look into <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html" rel="noopener nofollow ugc">ImageFolder</a> for images or <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.DatasetFolder.html#torchvision.datasets.DatasetFolder" rel="noopener nofollow ugc">DatasetFolder</a> for more generic stuff. In there you can define which files you want to load and your dataset will automatically have a label (0 to n, where n is the number of classes).</p>
<p>Your can also follow this tutorial to see how you can create your own custom Dataset class<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class</a></p>
Checking CUDA compatibility

mahmoodn: <p>Hi<br>
How can I find whether pytorch has been built with CUDA/CuDNN support? Is there any log file about that?</p>

ptrblck: <p>You could use <code>print(torch.__config__.show())</code> to see the shipped libraries or alternatively something like:</p>
<pre><code class="lang-python">print(torch.cuda.is_available())
print(torch.version.cuda)
print(torch.backends.cudnn.version())
</code></pre>
<p>would also work.</p>
About libtorch "model.to(device)"

salt0107fish: <p>Hi, I got a question when I Debug my c++ code.<br>
The code is just like following:</p>
<pre><code class="lang-auto">torch::DeviceType device_type;
device_type = torch::kCUDA;
torch::Device device(device_type);
…………
auto img_var = torch::from_blob(mRGB.data, dims, torch::kFloat32).to(device);
…………
modle_vlad.to(device);
modle_vlad.eval();
</code></pre>
<p>And the error is about “modle_vlad.to(device);”</p>
<pre><code class="lang-auto">  what():  _ivalue_ INTERNAL ASSERT FAILED at "../torch/csrc/jit/api/object.cpp":19, please report a bug to PyTorch. 
Exception raised from _ivalue at ../torch/csrc/jit/api/object.cpp:19 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x69 (0x7fff074b9b29 in /usr/local/libtorch1.8.2/libtorch/lib/libc10.so)
frame #1: torch::jit::Object::_ivalue() const + 0x2c8 (0x7fff8435dc38 in /usr/local/libtorch1.8.2/libtorch/lib/libtorch_cpu.so)
frame #2: &lt;unknown function&gt; + 0x3484f38 (0x7fff8435cf38 in /usr/local/libtorch1.8.2/libtorch/lib/libtorch_cpu.so)
frame #3: torch::jit::Module::to_impl(c10::optional&lt;c10::Device&gt; const&amp;, c10::optional&lt;c10::ScalarType&gt; const&amp;, bool) + 0x15a (0x7fff8434d28a in /usr/local/libtorch1.8.2/libtorch/lib/libtorch_cpu.so)
frame #4: torch::jit::Module::to(c10::Device, bool) + 0x37 (0x7fff8434e377 in /usr/local/libtorch1.8.2/libtorch/lib/libtorch_cpu.so)
…………
</code></pre>
<p>It is strange that I have no difficulty to transform the cv::Mat to GPU tensor , but it shows that some errors ocurr when I try to transform my model to GPU.  I used the same code to transform other model to GPU a few days before, and it had no problem.<br>
Could you please help me with the question?</p>

ptrblck: <p>In case you are seeing this error using the latest release, could you please create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> with a minimal, executable code snippet to reproduce the issue, please?</p>
Plot 4D tensor as image

danielmanu93: <p>Hi All, I’m trying to plot a randomly sampled image from my test data as shown in the code below but I get this error <code>TypeError: Invalid dimensions for image data</code>. The image tensor is a 4D tensor with shape (TestBatchSIze, Channels, height, width). Any ideas to help resolve this error and plot the data are welcomed.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f16770cb054588ab1883ab70d876c874222a6bc0.png" alt="1" data-base62-sha1="yryGPzNLOheiabiFZpii9uGluZa" width="644" height="202"></p>

ptrblck: <p><code>plt.imshow</code> expects a numpy array in the shape <code>[height, width, channels]</code>.<br>
Given your input has the shape <code>[batch_size, channels, height, width]</code> this should work:</p>
<pre><code class="lang-python">batch_size, channels, height, width = 2, 3, 224, 224
x = torch.randn(batch_size, channels, height, width)

# select a sample from the batch
img = x[0]
# permute to match the desired memory format
img = img.permute(1, 2, 0).numpy()
plt.imshow(img)
</code></pre>

danielmanu93: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>. The no. of channels in my image tensor is 10, hence when I use permute I still get the same error. I guess permute expects the no. of channels to be 3 or 4. Is there a way to plot if the no. of channels is more than 4?</p>

ptrblck: <p><code>permute</code> will work on any number of channels, as it’s only permuting the dimensions.<br>
<code>plt.imshow</code> expects an “image format”, i.e. a numpy array with either 3 channels (RGB), a single channel (grayscale), or no channel dimension (also grayscale or an arbitrary matrix which will be visualized using the specified colormap).<br>
10 channels do not represent a valid image format so you won’t be able to visualize it directly.<br>
I don’t know what this data represents, but you might want to create 10 subplots, where each one visualizes a single channel.</p>
LSTM memory consumption issue

Nathan_Goedeke: <p>Hello,<br>
I have a network that uses 8 LSTM layers of size 512 each, with a batch size of 1024 and a seq length of 512. The result seems to cause python to use 100% of memory and cause my PC to freeze up, even though I have upgraded to 32 GB of RAM.<br>
I am using a large batch size because I have a very large training dataset so it seems to make sense to get through the data quicker. When I make a smaller batch size and seq length then it doesn’t freeze up, but is unreasonably slow.</p>
<ol>
<li>Is there a way to have PyTorch not consume so much memory?</li>
<li>Would it help if I increased the RAM again? It seems like it made no difference</li>
<li>Is the size of the batches and / or architecture unreasonable? My limited experience with RNNs seems like the architecture isn’t that big, but maybe I’m mistaken</li>
</ol>
<p>Thanks,<br>
Nathan</p>

ptrblck: <p>Based on your description I guess that either the swap is used as your RAM is already full or the process gets indeed stuck and the OS should kill it.</p>
<ol>
<li>If your current training really needs this amount of memory, there wouldn’t be a workaround besides e.g. lowering the batch size or using <code>torch.utils.checkpoint</code> to trade compute for memory. Check if you are storing unnecessary data and could remove it.</li>
<li>Maybe, but you should check how much the memory usage increases depending on the batch size. E.g. check the RAM usage for a batch size of 1, 2, 3, … and try to estimate how much memory would be needed for your desired batch size.</li>
</ol>