akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
My test loss increase after 10 epochs but the train loss converged well

nikiguo93: <p>Hi I used the resnet50 and I found that after 10 epochs, my test loss increased(before that it decreased) but the train loss converged. Can anybody offer me some solutions to solve this problem?</p>

soikat_hasan: <p>Your network is overfitting. I believe, your dataset for training is small. If yes, try to use larger dataset.</p>

nikiguo93: <p>I don’t have bigger dataset. Are there some other solutions?</p>

soikat_hasan: <p>In this case, try to use data augmentation,  and use learning rate decay. However, I recommend you to share more details about your cureent setups (e.g., task, model, dataset properties, lr rate, etc). Its will be helpful to understand the problem and debug it.</p>
Index assignment in c++

7starsea: <pre><code class="lang-auto">x, x2 = torch.rand(2, 10), torch.rand(2, 3)
ind = torch.tensor([1,3,7], dtype=torch.long)
x[:, ind] = x2
</code></pre>
<p>could anyone tell me how to do last assignment in c++?<br>
I tried</p>
<pre><code class="lang-auto">x.index_select(x, 1, ind) = x2
</code></pre>
<p>but it does not work.</p>

7starsea: <p>the exact correct way is</p>
<pre><code class="lang-auto">x.index_put_({torch::indexing::Slice(), ind}, x2)
</code></pre>
<p>Thanks for letting me know <code>torch::indexing</code> apis.</p>

tom: <p>The direct way to make the initial code work is to use copy_ to a view.</p>
<pre><code class="lang-auto">x.index_select(1, ind).copy_(x2)
</code></pre>
<p>This works in both Python and C++ (but I think sometimes you used to need to assign the view to a Tensor to make it work).</p>

7starsea: <p>I just tested. <code>x.index_select(1, ind).copy_(x2)</code> does not work. I think <code>x.index_select(1, ind)</code> created a temporary tensor.</p>

tom: <p>Oh right. That only works with .select not indexing with multiple indices, sorry.</p>
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1

majlucaaa: <p>Hello I am having this error when I try to quantize this model of Unsupervised segmentation <a href="https://github.com/kanezaki/pytorch-unsupervised-segmentation-tip/blob/master/demo_ref.py" rel="noopener nofollow ugc">demo_ref.py</a>. For this I have been using the <a href="https://github.com/Xilinx/Vitis-AI/blob/master/tools/Vitis-AI-Quantizer/vai_q_pytorch/example/resnet18_quant.py" rel="noopener nofollow ugc">resnet18_quant.py</a> quantization example. I will copy again how the network is defined:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self,input_dim):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(input_dim, args.nChannel, kernel_size=3, stride=1, padding=1 )
        self.bn1 = nn.BatchNorm2d(args.nChannel)
        self.conv2 = nn.ModuleList()
        self.bn2 = nn.ModuleList()
        for i in range(args.nConv-1):
            self.conv2.append( nn.Conv2d(args.nChannel, args.nChannel, kernel_size=3, stride=1, padding=1 ) )
            self.bn2.append( nn.BatchNorm2d(args.nChannel) )
        self.conv3 = nn.Conv2d(args.nChannel, args.nChannel, kernel_size=1, stride=1, padding=0 )
        self.bn3 = nn.BatchNorm2d(args.nChannel)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu( x )
        x = self.bn1(x)
        for i in range(args.nConv-1):
            x = self.conv2[i](x)
            x = F.relu( x )
            x = self.bn2[i](x)
        x = self.conv3(x)
        x = self.bn3(x)
        return x
</code></pre>
<p>Can someone help me?</p>

Mukesh1729: <p>Hey print your target, input shape and see.</p>
<p>If you use a cross entropy loss function then,<br>
The shapes of input and target  should be like:</p>
<p>Example:<br>
Input: (1,1,160,160)<br>
Target:(1,160,160)<br>
Hope this helps</p>

majlucaaa: <p>Hello, yes I am using cross entropy loss function. when I call <code>loss = loss_fn(outputs, labels)</code> I get this error. If I print the size of outputs and labels I have this:<br>
size outputs:  torch.Size([3, 100, 224, 224])<br>
size labels:  torch.Size([3])<br>
So what should I do to have labels as [100,224,244]?</p>

Mukesh1729: <p>Hey,<br>
A torch.Size([3]) is just the batch size that you are passing through. So, to elaborate a bit more torch.Size([3, 100, 224, 224]) - 4 dimension<br>
3 - is batch size<br>
100 - is the number of channels<br>
224, 224, - image (w, h)</p>
<p>Exactly, your target should be torch.Size([3, 224, 224]).<br>
Take a look at the documentation of the CE loss for further refrence.</p>
<p>A small example</p>
<pre><code class="lang-auto">outputs = torch.rand(3, 100, 224, 224).float() # ex: outputs = model(input)
targets = torch.rand(3, 224, 224).long() # try to change it to torch.rand(3) and see

criterion = torch.nn.CrossEntropyLoss()
loss = criterion(input, targets)
print(loss)
</code></pre>
Dataloader adds additional unwanted dimension between __getitem__ and enumerate(dataloader)

4158ndfkvBHJ1: <p>Hello everybody,<br>
I have a dataloader, with a <code>__getitem__</code> function, that reads as follows:</p>
<pre><code class="lang-python">    def __getitem__(self, index):
        
        # pre-processing annotation
        with torch.no_grad():

            annotation = torch.zeros((self.image_size[0], self.image_size[1]))
            points = np.loadtxt(os.path.join(self.annotation_root, self.annotations[index]), delimiter=",")
            image = Image.open(os.path.join(self.image_root, self.images[index]))
            img_size = image.size

            # converting relative coordinates to absolute ones

            if len(points.shape) &lt; 2: 
                points = np.expand_dims(points, axis = 0)
            points = points[:,0:2]
            points[:,0] = np.rint(points[:,0]*img_size[0])
            points[:,1] = np.rint(points[:,1]*img_size[1])


            annotation[points[:,0], points[:,1]] = 1.0
            count = points.shape[0]

            # annotation is padded by receptive field size by default
            annotation = annotation.view((1,1,*annotation.size()))
            sum_annotation = F.conv2d(input = annotation,
                                      weight = self.weights,
                                      stride = self.stride,
                                      padding = self.patch_size)

        print("sum_annotation.size(): {}".format(sum_annotation.size()))

        return self.make_tensor(image), sum_annotation, count
</code></pre>
<p>nothing special about it, just some pre-processing for my annotations. Here, the print returns <code>sum_annotation.size(): torch.Size([1, 1, 921, 1041])</code>. However, doing</p>
<pre><code class="lang-auto">    for epoch in range(args.epochs):
        loss = 0
        for idx, elt in enumerate(train_dataloader):
            input, target, _ = elt
            print("target.size(): {}".format(target.size()))
</code></pre>
<p>gets me a strange print output: <code>target.size(): torch.Size([1, 1, 1, 921, 1041])</code>. Somewhere, an unwanted dimension gets added. I could do something ugly using <code>tensor.view</code> to fix this, but I would like to know if anybody has ever encountered this issue before.<br>
Any help would be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The <code>DataLoader</code> will add the batch dimension to the sample in <code>dim0</code>. Based on your description it seems that you are using <code>batch_size=1</code> which is why the additional dimension with a size of 1 is added.<br>
I don’t know what the other dimension represent, but if you are manually already trying to add a batch dimension, just remove it and let the <code>DataLoader</code> handle it.</p>

4158ndfkvBHJ1: <p>of course, that’s obvious now ! thanks, I had a brainfart … I needed the batch dimension for a pre-processing step involving <code>F.conv2d</code>, but I forgot to remove it before returning this tensor.</p>
Waveglow Cpu Inferencing

Hasan_Khan: <p>I am trying to produce the inference results of <strong>tacotron2</strong> and <strong>waveglow</strong> model on CPU. I have changed all the <strong>cuda tensors</strong> to <strong>cpu</strong> in <code>denoiser.py</code>, <code>glow.py</code> and all the files in which changes were required, But still I am getting this <strong>No CUDA GPUs are available error</strong>.</p>
<p>I have attached the screen shot of the error. Please help e figure out and resolve the problem.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e00709ad0e350ab5c5e079b44a960040f8e9976.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e00709ad0e350ab5c5e079b44a960040f8e9976" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e00709ad0e350ab5c5e079b44a960040f8e9976.png" alt="image" data-base62-sha1="4hpgwtM4eEn9sTEjIBS9JfP88Dk" width="690" height="333" data-dominant-color="F2F1F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">882×426 26.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t know what exactly you’ve changed, but this code works for me without a GPU:</p>
<pre><code class="lang-python">import torch

print(torch.cuda.is_available())

waveglow = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_waveglow', model_math='fp32')
waveglow = waveglow.remove_weightnorm(waveglow)
waveglow.eval()

tacotron2 = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tacotron2', model_math='fp32')
tacotron2.eval()

text = "hello world, I missed you so much"
utils = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tts_utils')
sequences, lengths = utils.prepare_input_sequence([text], cpu_run=True)

with torch.no_grad():
    mel, _, _ = tacotron2.infer(sequences, lengths)
    audio = waveglow.infer(mel)
audio_numpy = audio[0].data.cpu().numpy()
print(audio_numpy.shape)
</code></pre>
<p>after adding <code>map_location='cpu'</code> to the <code>torch.load</code> method in <code>entrypoints.py</code> for both models (since I wanted to use the pretrained models).</p>

Hasan_Khan: <p>Yeah, it worked for me too. Thanks!!</p>
How to change float variables in DataParallel forward function?

wandering007: <p>Example code  is shown as follows:</p>
<pre><code>import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable


class layer(nn.Module):
    def __init__(self):
        super(layer, self).__init__()
        self.fc = nn.Linear(10, 10)
        self.cnt = 0

    def forward(self, x):
        x = self.fc(x)
        self.cnt += 10
        return x

model = nn.DataParallel(layer(), device_ids=[0,1]).cuda()

x = Variable(torch.Tensor(10, 10)).cuda()
out = model(x)
print(model.module.cnt) # output: 0
</code></pre>
<p>The float variable <code>model.module.cnt</code> cannot be modified in forward() function for <code>nn.DataParallel</code> or the multi-gpu case, always resulting in zero. Is there any simple solution, e.g., put the addition operation on cpu?</p>
<p>The issue seems not conform to the official docs:</p>
<blockquote>
<p>Arbitrary positional and keyword inputs are allowed to be passed into DataParallel EXCEPT Tensors. All variables will be scattered on dim specified (default 0). Primitive types will be broadcasted, but all other types will be a shallow copy and can be corrupted if written to in the model’s forward pass. [<a href="http://pytorch.org/docs/master/nn.html#torch.nn.DataParallel" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch master documentation</a>]</p>
</blockquote>

Hiperdyne19012: <aside class="quote no-group" data-username="wandering007" data-post="1" data-topic="10101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wandering007/48/3458_2.png" class="avatar"> wandering007:</div>
<blockquote>
<p><code>.cuda()</code></p>
</blockquote>
</aside>
<p>Did you find a solution?</p>

wandering007: <p>Yes, registering <code>cnt</code> as the buffer of the module solves it.</p>
<pre><code class="lang-auto">self.register_buffer("cnt", torch.tensor(0))
</code></pre>

groffo: <p>Hi! Did you solved this problem?</p>
Deprecation warning with latest torch and opacus

xichens: <p>With the latest versions:</p>
<pre><code class="lang-auto">opacus==0.14.0
torch==1.9.0
</code></pre>
<p>I’m getting the following deprecation warnings when running the <a href="https://github.com/pytorch/opacus/blob/master/examples/cifar10.py" rel="noopener nofollow ugc">CIFAR10 example</a>:</p>
<pre><code class="lang-auto">/local_disk0/.ephemeral_nfs/envs/pythonEnv-4ed4226e-cb69-4f21-bbff-ba9e9ab8fc55/lib/python3.7/site-packages/torch/nn/modules/module.py:974: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
</code></pre>
<p>Is this the expected behavior? Should I be concerned with the warning saying <code>This hook will be missing some grad_input</code>?</p>
<p>Thank you!</p>

ffuuugor: <p>Hi!<br>
Yeah, that’s somewhat expected - we never migrated to <code>register_full_backward_hook </code> since it was released in PyTorch 1.8.<br>
We’ll fix that soon (thanks for raising this!). But in the meantime, as  far as I’m aware, this shouldn’t create any problems (at least with 1.9) - all our tests are passing and we never received any reports of this causing any issues</p>

NBu: <p>I got the same warning message with Pytorch 1.10</p>
RuntimeError: unexpected tensor scalar type

DL_aspirant: <p>I am getting an error saying <code>RuntimeError: unexpected tensor scalar type</code> while exporting my pytorch model to ONNX:</p>
<p>Could someone tell me what I’m missing here?<br>
Thanks!</p>

ptrblck: <p>The error could be raised e.g. in <a href="https://github.com/pytorch/pytorch/blob/4cb534f92ef6f5b2ec99109b0329f93a859ae831/torch/csrc/jit/passes/onnx/helper.cpp#L54-L83">these lines of code</a> which would point to an unknown <code>dtype</code>. Which <code>dtypes</code> are you using in your workflow and could you check, if this <code>dtype</code> is not in the supported list?</p>

Wilson: <p>Not sure if you have resolved the problem. I have come across the same error.</p>
<blockquote>
<p>File “/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py”, line 897, in _graph_op<br>
torch._C._jit_pass_onnx_node_shape_type_inference(n, _params_dict, opset_version)<br>
RuntimeError: unexpected tensor scalar type</p>
</blockquote>
<p>I try to print out the <em>n, _params_dict</em> here and I got</p>
<blockquote>
<p>%552 : Tensor = onnx::Constant[value={0}]()<br>
{}<br>
%553 : Tensor = onnx::Shape(%2)<br>
{}<br>
%554 : Tensor = onnx::Constant[value={0}]()<br>
{}<br>
%555 : Tensor = onnx::Constant[value={0}]()<br>
{}<br>
%556 : Tensor = onnx::Gather[axis=0](%553, %555)<br>
{}<br>
%557 : Tensor = onnx::Constant[value={4}]()<br>
{}<br>
%558 : Tensor = onnx::Constant[value={257}]()<br>
{}<br>
%559 : Tensor = onnx::Constant[value={-1}]()<br>
{}<br>
%561 : Tensor = onnx::Reshape(%0, %560)<br>
{}<br>
Is it due to Reshape()?</p>
</blockquote>

Wilson: <p>I have figure out my problem. it is due to complex number. You can refer to <a href="https://github.com/pytorch/pytorch/issues/59246" class="inline-onebox" rel="noopener nofollow ugc">Unexpected tensor scalar type with torch.onnx.export and ComplexFloat · Issue #59246 · pytorch/pytorch · GitHub</a>.</p>
<p>The error log comes from <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/serialization/export.cpp#L313" class="inline-onebox" rel="noopener nofollow ugc">pytorch/export.cpp at master · pytorch/pytorch · GitHub</a></p>
How to reset GPU status after OOM raising?

huahuanZ: <p>say I’m testing some codes and monitoring the GPU memory allocated with <code>torch.cuda.max_memory_allocated()</code><br>
Edit: I delete the misleading toy example.</p>
<p>The situation is that I run a benchmark over several different configurations and log their execution time, GPU memory footprint, etc.<br>
And the issue is when one of the configuration leads to CUDA OOM, the benchmark of next configuration always produces error memory stats (I know it’s wrong by exchanging the execution order). How can we avoid such thing?<br>
A typical error message like</p>
<pre><code class="lang-auto">...

T=150   U=20    V=5000  N=1     time=2.70       memory=243
T=150   U=20    V=5000  N=16    time=50.05      memory=3842
T=150   U=20    V=5000  N=32    time=128.20     memory=7684
T=150   U=20    V=5000  N=64    time=276.37     memory=15360
T=150   U=20    V=5000  N=128   error=CUDA out of memory. ...

T=1500  U=300   V=50    N=1     time=16.21      memory=23044   &lt;- Error occurs, this should be much smaller
T=1500  U=300   V=50    N=16    time=78.82      memory=5763
T=1500  U=300   V=50    N=32    time=209.08     memory=11520
T=1500  U=300   V=50    N=64    time=398.86     memory=23044
</code></pre>
<p>The repo can be found here <a href="https://github.com/maxwellzh/warp-rnnt" rel="noopener nofollow ugc">maxwellzh/warp-rnnt: CUDA-Warp RNN-Transducer (github.com)</a><br>
in <code>pytorch_binding/benchmark.py</code></p>

gphilip: <p>It is not clear to me what you want to do.</p>
<p>What do you mean by “resetting error states”? By “error memory stats” do you mean “incorrect memory stats”?</p>
<p>Are you saying that after an OOM error, <code>torch.cuda.max_memory_allocated()</code> returns a wrong value? If so, how do you know that this is a wrong value?</p>

huahuanZ: <p>Hi, <a class="mention" href="/u/gphilip">@gphilip</a> I edited topic, it should be more clear now.</p>
How to solve this error- Size Mismatch error

Hanen_Dh: <p>I am new to pyTorch and getting following Size Mismatch error when i try to run the DA-GAN:</p>
<p>size mismatch for module.conv1.filter.weight: copying a param of torch.Size([96, 3, 5, 5]) from checkpoint, where the shape is torch.Size([96, 1, 5, 5]) in current model.</p>

gphilip: <p>I think this is what is happening: Your model expects 3-channel images, and some input instance has only one channel.</p>

ptrblck: <p>The error seems to point to a shape mismatch while loading a <code>state_dict</code>, so my guess would be that the model architecture might have been changed after storing the <code>state_dict</code>.</p>
How to set batch size correctly when using multi-GPU training?

AndreaSottana: <p>Hi,<br>
I have a question on how to set the batch size correctly when using DistributedDataParallel.<br>
If I have N GPUs across which I’m training the model, and I set the batch size of the DataLoader to 16, would the effective batch size be 16 or 16 x N?</p>
<p>Here is a small worked example to make it clearer.</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
import torch.multiprocessing as mp
import torch.distributed as dist
from argparse import ArgumentParser
import os


class MyModel(nn.Module):
    def __init__(self, input_dim, inner_layer_1, inner_layer_2, output_dim):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, inner_layer_1)
        self.fc2 = nn.Linear(inner_layer_1, inner_layer_2)
        self.fc3 = nn.Linear(inner_layer_2, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.softmax(x, dim=1)
        return x


def train(gpu_number, n_epochs, model, train_data, optimizer, loss_fn, log_interval=2):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    torch.distributed.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=2,  # total number of gpus
        rank=gpu_number
    )

    sampler = DistributedSampler(train_data, num_replicas=2, rank=gpu_number)
    trainloader = DataLoader(train_data, batch_size=16, sampler=sampler)

    #torch.cuda.set_device(gpu_number)
    model = model.cuda(gpu_number)
    model = DDP(model, device_ids=[gpu_number], output_device=gpu_number)
    for epoch in range(n_epochs):
        for i, batch in enumerate(trainloader):
            inputs, labels = batch[:,:8].cuda(gpu_number), batch[:,-2:].cuda(gpu_number)
            optimizer.zero_grad()
            outputs = model.forward(inputs)

            loss = loss_fn(outputs, labels)
            loss.backward()
            optimizer.step()
    dist.barrier()

if __name__ == "__main__":
    train_data = torch.rand(30000, 100)
    n_epochs = 4
    learning_rate = 0.001
    model = MyModel(8, 800, 300, 2)
    loss_fn = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=learning_rate) 
    mp.spawn(train, nprocs=2, args=(n_epochs, model, train_data, optimizer, loss_fn, 2))

</code></pre>
<p>in this line <code>trainloader = DataLoader(train_data, batch_size=16, sampler=sampler)</code> I set the batch size to 16, but have two GPUs. What would be the equivalent / effective batch size? Would it be 16 or 32 in this case?</p>

huahuanZ: <p>The valid batch size is <code>16*N</code>. 16 is just the batch size in each GPU. During loss backward, DDP makes all-reduce to average the gradients across all GPUs, so the valid batch size is 16*N.</p>
LibTorch 1.8.2+cuda11.1 download link is invalid

HiroshiYAMA: <p><a href="https://download.pytorch.org/libtorch/cu111/libtorch-shared-with-deps-1.8.2%2Bcu111.zip" rel="noopener nofollow ugc">Download here (Pre-cxx11 ABI):</a></p>
<p><a href="https://download.pytorch.org/libtorch/cu111/libtorch-cxx11-abi-shared-with-deps-1.8.2%2Bcu111.zip" rel="noopener nofollow ugc">Download here (cxx11 ABI):</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8851edb35cda2b707b466b0027838f84ed68618.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8851edb35cda2b707b466b0027838f84ed68618" title="invalid_download_link_182+cu111"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8851edb35cda2b707b466b0027838f84ed68618_2_690x368.png" alt="invalid_download_link_182+cu111" data-base62-sha1="zsvCzl2HmN8wpJpzik7khbPNLfO" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8851edb35cda2b707b466b0027838f84ed68618_2_690x368.png, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8851edb35cda2b707b466b0027838f84ed68618.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8851edb35cda2b707b466b0027838f84ed68618.png 2x" data-dominant-color="F2E0DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">invalid_download_link_182+cu111</span><span class="informations">814×435 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HiroshiYAMA: <p>Already resolved.<br>
Now, I was able to download it.<br>
Thank you very much.</p>
How to do: [0,0,1,2,2,2,3,3] -> [2,1,3,2]?

Olafyii: <p>Sorry for the confusing title. Is there a neat way to do this: given an 1-d tensor, its value starts with 0, after some zeros it changes to one, then after some ones change to two… and I want to get a tensor indicating how many zeros, ones, twos… Assume that tensor[i] &lt;= tensor[j] iff i &lt;= j.</p>
<p>E.g. [0,0,1,2,2,2,3,3] → [2,1,3,2], there are two zeros, one one, three twos, and two threes, so the target tensor is [2,1,3,2].</p>

Olafyii: <pre><code class="lang-auto">a = torch.tensor([0,0,1,2,2,2,3,3])
(a == torch.arange(a.max()+1)[:, None]).sum(dim=1)
</code></pre>
<p>Found one way.</p>
Nan or Inf Error during Training

DSL: <p>Hello all,</p>
<p>I am working on a depth estimation model. I have extended this model with a discriminator as in <a href="https://arxiv.org/pdf/1611.07004.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1611.07004.pdf</a>. Now I get “Warning: NaN or Inf found in input tensor” all the time while training. I wonder if this has a negative effect on my training and what I can do about it, does anyone have any ideas or tips?<br>
The generator uses a pre-trained model and the discriminator is randomly initialized. I will now still initialize the weights as in the paper with the normal distribution. If that doesn’t help, my idea was to pre-train the discriminator and then later train everything in the composite.<br>
I would be very grateful for any suggestions or ideas.</p>

Sylvain_Ard: <p>try to decrease the learning rate</p>

AlphaBetaGamma96: <p>This will probably have an effect on training as you want neither <code>NaNs</code> nor <code>Infs</code> in your code at all. What you should do is first isolate what process is causing the <code>NaNs</code>/<code>Infs</code> to be created in the input. Secondly, which inputs exactly? I’d assume this is the inputs to your discriminator?</p>
<p>You can locate where the <code>NaNs</code> are occurring by using <code>torch.autograd.detect_anomaly</code> the docs for this function is here → <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.9.0 documentation</a></p>

DSL: <p>This command is great. I have now solved this differently and queried whether the value is infinite. I have also fixed the error. I used crossentropy with sigmoid function on my discriminator and afterwards it worked. I assume that my values from the discriminator were unstable at the beginning.</p>
GRU model does not free the cached memory automatically

IcarusWizard: <p>Hi, everyone. I find that GRU model does not free the cached memory automatically. Here is the code sample I use.</p>
<pre><code class="lang-python">import torch
data = torch.rand(998, 400, 23).cuda() 
# 1183M memory usage
model = torch.nn.GRU(23, 128).cuda()
# 1247M memory usage
with torch.no_grad():
    h, o = model(data)
# 2811M memory usage
torch.cuda.empty_cache()
# 1443M memory usage
</code></pre>
<pre><code class="lang-python">import torch
data = torch.rand(998, 400, 23).cuda() 
# 1183M memory usage
model = torch.nn.GRU(23, 128).cuda()
# 1247M memory usage
h, o = model(data)
# 2811M memory usage
torch.cuda.empty_cache()
# 2613M memory usage
</code></pre>
<p>Since I call the mode within the <code>no_grad</code> context, I expect no intermedium nodes will be created. But the result memory usage is no effect by the context. I wonder it is a feature or a bug for PyTorch? If it is a feature, how these cached memory will be used in the future?</p>

eqy: <p>I am a bit confused here, do you expect the memory usage to go down to what is needed for the bare PyTorch/CUDA context? The variables and model layers are still in scope so they are expected to use memory.</p>

IcarusWizard: <p>Sorry to cause the confusion. What I expect is that after I call <code>model.forward</code> with <code>no_grad</code>, only the outputs, i.e. <code>o</code> and <code>h</code>, occupy the memory and all the local variables inside the <code>forward</code> call is automatically freed. But as the example shows, I need to manually call <code>torch.cuda.empty_cache()</code> to free them.</p>

eqy: <p>Thanks for the clarification! I think this behavior suggests that even with <code>no_grad</code>, extra memory is allocated first and the deallocated in the caching allocator rather than being skipped. I wonder if this is true for other layers as well…</p>

IcarusWizard: <p>I have tried <code>GRU</code>, <code>LSTM</code>, <code>Linear</code> and <code>Conv2d</code>, only the recurrent models have this issue.</p>

ptrblck: <p>As <a class="mention" href="/u/eqy">@eqy</a> said, intermediate tensors must be computed at one point even if they are released afterwards. Otherwise you wouldn’t be able to pass any forward activations to the next operation. Since you are observing the usage of intermediates in RNNs, my guess is that the intermediate forward activations are computed before being passed to the next time step and released afterwards. This would also explain, why the memory is in the cache (not allocated).</p>
Load openvino for image segmentation

thalia27: <p>Hello everyone,</p>
<p>I am currently trying to speed up the performance of my pytorch model. This is what I did to load my pth file.</p>
<pre><code class="lang-auto">DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
self.model = UNET(in_channels=1, out_channels=1).to(DEVICE)
load_checkpoint(torch.load(config["load_checkpoint"], map_location=DEVICE), self.model)
 model.eval()
        img_tensor = tfms(img).unsqueeze(0)
        with torch.no_grad():
            preds = torch.sigmoid(model(img_tensor))
            preds = (preds &gt; 0.5).float()
            tensor_to_pil = transforms.ToPILImage()(preds.squeeze_(0))


</code></pre>
<p>I read in some articles, we can speed up the performance by using openvino.<br>
Here I tried</p>
<pre><code class="lang-auto">ie = IECore()
# Loading IR files
net = IENetwork(model="Unet.xml", weights=Unet.bin")
# Loading the network to the inference engine
self.mod = ie.load_network(network=net, device_name="CPU")
img_tensor = tfms(img).unsqueeze(0)
with torch.no_grad():
            preds = torch.sigmoid(mod(img_tensor))
            preds = (preds &gt; 0.5).float()
            tensor_to_pil = transforms.ToPILImage()(preds.squeeze_(0))

</code></pre>
<p>and I got an error</p>
<pre><code class="lang-auto">TypeError: 'openvino.inference_engine.ie_api.ExecutableNetwork' object is not callable
</code></pre>
<p>how can I solve this?<br>
BR,<br>
Thalia</p>
Is it necessary to Create a Custom dataset? what are the pros?

Aymen_Sekhri: <p>Hello Everyone,<br>
Actually, I followed many Pytorch tutorials and I found that most of them speak about how to implement Custom Datasets and I am wondering why they are so important?<br>
Especially in computer vision where All datasets follow certain organizational patterns and there are built-in functions to load the data easily (in torchvision).<br>
So what makes me choose to implement a custom dataset?<br>
Thank you in advance</p>

ptrblck: <aside class="quote no-group" data-username="Aymen_Sekhri" data-post="1" data-topic="131196">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aymen_sekhri/48/32759_2.png" class="avatar"> Aymen_Sekhri:</div>
<blockquote>
<p>Especially in computer vision where All datasets follow certain organizational patterns and there are built-in functions to load the data easily (in torchvision).</p>
</blockquote>
</aside>
<p>While some public datasets follow a specific pattern (and thus can be loaded using a <code>torchvision</code> dataset), a custom <code>Dataset</code> gives you the ability to load your custom dataset using any logic you want. I.e. you don’t have to follow a specific folder structure, can apply transformations using different conditions, could create the target using a custom logic etc.</p>
Improve results

Sylvain_Ard: <p>Hello !<br>
I succeeded to train my own dataset with code <a href="https://github.com/WXinlong/SOLO" class="inline-onebox" rel="noopener nofollow ugc">GitHub - WXinlong/SOLO: SOLO and SOLOv2 for instance segmentation, ECCV 2020 &amp; NeurIPS 2020.</a> with soloV2 and ResNet 101<br>
But the results are relatively bad.<br>
How can I improve the results please ?<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>nobody has any idea ?</p>

ptrblck: <p>Please don’t bump a topic after 3 hours.<br>
As a general advice, I would try to overfit a small data sample by playing around with some hyperparameters, and once this is done scale up the use case again.<br>
Also, some (open) ML courses might be helpful, e.g. you might want to take a look into the FastAI course.</p>
The test inputs have different shape from the trained model, while I used 32 batches for training the model I use just one batch for testing

AM.MO: <p>this the model config.<br>
CNN(<br>
(hidden_layers): Sequential(<br>
(0): Conv2d(1, 32, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))<br>
(1): ReLU()<br>
(2): Conv2d(32, 32, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))<br>
(3): ReLU()<br>
(4): Conv2d(32, 32, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))<br>
(5): ReLU()<br>
(6): Conv2d(32, 32, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))<br>
(7): ReLU()<br>
)<br>
(output_layer): Conv2d(32, 1, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))<br>
)</p>
<p>and my test inputs shape is [1,90,90]<br>
how can I fit the test input to evaluate the model??</p>

ptrblck: <p>PyTorch models don’t have a shape dependency regarding the batch dimension, so you would be able to pass the test inputs with a batch size of <code>1</code> to the model.</p>
Update 0-dim tensor value

pallgeuer: <p>How do you update the value of a 0-dim tensor in-place?</p>
<pre><code>import torch
t = torch.tensor(0, device='cuda')

# Now we want to set the value to 8 without creating a new tensor...
t[0] = 8  # &lt;-- Bad: Fails with IndexError
t = 8  # &lt;-- Bad: t is an int, not a Tensor, after this
</code></pre>
<p>What is the official/recommended way I’m supposed to achieve this?</p>

gphilip: <aside class="quote no-group" data-username="pallgeuer" data-post="1" data-topic="131047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pallgeuer/48/16079_2.png" class="avatar"> pallgeuer:</div>
<blockquote>
<p>the value of a 0-dim tensor</p>
</blockquote>
</aside>
<p>Can a tensor which has a zero for the length of any dimension, contain values?</p>

pallgeuer: <p>Yes it can and is not uncommon, try the code out. 0-dim basically means it is a single scalar value and not a 1-dim list that currently only happens to contain a single value.<br>
Note that 0-dim means that is has no dimensions, as opposed to one of the dimensions having a size of 0.</p>

gphilip: <aside class="quote no-group" data-username="pallgeuer" data-post="3" data-topic="131047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pallgeuer/48/16079_2.png" class="avatar"> pallgeuer:</div>
<blockquote>
<p>Yes it can and is not uncommon, try the code out.</p>
</blockquote>
</aside>
<p>Got it, I mistook what you said for a tensor which has zero as one of the dimensions.</p>
<p>The following seems to work for your problem:</p>
<pre><code class="lang-auto">t = torch.tensor(5)
print(t)
print(id(t))
t.copy_(torch.tensor(6))
print(t)
print(id(t))
</code></pre>

stevenjin8: <p>Another option might be <code>t.data = tensor(8)</code>.</p>

pallgeuer: <p>Yes, both those seem to be options. I was hoping/thinking there surely must be a way to do it though without having to create a new tensor in the process.<br>
When creating that new tensor I would have to be careful about dtype and device (at least when assigning to t.data?)?</p>

stevenjin8: <p>You’re right. Updating <code>t.data</code> also makes the corresponding updates to the <code>dtype</code>. Sorry I couldn’t be of more help.</p>

pallgeuer: <p>After more doc-searching I have settled for:</p>
<pre><code>t.fill_(8)
</code></pre>
<p>I guess this will have to do until there is an official intended way to update the value of a 0-dim tensor.</p>
MLPerf pytorch resnet50 error：decoded = cast(optional[int], proto.parsefromstring(s)) google.protobuf.message.decodeerror: error parsing message

zzsunshine: <p>Hi all,</p>
<p>I want to use MLPerf for model inference. I use imagenet and resnet50 model here.<br>
The resnet50 model is download from: <a href="https://zenodo.org/record/4588417#.YTXHZp5KjGI" class="inline-onebox" rel="noopener nofollow ugc">Resnet-50 v1.5 PyTorch model | Zenodo</a></p>
<p>After running the command "</p>
<pre><code class="lang-auto">./run_local.sh pytorch resnet50 gpu --count 100 --time 60 --scenario SingleStream
</code></pre>
<p>The error is coming:<br>
decoded = cast(optional[int], proto.parsefromstring(s)) google.protobuf.message.decodeerror: error parsing message</p>
<p>I try to solve it but failed after several tries, any help would be appreciated!</p>
Why are powers of 2 often used?

Superklez: <p>More specifically, why are the kernel sizes and number of channels in convolutional kernels powers of 2? Why are batch sizes also powers of 2? I vaguely recall an explanation stating that it’s because computers compute faster this way, but can anyone please suggest research papers or articles as to why this is the case? I come from a non-CS background, so please bear with me.</p>

googlebot: <p>Say you’ve chosen 2^n size of something initially. If you decrease it, it may sometimes not help at all - you usually have 2^k parallelism cores for SIMD (same instruction multiple data) processing, and with regard to saved memory - there are downsides with these “unaligned” zones of arbitrary size too. If you increase the size to 2^n+1 instead, this usually results in a new memory block allocation, so it is not efficient in many cases.</p>
Is torch.clamp the same as torch.nn.functional.hardtanh?

Liangzhi_Tang: <p>The following codes give the same results. Does it mean torch.clamp is totally the same as torch.nn.functional.hardtanh both in training and test phase?</p>
<pre><code>x1 = torch.randn(10)
x2 = x1.detach().clone()

x1.requires_grad_()
x2.requires_grad_()

y1 = torch.nn.functional.hardtanh(x1)
y2 = torch.clamp(x2, -1.0, 1.0)

l1 = y1.sum()
l2 = y2.sum()

l1.backward()
l2.backward()

assert (x1.grad - x2.grad).abs().sum() == .0
assert l1 == l2</code></pre>

gphilip: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Hardtanh.html#torch.nn.Hardtanh" rel="noopener nofollow ugc">The definition of <code>torch.Hardtanh</code></a> matches the way you clamped <code>x2</code>. For these choices of <code>min,max</code>, <code>torch.clamp</code> seems to be identical to <code>torch.Hardtanh</code>.</p>

Liangzhi_Tang: <p>Hi, what do you mean by “clamped x2”. I thought if min and max are the same, it means the behavior of clamp and hardtanh are identical, am I right?</p>

gphilip: <aside class="quote no-group" data-username="Liangzhi_Tang" data-post="3" data-topic="131027">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liangzhi_tang/48/15160_2.png" class="avatar"> Liangzhi_Tang:</div>
<blockquote>
<p>what do you mean by “clamped x2”</p>
</blockquote>
</aside>
<p>I meant this line of your code:</p>
<pre><code class="lang-auto">y2 = torch.clamp(x2, -1.0, 1.0)
</code></pre>
<aside class="quote no-group" data-username="Liangzhi_Tang" data-post="3" data-topic="131027">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liangzhi_tang/48/15160_2.png" class="avatar"> Liangzhi_Tang:</div>
<blockquote>
<p>if min and max are the same, it means the behavior of clamp and hardtanh are identical, am I right?</p>
</blockquote>
</aside>
<p>Yes, this is what it looks like to me.</p>

Liangzhi_Tang: <p>~<br>
So the implementation of hardtanh and clamp should be completely the same (both the forward and the backward) ?</p>

gphilip: <p>I don’t know. You could look at the source code for both, or wait for someone knowledgeable to answer.</p>
[WinError 1455] caffe2_detectron_ops_gpu.dll or one of its dependencies

Sarmad_GTU: <p>[WinError 1455] The paging file is too small for this operation to complete. Error loading “C:\Users\gtu\anaconda3\envs\Deeplearning\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.</p>
<p>How can I overcome this error?</p>

Sarmad_GTU: <p>Any solution for these problem</p>
Issue With number of workers

Marcus_Brown: <p>Hello.</p>
<p>I notice a strange behavior when training.<br>
If for instance, I have a batch size of N=(1, 2, 3, …) and 8 workers (or more),</p>
<p>I notice that during training, the loop pauses for a few seconds every 8th iterations. As if that pause is somehow used to load data or something of the like.</p>
<p>This behavior appears on a GTX 1080TI GPU.</p>
<p>The same code when executed on a TITAN RTX  does not exhibit the above behavior.</p>
<p>Has someone else encountered this issue? could this be an issue with my computer?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Marcus_Brown: <pre><code class="lang-auto">for epoch in range(1 + epoch, NUM_EPOCHS + 1):

      for tensors in train_loader:

        input_image = tensors['input_raw']
        gt_image = tensors['target_rgb']

        iteration += 1
        batch_size = gt_image.size(0)


        if torch.cuda.is_available():
            input_image = input_image.to(device)
            gt_image = gt_image.to(device)


        network.zero_grad()

        prediction_tensor, features = network(input_image)

        loss_target_second_tensor = l1_loss(prediction_tensor, gt_image)


        loss = loss_target_second_tensor   
        loss.backward()
        opt.step()

</code></pre>

ptrblck: <p>The issue sounds indeed like a data loading bottleneck similar to <a href="https://discuss.pytorch.org/t/strange-behavior-in-data-loader-with-workers/83769/2">this one</a> so you could profile your code to isolate the bottleneck further.<br>
As a quick test, you could replace the data loading with a single random tensor and let the model train on it. If the periodic slowdown is gone, it would point towards the data loading.</p>
Effect of batch size on the model accuracy

Sarmad_GTU: <p>Hi all<br>
I want to ask about the effect of batch size on the model accuracy.<br>
I have two semantic segmentation models I want to compare the result of each one of them but the batch sizes are differents<br>
batch size of the first one is 8<br>
batch size of the second one is 4 (i put 4 because I got out of memory during training)<br>
can this be a fair comparation or not ???</p>

tom: <p>There are many aspects of this and if you present results, you might also offer the first model with batch size 4. To my mind, it is important mention that the reason to compare 8 vs. 4 is memory (which, of course, makes it fair because you compare models trained in the same resource constraints, at least if it is indeed infeasible / nontrivial to use batch size 8).</p>
<p>Some thoughts:</p>
<ul>
<li>One line of thought to follow is that minibatch gradients are an stochastic estimate of the true gradient (over the entire dataset or even beyond). With this thinking, smaller batch sizes give more noisy gradient estimates. But is it good or bad?
<ul>
<li>More noise can be viewed as a form of regularization,</li>
<li>but the conventional wisdom is that the noise hurts, in particular towards the end of the training (when the gradient signal tends to be lower).</li>
</ul>
</li>
<li>The statistics used by training-mode batchnorm can be more extreme for smaller batches.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

Sarmad_GTU: <p>Thank you Thomas<br>
i appreciate that</p>
What does dim=-1 mean in torch.cat?

dcl: <p>The output of<br>
torch.cat((x, x, x), -1) and torch.cat((x, x, x), 1) seems to be the same but what does it mean to have a negative dimension. It is not mentioned in pytorch documentation that int needs to be non-negative.</p>
<p>Can someone explain what does it mean?</p>

dazzle-me: <p>Sure.</p>
<p>I guess your <code>x</code> tensor has a shape of length 2.</p>
<p>In torch, <code> dim = -1</code> means that the operation has to be performed along <strong>last</strong> dimension, and I think that is why  <code> torch.cat((x, x, x,) -1) == torch.cat((x, x, x,), 1)</code><br>
(not strictly because it’s links and something but you got the idea)  in your example.</p>
<p>For better understanding :</p>
<pre><code class="lang-auto">x = torch.Tensor([[1, 2, 3]])
x.shape
&gt; torch.Size([1, 3])

torch.cat((x, x, x), 1).shape
&gt; torch.Size([1, 9])

torch.cat((x, x, x), -1).shape
&gt; torch.Size([1, 9])
</code></pre>
<p>Hope it makes things clear for you.</p>

dcl: <p>Thanks, that’s helpful.</p>

maqboolurrahim: <p>Adding more explanation to <a class="mention" href="/u/dazzle-me">@dazzle-me</a> 's answer. I learned -ive dimension like this:<br>
-ive dimension can come in handy when we are too lazy to figure out tensor’s dimension and then<br>
-1 is last dimension and -2 is the second last dimension and then so on</p>
When to create custom dataset class

carlo_david: <p>what is the purpose of creating a custom dataset class if we can use ImageFolder from torchvision package</p>

albanD: <p>Hi,</p>
<p>For people that do not work with images for example.<br>
Or for people that do but do not store images on the disk in folders the same way ImageFolder assumes it should be.</p>

carlo_david: <p>thanks for the reply, what do you mean do not work with images? like text and dataframes?</p>

albanD: <p>Yes like text or audio or video.</p>

Aymen_Sekhri: <p>So if I have a dataset that follows the same way ImageFolder assumes, then can I use it, in professional work?</p>
Is there an efficient way to compare two different torch.profilers?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’m currently comparing my code by using a custom autograd function and I’ve noticed that my function is considerably slower than pytorch’s default function. (the function of choice is <code>torch.linalg.det</code>).</p>
<p>I’ve been using <code>torch.profier</code> to compare how my code runs between using <code>torch.linalg.det</code> and my <code>CustomDet</code>. Is there any efficient way to compare between 2 different profilers? (i.e. compare number of calls between versions, or perhaps similar operation called between them and different operations called between them?</p>
<p>Any help would be greatly appreicated!</p>