akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CIFAR 10 dataset

username: garvitkhera
cooked: <p>The CIFAR 10 dataset runs up till 12000 images and then a new epoch starts that too runs till 12000 images. Can anyone explain to me why it doesn’t go till 50,000 images as the training data contains 50000 images and not 12000</p>

username: Oli
cooked: <p>My CIFAR10 dataset has 50k images in the training set. What does it say if you print(len(data_loader)) where dataloader is your training data. Perhaps your batch size is 5 -&gt; 12*5=60</p>
An easy question about autograd

username: Audrey
cooked: <p>Hi, I am a newbie to PyTorch and have a question about autograd. I am following a tutorial like this to calculate the gradients. :</p>
<pre><code class="lang-python">x = torch.randn(2,2, requires_grad=True)
y = x**2
z = y.mean()
z.backward()
print(x.grad)
</code></pre>
<p>But when I run:</p>
<pre><code class="lang-python">y.backward()
</code></pre>
<p>Error appears! So why is it not allowed to use .backward() on tensor <code>y</code>？</p>

username: albanD
cooked: <p>Hi,</p>
<p>The error message states that you should provide it with the grad output because it is not a scalar right?</p>
<p>The thing is that autograd computes reverse mode AD, which basically performs a matrix product between a given vector v and the Jacobian of the function.<br>
This vector v is what backward expects as input.<br>
For the particular case where your function outputs a scalar value, then v is of size 1 and if it contains the value 1, then what you get is the Jacobian of your function (derivatives).<br>
For the case where there is more outputs (in your case, your y is of size &gt; 1), depending on v, you will get a weighted sum of the rows of the Jacobian. Unfortunately, there is no natural default to use in this case and so we require the user to provide v.</p>

username: Audrey
cooked: <p>Thanks for your helpful detailed explanation! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
DataParallel and manually modifying parameters

username: mraggi
cooked: <p>Hello,</p>
<p>I’d like to train my model on multiple GPUs but unfortunately I’m getting a massive validation error (but not so when only doing one gpu) after even 1 epoch.</p>
<p>I think the reason is that I manually modify some of the model parameters, after the optimization step:</p>
<pre><code class="lang-python">torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')

...

model = nn.Sequential(...).cuda()
dmodel = nn.DataParallel(model)

...
    loss = criterion(dmodel(x),y)
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        deterministic_modify(model[17]) # I need to manually modify some weights.
</code></pre>
<p>I’m guessing there is a sync problem, because if I do this on a single gpu, things work as expected. But on multiple gpu’s I get terrible validation error.</p>
<p>The way I understand nn.DataParallel works (but please correct me if I’m wrong) is that it’s a wrapper, and each gpu has a copy of the model, and nn.DataParallel splits the batch into two, gives each gpu half, computes the gradients, and then, somehow, sync’s the model in both gpus (how?).</p>
<p>Thanks.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="mraggi" data-post="1" data-topic="57432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mraggi/48/7200_2.png" class="avatar"> mraggi:</div>
<blockquote>
<p>The way I understand nn.DataParallel works (but please correct me if I’m wrong) is that it’s a wrapper, and each gpu has a copy of the model, and nn.DataParallel splits the batch into two, gives each gpu half, computes the gradients, and then, somehow, sync’s the model in both gpus (how?).</p>
</blockquote>
</aside>
<p>Above is correct except the model sync part. In the forward pass, it replicates the model to all devices, creates one thread per device, scatters (uses broadcast) the input to all devices (so that each thread exclusively works on one model replica with a input split), and finally gathers the output from all threads, and uses that as the return value of the forward function.</p>
<p>So after the forward pass, the autograd graph will contain multiple replicas of the original model, which all points back to the same original model. In the backward pass, each model replica will compute there own gradients, then, as they all have autograd edges point back to the original model, those gradients on different replicates will also accumulate into the same original module. So, it is not synchronizing across replicas, instead all gradients from all replicates are accumulated into the original module.</p>
<p>I noticed that the code snippet above calls init_process_group, which is required for DistributedDataParallel but not necessary for DataParallel. And DistributedDataParallel indeed does the gradient sync across multiple processes, and which should be faster than DataParallel.</p>
How to get deterministic behavior with DistributedDataParallel?

username: wzn0828
cooked: <p>Hello, my code has deterministic behavior without DistributedDataParallel, however, not deterministic with DistributedDataParallel.</p>
<p>My code for deterministic behavior is:</p>
<p>cudnn.benchmark = False<br>
cudnn.deterministic = True<br>
random.seed(123)<br>
torch.manual_seed(123)<br>
torch.cuda.manual_seed_all(123)<br>
torch.utils.data.DataLoader(…, worker_init_fn=random.seed)</p>
<p>And my launch command:<br>
python - torch.distributed.launch <br>
–nproc_per_node=4 <br>
–master_ports=$((RANDOM + 10000)) <br>
train.py</p>
<p>Does the DistributedDataParallel need more tricks to get deterministic behavior?</p>

username: mrshenli
cooked: <p>DistributedDataParallel should be deterministic. All it does is applying allreduce to sync gradients across processes. Can you check if the data loader is providing deterministic inputs for you?</p>
A question about torchvision.transforms.ToTensor

username: Reza_Akbari_Movahed
cooked: <p>Hello Everyone.<br>
I have a question about torchvision.transforms.ToTensor.<br>
In the Documentation of this Command has said that :<br>
Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0].</p>
<p>My question is How to this command scale the values in the range [0, 255] to the range of [0.0, 1.0] ?</p>

username: ptrblck
cooked: <p>You can find the code <a href="https://github.com/pytorch/vision/blob/master/torchvision/transforms/functional.py#L32" rel="nofollow noopener">here</a>. Basically, you divide by <code>255</code>.</p>

username: Reza_Akbari_Movahed
cooked: <p>Thanks a lot for your answer.</p>

username: Nitesh
cooked: <p>What if I want to convert an image into a tensor but not in the range [0, 1]? Is there any solution to that?</p>

username: ptrblck
cooked: <p>You could most likely use <code>torch.from_numpy</code>, as most libraries load images as numpy arrays.</p>
Torch.bool + nn.DataParallel + >1 GPU = RuntimeError: Unconvertible NCCL type

username: pierrecurie
cooked: <p>Python 3.6.9<br>
pytorch 1.2.0<br>
I’m trying to implement a modified Conv2d (long story), so I subclassed it.  However, I wish to save some additional parameters (discrete, rarely changes) in the state_dict, so I have extra nn.Parameter(…, requires_grad=False).  Code stub:</p>
<pre><code class="lang-auto">class test(nn.Conv2d):
    def __init__(self, **kwargs):
        super(test, self).__init__(**kwargs)
        self.foo = nn.Parameter(torch.tensor(False), requires_grad=False)
    def forward(self, x):
        return F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)

net = nn.Sequential(test(in_channels=3, out_channels=8, kernel_size=224),nn.Flatten())
net = nn.DataParallel(net)
</code></pre>
<p>For the forward pass, this works fine.  However, for the backwards pass, if there are &gt;1 GPU, I get:</p>
<pre><code class="lang-auto">...
  File "/workspace/pytorch_mdk/train.py", line 86, in train
    loss.backward()
  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 32, in backward
    return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 43, in forward
    return comm.reduce_add_coalesced(grads, destination)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 121, in reduce_add_coalesced
    flat_result = reduce_add(flat_tensors, destination)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 77, in reduce_add
    nccl.reduce(inputs, outputs, root=nccl_root)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/nccl.py", line 60, in reduce
    torch._C._nccl_reduce(inputs, outputs, root, op, streams, comms)
RuntimeError: Unconvertible NCCL type

</code></pre>
<p>Adding some print statements to torch/cuda/nccl.py, we can see that it’s given a bunch of tensor([False], device=‘cuda:0’), tensor([False], device=‘cuda:1’), etc.  The C code that it calls apparently does not like torch.bool, so it dumps the above RuntimeError.</p>
<p>Is this a bug?  Am I doing it wrong?<br>
In the short term, I’ll probably have to change bool parameters to torch.uint8 or something.</p>

username: pierrecurie
cooked: <p>For my use case, I think the correct answer is to use <code>self.register_buffer</code> instead of nn.Parameter.</p>
<p>As for the title, I don’t know if that combination has any use cases (and hence does not need to be supported) or is a bug.  IMO, it should at least be documented with a warning of some sort.</p>
Mapping the Label Image to Class Index For Semantic Segmentation

username: kenneth_maanika
cooked: <p>Hi<br>
I have some doubts in mapping colors to class index<br>
I have label images (raw pixel values ranging from 0 to 1) and visually there are three classes (black , green, red color). I want to create masks from these label images to feed it to my Segmentation model (which uses cross entropy loss).</p>
<p>After looking at some code from a forum post</p>
<p><em>Create mapping</em><br>
<em>Get color codes for dataset (maybe you would have to use more than a single</em><br>
<em>image, if it doesn’t contain all classes)</em></p>
<p>target = torch.from_numpy(target)<br>
colors = torch.unique(target.view(-1, target.size(2)), dim=0).numpy()</p>
<p>implementing this i got colors to have a shape (7824, 3) meaning that there are 7824 different colors right ?</p>
<p>could i have some guidance  on how to use this to create masks for all my label image which has only the class index (black class 0, green class 1, red class 2)</p>

username: ptrblck
cooked: <p>I assume you are referring to <a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">this post</a>.</p>
<p>Try to adapt the code to your use case and make sure you are dealing with the same data shapes (i.e. check for the same dimension layout etc.).<br>
One possible reason for a lot of unique colors could be the usage of an interpolation method while resizing other than nearest neighbors.</p>

username: kenneth_maanika
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. I will give this a try !</p>
Translate and Rotate torch.autograd.Variable() from regressed outputs of ConvNet

username: ru6928
cooked: <p>I would like to translate and rotate a torch.autograd.Variable() using the output of a ConvNet and calculate the loss for back propagation from translated+rotated images and input fixed image. If I convert it to numpy and translate, the loss.backward() does not work. How do I solve this problem?</p>
Replace two sequential conv layers with one

username: XW324
cooked: <p>In theory, two circulant convolution layers(or more) should be replaceable by a single conv layer by assosiative property. Although circulant conv might introduce bad boundary artifacts, have anyone thought about using the assosiative property to reduce the size of neural nets?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Won’t the non-linearities prevent you from doing it?</p>
<p>Even without non-linearities, you could do it for evaluation, but the behavior for training would be different as you would have different degrees of freedom for your weights.</p>
F.unfold behavior

username: gslaller
cooked: <p>I have the following code:</p>
<pre><code class="lang-auto">x = (torch.arange(16)+1).reshape(1,1,4,4).float()
im2col = F.unfold(x, (2,2))
</code></pre>
<p>with the <code>x</code> being:</p>
<pre><code class="lang-auto">1   2  3   4
5   6  7   8
9  10 11  12
13 14 15  16
</code></pre>
<p>I expect the this to have an output of</p>
<pre><code class="lang-auto">[[1,2,5,6],
 [2,3,6,7],
 [3,4,7,8],
 .....]
</code></pre>
<p>but it is giving me the output of kernel size 3:</p>
<pre><code class="lang-auto">[[1,2,3,5,6,7,9,10,11],
 [2,3,4,6,7,8,10,11,12],
 .....]
</code></pre>
<p>Am I thinking wrong about the unfold operation? and is this the expected behavior? When so, can you elaborate?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The only difference between what you expect and what you get is a transpose. The expected use for convolution is to do mm(w, unfold(input)).<br>
You can see <a href="https://github.com/pytorch/pytorch/blob/ede38bd743eaa7d4312cd30956d0f76e7a9c4c4f/aten/src/THNN/generic/SpatialConvolutionMM.c#L119" rel="nofollow noopener">here</a> that the unfold is called and <a href="https://github.com/pytorch/pytorch/blob/ede38bd743eaa7d4312cd30956d0f76e7a9c4c4f/aten/src/THNN/generic/SpatialConvolutionMM.c#L135" rel="nofollow noopener">here</a> the mm is done with the weights.</p>

username: gslaller
cooked: <p>My bad, but thanks for helping out.</p>