akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gibbs Sampling in Variational Autoencoders

Achint_Kumar: <p>I am trying to implement a variation of VAEs in which the latent space variable z is derived from a Gibbs sampling routine. To generate z, I initialize it by sampling from a gaussian distribution. Then, I run the Gibbs sampling procedure 2000 times to get past the burn-in period. Then I again sample from the Gibbs sampler routine to get the z which I send to the decoder of the VAE. The problem is that I am unable to do backpropogation through the VAE anymore. I defined a variable flag_initialize=1 in the init function, and made it run the first time. Then, I had hoped to use that as the initial value for all further computations. The code in the forward function is the following:</p>
<pre><code class="lang-auto">if self.flag_initialize==1:
            z1_prior,z2_prior = self.gibbs.initialize_prior_sample(self.g11,g22,self.g12)
            self.flag_initialize=0
            self.z1_prior =z1_prior
            self.z2_prior =z2_prior
        self.z1_prior,self.z2_prior = self.gibbs.prior_sample(self.z1_prior,self.z2_prior,self.g11,g22,self.g12)
</code></pre>
<p>When I run the code, I get an error which reads,</p>
<p>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.</p>
<p>Any help on how I can do a backpropogation with Gibbs sampling? Thank you!</p>
Possible to skip bad items in data loader?

deepcode: <p>I have some rows in my data that are bad . Is it possible to skip or return None for bad data? I’ve tried returning None, but it dies in the pipeline.  Is it possible to do this kind of functionality without modify the core pytorch libraries?</p>
<p>Traceback (most recent call last):<br>
File “<a href="http://train.py" rel="nofollow noopener">train.py</a>”, line 125, in <br>
main(args)<br>
File “<a href="http://train.py" rel="nofollow noopener">train.py</a>”, line 53, in main<br>
for i, (images, captions, lengths) in enumerate(data_loader):<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 212, in <strong>next</strong><br>
return self._process_next_batch(batch)<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 239, in _process_next_batch<br>
raise batch.exc_type(batch.exc_msg)<br>
OSError: Traceback (most recent call last):<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 41, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 41, in <br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/home/jtoy/sandbox/sketchnet/pytorch/data_loader.py”, line 44, in <strong>getitem</strong><br>
image = Image.open(os.path.join(path, “image.jpg”)).convert(‘RGB’)<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/PIL/Image.py”, line 844, in convert<br>
self.load()<br>
File “/home/jtoy/anaconda3/lib/python3.6/site-packages/PIL/ImageFile.py”, line 226, in load<br>
"(%d bytes not processed)" % len(b))</p>

smth: <p>this thread might help you <a href="https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/4">Questions about Dataloader and Dataset</a></p>

deepcode: <p>Exactly what I needed to fix the issue, thank you!</p>

msamogh: <p><a class="mention" href="/u/deepcode">@deepcode</a> <a class="mention" href="/u/smth">@smth</a> Check out <a href="https://github.com/msamogh/nonechucks" rel="nofollow noopener">nonechucks</a> - a library I wrote for PyTorch that allows you to do exactly that (and more)!</p>

sascha10000: <p>Thanks, was totally searching for a simple way to filter without getting deeper into the Dataloaders.</p>

aksg87: <p><a class="mention" href="/u/msamogh">@msamogh</a></p>
<p>nonechucks doesn’t allow iteration over very large datasets though due to the memory leak issue. Would be great if this was resolved!</p>
What is the parameter `require_forward_param_sync` in DistributedDataParallel for

yhcc: <p>Hi, I am curious why this require_forward_param_sync is set to True in the DistributedDataParallel. After I manually set it to False, the multi-gpus in one node speeds up a lot. Since the gradients of each replica have been synchronized, why do we need to synchronize the parameters in the forward?</p>

rvarm1: <p>One of the things this flag controls is whether to broadcast model <code>buffers</code> in each iteration to ensure that module buffers are synchronized across processes. You can evaluate the speed up of your model with <code>broadcast_buffers=False</code> and remove it if model accuracy permits.</p>

yhcc: <p>Thanks for your reply, by setting broadcast_buffers to False indeed achieve the same speed up. Therefore, the broadcast_buffers can be safely set to False if anyone of the following holds<br>
(1) there is no buffer in the model;<br>
(2) the buffer will never be updated.<br>
is this correct?</p>

rvarm1: <p>Are you sure there are no buffers in your module? If that is the case it is surprising that you’re seeing a speedup because this synchronization would be a noop (there are no buffers to synchronize).</p>
<p>Can you confirm this by checking the <code>self.named_buffers()</code> attr of your nn.Module you’re passing into ddp?</p>

yhcc: <p>There are some small buffers in my model, but they are just some constant matrix, the synchronization might take some time.<br>
Another interesting speedup also confuses me a lot, in the Megatron-LM (<a href="https://github.com/NVIDIA/Megatron-LM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/Megatron-LM: Ongoing research training transformer language models at scale, including: BERT &amp; GPT-2</a>) package, the model will be explicitly converted into the Float16, by calling something like model.half() before training the model with fp16 enabled, and by doing so the backward time will be largely reduced (e.g. the time takes to finish loss.backward()). I think the backward time should be agnostic to whether the model is float16 or not since the gradients are fp16 when the fp16 is enabled. Do you have  any hint about why this may happen?</p>
How do I insert a layer to a pretrained model?

Superklez: <p>I’m using the MobileNet v3 model from torchvision but I want to change the padding of every convolutional layer to be causal in the “width” dimension. To do this, I might need to insert <code>nn.ZeroPad</code> right before every convolutional layer. For example, I want to go from</p>
<pre><code class="lang-auto">nn.Conv2d(1, 3, kernel_size=3, padding=1)
</code></pre>
<p>to something like</p>
<pre><code class="lang-auto">nn.ZeroPad2d((2, 0, 1, 1))
nn.Conv2d(1, 3, kernel_size=3)
</code></pre>
<p>Edit: What I’ve tried doing is</p>
<pre><code class="lang-auto">def adjust_model(model):
    for child_name, child in model.named_children():
        if isinstance(child, nn.Conv2d):
            padding = (2 * child.padding[1], 0,
                child.padding[0], child.padding[0])
            child.padding = (0, 0)
            setattr(model, "CausalConv2d", nn.Sequential(
                nn.ZeroPad2d(padding),
                child
            ))
        else:
            adjust_model(child)
</code></pre>
<p>but returns an error saying</p>
<pre><code class="lang-auto">RuntimeError: OrderedDict mutated during iteration
</code></pre>

Superklez: <p>Solved it using the function below</p>
<pre><code class="lang-auto">def adjust_model(model):
    for child_name, child in model.named_children():
        if isinstance(child, nn.Conv2d):
            causal_padding = (2 * child.padding[1], 0, child.padding[0],
                child.padding[0])
            child.padding = (0, 0)
            setattr(model, child_name, nn.Sequential(
                nn.ZeroPad2d(causal_padding),
                child
            ))
        else:
            adjust_model(child)
</code></pre>

Karelle: <p>You can simply keep adding layers in a sequential model just by calling add method. The other is functional API, which lets you create more complex models that might contain multiple input and output.</p>
<p><a href="https://www.dgcustomerfirst.run/" rel="noopener nofollow ugc">www.dgcustomerfirst.com</a></p>
Implemented cv2 filter2D but output is wrong

srinivirt: <p>Read about kernel correlation and implemented cv2 filter2D but for an example test input and kernel, the output produced does not match output from cv2.filter2D(). What am I missing here?</p>
<pre><code class="lang-auto"># input two tensors, images is 3d, kernel 2d
def my_filter2D(image, kernel):
    kx, ky = kernel.shape
    x_n = kx//2
    y_n = kx//2
    padded_image = torch.nn.functional.pad(image, (0, 0, ky//2, ky//2, kx//2, kx//2))
    filtimg = torch.Tensor(image)
    px, py, nc = padded_image.shape
    for x in range(px):
        for y in range(py):
            for c in range(nc):
                if x - x_n &lt; 0:
                    continue
                if x + x_n &gt;= px:
                    continue
                if y - y_n &lt; 0:
                    continue
                if y + y_n &gt;= py:
                    continue
                total = 0.0
                image_n = padded_image[x-x_n:x+x_n+1, y-y_n:y+y_n+1,c]
                for x_i in range(kx):
                    for y_i in range(ky):
                        total += image_n[x_i,y_i]*kernel[x_i,y_i]
                filtx = x-x_n
                filty = y-y_n
                filtimg[filtx,filty,c] = total
    return filtimg
</code></pre>
<p>Here’s sample input and kernel from a stackoverflow response:</p>
<pre><code class="lang-auto">import numpy as np
import cv2


a = np.array([[0.0,0.0,0.0,0.0,0.0],
              [0.0,0.0,0.0,0.0,0.0],
              [10.0,0.0,0.0,0.0,0.0],
              [20.0,20.0,20.0,0.0,0.0],
              [30.0,30.0,30.0,30.0,30.0]], dtype=np.float32)

kernel = np.ones((3,3), dtype=np.float32)

filtered_a = cv2.filter2D(a, -1, kernel)
</code></pre>
<p>The output of the above cv2.filter2D() call produces:</p>
<pre><code class="lang-auto">array([[  0.,   0.,   0.,   0.,   0.],
       [ 10.,  10.,   0.,   0.,   0.],
       [ 70.,  70.,  40.,  20.,   0.],
       [160., 160., 130., 110.,  90.],
       [210., 210., 170., 130.,  90.]], dtype=float32)
</code></pre>
<p>But the output from my filter2D implementation does not match the above from cv2.filter2D() - how to fix the code above?</p>
<pre><code class="lang-auto">   a = np.array([[0.0,0.0,0.0,0.0,0.0],
              [0.0,0.0,0.0,0.0,0.0],
              [10.0,0.0,0.0,0.0,0.0],
              [20.0,20.0,20.0,0.0,0.0],
              [30.0,30.0,30.0,30.0,30.0]], dtype=np.float32)
    testkernel = np.ones((3,3), dtype=np.float32)
    a3d = a.reshape(5,5,1)
    my_filter2d_a = my_filter2D(torch.from_numpy(a3d), torch.from_numpy(testkernel))

and output:
tensor([[  0.,   0.,   0.,   0.,   0.],
        [ 10.,  10.,   0.,   0.,   0.],
        [ 50.,  70.,  40.,  20.,   0.],
        [110., 160., 130., 110.,  60.],
        [100., 150., 130., 110.,  60.]])
</code></pre>
<p>Please redirect me the correct algorithm for to implement cv2 filter2D().</p>
Replace the Fully connected layer of my model by an SVM

rubijade: <p>Hello everyone<br>
Can someone help me with this? I want to replace the FC layers (f1 and f2) of my model with  SVM but I really don’t know how to. Need your guidance. Thank you in advance</p>
<p>This is the model:</p>
<pre><code class="lang-auto">import torch
import torchvision # torch package for vision related things
import torch.nn.functional as F  # Parameterless functions, like (some) activation functions
import torchvision.datasets as datasets  # Standard datasets
import torchvision.transforms as transforms  # Transformations we can perform on our dataset for augmentation
from torch import optim  # For optimizers like SGD, Adam, etc.
from torch import nn  # All neural network modules
from torch.utils.data import DataLoader  # Gives easier dataset managment by creating mini batches etc.
from tqdm import tqdm  # For nice progress bar!

class CNNskl(nn.Module):
    def __init__(self, in_channels=3, num_classes=120):
        super(CNNskl, self).__init__()
        self.conv1 = nn.Conv2d(
            in_channels=in_channels,
            out_channels=128,
            kernel_size=3,
            stride=1,
            padding=2,
        )
        self.maxpool1 = nn.MaxPool2d(kernel_size=2)
        self.conv2 = nn.Conv2d(
            in_channels=128,
            out_channels=64,
            kernel_size=(3, 3),
            stride=1,
            padding=2,
        )

        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        self.conv3 = nn.Conv2d(
            in_channels=64,
            out_channels=32,
            kernel_size=3,
            stride=2,
            padding=2,
        )
        self.maxpool3 = nn.MaxPool2d(kernel_size=2)
        self.fc1 = nn.Linear(9248, 200)
        self.fc2 = nn.Linear(200, num_classes)
        self.dropout = nn.Dropout(0.5)

    def forward(self, x):
        # 64, 49, 100, 3
        # 64, 1200, 4, 9, 3
        
        # b, n, m, z, h, w = x.shape
        # x = x.view(b, z, h*n*m, w)
        b, n, H, W = x.shape
        #print(x.shape)
        x = self.conv1(x)
        x = F.relu(x)
        x = self.maxpool1(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.maxpool2(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.maxpool3(x)
        #b1, c, h1, w1 = x.shape proposition de yoik
        x = x.reshape(x.shape[0], -1)
        #x = self.dropout(x)
        #x = x.view(b1*c*h1*w1, -1) cest ce que yoik a ppropose
        #x = x.mean(dim=1) proposition de yoik
        x = self.fc1(x)
        x = F.relu(x)
        #x = self.dropout(x)
        #print(x.shape)
        x = self.fc2(x)
        #print(x.shape)
        return x
</code></pre>
How to use print_lr in the lr_scheduler?

seyeeet: <p>Im trying to use the build in function for printing the lr in my schedule</p>
<pre><code class="lang-auto">scheduler = StepLR(optimizer, step_size=3, gamma=0.1)
</code></pre>
<p>I see that I can use <code>print_lr(is_verbose, group, lr, epoch=None)</code> to see the lr? but what every I do it shows the same thing, should not it be different for diferent epoch?<br>
e.g.<br>
I tried:<br>
<code>scheduler.print_lr(True,optimizer,args.lr,epoch=100)</code></p>
<p>and</p>
<p><code>scheduler.print_lr(True,optimizer,args.lr,epoch=10)</code></p>
<p>and both gives similar output?</p>

ptrblck: <p>I don’t think this method is supposed to be exposed, as it’s just printing the input.<br>
Based on <a href="https://github.com/pytorch/pytorch/blob/0dc98728bc06e1e0a553b8ad4ed2591b688cc72f/torch/optim/lr_scheduler.py#L105-L114">this code</a>:</p>
<pre><code class="lang-python">    def print_lr(self, is_verbose, group, lr, epoch=None):
        """Display the current learning rate.
        """
        if is_verbose:
            if epoch is None:
                print('Adjusting learning rate'
                      ' of group {} to {:.4e}.'.format(group, lr))
            else:
                print('Epoch {:5d}: adjusting learning rate'
                      ' of group {} to {:.4e}.'.format(epoch, group, lr))
</code></pre>
<p>it seems to be a convenient method to print the lr updates in case you are using the <code>verbose</code> option.</p>
<p>I think you should use <code>scheduler.get_last_lr()</code> instead.</p>
Learning correlation parameters of two signals

daver: <p>Hey folks,</p>
<p>Suppose that I have two signals, where one is basically a delayed and scaled version of the other, plus some noise for fun. I am trying to learn that delay and scale.</p>
<p>So, my input features to my model are the last N samples of the first signal, and I want to create/learn a model that will predict the second signal from the recent samples of the first.</p>
<p>The scale part is simple enough to learn, as I can just create a pytorch variable for it, and just multiply my input features by that value in my module’s forward function. I’m not sure how to code up the delay part in a way that makes that parameter learnable. It is essentially having to choose which sample of the input to output (or, because that delay is a float value, it would have to interpolate between two adjacent samples).</p>
<p>So, what is the best way to achieve what I want? I can achieve this with other packages (say, a nonlinear optimizer, or using scikit’s signal processing functions), but I am wondering how to do it in pytorch.</p>
Use of nn.ReflectionPad2d(1)

Arsham_mor: <p>I was looking for simple example of <a href="https://paperswithcode.com/method/siamese-network" rel="noopener nofollow ugc">Siamese Networks</a> and I found this <a href="https://hackernoon.com/facial-similarity-with-siamese-networks-in-pytorch-9642aa9db2f7" rel="noopener nofollow ugc">article</a> at hackernoon.</p>
<p>input batch first go through <code>nn.ReflectionPad2d(1)</code> as shown below:</p>
<pre><code class="lang-python">class SiameseNetwork(nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.cnn1 = nn.Sequential(
            nn.ReflectionPad2d(1),
            nn.Conv2d(1, 4, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(4),
            nn.Dropout2d(p=.2),
            
            nn.ReflectionPad2d(1),
            nn.Conv2d(4, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),
            nn.Dropout2d(p=.2),

            nn.ReflectionPad2d(1),
            nn.Conv2d(8, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),
            nn.Dropout2d(p=.2),
        )

        self.fc1 = nn.Sequential(
            nn.Linear(8*100*100, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 5)
        )

    def forward_once(self, x):
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

    def forward(self, input1, input2):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2
</code></pre>
<ol>
<li>why do siamese network (or the author’s approach) need padding before feeding into convolutions ?</li>
<li>what is the point of using <code>ReflectionPad</code> instead of zero padding ?</li>
</ol>

ptrblck: <ol>
<li>I guess the <code>ReflectionPad2d</code> layers were added as <code>nn.Conv2d</code> supported zero padding only in the past (in new PyTorch versions you can specify the <code>padding_mode</code>).</li>
<li>I don’t know if the author has explained this architecture in a research paper, but would guess that this padding type worked better than zero padding based on their experiments.</li>
</ol>

Shawn_Zhuang: <p>Hi<br>
do you know any paper explaining the contribution of ReflectionPad instead of ZeroPad?<br>
or can I say in most cases ReflectionPad is better than ZeroPad?</p>

ptrblck: <p>No, I don’t have a reference and also cannot claim one approach is superior.</p>
Regression based Importance feature

sushmit_roy: <p>How can I use a permutation-based importance feature using captum as an example</p>
<p><a href="https://scikit-learn.org/stable/modules/permutation_importance.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://scikit-learn.org/stable/modules/permutation_importance.html</a></p>

Narine: <p>Yes, You can use captum.attr.FeaturePermutation</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://captum.ai/api/feature_permutation.html">
  <header class="source">
      <img src="https://captum.ai/img/captum.ico" class="site-icon" width="256" height="256">

      <a href="https://captum.ai/api/feature_permutation.html" target="_blank" rel="noopener nofollow ugc">captum.ai</a>
  </header>

  <article class="onebox-body">
    <img src="https://captum.ai/img/captum-icon.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://captum.ai/api/feature_permutation.html" target="_blank" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a></h3>

  <p>Model Interpretability for PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


sushmit_roy: <p><a class="mention" href="/u/narine">@Narine</a> Thanks for your reply. I did see we can use it, however, I was not able to understand the implementation. For example, if we look at the document <a href="https://captum.ai/api/feature_permutation.html" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a> I do not see how we can pass the targets. I am making a comparison between <a href="https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html#sklearn.inspection.permutation_importance" class="inline-onebox" rel="noopener nofollow ugc">sklearn.inspection.permutation_importance — scikit-learn 0.24.2 documentation</a>. How do we pass the scoring criteria? If you have any working example that you can share will be great</p>

Narine: <aside class="quote no-group" data-username="Narine" data-post="2" data-topic="131290">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/narine/48/16927_2.png" class="avatar"> Narine:</div>
<blockquote>
<p><a href="https://captum.ai/api/feature_permutation.html" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a></p>
</blockquote>
</aside>
<p><a class="mention" href="/u/sushmit_roy">@sushmit_roy</a>, if you want to do something  similar to scikitlearn, you can leave target=None and set <strong>forward_func</strong> as a scoring function.</p>
<p><strong>forward_func</strong> can be something like</p>
<pre><code class="lang-auto">def my_forward_func(inputs, labels):
   return score(model(input), labels) # here score should be returned in form of a scalar tensor.
</code></pre>
<p><strong>labels</strong> can be passed as <strong>additional_forward_args</strong> to the attribute method. Let me know if it works for you.</p>
Readings for improve my knowledge in deep learning

Sylvain_Ard: <p>Hello !<br>
Have you some books to advise tome to better understand deep learning and pytorch please ?<br>
Thank you<br>
Best regards</p>

ptrblck: <p>I can recommend <a href="https://www.manning.com/books/deep-learning-with-pytorch">Deep Learning With Pytorch</a> by <a class="mention" href="/u/elistevens">@elistevens</a> <a class="mention" href="/u/lantiga">@lantiga</a>, and <a class="mention" href="/u/tom">@tom</a>, which was also promoted a while ago on the official PyTorch website.</p>

Sylvain_Ard: <p>thank you<br>
best regards</p>
Torch not working with CUDA 11.0, returns false for is_available()

Aiman_Shivani: <p>I have remote access via SSH to a user(not root) in an NVIDIA Tesla A100 40GB which i want to use for training purposes. For which i require to use pytorch enabled CUDA.  I am using conda for installing it in a virtual env .The nvidia-smi on the device shows:</p>
<blockquote>
<p>NVIDIA-SMI 450.142.00   Driver Version: 450.142.00   CUDA Version: 11.0</p>
</blockquote>
<p>So i tried to install first pytorch for cuda with following command i found on the pytorch website</p>
<blockquote>
<p>conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch</p>
</blockquote>
<p>But it returned torch.cuda.is_available() to be false. Same was the case when i tried to install cuda 10.2 or cuda 11.1 with commands available on the <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">website here</a> but still returned to be false.</p>
<p>If i run the command</p>
<blockquote>
<p>/usr/local/cuda-11.0$ cat version.txt</p>
</blockquote>
<p><em>i get the output as follows</em>: - <strong>CUDA Version 11.0.207</strong></p>
<p>And if i run the command</p>
<blockquote>
<p>dpkg -l | grep cuda-toolkit</p>
</blockquote>
<p>I get the output as :-</p>
<blockquote>
<p>rc  cuda-toolkit-11-3                          11.3.1-1                              amd64        CUDA Toolkit 11.3 meta-package<br>
rc  cuda-toolkit-11-3-config-common            11.3.109-1                            all          Common config package for CUDA Toolkit 11.3.<br>
rc  cuda-toolkit-11-config-common              11.3.109-1                            all          Common config package for CUDA Toolkit 11.<br>
rc  cuda-toolkit-config-common                 11.3.109-1                            all          Common config package for CUDA Toolkit.</p>
</blockquote>
<p>Please help!</p>

ptrblck: <p>Make sure the NVIDIA driver is correctly installed and in case you have a local CUDA toolkit installed, build and run some CUDA examples.<br>
Once this is verified, install the latest PyTorch binary using the CUDA11.1 or 11.3 runtime.</p>
How can i gather tensors from all gpus in one machine(Windows with gloo backend))?

lirx02: <p>I use only one machine with multigpu to train. In init_process_group(),  i set world_size==1 and rank==0. I am not very sure that whether it is right for multigpu in one node. It seems that the gpu usage is fine(about 100% in 2 gpu). But when i want to gather the same tensor in different gpus, dist.gather and dist.all_gather doesn’t work, the error like below when running dist.gather:<br>
ValueError: ProcessGroupGloo::gather: Incorrect output list size 2. Output list size should be 1, same as size of the process group.</p>
<p>The environment is Windows, torch 1.7.1, gloo backend, cuda10.2.</p>

rvarm1: <p>Hi, if you’re aiming to do distributed training across 2 GPUs you’d want to set world_size to 2 and the corresponding ranks would be 0 and. 1.</p>
<p>The GPU utilization is at 100% across both GPUs possibly because both GPUs are operating independently with no distributed coordination between them.</p>
<p>Your call to <code>all_gather</code> is the right approach and the error message confirms that world_size is set incorrectly. Fixing the world_size issue should unblock your use case.</p>
How to run torch DDP model with multiple gpus in one machine?

lirx02: <p>I have 2 gpus in one machine for example. When using DistributedDataParallel, i need to set init_process_group. In TORCH.DISTRIBUTED doc I find an example like below:</p>
<p>For example, if the system we use for distributed training has 2 nodes, each of which has 8 GPUs. On each of the 16 GPUs, there is a tensor that we would like to all-reduce. The following code can serve as a reference:</p>
<p>Code running on Node 0</p>
<p>import torch<br>
import torch.distributed as dist</p>
<p>dist.init_process_group(backend=“nccl”,<br>
init_method=“file:///distributed_test”,<br>
world_size=2,<br>
rank=0)<br>
tensor_list = []<br>
for dev_idx in range(torch.cuda.device_count()):<br>
tensor_list.append(torch.FloatTensor([1]).cuda(dev_idx))</p>
<p>dist.all_reduce_multigpu(tensor_list)<br>
Code running on Node 1</p>
<p>import torch<br>
import torch.distributed as dist</p>
<p>dist.init_process_group(backend=“nccl”,<br>
init_method=“file:///distributed_test”,<br>
world_size=2,<br>
rank=1)<br>
tensor_list = []<br>
for dev_idx in range(torch.cuda.device_count()):<br>
tensor_list.append(torch.FloatTensor([1]).cuda(dev_idx))</p>
<p>dist.all_reduce_multigpu(tensor_list)</p>
<p>Like this example, i think if i have only one node, i should set world_size==1 and rank==0 in init_process_group. It works because i can see 2 gpus usage is about 100%. But it seems that all of the collective functions like all_reduce/all_reduce_multigpu do not work well.</p>
<p>If i set world_size=2 and rank=local_rank( the arg in torch.distributed.launch) in init_process_group. The collective functions work well, and  I can collective tensors in different gpus. But, the gpu usage is very low, and loss.backward() runs very slow. The gpu usage drop to 0% sometimes when running loss.backward().</p>
<p>The environment is cuda10.2, torch1.7.1, windows with gloo backend.</p>
<p>So which setting is right? world_size==1 with rank==0 or world_size==2 with changing rank according to local_rank. If the first one is right, how can i collective tensors infomation in different gpus.</p>
<p>Thanks a lot.</p>

rvarm1: <p>Hi, world_size = 1 and rank = 0 wouldn’t really work for distributed training as we generally want to train with &gt; 1 GPU (i.e world size &gt; 1). In particular # of GPUs (not necessarily # of nodes) is usually used as the world size.</p>
<p>Regarding slowness in loss.backward(), can you provide a repro of that (it appears that your code snippet is an allreduce above)? In general, loss.backward() will trigger additional allreduces during the backwards pass to synchronize parameter gradients, but especially for 2 GPUs we don’t expect this to add significant overhead.</p>
Add reshape nn.Module

dashesy: <p>Looking at the <code>torchvision.models.resnet34</code> this is forward:</p>
<pre><code class="lang-python">class ResNet(nn.Module):

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)

        return x
</code></pre>
<p>This means <code>resnet34</code> could have been</p>
<pre><code class="lang-auto">class ResNet(nn.Sequential):
</code></pre>
<p>If it was not for the reshape. Then manipulating it would have been more straightforward and we would not need to treat it differently. <code>resnet34</code> is just an example, but in general it would be nice to also have a simple <code>reshape</code> <code>nn.module</code> and use it instead of re-implemeting <code>forward</code>.</p>

ptrblck: <p>We have a discussion <a href="https://github.com/pytorch/vision/issues/720" rel="nofollow noopener">here</a>, so maybe you would want to explain your use case there, too. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

dashesy: <p>Thanks. I added to that thread. It is a simple module, but if it is part of <code>torch.nn</code> the likelyhood of simppler implementation would be higher.</p>
<p>Relying on <code>forward</code> is good if you own the code, or if there is no standard layers you want to manipulate. But if for example you want to take <strong>official</strong>  <code>resnet34</code>  and do something to it, suddenly you realize that some part of the model is in <code>forward</code> that could have been in <code>__init__</code> as a module. Manipulating a dict is simpler that code.</p>

leimao: <p>I would like to have Reshape as as <code>nn.Module</code> because then I can use it in <code>nn.Sequential</code> easily.</p>

ptrblck: <p>Would the implementation in the GitHub issue work for you or are you seeing any issues with it?</p>

leimao: <p>It definitely will work. Some people might prefer to have it as part of the official PyTorch API.</p>
How to prevent the increase of the memory with a tensor accumulating

hilbert_deng: <p>In my case, before the iteration I got several tensor which will be calculated through the whole training stage. As I noticed, the memory keeps increasing, even though I use .cpu(). The code structure is like allowings:<br>
tensor_for_accumulate = [ ]<br>
for iter in range(10):<br>
…<br>
tensor_current = …<br>
tensor_for_accumulate += tensor_current<br>
I found that the length and dtype is the same during the training. Thus, I can’t understand why the memory is increasing all the time. <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=10" title=":cry:" class="emoji" alt=":cry:"></p>

ptrblck: <p>If <code>tensor_current</code> is attached to the computation graph (i.e. if its <code>.grad_fn</code> attribute returns a valid function) then all computation graphs will be stored with <code>tensor_current</code> in <code>tensor_for_accumulate</code>. Pushing <code>tensor_current</code> to the CPU doesn’t change anything if the operations were performed on the GPU beforehand, since all intermediate will still be on the GPU.<br>
I’m not familiar with your use case and don’t know if you want to store the computation graph. If not, <code>detach()</code> the tensor before accumulating it.</p>

hilbert_deng: <p>Thanks a lot!!! detach() is work for and I check out the reason is exact what you mentioned above. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>
JIT error on type annotation

khatrishubham: <p>Any clues what might be causing this JIT issue? I would assume it should detect the Generator annotation. This is linked to following PR<br>
<a href="https://github.com/pytorch/pytorch/pull/65139" rel="noopener nofollow ugc">Adding RNG to model initializers</a></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/test_jit.py", line 14072, in test_nn_init
    cu = torch.jit.CompilationUnit(code)
  File "/opt/conda/lib/python3.6/site-packages/torch/jit/_recursive.py", line 838, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/opt/conda/lib/python3.6/site-packages/torch/jit/_script.py", line 1311, in script
    qualified_name, ast, _rcb, get_default_args(obj)
RuntimeError: 
Unknown type name 'torch.Generator':
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/init.py", line 122
def uniform_(tensor: Tensor, a: float = 0., b: float = 1., generator: torch.Generator = None) -&gt; Tensor:
                                                                      ~~~~~~~~~~~~~~~ &lt;--- HERE
    r"""Fills the input Tensor with values drawn from the uniform
    distribution :math:`\mathcal{U}(a, b)`.
'uniform_' is being compiled since it was called from 'test'
  File "&lt;string&gt;", line 4
def test(a):
    # type: (Tensor)
    return torch.nn.init.uniform_(a)
           ~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>

ptrblck: <p>I think asking in your PR might be a better place to fix potentially related issues or is the error unrelated to your changes?</p>

khatrishubham: <p>The error doesn’t really arise from the changes. It is really type annotation, removing which pylint complains and keeping it the jit complains <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
Help with understanding image trandofrmation

Gabriella_m: <p>Hi I want to ask somebody if can explain to me what these two lines of code do with image and how can i take this transofrmation back.<br>
mask_np have shape - (3, 384, 512)</p>
<pre><code class="lang-auto">mask_np = numpy.array(masks[group_idx][image_idx]).mean(axis=0).astype(int)
mask_one_hot = numpy.eye(self.classes_count)[mask_np]

</code></pre>

ptrblck: <p>It’s a bit hard to certainly explain what this code is doing, as all objects are undefined.<br>
However, based on the used operations my guess is that <code>mask_np</code> would contain class indices, while <code>mask_one_hot</code> represents the one-hot encoded mask as seen here:</p>
<pre><code class="lang-python"># setup
classes_count = 10
nb_samples = 3
# this is most likely containing the class indices
mask_np = np.random.randint(0, classes_count, (nb_samples,))
print(mask_np)
&gt; [5 4 2]

# this is most ikely creating a one-hot encoding for the class indices
mask_one_hot = np.eye(classes_count)[mask_np]
print(mask_one_hot)
&gt; [[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
   [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
   [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]]

# argmax returns the class indices back
print(np.argmax(mask_one_hot, axis=1))
&gt; [5 4 2]
</code></pre>
<p>I don’t know what <code>masks</code> is containing and how the <code>mean</code> in <code>axis=0</code> and the transformation to <code>int</code> is creating the class indices though, so would need to see the input array.</p>

Gabriella_m: <p>This post was flagged by the community and is temporarily hidden.</p>
How to retrain a custom trained model on less number of classes?

Aliktk: <p>Hello seniors and expertise,<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>  am working on Chest X-rays and I want to detect and classify different thoric diseases. I trained a model on a Large NIH dataset but the AUC is not good. For improving the model I am going to retrain the trained model with fewer classes because the old dataset and new dataset classes are not matched 100%.<br>
I have only 9 classes that are matched with each other.<br>
For this is I’m confused that how I can retrain the model?<br>
<strong>Question:</strong><br>
Is it possible to train the model?<br>
How much is it worth to my model in terms of performance?<br>
I trained my model on <strong>PNG</strong> images whether the new dataset is <code>.dicom</code> images which having numpy arrays. So Does it reduce the performance?<br>
Please help regarding this.<br>
Thanks</p>
Pytorch calculation precision

Sam-gege: <p>Hi I’m fairly new to pytorch, plz correct me if I’m wrong.</p>
<p>My question is, does pytorch calculates everything in float64 but saves its parameters using float32? I heard pytorch prefers float32 for its speed and memory efficiency, so I wrote my own autograd using numpy and set its dtype to float32 as well.</p>
<p>However, when compared the result with pytorch, it varies by a bit: (for example, the final loss comparison is: numpy autograd:<strong>0.0059468</strong>58786046505 vs pytorch autograd: <strong>0.0059468</strong>69496256113).</p>
<p>When I used float64 in numpy, the result became almost identical to pytorch: (numpy autograd: <strong>0.953232780278648</strong>1 vs pytorch autograd: <strong>0.953232780278648</strong>4)</p>
<p>I can’t seem to find an answer elsewhere on this topic, thanks for any help!</p>

ptrblck: <aside class="quote no-group" data-username="Sam-gege" data-post="1" data-topic="132372">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam-gege/48/42338_2.png" class="avatar"> Sam-gege:</div>
<blockquote>
<p>does pytorch calculates everything in float64 but saves its parameters using float32?</p>
</blockquote>
</aside>
<p>No, PyTorch uses <code>float32</code> by default for the parameter storage and all calculations.<br>
If you are using mixed-precision training via <code>torch.cuda.amp</code> then some operations will be performed with <code>float16</code> inputs/outputs, but the computation will still be performed in <code>float32</code>.</p>
<p>The difference might come from the limited floating point precision and a different order of operations between PyTorch and numpy.</p>