akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch distributed loss.backward errors out with CudnnBatchNormBackward, inplace operation

James_W: <p>I have some simple model code, transfer learned from resnet, and when I run it without distributed, everything works fine. however, when I try it in ditributed mode, I get this weird error:</p>
<p><code>Error detected in CudnnBatchNormBackward.</code></p>
<p>and then:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>The model in question is a vanilla resnet, which is loaded like so:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        backbone = models.resnet50(pretrained=True)
        self.backbone = torch.nn.SyncBatchNorm.convert_sync_batchnorm(backbone)

    def forward(self, x):
        y1 = self.backbone(x)
        y_n = F.normalize(z1, dim=-1)
        return y_n
</code></pre>
<p>and the training loop looks like so:</p>
<pre><code class="lang-auto">    for batch_idx, ((img1, img2), _) in enumerate(train_loader):

        if args.gpu is not None:
            img1 = img1.cuda(args.gpu, non_blocking=True)
            img2 = img2.cuda(args.gpu, non_blocking=True)

        optimizer.zero_grad()

        out_1 = model(img1)
        out_2 = model(img2)

        loss = contrastive_loss_fn(out1,out2)

        loss.backward()

        optimizer.step()
</code></pre>
<p>and it errors out at <code>loss.backward</code> in essence. I suspect it is because I run the model twice, but I am unsure of this.</p>
<p>Any pointers would be great… Havent been able to troubleshoot this for days now!</p>
<p>PS: I have tried cloning the outputs and using SyncedBatchnorm… neither seems to help!</p>

pbelevich: <p>I can suggest to set <code>broadcast_buffers=False</code> in the DDP module constructor as was mentioned <a href="https://github.com/pytorch/pytorch/issues/22095#issuecomment-505099500" rel="noopener nofollow ugc">here</a></p>

James_W: <p>I spent about 6 days hunting for this solution <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"><br>
Thank you VERY much <a class="mention" href="/u/pbelevich">@pbelevich</a> genius stuff!<br>
|I just have a follow up question (it all works fine now with the buffer flag): what does this do to performance? are there any gotchas I need to be aware of?<br>
Thank you.</p>
Make a translation on x

Mathieu_Grosso: <p>hello, i want to translate an image on the x axis. I have already made a function that computes translation with only one value (every item are translated with the same value) and thus the GT of the disparity is constant (at each pixel of x the image is moved by the same value).<br>
I would like to do the same thing but with a value different at each pixel but I am struggling on how to do it.<br>
Example:</p>
<pre><code class="lang-auto">
def concat_translatex_batch(batch, translate_x_max, scale, num_times):
    # print('translate_x_max',translate_x_max)
    translate_x_max_img = translate_x_max
    ic(batch)
    translate_x_max_evt = round(translate_x_max_img / scale)

    # print('translate_x_max_evt',translate_x_max_evt)
    newbatch = {}
    x_list = []
    for key in batch:
        if key == 'input':
            # ic(key)
            input_aug = []
            for i in range(num_times):
                input_aug_i = torch.ones_like(batch[key])
                # ic(input_aug_i)
                for b in range(batch[key].shape[0]):
                    # triangular distribution
                    # half_max_transl = translate_x_max_evt // 2
                    # rest_transl = translate_x_max_evt - half_max_transl
                    # x = random
                    x = random.randint(-translate_x_max_evt,
                                       translate_x_max_evt)  # + random.randint(-half_max_transl, half_max_transl)
                    # x = torch.randint(-translate_x_max_evt,translate_x_max_evt,size=(8, 3, 1080, 1868))
                    # assert -translate_x_max_evt &lt;= x and x &lt;= translate_x_max_evt
                    if x &gt; 0:
                        input_aug_i[b, :, :, :-x] = batch[key][b, :, :, x:]
                        ic(input_aug_i)
                    elif x == 0:
                        input_aug_i[b] = batch[key][b]
                    else:
                        input_aug_i[b, :, :, abs(x):] = batch[key][b, :, :, :-abs(x)]
                    x_list.append(x)
                    # ic(x_list)
                input_aug.append(input_aug_i)
                # ic(input_aug)

                # print(i,x,input_aug_i.max(),input_aug_i.min())
            newbatch[key] = torch.cat(input_aug, dim=0)[:, :, :, translate_x_max_evt:-translate_x_max_evt]
            # ic(newbatch[key])
        else:
            newbatch[key] = torch.cat(
                [batch[key][:, :, :, translate_x_max_img:-translate_x_max_img] for _ in range(num_times)], dim=0)
        
    return newbatch, x_list


</code></pre>

eqy: <p>Can this be achieved with a slicing operation?</p>
<pre><code class="lang-auto">a = torch.randn(1, 3, 224, 224)
shift = n
b = torch.zeros(1, 3, 224, 224)
if n &gt;= 0:
  b[:,:,:,shift:] = a[:,:,:,0:224-shift]
else:
  b[:,:,:,0:224+shift] = a[:,:,:-shift:] 
</code></pre>
Doubt about a loss function

Abhranta: <p>Can anyone help me out with how I can write a loss function to spread the output tensors of a network uniformly ? Let’s say my network gives a tensor [[x1] , [x2] , [x3] , …[xn]] . I want to spread these points evenly between 0 to 1 . How can i achieve this ?</p>
<p>P.S. I am sorry because I think this question is more suitable for Stack Overflow but I like the pytorch community here much better. So, I am asking this here.</p>

eqy: <p>Assuming you want the distribution to be uniform, it sounds like you might want something like KL Divergence loss:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html#:~:text=The%20Kullback-Leibler%20divergence%20loss%20measure%20Kullback-Leibler%20divergence%20is,the%20space%20of%20%28discretely%20sampled%29%20continuous%20output%20distributions." rel="noopener nofollow ugc">KLDivLoss — PyTorch 1.9.0 documentation</a></p>
Using linear layer in torch.autograd.Function

yash11: <pre><code class="lang-python">net = nn.Linear(out_, in_, bias=False)

class Function(T.autograd.Function):

    @staticmethod
    def forward(ctx, input, net):
 
        output = net(input)
        ctx.save_for_backward(net, input)
        return output

    @staticmethod
    def backward(ctx, grad_output):...
</code></pre>
<p>How to use a net inside autograd.Function? Is it possible? What are my options?</p>

tom: <p>What are you trying to achieve?</p>
<p>While it is possible (you’d want to assign the net to ctx._net instead of using <code>save_for_backward</code>, probably), it certainly is not advisable to do so.<br>
A more natural way would be to pass the net.weight as an argument to the forward (and, if you don’t want to differentiate with respect to that, return None).</p>
<p>Best regards</p>
<p>Thomas</p>
Use BatchNorm directly on input

dcaffo98: <p>Hi, I was wondering whether it could be useful or harmful to apply <em>batch normalization</em> directly on the input of a nn. Example of a simple network:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.bn0 = nn.BatchNorm1d(128)
        self.fc1 = nn.Linear(128, 4096)
        self.bn1 = nn.BatchNorm1d(4096)
        self.fc2 = nn.Linear(4096, 4096)
        self.bn2 = nn.BatchNorm1d(4096)
        self.fc3 = nn.Linear(4096, 10)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        normalized_input = self.bn0(x)
        h = self.relu(self.bn1(self.fc1(normalized_input)))
        h = self.relu(self.bn2(self.fc2(h)))
        h = self.fc3(h)
        return h
</code></pre>
<p>Suppose I have a huge trainset whose <em>mean</em> and <em>std</em> are unknown (for any possible reason), so I cannot normalize the dataset. By applying a <em>batch norm</em> even before feeding data to the first layer, I should be able to normalize my data anyway. To me it sounds plausible, but I haven’t ever seen a network such that. What are the possible implications of such an approach?</p>

ayalaa2: <p>Essentially the answer is yes. At train-time, mean and std will be estimated given the batch. Therefore a larger batch size will be more effective for this technique. Given enough training, you’ll probably have good estimates of your mean and std for your distribution.</p>
<p>However, I don’t see many circumstances where you <em>couldn’t</em> directly compute the mean and std. Even if the dataset is massive, you can compute mean/std precisely without loading everything up in memory. You can do one pass at the dataset to precisely compute the mean and perform another pass to compute the std.</p>
SGD momentum formula error?

PhysicsIsFun: <p>Greetings,</p>
<p>I am confused by PyTorchs usage of the momentum SGD method.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD</a><br>
The formulas state that</p>
<pre><code class="lang-auto">v_{t+1} = \mu * v_{t} + g_{t+1} 
p_{t+1} = p_{t} - lr  * v_{t+1}
</code></pre>
<p>The documentation remarks that this differs from the original formula used by Sutskever et al. (<a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiJ863HoN7xAhXghv0HHb1eAqcQFjACegQIBhAD&amp;url=https%3A%2F%2Fwww.cs.toronto.edu%2F~fritz%2Fabsps%2Fmomentum.pdf&amp;usg=AOvVaw2WyMzRSQj9QR7Dxzx8pwnk" rel="noopener nofollow ugc">download</a> ) in terms of the application of the learning rate. However, it also differs by the fact that PyTorch <strong>subtracts</strong> the velocity from the parameter, instead of adding it.</p>
<p>Shouldn’t the formula be the following?</p>
<pre><code class="lang-auto">v_{t+1} = \mu * v_{t} - g_{t+1} 
p_{t+1} = p_{t} + lr * v_{t+1}
</code></pre>
<p>Note that in all cases, \mu &gt; 0 is assumed.</p>
<p>Regards</p>

KFrank: <p>Hi Physics!</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="126541" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>However, it also differs by the fact that PyTorch <strong>subtracts</strong> the velocity from the parameter, instead of adding it.</p>
</blockquote>
</aside>
<p>The two formulations are equivalent.  Let’s call the “velocity” in the<br>
first, pytorch, formulation <code>vPytorch_{t}</code> and in your second proposed<br>
version <code>vPhysics_{t}</code>.  The two formulations only differ in a redefinition<br>
of <code>v</code>, namely <code>vPhysics_{t} = -vPytorch_{t}</code>, that drops out of the<br>
final calculation of <code>p_{t}</code>.</p>
<p>Now you might prefer to call the thing that you <em>add</em> to your “position”,<br>
<code>p_{t}</code>, a “velocity,” so you might prefer the second, <code>vPhysics</code>,<br>
formulation.  But that is purely a semantic or stylistic choice – again,<br>
the two formulations are mathematically equivalent.</p>
<p>Best.</p>
<p>K. Frank</p>

PhysicsIsFun: <p>Thank you Frank, I was seeing things…</p>
Torchserve spawning 30k+ threads but never closing them

maligree: <p>Hey everyone,</p>
<p>We’re currently looking into deploying our first torchserve service. As some simple tests, we’ve loaded it up with two workflows, did some load testing with <code>ab</code> and we’re seeing the <code>torchserve</code> process creating lots and lots of threads (we’re talking 30k+, about 3 per request), never cleaning them up. Eventually it leads to <code>pthread</code> errors and broken requests.</p>
<p>We’re super new to this - any tips on what to look for, whic configuration parameters could influence this and so on would be super apprecianted. I can share parts of the configuration, but please let me know which parameters are of most importance.</p>
<p>Bonus info:</p>
<ul>
<li>torchserve 0.4.2 and 0.4.0 tested, same behavior</li>
<li>using the REST interface for now for inference</li>
<li>adjusted netty threads as well as job queue size and even relatively small values (2 and 100) don’t seem to limit the number of threads spawned.</li>
</ul>
<p>Thanks!</p>
PyTorch/Tensorflow impact of xavier and kaming_uniform weights initialization

mustansarsaeed: <p>Hi, I am using the following model for training the network:</p>
<pre><code>class FemnistNet(nn.Module):
    def __init__(self):
        super(FemnistNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2) ##output shape (batch, 32, 28, 28)
        self.pool1 = nn.MaxPool2d(2, stride=2, ) ## output shape (batch, 32, 14, 14)
        
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2) ##output shape (batch, 64, 14, 14)
        self.pool2 = nn.MaxPool2d(2, stride=2) ## output shape (batch, 64, 7, 7)
        
        self.fc1 = nn.Linear(3136, 2048)
        self.fc2 = nn.Linear(2048 ,62)
        
    def forward(self, x):
        x = x.view(-1, 1, 28, 28)
        x = self.conv1(x)
        x = th.nn.functional.relu(x)

        x = self.pool1(x)

        x=self.conv2(x)
        x = th.nn.functional.relu(x)
        
        x = self.pool2(x)
        
        x = x.flatten(start_dim=1)
        
        x = self.fc1(x)
        l1_activations = th.nn.functional.relu(x)
        
        x = self.fc2(l1_activations)

        x = x.softmax()

        return x, l1_activations
</code></pre>
<p>Default initializations of <code>weights</code> is <code>kaiming_uniform</code>. It trains the model well. When I initializes the <code>weights</code> using <code>xavier</code> as <code>th.nn.init.xavier_uniform_(self.fc1.weight)</code> then model parameters become <code>nan</code> for <code>dense/linear</code> layers. What is the impact of <code>weights</code> initialization distribution? Why weights become <code>nan</code> in <code>th.nn.init.xavier_uniform_(self.fc1.weight)</code>?</p>
<p>Different distributions work well in <code>Tensorflow</code>. I don’t experience <code>NaN</code>s in <code>Tensorflow</code>.</p>
Save individual images generated by DCGAN

ArshadIram: <p>Hi,<br>
Below I am sharing code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/750d84b8022e7368818c07fb3d22d6ba63538567.png" data-download-href="https://discuss.pytorch.org/uploads/default/750d84b8022e7368818c07fb3d22d6ba63538567" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/750d84b8022e7368818c07fb3d22d6ba63538567_2_690x314.png" alt="image" data-base62-sha1="gHuPhzgVpCU0i7f4KN2PFESlAdV" width="690" height="314" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/750d84b8022e7368818c07fb3d22d6ba63538567_2_690x314.png, https://discuss.pytorch.org/uploads/default/original/3X/7/5/750d84b8022e7368818c07fb3d22d6ba63538567.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/750d84b8022e7368818c07fb3d22d6ba63538567.png 2x" data-dominant-color="2F2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">972×443 39.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When adding this highlighted line for saving the images only three images are saved.</p>
<p>Image0<br>
Image500<br>
Image 619</p>
<p>Can anyone please guide how to save all the generated images (fake) to the directory?</p>

zetyquickly: <p>Hi, <a class="mention" href="/u/arshadiram">@ArshadIram</a></p>
<p>I suspect that it is saving only 3 images due to the highlighted line within a if statement that is called three times. Try to indent left that line that might help</p>
Setting up the test data environment for IMAGENET on Colab

jarvico: <p>Hi,</p>
<p>I need to use a small subset of IMAGENET test data to conduct some experiments (usual classification task) for my project. I will be using Google COLAB environment.</p>
<p>I am struggling to setup the test data environment.</p>
<p>I wonder if anyone can help me to find scripts which I can run on a google colab notebook to download the IMAGENET test images, arrange all folder structure in an appropriate manner and use torchvision.datasets.ImageNet class with proper data loader functionality. Actually around 5-10K of test image is enough for my experiment.</p>
<p>I need to handle the data loading issue just like below. And I think, I need to download the test images and construct the necessary folder structure so that Pytorch torchvision.datasets.ImageNet class can work properly.</p>
<pre><code class="lang-auto">test_data = datasets.CIFAR10(root='/CIFARDATA', train=False, download=True, transform=transform)
test_loader = DataLoader(test_data, batch_size=batch_size, shuffle=True)
</code></pre>
<p>I would appreciate if you can help me with the scripts/steps necessary for this.</p>
<p>Best Regards…</p>
Using Pixel Weight Map with BCEWithLogitsLoss

John_Doe: <p>Hello,</p>
<p>I am trying to implement a BCEWithLogitsLoss version with a different weight for some pixels. The goal is to have good borders in a segmentation task.</p>
<p>The outputs of my net are logits.</p>
<p>My weight map have the same size as my target. (For example [batch_size, height, width]).</p>
<p>My implementation currently looks like this :</p>
<pre><code class="lang-auto">def BCE_with_logits_loss_pixel_map(pred, target, pix_map):
    
    # check that pred, target and map have the same dimension
    assert pred.size() == target.size() == pix_map.size(), "Loss Components don't have the same size"
    
    sig_pred = torch.sigmoid(pred)
    unweighted_loss = target * torch.log(sig_pred) + (1 - target) * torch.log(1 - sig_pred)
    weighted_loss = -(pix_map * unweighted_loss)
    mean_weighted_loss = weighted_loss.mean()
    
    return mean_weighted_loss

batch_size = 8
out_channels = 1
W = 128
H = 128
logits = torch.FloatTensor(batch_size, H, W).normal_()
target = torch.LongTensor(batch_size, H, W).random_(0, out_channels)
weights = torch.FloatTensor(batch_size, H, W).random_(1, 3)

loss_val = BCE_with_logits_loss_pixel_map(logits, target, weights)


</code></pre>
<p>I would like to know if there are any mistake in my code ?</p>
<p>Thank you.</p>

KFrank: <p>Hi John!</p>
<aside class="quote no-group quote-modified" data-username="John_Doe" data-post="1" data-topic="128421" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/91b2a8/48.png" class="avatar"> John_Doe:</div>
<blockquote>
<p>My implementation currently looks like this :</p>
<pre><code class="lang-auto">    unweighted_loss = target * torch.log(sig_pred) + (1 - target) * torch.log(1 - sig_pred)
    weighted_loss = -(pix_map * unweighted_loss)
    mean_weighted_loss = weighted_loss.mean()
</code></pre>
</blockquote>
</aside>
<p>I would recommend that you use <code>BCEWithLogitsLoss</code>’s <code>weight</code><br>
argument:</p>
<pre data-code-wrap="python"><code class="lang-python">    mean_weighted_loss = torch.nn.BCEWithLogitsLoss (weight = pix_map) (pred, target)
</code></pre>
<p>As a further comment:  You can certainly write your own version of<br>
<code>BCEWithLogitsLoss</code>, but if you do, you should, for reasons of<br>
numerical stability, use <code>logsigmoid()</code> instead of <code>sigmoid()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">    log_sig_pred = torch.nn.functional.logsigmoid (pred)
    log_sig_neg_pred = torch.nn.functional.logsigmoid (-pred)
    unweighted_loss = target * log_sig_pred + (1 - target) * log_sig_neg_pred
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

John_Doe: <p>Thank you very much for your help K. Frank,<br>
I didn’t saw the weight parameter in the torch.nn.BCEWithLogitsLoss function.<br>
It sure is easier than to do my own function.<br>
Also thanks for the tip regarding the numerical stability for sigmoid.<br>
Have a great day !</p>
Inference getting slower with TensorDateset training?

WoodezXJ: <p>Hello, I am using GoogleColab to train a generative model and find that moving the data to GPU is the bottleneck of the training. Since Colab provides sufficient RAM, I did the following:</p>
<ol>
<li>put all the training images in one NumPy array</li>
<li>create a tensor from it</li>
<li>move the tensor to GPU</li>
<li>create a TensorDataset from the tensor</li>
<li>feed TensorDataset to dataloader</li>
</ol>
<p>As you can guess the cost of moving data to GPU is significantly decreased, while the inference step starts to cost almost the same amount of time, and what is even weird is that at the beginning of each epoch, the inference is almost the same speed, and within 3 batches it totally slows down…</p>

WoodezXJ: <p>Okay so I managed to locate one operation during the loss construction, and the time cost of it does not change after I switched to TensorDataset, while if I have it simplified the inference time strangely decrease to the previous level, which is even confusing now.</p>
<p>To better explain this, this is a piece of the sample code:</p>
<pre><code class="lang-auto"># shape of input and target: (batch_size,3,256,256)
for input,target in train_loader:
    
    Generative_Model.zero_grad()
    predict = Generative_Model(input)

    # shape of m_predict and m_target: (batch_size,1,256,256)
    m_predict = predict[:,0,:,:]+predict[:,1,:,:]+predict[:,2,:,:]
    m_predict = torch.reshape(input,(input.shape[0],1,input.shape[2],input.shape[3]))
    m_target = target[:,0,:,:]+target[:,1,:,:]+target[:,2,:,:]
    m_target = torch.reshape(input,(input.shape[0],1,input.shape[2],input.shape[3]))

    # kernels are prefixed tensor
    # shape of kernels: (30,1,200,200)
    m_predict = torch.nn.functional.conv2d(m_predict,kernels,padding="same")
    m_target = torch.nn.functional.conv2d(m_target,kernels,padding="same")
    
    loss = torch.sum(m_predict)+torch.sum(m_target)
    
    loss.backward()
    Generative_Model_optimizer.step()
</code></pre>
<p>So with the dataset of train_loader changed to TensorDatasest, the cost of <code>predict = Generative_Model(input)</code> increases from 0.01s to 0.88s,</p>
<p>while if I change<br>
<code>m_predict = torch.nn.functional.conv2d(m_predict,kernels,padding="same")</code><br>
<code>m_target = torch.nn.functional.conv2d(m_target,kernels,padding="same")</code><br>
to<br>
<code>m_predict = torch.nn.functional.conv2d(m_predict,kernels)</code><br>
<code>m_target = torch.nn.functional.conv2d(m_target,kernels)</code><br>
Although the computation result is no longer my intention, the cost of <code>predict = Generative_Model(input)</code> falls back to 0.01s, and in both cases these two <code>conv2d</code> operation cost less than 0.01s…</p>
FCN8s, first time need a hand!

Ian_Scott: <p>Hi</p>
<p>I new to CNNs and pytorch, so I apologies if my question is obvious.  I am doing a MSc project in semantic segmentation and trying to write an FCN8 (vgg16) model that’s true to original paper.  What I am struggling with is the idea that any input size image can be used?</p>
<p>What I have is a kernal of 7x7 at “fc6”.  Now I assume the original inputs for the model in the paper would have reduced to 7x7x4096 at this point?  If this was the case then it would produce a 1x1x4096?</p>
<p>I do not know what size input they were actually using (if you know please let me know!) but, my input is currently 256x256 leading to a pool5 size of 15x15.  This means fc6, with its 7x7 kernel drops it to a 9x9x4096.  Is this ok, or do I need to change fc6 to a kernel of 15x15 in order to get down to a 1x1?</p>
<p>So I guess the question is, can any input size be used with the original architecture or do you have to adjust kernel sizes in order to ensure a 1x1x4096 after fc6?</p>
<p>Thanks in advance<br>
Ian</p>

tom: <p>No, you don’t need to go to 1x1. In fact, there is a lower bound what sizes you can input (because you need to stay &gt;= 1x1).<br>
Similar for UNets.</p>

Ian_Scott: <p>Thanks Thomas, this was my hunch.  So in fact we end up with a feature map of at least 1x1 which we then up sample from, and the size of this feature map is related to the input image size.  The question is then does it perform best with a certain size?  I have made 256x256 tiles of a 4000x4000 satellite patch, would I be better using a different tiles size?</p>
<p>On a side note, is 256 tiles of this size enough to train on?  I have seen a lot of examples using transfer learning and saying it is important to do, however I have 22 channels in my data, so I assume I can’t make use of the vgg16 pre trained model as that will be just 3?</p>

tom: <p>First thing is to try… You probably want rather aggressive augmentation and maybe see if you can get more unlabeled data.</p>
<p>Regarding transfer learning: You could try to duplicate and scale down the three channel weights in the first layer or add just a little noise, but one of the key observations is that transfer learning works best when the pretraining inputs are related to the final purpose (e.g. imagenet pretraining and then using it for medical imaging is not as useful as pretraining on medical imaging to start with).</p>
<p>Regarding the input sizes: So performance will degrade if you dramatically change the resolution but you can get by with changing the size.</p>
How to randomly select one image out of the five images produced by FiveCrop?

nikhil6041: <p>I am trying to add fivecrop to my torchvision.transforms pipeline but as per the doc it returns me a tuple of 5 instead of all the 5 images i want to randomly select one of them and then feed it to the network . If there is any way to do this?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>  Would you have a look at it?</p>

ptrblck: <p>I’m unsure if I understand the issue correctly, as a <code>tuple</code> of 5 images would return all images.<br>
To randomly select one of them, you could use <code>torch.randperm</code> and use it to index the returned images:</p>
<pre><code class="lang-python">transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.FiveCrop(size=(10, 10))
])

x = torch.randn(3, 24, 24)
out = transform(x)
out_select = out[torch.randperm(len(out))[0]]
</code></pre>

Pederson390: <p><em>from</em> <strong>future</strong> import division import torch import math import <em>random from</em> input <em>image</em> <a href="https://www.epayitonline.us/" rel="noopener nofollow ugc">,</a> so basically the <em>images generated</em> by each crop are different.</p>

nikhil6041: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> it solves what i was looking for.</p>
CUDA out of memory when calling model.state_dict()

bily: <p>I have been training a very large model which occupies over 90% GPU memory. The training process works fine until calling model.state_dict() to save checkpoints. It throws CUDA out of memory error.</p>
<p>The same problem has been reported in:</p><aside class="quote" data-post="4" data-topic="2219">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bfortuner/48/315_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-cuda-runs-out-of-memory-when-calling-torch-save/2219/4">Why CUDA runs out of memory when calling torch.save?</a> 
  </div>
  <blockquote>
    +1. I had the same problem and moving the model to cpu before saving also worked.
  </blockquote>
</aside>
<aside class="quote" data-post="2" data-topic="6444">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/utsav/48/3043_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/out-of-memory-during-torch-save/6444/2">Out of memory during torch.save</a> 
  </div>
  <blockquote>
    Shifting the model to CPU before saving and moving it back to GPU after saving solved it.
  </blockquote>
</aside>

<p>The suggested fix is to move the model to CPU device before saving checkpoints and then move the model back to GPU. However, I use distributed training and the training program freezes after model.cuda().</p>
<p>Any help is appreciated!</p>
How to quantize a model by layer

zooman: <p>By reading the tutorial by pytorch, it seems dynamic quantization only offers to quantize model by op type, correct me if i am wrong, and static quant requires modification to the floating model by adding quantStub and dequantstub…</p>
<p>Is there a way to simply just quantize a model one layer at a time? The approach I can think of is to use static quant, modify the model first, and instead of using a global model.config, just add model.layerx.config to the layers I want to quantize…</p>
<p>Is there any other way available without messing with the model since there are many models out there and different implementations…</p>
Entropy Objective Related to A Beta Distribution

mortazavi: <p>Read this entry:</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Entropy_(information_theory)">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)" target="_blank" rel="noopener nofollow ugc">Entropy (information theory)</a></h3>

<p>
 In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable 
  
    
      
        X
      
    
    {\displaystyle X}
  
, which takes values in the alphabet 
  
    
      
        
          
            X
          
        
      
    
    {\displaystyle {\mathcal {X}}}
  
 and is distributed according to 
  
    
      
        p
        :
       ...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Examine the following:</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
from torch.distributions import Beta<br>
m = Beta(torch.tensor([5.0]), torch.tensor([5.0]))<br>
m.entropy()<br>
tensor([-0.4806])</p>
</blockquote>
</blockquote>
</blockquote>
<p>For discrete probabilities one expects -p*log(p) to always integrate to a positive number.</p>
<p>Many RL algorithms contain policies defined over discrete action domains and most (e.g., A2C, PPO, etc.) contain an entropy component in their objective function.</p>
<p>When entropy is positive, we expect D-RL to train policy distribution towards a distribution with less entropy as successful action distribution is discovered (assuming we start with maximum entropy).</p>
<p>Is this trend during training still the case when one uses continuous probabilities (assuming there’s some kind of convergence)?</p>
<p>Mathematical intuition says that the answer to this question should be positive . . . However, are there any RL problems where this may not the guaranteed . . . I’m asking this question in the RL group simply out of curiosity.</p>
<p>(By the way, I’ve also noticed – and I may be wrong here – that the entropy beta has no impact on the solution when dealing with continuous action spaces in pytorch whose distribution is modeled with pytorch’s beta distribution .  . .  Could it be that with pytorch’s beta distribution, the result of the entropy computation is already “detached”? . . . Changing the value of the entropy beta seems to make no difference to the solution . . . When the same action space is discretized into a categorical, this is not the case and entropy beta makes a difference. I hope my use of “entropy beta” in this parenthesis is not confused with the “beta distribution” or its entropy. Entropy beta is usually the multiplier of action distribution’s entropy in the rl algorithms’ objective.)</p>
Help me how is how w.grad is being calculated here

saurav23: <p>import numpy as np<br>
import torch</p>
<h1>
<a name="input-temp-rainfall-humidity-1" class="anchor" href="#input-temp-rainfall-humidity-1"></a>Input (temp, rainfall, humidity)</h1>
<p>inputs = np.array([[73, 67, 43],<br>
[91, 88, 64],<br>
[87, 134, 58],<br>
[102, 43, 37],<br>
[69, 96, 70]], dtype=‘float32’)</p>
<h1>
<a name="targets-apples-oranges-2" class="anchor" href="#targets-apples-oranges-2"></a>Targets (apples, oranges)</h1>
<p>targets = np.array([[56, 70],<br>
[81, 101],<br>
[119, 133],<br>
[22, 37],<br>
[103, 119]], dtype=‘float32’)</p>
<h1>
<a name="convert-inputs-and-targets-to-tensors-3" class="anchor" href="#convert-inputs-and-targets-to-tensors-3"></a>Convert inputs and targets to tensors</h1>
<p>inputs = torch.from_numpy(inputs)<br>
targets = torch.from_numpy(targets)<br>
print(inputs)<br>
print(targets)</p>
<h1>
<a name="weights-and-biases-4" class="anchor" href="#weights-and-biases-4"></a>Weights and biases</h1>
<p>w = torch.randn(2, 3, requires_grad=True)<br>
b = torch.randn(2, requires_grad=True)<br>
print(w)<br>
print(b)</p>
<p>def model(x):<br>
return x @ w.t() + b</p>
<h1>
<a name="generate-predictions-5" class="anchor" href="#generate-predictions-5"></a>Generate predictions</h1>
<p>preds = model(inputs)<br>
print(preds)</p>
<h1>
<a name="msemean-squared-error-loss-6" class="anchor" href="#msemean-squared-error-loss-6"></a>MSE(mean squared error) loss</h1>
<p>def mse(t1, t2):<br>
diff = t1 - t2<br>
return torch.sum(diff * diff) / diff.numel()</p>
<h1>
<a name="compute-loss-7" class="anchor" href="#compute-loss-7"></a>Compute loss</h1>
<p>loss = mse(preds, targets)<br>
print(loss)</p>
<h1>
<a name="compute-gradients-8" class="anchor" href="#compute-gradients-8"></a>Compute gradients</h1>
<p>loss.backward()</p>
<h1>
<a name="gradients-for-weights-9" class="anchor" href="#gradients-for-weights-9"></a>Gradients for weights</h1>
<p>print(w)<br>
print(w.grad)</p>
<p><strong>Help me how is how w.grad is being calculated here</strong></p>

Varal7: <p>Hi,</p>
<p>This tutorial should be helpful: <a href="https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Differentiation with torch.autograd — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
Torch.onnx.export overwrites param names after conv+bn fusion

roldja: <p>pytorch v1.8.1</p>
<p>I am exporting a simple model to ONNX and noticed that a [conv + BN + relu] block parameters were renamed. After stepping through the export code, I find the offending change happening within <code>torch._C._jit_pass_onnx_eval_peephole</code>, and presumably during <a href="https://github.com/pytorch/pytorch/blob/e845b0ab515fc79406a226f4e8acf16881c27b8b/torch/csrc/jit/passes/onnx/eval_peephole.cpp#L37" rel="noopener nofollow ugc">conv+bn fusion here</a>.</p>
<p>after peephole:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5702df8967912e6667e31215daf1e5971df314ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/5702df8967912e6667e31215daf1e5971df314ad" title="image (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5702df8967912e6667e31215daf1e5971df314ad_2_690x350.png" alt="image (2)" data-base62-sha1="cpJH8jqo9FQfrtDTg2poZHi9Dyt" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5702df8967912e6667e31215daf1e5971df314ad_2_690x350.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5702df8967912e6667e31215daf1e5971df314ad_2_1035x525.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5702df8967912e6667e31215daf1e5971df314ad_2_1380x700.png 2x" data-dominant-color="2D3036"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (2)</span><span class="informations">1792×910 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>parameters ‘18’ and ‘19’ are introduced after peephole, and later replace “conv_bn_and_relu” parameter names in the final graph. Is it possible to retain the “conv_bn_and_relu” parameter name in the graph?</p>
Apply different function/module to each slice of a tensor along certain dimension

wasabi: <p>I have a list of modules f = nn.ModuleList[…] of length n and an input tensor x of shape say n*b. What will be the most efficient of of computing a new tensor y=[f_1(x_1) … f_n(x_n)]? It seems that the closest thing is <a href="https://pytorch.org/tutorials/prototype/vmap_recipe.html" rel="noopener nofollow ugc">vmap</a> but does it only apply the same function/module?</p>
Custom loss function with a gradient operation

vicramr: <p>I am attempting to implement adversarial training for the Fast Gradient Sign Method, which is a widely-used adversarial attack (see section 6 of <a href="https://arxiv.org/abs/1412.6572" class="inline-onebox" rel="noopener nofollow ugc">[1412.6572] Explaining and Harnessing Adversarial Examples</a>). This involves creating a custom loss function which includes a gradient operation. What is the proper way to do this? I don’t believe it is sufficient to use <code>.backward</code> inside the loss computation, because this would set the <code>.grad</code> attribute on all my tensors as a side effect.</p>
<p>From reading the documentation, it seems that the function I need is <code>torch.autograd.grad</code>, but the docs don’t clearly specify what side effects this function has. Will it modify the <code>.grad</code> attributes of my tensors?</p>

Varal7: <p>Calling <code>torch.autograd.grad</code> will not accumulate gradients in your tensors, but will free the computation graph.<br>
Can your problem be solved by using a strategically-placed <code>.detach</code> somewhere in your loss computation?</p>

vicramr: <p>Thanks for your reply! I did consider using <code>.detach</code> but it doesn’t seem to fit my use case, as I need to backpropagate through the gradient operation. I ended up using <code>torch.autograd.grad</code> with <code>retain_graph=True</code> and <code>create_graph=True</code> and it appears to be doing the right thing.</p>
<p>I do have one follow-up question: in <code>torch.autograd.grad</code>, what is the <code>grad_outputs</code> argument used for? The description in the docs is unclear to me. I’m leaving it as the default value and my code appears to be working fine.</p>
How to count model parameters?

blackhole: <p>how to count model parameters in pytorch, torchstat packages didn’t update for a long time, meet some error when i use it.</p>

mariosasko: <p>Hi,</p>
<p>you can count them as follows:</p>
<pre><code class="lang-python">num_params = sum(param.numel() for param in model.parameters())
</code></pre>
<p>or:</p>
<pre><code class="lang-python">num_params = sum(param.numel() for param in model.parameters() if param.requires_grad)
</code></pre>
<p>to only consider trainable parameters.</p>

blackhole: <p>thank you!! if i want to see parameters in each layer, how to do it?</p>

safakkbilici: <p>Maybe listing all modules in a model can be helpful if you want to see parameters in each layer:</p>
<pre><code class="lang-python">for name, module in model.named_modules():
     print(name, sum(param.numel() for param in module.parameters()))
</code></pre>
<p>It will print all modules and modules’ number of parameters including activation functions or dropout.</p>

Vrushank98: <p>You can use <strong>torchsummary</strong> library for that.</p>
<pre><code class="lang-auto">from torchsummary import summary
print(summary(model_name,  (3,224,224) , 'cpu'))
</code></pre>
Slow training of seq2seq models

FAhtisham: <p>Hello everyone,</p>
<p>I have been trying to reuse the seq2seq model (<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.9.0+cu102 documentation</a>) for the purpose of DNA sequence translation that are usually very long (appx 1275 words in a single sequence). But the problem is that it takes alot of time train such a model, as the model itself is very simple in nature (GRUs + Luong Attention). If I try to see the gpu usage it always say 10-15 percent. How can I make it use more gpu and accelerate the training process ?</p>
<p>Thanking you in advance for any leads</p>
<p>PS: The GPU usage for this example (<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.9.0+cu102 documentation</a>) is also very less but the computations are very fast bcz the sentence length is only 10.</p>
Getting the flattened image from a pretrained model

upasana_siva: <p>How do I get a flattened image from a pretrained model (VGG16). I need to put this flattened image as an input to a simple perceptron and then train. Kindly help me with this part.</p>
<pre><code class="lang-auto">#Function to get pre-trained model
from torchvision import models
import torch.nn as nn
def get_pretrained_model(model_name):
    """Retrieve a pre-trained model from torchvision

    Params
    -------
        model_name (str): name of the model (currently only accepts vgg16 and resnet50)

    Return
    --------
        model (PyTorch model): cnn

    """

    if model_name == 'vgg16':
        model = models.vgg16(pretrained=True)

        # Freeze early layers
        for param in model.parameters():
            param.requires_grad = False
        n_inputs = model.classifier[6].in_features

        # Add on classifier
        model.classifier[6] = nn.Sequential(
            nn.Linear(n_inputs, 512), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(512, 5))

    elif model_name == 'resnet50':
        model = models.resnet50(pretrained=True)

        for param in model.parameters():
            param.requires_grad = False

        n_inputs = model.fc.in_features
        model.fc = nn.Sequential(
            nn.Linear(n_inputs, 512), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(512, 5))
    return model
</code></pre>
<pre><code class="lang-auto">#getting Pre-trained model
    model = models.vgg16(pretrained=True)
    # Freeze model weights
    for param in model.parameters():
        param.requires_grad = False
    
    n_inputs = model.classifier[6].in_features

    # Add on classifier
    model.classifier[6] = nn.Sequential(
        nn.Linear(n_inputs, 512), nn.ReLU(), nn.Dropout(0.4),
        nn.Linear(512, 5))

    model = get_pretrained_model('vgg16')
    print(model.forward)
    criterion = torch.nn.CrossEntropyLoss()
    n_class=5
    optimizer = torch.optim.Adam(model.parameters())
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min' if n_class &gt; 1 else 'max', patience=2)
</code></pre>

Manuel_Alejandro_Dia: <p>You could use the <a href="https://pytorch.org/docs/stable/generated/torch.flatten.html" rel="noopener nofollow ugc">flatten operation</a> of tensors!</p>
<pre><code class="lang-auto">input = torch.ones((1,3,100,100))
flat_input = input.flatten(input, start_dim=1)
</code></pre>
<p>Keep an eye on the <code>start_dim</code> parameter, so you keep the number of batches.</p>
How do I train DenseNet in PyTorch?

Tclha: <p>I’m working on a project to classify a series of images (X-Rays) using ‘Densenet-121’ in PyTorch.</p>
<p>I have a folder containing the images in png format, and a csv file that contains the image file names along with their labels (and other data as well).</p>
<p>I was told to follow this tutorial, and replace the sample dataset they use with my own, and to change the classifier they used to DenseNet-121: <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Quickstart — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>I don’t really understand where I’m supposed to change the code to use a DenseNet classifier instead. The PyTorch tutorial for DenseNet (<a href="https://pytorch.org/hub/pytorch_vision_densenet/" class="inline-onebox" rel="noopener nofollow ugc">Densenet | PyTorch</a>) uses a pretrained DenseNet model, whereas I will need to train it myself, but I’m not sure where to start with this.</p>
<p>To be honest, I’d be totally fine just to setup a simpler neural network than densenet, just so I have something to show for now, but I’m a bit overwhelmed with all the tutorials and now really sure how to get my own dataset/labels into PyTorch and use a classifier on it.</p>
<p>Could someone point me in the right direction or assist me with this?</p>

Manuel_Alejandro_Dia: <p>You can replace the code for the network in the <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html#creating-models" rel="noopener nofollow ugc">Creating Models</a> part for:</p>
<pre><code class="lang-auto">model = torch.hub.load('pytorch/vision:v0.10.0', 'densenet121', pretrained=True)
model.eval()
</code></pre>
<p>Remember to change the last layer though!</p>
Binary 2D mask implementation

nerdk312: <p>Hi,</p>
<p>I have two sets of labels, for example<br>
labels_a = torch.Tensor([0,1,2,3]) with a shape of  torch.size([4])<br>
labels_b = torch.Tensor([0,1,2,3,0,1,3,1,2,3,2]) with shape of torch.size([11])</p>
<p>I was wondering if there is an efficient way to make a binary mask of shape [labels_a_size,labels_b_size] which corresponds to [4,11] in which mask[i,j] = 1 if labels_a[i] == labels_b[j] and 0 otherwise?</p>
<p>Also, this is my first post so let me know if there is anything I can improve upon</p>

tom: <p><a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">Broadcasting</a> is the superpower you need</p>
<pre><code class="lang-python">mask = (labels_a[:, None] == labels_b[None, :])
</code></pre>
<p>Beast regards</p>
<p>Thomas</p>
CUDA error out of memory

saran: <p>every time when I am running my code its run smoothly for the first time but when I am running it for second or 3rd time its give me this error I don’t think  its memory problem otherwise it couldn’t run for the first time too maybe its cache problem or something like that i am not sure any ways how i can clear my cache after every run</p>

safakkbilici: <p>Normally <code>torch.cuda.empty_cache() </code> clears cache as stated in <a href="https://pytorch.org/docs/stable/generated/torch.cuda.empty_cache.html" rel="noopener nofollow ugc">documentation</a>. The documentation also stated that <code>it doesn’t increase the amount of GPU memory available for PyTorch</code>. I’ve been dealing with same problem on colab, the problem can be related with its garbage collector or something like that. Removing variables (e.g. <code>del model</code>) or a factory reset would be a precise solution.</p>
Memory assignment in loadbalance partition

Levin: <p>In loadbalance partition algorithm,</p>
<ul>
<li>For checking if function fits into the single device it checks if all constants fit into device memory (comparing <code>module_-&gt;getConstantsSize()</code> to device memory)</li>
<li>For checking if node fits into any device it compares available memory device with sums of the memory size of all the inputs of that node (using <code>getNodeMemUsage()</code>)</li>
<li>While updating memory of device if node fits, it consider memory of both placeholder and constants(in <code>updateGraphMemInfoByAddingNode()</code>)</li>
</ul>
<p>What’s the idea or intuition behind all three points?</p>