akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
MultiheadAttention / transformer with batch first

username: Dee
cooked: <p>Is it possible to add option <code>batch_first = True</code> for MultiheadAttention and transformer modules like it is done for RNN ?<br>
<code>Batch x Sequence x Embedding</code><br>
I find it more understandable and intuitive when batch is first in the input</p>

username: 19951024
cooked: <p>I can’t agree with u anymore!</p>

username: ptrblck
cooked: <p>We have an issue <a href="https://github.com/pytorch/pytorch/issues/25100" rel="nofollow noopener">here</a> so feel free to post your opinion there.<br>
I’m not sure, why it was closed.</p>

username: omveer_sharma
cooked: <p>yes, we can do this. i have tried this and successfully implemented the codeBut I i face one problem, i can only run code on the CPU, while using the cuda compiler shows error regarding batch first command.</p>
One model with multiple outputs vs. multiple single output models

username: 20100
cooked: <p>GIVEN<br>
Given a requirement to classify a text (document) in each of 7 set of labels (3 binary sets, 3 multi-label sets of 4 to 20 labels each and one set of 1500 labels).</p>
<p>GOAL<br>
With the goal of reducing computing costs and data-scientist-efforts, I am considering building one model that will predict each of the 7 classes (multi-output model) versus building and fine-tuning 7 different deep-learning models. I adopted this approach in the past but I find it difficult to find good discussions of the pros and cons of each approach.</p>
<p>YOUR INPUT<br>
I would appreciate anyone advice of the criteria they would use to decide to choose between one multi-output vs. several single-output models. I understand that the specifics of each deep-learning situation matter in the decision process but what I am after is overarching questions to ask / decision criteria.</p>
<p>MY HYPOTHESIS<br>
Here one of my take. Since transfer learning with transformers is generally performing well - even when freezing the weights of an underlying pre-trained transformer - one can assume that by adding 7 independent linear layers on top of shared transformer layers, the model accuracy should not decrease significantly given a large supervised training dataset.</p>
<p>Thank you in advance for your inputs.</p>
Low performance of transferring tensor to CUDA

username: AlexT
cooked: <p>Hello Team, i using Libtorch with CUDA support. I get lowest performance on transferring memory to CUDA operation. It cost about 2.60s.<br>
<strong>tensor_image = tensor_image.to(torch::kCUDA);</strong><br>
How can i improve this operation, check it or  profile?</p>
<pre><code class="lang-auto">  clock_t tROIset = clock();
  cv::Rect myROI(30, 10, 400, 400);
  printf("ROI set.\n Time taken: %.2fs\n", (double)(clock() - tROIset)/CLOCKS_PER_SEC);

  clock_t tLoadOpenCV = clock();
  cv::Mat img = cv::imread("test_data/image.png");     // 600x900  
  printf("loader of Tensor.\n Time taken: %.2fs\n", (double)(clock() - tLoadOpenCV)/CLOCKS_PER_SEC);
  
  clock_t tTester = clock();
  tensor_image = tensor_image.to(torch::kCUDA);
  printf("CPU-GPU transfer.\n Time taken: %.2fs\n", (double)(clock() - tTester)/CLOCKS_PER_SEC);
</code></pre>
<p>Result of execution:</p>
<p>ROI set in Tensor .               Time taken: 0.00s<br>
loader of Tensor.                  Time taken: 0.01s<br>
<strong>CPU-GPU Tensor transfer.  Time taken: 2.60s</strong></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e19dd67f4e4ce37cc3985167d3b51d7b6bb71b68.png" alt="изображение" data-base62-sha1="wbTB0FKX1ahoKRN5v8q3sPEzcfe" width="646" height="307"><br>
<strong>PyTorch version: 1.10.0</strong><br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Wed_Oct_23_19:24:38_PDT_2019<br>
<strong>Cuda</strong> compilation tools, <strong>release 10.2, V10.2.89</strong></p>

username: AlexT
cooked: <p>How can i use non blocking memory transfering here ?<br>
non_blocking=True, or something like this…</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the timers. Based on your code snippet you might profile the CUDA context initialization etc. which would increase the measured time.</p>
<p>Yes, <code>non_blocking=True</code> would allow the transfer to be executed in the background. Profiling it could still show the same time as the advantage would be a potential overlap with other workloads.</p>

username: AlexT
cooked: <p>How can i set this parametrs ( <code>non_blocking=True</code>) in C/C++ code?<br>
Are there examples?</p>

username: ptrblck
cooked: <p>The <code>to()</code> operations as well as e.g.<code>copy_</code> accept the <code>non_blocking</code> argument and an example was posted in your <a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234/24">other question</a>.</p>

username: AlexT
cooked: <p>Thanks Patrick, for the example, it seems things have moved)</p>
How to load saved model in google colab?

username: Sumera_Rounaq
cooked: <p>I saved my model like this:</p>
<pre><code class="lang-auto">torch.save({
            'epoch': all_epochs,
            'Gmodel_state_dict': gen.state_dict(),
            'Dmodel_state_dict': crit.state_dict(),
            'G_optimizer':gen_opt.state_dict(),
            'D_optimizer':crit_opt.state_dict()
            }, 'WGAN-GP-lr1-0.0002.pt')
</code></pre>
<p>For loading the model I wrote these lines of code:</p>
<pre><code class="lang-auto">checkpoint = torch.load('WGAN-GP-lr1-0.0002.pt')
gen.load_state_dict(checkpoint['Gmodel_state_dict']) # Load Generator model
crit.load_state_dict(checkpoint['Dmodel_state_dict']) # load Discriminator model
gen_opt.load_state_dict(checkpoint['G_optimizer']) # Load Generator's optimizer
crit_opt.load_state_dict(checkpoint['D_optimizer']) # Load Discriminator's optimizer
previous_epochs = checkpoint['epoch']
epoch_flag = True
print("Number of Epochs after training:",previous_epochs)
</code></pre>
<p>During loading the model I am getting this error:</p>
<p><strong>RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory</strong></p>
<p>I don’t have any idea why this error is occurring and how would I resolve it. Kindly guide me. Any help shall be highly appreciated.</p>
<p>Note: I am loading and saving the model in google colab.</p>

username: Sumera_Rounaq
cooked: <aside class="quote no-group" data-username="Sumera_Rounaq" data-post="1" data-topic="141652">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sumera_rounaq/48/30432_2.png" class="avatar"> Sumera_Rounaq:</div>
<blockquote>
<p>PytorchStreamReader failed reading zip archive: failed finding central directory</p>
</blockquote>
</aside>
<p>I have question in this regard<br>
If we save model on gpu machine and then want to load that model on google colab why it throws “PytorchStreamReader failed reading zip archive: failed finding central directory” error?</p>

username: ptrblck
cooked: <p>This error could be raised if the checkpoint file is corrupt and thus the reading fails. Could you check the file size of the stored checkpoint and the one used in Colab? Also, try to save a new checkpoint and load it in your other setup.</p>
Does NCCL allreduce use fp16?

username: zzibc
cooked: <p>Hello，<br>
I tried to reduce communication bottleneck by using auto mixed precision training. I also used <code>torch.profiler</code> to check the training performance.</p>
<p>In the timeline, I noticed that the all-reduce kernel is <code>ncclAllReduceRingLLKernel_sum_f32</code>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1732760bc372597e626bcac726b612c3a7a94787.png" alt="image" data-base62-sha1="3jd5QlXAxEvClxV2p0pSgqztKEn" width="258" height="26"></p>
<p>Does it mean the fp16 gradients are casted to fp32 to be all-reduced?  Does something like <code>ncclAllReduceRingLLKernel_sum_f16</code> exist?</p>
<p>My GPU is P100. PyTorch version is 1.9.1+cu111.</p>

username: ptrblck
cooked: <p>The parameters are stored in <code>float32</code> using the automatic mixed-precision util. and thus also the gradients. NCCL thus communicates them in <code>float32</code>, too. If you are calling <code>.half()</code> on the model directly and thus apply a pure <code>float16</code> training, NCCL should communicate in <code>float16</code>.</p>

username: zzibc
cooked: <p>Thanks ptrblck! I’ll try <code>half()</code>.</p>

username: ptrblck
cooked: <p>Sure, but be careful about the training stability, as <code>amp</code> makes sure that <code>float32</code> is used in layers where it’s needed.</p>

username: zzibc
cooked: <p>If I use <code>torch.cuda.amp.autocast</code> for mixed precision training, and I call <code>model.half()</code> to ensure that NCCL communicate in <code>float16</code>, do I need to call <code>float()</code> manually at some layers(like <code>BatchNorm</code>)?</p>

username: ptrblck
cooked: <p><code>autocast</code> doesn’t expect a manual <code>.half()</code> call so you should either use <code>amp</code> or your manual mixed-precision recipe.</p>

username: epoch
cooked: <p>Hi all,<br>
I am confused here FP32 parameters are stored indeed as the master copy, but why not the gradients appear as FP16 to store and transfer to reduce the load as <a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html" rel="noopener nofollow ugc"><em>Nvidia Apex Philosophy</em></a> details? Is there any tradeoff between the implementation and design of torch autocast?<br>
Any feedback will be appreciated!</p>

username: ptrblck
cooked: <p><code>torch.cuda.amp</code> is comparable to the deprecated <code>O1</code> <code>opt_level</code> in apex, not <code>O2</code>, as the former was considered to be more stable.</p>
Why doesn't the accuracy change when I train

username: hunter137
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/1458f1c229e74d6444ce097acb6749b32965afc6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1458f1c229e74d6444ce097acb6749b32965afc6" title="e7f09ecfddac61e5f6228c97258c7d3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1458f1c229e74d6444ce097acb6749b32965afc6_2_367x500.jpeg" alt="e7f09ecfddac61e5f6228c97258c7d3" data-base62-sha1="2U06VUKfyNh7JphkxKtQvGcSKqO" width="367" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1458f1c229e74d6444ce097acb6749b32965afc6_2_367x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1458f1c229e74d6444ce097acb6749b32965afc6_2_550x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1458f1c229e74d6444ce097acb6749b32965afc6_2_734x1000.jpeg 2x" data-dominant-color="372D30"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">e7f09ecfddac61e5f6228c97258c7d3</span><span class="informations">1170×1592 95.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/hunter137">@hunter137</a> there are multiple reasons why this might happen. Lack of convergence etc. Please provide your code and data link so that people can take a look</p>
Select rows from a 2D tensor

username: Deeply
cooked: <p>I have a tensor  <code>x</code> :</p>
<pre><code class="lang-auto">x.shape
torch.Size([90, 50])
dtype=torch.float64, device='cuda:0'
</code></pre>
<p>and I need to select rows defined by the list:</p>
<pre><code class="lang-auto">loc = [0, 0, 0, 1, 0, 1, 1, 0, 1,..., 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1]
# len(loc) is 90
</code></pre>
<p>and store the selected rows into tensor <code>y</code><br>
I think there is this function  torch.masked_scatter_(), but I did not manage to use it successfully!</p>

username: Diego
cooked: <p>You can do it similar as you would do it with numpy indexing. Like so:</p>
<pre><code class="lang-auto"> import torch
 x = torch.rand(5,4)
 loc = torch.ByteTensor([0,1,0,0,1])
 y = x[loc]
</code></pre>
<p>Storing in y the 2nd and 5th rows of the x tensor as indicated by the ones in your loc tensor. Is this what you need?</p>

username: Deeply
cooked: <aside class="quote no-group" data-username="Diego" data-post="2" data-topic="20839">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego/48/6989_2.png" class="avatar"> Diego:</div>
<blockquote>
<p>import torch x = torch.rand(5,4) loc = torch.ByteTensor([0,1,0,0,1]) y = x[loc]</p>
</blockquote>
</aside>
<p>Yes  Diego, it worked. Thanks a bunch.</p>

username: mhadnanali
cooked: <p>import torch<br>
xy = torch.rand(5,4)<br>
loc = torch.tensor([False,True,False,False,True])<br>
y = xy[loc]<br>
print(y)</p>
<p>Worked for me like this. Hope maybe useful for anyone else. Thanks to <a class="mention" href="/u/diego">@Diego</a></p>
RAM memory consumption in compressed models

username: fnak
cooked: <p>I am running the following compression/optimization algorithms on my model:<br>
1- pruning<br>
2- fusion<br>
3- quantization</p>
<p>I am using a profiler to monitor the memory usage on the models as follows:</p>
<pre><code class="lang-auto">prof = torch.profiler.profile(
        activities=[torch.profiler.ProfilerActivity.CPU],
        on_trace_ready=torch.profiler.tensorboard_trace_handler(logPath),
        profile_memory=True,
        record_shapes=True,
        with_stack=True)

prof.start()

for image, label in test_loader:

    with torch.no_grad():
        output, min_distances = model(image)
    all_preds = torch.cat((all_preds, output)  ,dim=0)
    all_labels = torch.cat((all_labels, label) ,dim=0)

    prof.step()

prof.stop()
</code></pre>
<p>Weirdly enough, I am monitoring the peak memory (RAM) usage of the aforementioned models (originally trained model, pruned model, fused and quantized model) and it seems to get higher in every single one.</p>
<p>While I can see the file size difference and inference speed of the quantized model, it does however have a higher peak memory usage.</p>
<p>Question 1: Is this a normal behavior? because I understand that pruning in pytorch might not have any effect on the model, but quantization should.<br>
Question 2: Am I measuring memory (RAM) usage correctly? or is there a better way to do it?</p>
<p>Thanks</p>
Using torch::from_blob(...) with const data

username: robeast
cooked: <p>Hello,<br>
due to infrastructural limitations, I need to use a <code>const</code> input to <code>from_blob</code> to create a <code>torch::tensor</code>. Minimal reproducible example:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

int main ()
{
    const float array[] = {1.23, 2.34, 3.45, 4.56, 5.67};
    auto options = torch::TensorOptions().dtype(torch::kFloat32);
    torch::Tensor t1 = torch::from_blob(array, {5}, options);
    std::cout &lt;&lt; t1 &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>This raises however a compilation error: <code>invalid conversion from ‘const void*’ to ‘void*’ [-fpermissive]</code>. Of course I can cast away the const-ness of the array, but is there a way to avoid this? Can I somehow use <code>from_blob</code> (or similar)  with const data input?</p>
<p>As always, help is highly appreciated… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thank you &amp; Best regards,<br>
RB</p>

username: hoyden
cooked: <p>You can use std::copy_n as below:</p>
<pre><code class="lang-auto">const float array[] = {1.23, 2.34, 3.45, 4.56, 5.67};
auto options = torch::TensorOptions().dtype(torch::kFloat32);
torch::Tensor t1 = torch::empty({5}, options);
std::copy_n(array, 5, t1.contiguous().data_ptr&lt;float&gt;());
</code></pre>

username: robeast
cooked: <p>Hi <a class="mention" href="/u/hoyden">@hoyden</a>, thank you very much for your suggestion. Yes, you are right. I did a similar workaround with a simple memcpy(). The only issue is, that such a a memory copy per input and output tensor hurts runtime execution performance. I will apply this workaround, if there is no other way to deal with const arrays. But is there? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thank you very much &amp; Best regards,<br>
RB</p>

username: hoyden
cooked: <p>I only know this one way to solve your problem. BTW, why you could only use a <code>const</code> input?</p>

username: robeast
cooked: <p>Hi <a class="mention" href="/u/hoyden">@hoyden</a>, thanks once again for your answer. It’s a bit tricky. We need to extend the ONNX Runtime (ORT) with operators that are not yet present. This can be done since ORT has a pretty nice C/C++ extension mechanism.</p>
<p>To save development time/effort we are basically using Libtorch in the ORT custom operators, and thus just need to interface the input/output data formats between ORT-&gt;Torch-&gt;ORT. However, the inputs, as defined by ORT come always as const float*. We don’t have a saying in this. We can always cast away the constedness, but that is also not the nicest thing…</p>
Volatile gpu-util is 100 and no progress

username: Oktai15
cooked: <p>I have Unet, size_batch = 16 and images that have (512, 512, 3) shape.<br>
My training doesn’t progress and nvidia-smi shows me that both my GPU’s have 100 volatile gpu-util. I waited around 30 minutes no one batch was progress…</p>
<p>What happened?</p>

username: jianchao-li
cooked: <p>Hello, <a class="mention" href="/u/oktai15">@Oktai15</a>. I also came across a similar problem while training a segmentation network using <a href="https://github.com/CSAILVision/semantic-segmentation-pytorch" rel="nofollow noopener">semantic-segmentation-pytorch</a>. After running for some random number of epochs, all volatile GPU util becomes 100% and the training seems to be stuck.</p>
<p>Did you fix the problem or have any thoughts about it?</p>

username: adhi54
cooked: <p>Hello, <a class="mention" href="/u/oktai15">@Oktai15</a> , Me too have a similar problem, any info about it?</p>

username: LucasX
cooked: <p>I came across the same problem, did you solve it ?</p>

username: Olivier-CR
cooked: <p>same problem today. Script working fine without DDP. When using DDP, all 4 GPUs are 100% and no progress (almost no CPU activity either, and GPU at mid-power,)</p>
Custom gradient with different forward and backward paths

username: Mohamad7
cooked: <p>Hi everyone,</p>
<p>I want to implement a ML optimization while having different forward and backward function for one module in my computational graph. I got the intuition from <a href="https://discuss.pytorch.org/t/custom-backward-pass/83433">this</a> link, but when running the code, gradients can be passed for the module  but the loss won’t be decreased (converged). I bring a scheme of my code in the following for more details:</p>
<pre><code class="lang-auto">class Forward_function(torch.nn.Module):
    def __init__(self):
        super(Forward_function, self).__init__()

    def forward(self, input1, input2):
        ...
        return output_forward

class Backward_function(torch.nn.Module):
    def __init__(self):
        super(Backward_function, self).__init__()

    def forward(self, input1, input2):
        ...
        return output_backward
</code></pre>
<p>Here the forward function is not differentiable (cannot pass gradients) and we intend to use the backward function for the backpropagation. So, we want to have the “output_forward” as the output of forward path, and take gradients from “output_backward” in the backward path. Hence, the final module would be:</p>
<pre><code class="lang-auto">forward_class = Forward_function()
backward_class = Backward_function()

class Forward_Backward(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input1, input2):
        with torch.enable_grad():
            output = backward_calss.forward(input1, input2)
            ctx.save_for_backward(output, input1, input2)
        return forward_calss.forward(input1, input2)
    @staticmethod
    def backward(ctx, grad_output):
        output, input1, input2, = ctx.saved_tensors
        input1_grad = torch.autograd.grad(output, input1, grad_output)
        input2_grad = torch.autograd.grad(output, input2, grad_output)
        return input1_grad[0], input2_grad[0]
</code></pre>
<p>For more details, input1_grad and input2_grad are tuples and their gradients lies in their first index. Also, for computing gradients we chose to use “torch.autograd.grad” instead of “torch.autograd.backward”, since in the middle of backwarding from loss function (when calling loss.backward), output.backward function calculates the gradients for all leaf variable again from the “Forward_Backward” module to the beginning of the graph. It seems like we are calling another .backward function in the middle of loss.backward, and it seems to be wrong to proceed in this way.</p>
<p>Does anyone find what is the problem with my code? Could you please help me to get rid of this problem?</p>
<p>Thanks a lot for your help in advance!</p>
Quantization for VGG model

username: Shashank_Shekhar
cooked: <p>For the below code from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py</a> I was trying to implement quantization. Since presently it doesn’t has Conv, BatchNorm and Relu as a single block, for classification class what should be the fuse_model.</p>
<p><code>cfg=[64,"M",128,"M",256,256,"M",512,512,"M",512,512,"M"]</code></p>
<pre><code class="lang-auto">def vgg_make(cfg,batch_norm):
    emp=[]
    inp_c=3
    for i,val in enumerate(cfg):
        if val=="M":
            emp+=[nn.MaxPool2d(kernel_size=(2,2), stride=2)]
        else:
            conv=nn.Conv2d(inp_c, val, kernel_size=(3,3), padding=1)
            if batch_norm:
                emp+=[conv, nn.BatchNorm2d(val), nn.ReLU(inplace=True)]
            else:
                emp+=[conv, nn.ReLU(inplace=True)]
            inp_c=val
    return nn.Sequential(*emp)
</code></pre>
<pre><code class="lang-auto">backbone=vgg_make(cfg, batch_norm=True)
</code></pre>
<pre><code class="lang-auto">classifer = nn.Sequential(nn.Linear(512*7*7,4096),
                          nn.ReLU(inplace=True),
                          nn.Dropout(),
                          nn.Linear(4096,4096),
                          nn.ReLU(inplace=True),
                          nn.Dropout(),
                          nn.Linear(4096,10))
</code></pre>
<pre><code class="lang-auto">class classfication(nn.Module):
    def __init__(self, backbone, classifer):
        super().__init__()
        self.backbone=backbone
        self.classifer=classifer
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
    def forward(self, x):
        x=self.quant(x)
        x=self.backbone(x)
        x=torch.flatten(x, 1)
        x=self.classifer(x)
        x=self.dequant(x)
        return x
    
    def fuse_model(self):
        for m in self.modules():
            pass
</code></pre>

username: Shashank_Shekhar
cooked: <p><strong>Unable to fuse convolution, batchnorm and relu.</strong></p>
<p>Each entry in the module list looks something like</p>
<pre><code class="lang-auto">  (0): ConvBNReLU(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU6(inplace=True)
  )
</code></pre>
<p>At the time of fusion using below code i get the error.</p>
<pre><code class="lang-auto"> def fuse_model(self):
     for m in self.modules():
        if type(m) == ConvBNReLU:
           fuse_modules(m, ['0', '1', '2'], inplace=True)
</code></pre>
<pre><code class="lang-auto">NotImplementedError: Cannot fuse modules: (&lt;class 'torch.nn.modules.conv.Conv2d'&gt;, &lt;class 'torch.nn.modules.batchnorm.BatchNorm2d'&gt;, &lt;class 'torch.nn.modules.activation.ReLU6'&gt;)
</code></pre>
<p>Please let me know the way around.</p>

username: Raghav_Gurbaxani
cooked: <p>did you find a way around this ? having the same problem</p>

username: Ziwen_Lu
cooked: <p>Try replace ReLU6 with ReLU, as quantization does not currently support ReLU6.</p>
How to force the weights of a Conv layer to be positive?

username: Ximeng
cooked: <p>Hi,</p>
<p>I tried to use several convolution layers as learnable filters or smoothers to process time series data. However, I found the some of the weights are still negative even after convergence. And this is definitely not a good property for a smoother, so I want to force the all the weights to be positive.</p>
<p>What I am currently doing is to build a weight tensor x with require_grads=True manually, and envelop it with a softplus function. Just curious if there is any more efficient way to handle it? Thank you!</p>
<p>Best,</p>

username: googlebot
cooked: <p>I don’t think so, all included optimizers work on real domain, so unrestricted &lt;-&gt; positive mapping seems unavoidable.</p>

username: tom
cooked: <p>The fancy way PyTorch offers for this is parametrizations, but I think for positive you still need to roll your own transformation (but it could be easy with exp or softplus or so).<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html?highlight=parametrization#torch.nn.utils.parametrize.register_parametrization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html?highlight=parametrization#torch.nn.utils.parametrize.register_parametrization</a></p>
<p>Best regards</p>
<p>Thomas</p>

username: Ximeng
cooked: <p>That perfectly solve my problem. Many Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=10" title=":grin:" class="emoji" alt=":grin:"></p>
Torch::empty taking significant amount of CPU time

username: tgale96
cooked: <p>I have a model that uses a number of custom CUDA operations. When benchmarking/profiling one of the core modules I’m seeing that roughly one out of every ten runs torch::empty is taking as much as 200us on CPU per call, which across all of the 19 calls that happen in one forward pass of my model is making up 85% of the CPU time. All of the operations in the module are on GPU, but these empty tensor creation calls are the primary bottleneck for the runs where this is happening.</p>
<p>Why is this happening? I expect these to be essentially free. Is there a workaround?</p>
<p>I’m using pytorch 1.8 with CUDA 11.2 in the standard <a href="https://github.com/NVIDIA/Megatron-LM" rel="noopener nofollow ugc">Megatron</a> container (NGC’s pytorch 20.12).</p>
Why is the dropout layer affecting all values, not only the ones set to zero?

username: nicolas-gervais
cooked: <p>The dropout layer from Pytorch changes the values that are not set to zero. Using Pytorch’s documentation example: (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout" rel="nofollow noopener">source</a>):</p>
<pre><code class="lang-python">import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.ones(5, 5)
</code></pre>
<pre><code class="lang-python">print(input)
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])
</code></pre>
<p>Then I pass it through a <code>dropout</code> layer:</p>
<pre><code class="lang-python">output = m(input)
print(output)
tensor([[0., 0., 2., 2., 0.],
        [2., 0., 2., 0., 0.],
        [0., 0., 0., 0., 2.],
        [2., 2., 2., 2., 2.],
        [2., 0., 0., 0., 2.]])
</code></pre>
<p>The values that aren’t set to zero are now 2. Why?</p>

username: ptrblck
cooked: <p>You need to scale the activations either</p>
<ul>
<li>during testing with the keep probability or</li>
<li>during training with <code>1/keep_prob</code>
</li>
</ul>
<p>Otherwise the expected activation values will have a different range and your model will perform poorly.<br>
PyTorch uses the second approach.</p>

username: seyeeet
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> can you provide a little more explanation on why we need to do one of the two solutions that you mentioned and why otherwise it will cause the poor performance?</p>

username: ptrblck
cooked: <p>The expected value range of the output activations wouldn’t match, if you are dropping activations during training with the specified <code>p</code> value and are using them during evaluation.<br>
From the <a href="https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf">dropout paper</a>:</p>
<blockquote>
<p>At test time, it is not feasible to explicitly average the predictions from exponentially<br>
many thinned models. However, a very simple approximate averaging method works well in<br>
practice. The idea is to use a single neural net at test time without dropout. The weights<br>
of this network are scaled-down versions of the trained weights. If a unit is retained with<br>
probability p during training, the outgoing weights of that unit are multiplied by p at test<br>
time as shown in Figure 2. This ensures that for any hidden unit the expected output (under<br>
the distribution used to drop units at training time) is the same as the actual output at<br>
test time.</p>
</blockquote>
Memory usage in forward/backward

username: thyeros
cooked: <p>I’m playing with torch.autograd.graph.saved_tensors_hooks to compress the tensors. I found during the process, the following two codes look to have different memory usage (assuming x.size()&gt;&gt; b.size()).</p>
<pre><code class="lang-auto">case a)
   x = f(a)
   y = matmul(x.T, b)
   z = matmul(x, c)

case b)
   x = f(a)
   y = matmul(b.T, x)
   z = matmul(x, c)
</code></pre>
<p>It looks the case b) keeps using the same x in forward of y,z (verified with id()), while the case a) creates a new instance to save x.T (verified with id()). In that sense, can I say b) is more memory efficient?</p>
<p>Also as a related question, it looks pytorch doesn’t duplicate the same tensor for the forward/backward by looking at the python id of tensors. Is this the right way to check if two tensors point to the same memory allocation?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="thyeros" data-post="1" data-topic="141411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thyeros/48/12827_2.png" class="avatar"> thyeros:</div>
<blockquote>
<p>In that sense, can I say b) is more memory efficient?</p>
</blockquote>
</aside>
<p>No, as the transposes should be views and the memory allocations should thus also be identical as seen via <code>print(torch.cuda.memory_summary())</code>.</p>

username: thyeros
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , but does it hold true for the backward case? Doesn’t autogrid make independent copies of b and b.T? Please correct me if I’m wrong. But if I’m right, then</p>
<p>case a) stores:  x, x.T, b, c<br>
case b) stores: x, b.T, b, c</p>
<p>Since x.size()&gt;&gt; b.size(), case a) needs more memory for backward?</p>

username: ptrblck
cooked: <p>No, this should also not be the case and you could profile the memory usage via <code>torch.cuda.memory_summary()</code> to compare both approaches.</p>
<pre><code class="lang-python">print(torch.cuda.memory_summary())
x = torch.randn(1024, 1024).cuda().requires_grad_(True)
b = torch.randn(1024, 1).cuda().requires_grad_(True)
c = torch.randn(1024, 512).cuda().requires_grad_(True)


# y = torch.matmul(x.T, b)
# print(torch.cuda.memory_summary())
# y.mean().backward()
# print(torch.cuda.memory_summary())
# z = torch.matmul(x, c)
# z.mean().backward()
# print(torch.cuda.memory_summary())


y = torch.matmul(b.T, x)
print(torch.cuda.memory_summary())
y.mean().backward()
print(torch.cuda.memory_summary())
z = torch.matmul(x, c)
z.mean().backward()
print(torch.cuda.memory_summary())
</code></pre>
<p>Both methods show the identical allocations.</p>

username: thyeros
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> really appreciated!</p>
Memory parity with JAX attention

username: gahdritz
cooked: <p>Our attention module is essentially standard multi-head attention, just with arbitrarily many bias tensors. It’s approximately as follows:</p>
<pre><code class="lang-auto">a = torch.matmul(q, k)

for b in biases:
    a += b

a = softmax(a, dim=-1)

o = torch.matmul(a, v.transpose(-1, -2))
</code></pre>
<p>where <code>q</code>, <code>k</code>, and, <code>v</code> are <code>[batch_dim, heads, Q/K/V, C]</code> query, key, and value tensors, respectively, and biases is a list of up to 2 bias tensors (e.g. a mask), all of which broadcast along at least one dimension such that their sizes are insignificant (they’re much smaller than the full <code>[batch_dim, heads, Q, K]</code> attention logit matrix <code>a</code>, in any case). During the backwards pass, since <code>batch_dim</code>, Q and K are &gt;&gt; C, memory consumption is dominated by the first matrix multiplication and the softmax, which together contribute (<strong>2</strong> * <code>batch_dim</code> * <code>heads</code> * Q * K * (2 bytes)) to the module total. Equivalent JAX code requires just (<code>batch_dim</code> * <code>heads</code> * Q * K * (2 bytes)), half as much.</p>
<p>torch doesn’t have anything like an in-place softmax, and for various reasons I’ve been unable to shrink the memory requirements using keops (apparently incompatible with the bias tensors) or by manually building an in-place softmax myself (breaks auto-differentiation).</p>
<p>Is there a (<code>batch_dim</code> * <code>heads</code> * Q * K * (2 bytes)) solution in torch?</p>
ResNet152 - Flipping features vs Flipping video then computing features

username: Wilan
cooked: <p>I’m currently using ResNet152 to extract features from a video, where the extracted features are of shape <code>(1842, 2048)</code>. I would like to also extract features for the same video, but horizontally flipped. I see 2 ways to do so:</p>
<ol>
<li>Flip the video horizontally, then re-compute features</li>
<li>Flip the ResNet152 features for each frame from the already-computed features on the original video</li>
</ol>
<p>The 2nd option is attractive when considering computing resources - how would that be achieved? Are ResNet152 features able to be “flipped” like this? If so, I’m guessing it can be done with <code>torch.flip</code>?</p>
<p>Thank you for reading! Hoping someone can guide me in the right direction.</p>

username: ptrblck
cooked: <p>I’m not sure but don’t think that flipping the intermediate features would work.<br>
If you only flip the input image, e.g. the conv kernels would not be flipped and the intermediate activations (as well as the ouput) would be different. Think about kernels used as edge detectors which work on a flipped image, so I would probably try to use the first approach of flipping the inputs.</p>

username: Wilan
cooked: <p>Thanks so much for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>! It aligns with what I found when comparing my results from the 2 options (just wanted to be sure).</p>
How to get RGB value of a value from model output in Pytorch

username: KURUVILLA_ABRAHAM
cooked: <p>Hello Team,</p>
<p>After getting model output i need to get dominant color of an image from(i.e RGB value of image) to calculate loss with the RGB value .</p>
<p>Can you please suggest how to do in pytorch ,I could do the same using open cv.<br>
But how to via Tensors</p>
Caffe2:TypeMeta

username: afshin67
cooked: <p>What is the usage of <code>Caffe2:TypeMeta</code>?<br>
I found this description:<br>
<code>TypeMeta is a thin class that allows us to store the type of a container such as a blob, or the data type of a tensor, with a unique run-time id. It also stores some additional data such as the item size and the name of the type for run-time inspection. Definition at line 324 of file typeid.h.</code><br>
But, really I am not sure how does it work.<br>
In the following example, it gets <code>z.dtype()</code> which is <code>at:Tensor</code> and then it saves it in <code>dtype</code> as <code> caffe2::TypeMeta</code>.</p>
<pre><code>    caffe2::TypeMeta dtype;
    z = torch::rand({1,4});
    dtype = z.dtype();
</code></pre>
<p>Not sure what is the point of that and how it is possible to get the data back from <code>dtype</code> as it is done in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/adam.cpp at master · pytorch/pytorch · GitHub</a></p>
Weights initialization

username: pascalm
cooked: <p>Hello,</p>
<p>I’m a bit confused about weight initialization. In my neural network I use: BatchNorm1d, Conv1d, ELU, MaxPool1d, Linear, Dropout and Flatten.</p>
<p>Now I think only Conv1D, Linear and ELU have weights right? In particular:</p>
<p>Conv1D: Has weights for the weighted sum it uses.<br>
ELU: Has alpha as a weight<br>
Linear: Weights represent basically the transformation matrix</p>
<p><strong>Question 1:</strong><br>
Now all those weights need to be set to something in the beginning. I know that for symmetrical activation functions, one uses Xavier and for thinks like ReLU (and I guess ELU) one uses Kaiming to set these weights. Correct?</p>
<p><strong>Question 2:</strong><br>
What is used for the weights for Linear?</p>
<p><strong>Question 3:</strong><br>
What is used for Conv1D weights?</p>
<p><strong>Question 4:</strong><br>
What are the default weights set by pyTorch? I guess they are:</p>
<ul>
<li>Linear: alpha: float = 1.</li>
<li>Conv1D: U(-sqrt(k), sqrt(k)) with k = groups / (Cin*kernel siye) whereas k = 1 by default.</li>
<li>ELU: alpha = 1.0</li>
</ul>
<p>Correct?</p>
<p><strong>Question 5:</strong><br>
Do people set the weights only at the beginning or are there usecases where one does it while training?</p>
<p><strong>Question 6:</strong><br>
What is the correct way of initialize weights?</p>
<p>Thanks in advance</p>

username: JamesWright
cooked: <p>In general, I highly recommend looking into <a href="https://pytorch.org/docs/1.9.1/index.html" rel="noopener nofollow ugc">PyTorch’s documentation</a>, e.g. information about weight initialisation for <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">torch.nn.Linear</a> can be found under <em>Variables</em> section. Same information can be found also in the source code - see <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L92-L100" rel="noopener nofollow ugc">reset_parameters()</a> method of <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L39" rel="noopener nofollow ugc">Linear</a> class.</p>

username: JamesWright
cooked: <p>For correct way of initialising weights, see <a href="https://pytorch.org/docs/stable/nn.init.html" rel="noopener nofollow ugc">torch.nn.init</a>.</p>
<p>The example with <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Conv2D</a>, would be:</p>
<pre><code class="lang-python">conv = torch.nn.Conv2d(16, 33, 3)
torch.nn.init.xavier_uniform_(conv.weight)
</code></pre>

username: pascalm
cooked: <p>Oh I did check all that, that’s how I came up with the above information in the post. I actually ended up doing what you did, I was just very unsure what people usually do and how.</p>

username: pascalm
cooked: <p>also this is outdated btw. one should use <code>xavier_uniform_()</code></p>

username: JamesWright
cooked: <p>You are correct, I’ve fixed it</p>

username: seyeeet
cooked: <p>I am also interested in knowing the answer of those questions…</p>
RuntimeError during compilation on different computer

username: guillefix
cooked: <p>I have a module that is running fine on one computer, but on another computer it throws this error</p>
<pre><code class="lang-auto">  File "/home/guillefix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: __nv_nvrtc_builtin_header.h(78048): error: function "operator delete(void *, size_t)" has already been defined

__nv_nvrtc_builtin_header.h(78049): error: function "operator delete[](void *, size_t)" has already been defined

2 errors detected in the compilation of "default_program".

nvrtc compilation failed: 

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)


template&lt;typename T&gt;
__device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a &gt; b ? a : b);
}

template&lt;typename T&gt;
__device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a &lt; b ? a : b);
}

extern "C" __global__
void fused_tanh_mul_add__14203776399538843293(float* tv_, float* tb0_2, float* tv__, float* tv___, float* tv____, float* aten_log, float* aten_cat) {
{
if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)&lt;8ll ? 1 : 0) {
    aten_cat[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = (((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)&lt;4ll ? 1 : 0) ? (__ldg(tv__ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) / (1.f / (1.f + (expf(0.f - ((__ldg(tv___ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) * (tanhf(__ldg(tv____ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)))) + 2.f)))) + 0.1192029193043709f) - (__ldg(tb0_2 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) : __ldg(tv_ + ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) - 4ll));
  }if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)&lt;4ll ? 1 : 0) {
    float v = __ldg(tv___ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
    float v_1 = __ldg(tv____ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
    aten_log[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = logf(1.f / (1.f + (expf(0.f - (v * (tanhf(v_1)) + 2.f)))) + 0.1192029193043709f);
  }}
}
</code></pre>
<p>The computer in which it works has cuda 11.4, and torch 1.10.0a0+git36449ea, and the comptuer in which it doesnt work has cuda 11.5 and torch 1.10.1+cu102 (though I also tried with 1.10.0+cu102). So it seems like quite a similar set up. Any idea about why it’s failing?</p>

username: guillefix
cooked: <p>It worked after updating the Nvidia Toolkit to version 11.6!</p>
RuntimeError: One of the differentiated Tensors does not require grad

username: ToxicDoubleH
cooked: <p>I want a batchnorm with no bias. so I write like this:</p>
<pre><code class="lang-auto">self.bn = BatchNorm1d(2048)
self.bn.bias.requires_grad_(False)  # no shift
</code></pre>
<p>And what I want to do is meta-learning, I need to calculate the grad, so I write</p>
<pre><code class="lang-auto">grad_info = torch.autograd.grad(loss_meta_train, self.model.module.params(), create_graph=True)
</code></pre>
<p>to require backward, but I get this error:</p>
<pre><code class="lang-auto">RuntimeError: One of the differentiated Tensors does not require grad
</code></pre>
<p>the Tensor is BatchNorm’s bias, how can I solve this question</p>

username: thecho7
cooked: <p>What about using <em>allow_unused=True</em>?</p>
<pre><code class="lang-auto">grad_info = torch.autograd.grad(loss_meta_train, self.model.module.params(), create_graph=True, allow_unused=True)
</code></pre>
<p>The question is quite similar to <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-differentiated-tensors-appears-to-not-have-been-used-in-the-graph-set-allow-unused-true-if-this-is-the-desired-behavior/43679" class="inline-onebox">RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior</a></p>
How to generate a color using Torch

username: KURUVILLA_ABRAHAM
cooked: <p>Hello Team,</p>
<p>Have a task to pass a Target color ,How to define a color in torch ?<br>
right now i am using the same .</p>
<p>Target_color = torch.randn(1,256).to(device)</p>
<p>But i am not sure what color is this</p>
<p>so my task is</p>
<p>Example - Passing a color red</p>
<p>Pass color red to Torch Tensor</p>
<p>Please let me know on the same .</p>

username: ptrblck
cooked: <p>I don’t know what the actual use case is, but take a look at the <a href="https://en.wikipedia.org/wiki/RGB_color_spaces">RGB color space</a> and how a color is defined in it (i.e. via 3 <code>uint8</code> values). Then create a tensor with the desired color by assigning the R, G, and B value to it.</p>

username: KURUVILLA_ABRAHAM
cooked: <p>hello patrick,</p>
<p>For Example this is the code i am using generating using numpy</p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt</p>
<h1>
<a name="create-a-blank-300x300-red-image-1" class="anchor" href="#create-a-blank-300x300-red-image-1"></a>Create a blank 300x300 red image</h1>
<p>image = np.zeros((200, 200, 3), np.uint8)</p>
<h1>
<a name="fill-image-with-red-colorset-each-pixel-to-red-2" class="anchor" href="#fill-image-with-red-colorset-each-pixel-to-red-2"></a>Fill image with red color(set each pixel to red)</h1>
<p>image[:] = (255, 0, 0)</p>
<p>It is a red color</p>
<p>Now i want to use this color information in torch ,How can i do that ?</p>

username: ptrblck
cooked: <p>If you want to use the numpy array as a tensor, you could use <code>tensor = torch.from_numpy(image)</code>. Alternatively, if you want to create the same “red tensor” this should work:</p>
<pre><code class="lang-python">image = torch.zeros(200, 200, 3, dtype=torch.uint8)
image[:, :, 0] = 255
</code></pre>

username: KURUVILLA_ABRAHAM
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="141436">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">image = torch.zeros(200, 200, 3, dtype=torch.uint8)
image[:, :, 0] = 255
</code></pre>
</blockquote>
</aside>
<p>Thanks a Lot Patrick this worked for me .</p>