akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I change this for loop operation to matrix operation?

username: jeongHwarr
cooked: <p>This task is about classification.</p>
<p>I want the model to make predictions with priority information, not just probability, when predicting a true label.</p>
<p>For example, there are 6 classes and I have the priority information like this  [0,0,0,1,1,2], threshold: 0.5</p>
<p>Lower number means more important class!</p>
<p>If the difference in probability between the label with more important priority and the current prediction label not exceed the threshold, I change the current prediction label to a label with more important priority.</p>
<p>For example, probability from models [0.3, 0.2, 0.15, 0.15, 0.1, 0.1] and priority [1,0,1,1,1,1], threshold:0.2.</p>
<p>Normally, the predicted class should be 0th, but it is changed to 1th according to priority information.<br>
(1th class is more important class and 0.3-0.2&lt;0.2)</p>
<p>I implemented this logic with the for loop below, but I think it can be optimized with matrix operation. How can I do it?</p>
<pre><code class="lang-auto">def modify_predResult_with_priority(prob, priority, threshold):
    #prob: [batch_size, num_cls]
    #priority: [num_cls] ex) 6class: [0,0,0,1,1,2]
    #threshold: 0.5
   
    sorted_prob, sorted_prob_cls_index = prob.sort(descending=True)
    sorted_prob_priority = torch.tensor(priority)[sorted_prob_cls_index]

    candidate_cls = sorted_prob_cls_index[:,0]
    candidate_prob = sorted_prob[:,0]
    candidate_priority = sorted_prob_priority[:,0]

    for batch in range(sorted_prob.shape[0]):
        for idx in range(sorted_prob.shape[1]):
            if idx==0:
                continue
            if ((candidate_prob[batch] - sorted_prob[batch][idx]) &gt; threshold):
                break
            if (sorted_prob_priority[batch][idx] &lt; candidate_priority[batch]):
                candidate_prob[batch] = sorted_prob[batch][idx]
                candidate_cls[batch] = sorted_prob_cls_index[batch][idx]
                candidate_priority[batch] = sorted_prob_priority[batch][idx]
    return candidate_cls
</code></pre>

username: zilong
cooked: <p>Maybe like this?</p>
<pre><code class="lang-auto">#Get idx which position needs priority update
idxs = prob &gt; threshold

# Update the priority information
priority[idxs] += 1
</code></pre>
Libtorch.so aarch64

username: dankirlin
cooked: <p>Hey all,</p>
<p>I found this post on the Pytorch website that mentions aarch64 builds.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/prototype-features-now-available-apis-for-hardware-accelerated-mobile-and-arm64-builds/#arm64-builds-for-linux">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/prototype-features-now-available-apis-for-hardware-accelerated-mobile-and-arm64-builds/#arm64-builds-for-linux" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/prototype-features-now-available-apis-for-hardware-accelerated-mobile-and-arm64-builds/#arm64-builds-for-linux" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Would someone be able to point me to the linux aarch64 build for libtorch.so?</p>
Simple Custom QAT .backward() function

username: Lfiske1
cooked: <p>I have a model which I need to quantize down to a very low bit depth (3-5 bits) so I can not use the Quantize package which only supports 8 bit numbers.</p>
<p>My model can be called in two ways: Model(x) and Model.quantized_forward(x) which involves a not differentiable round operation. I want to implement simple Quantization aware L2 loss function of the form:</p>
<p>MSE(x) = || Model(x) - Target ||^2<br>
Grad(MSE(x)) = Grad(Model(x))^T (Model.quantized_forward(x) - Target)</p>
<p>I am very confused about how to write this backwards function for something like this which uses the gradient of the model as well as a custom part. Can anyone help me get unstuck?</p>
<p>Thank you!</p>
Backward pass for this setting

username: conv8d
cooked: <p>I have 2 models <code>m1</code> and <code>m2</code> and 2 losses <code>l1</code> and <code>l2</code><br>
I want to train both models jointly.<br>
<code>m1</code> needs to be updated using <code>(l1 + l2)</code> and <code>m2</code> only by <code>l2</code>.<br>
To achieve this, I have 2 optimizers <code>o1</code> (for <code>m1</code>'s parameters) and <code>o2</code> (for <code>m2</code>'s parameters)<br>
Is the following approach correct for achieving what I want?</p>
<pre><code class="lang-auto">y1 = m1(x1)
y2 = m2(x2)
loss1 = l1(y1, y) # y is ground truth
loss2 = l2(y1, y2) # l2 works on the outputs for m1 and m2
loss = loss1 + loss2 
loss.backward()
o1.step()
o2.step()
</code></pre>
<p>Here, I make the assumption that <code>d loss / d m2</code> will be same as <code>d loss2 / d m2</code></p>

username: InnovArul
cooked: <p>This seems to be correct.</p>
How to Load a model_dict saved in GPU on cpu

username: alessandro_mondin
cooked: <pre><code class="lang-auto"># original saved file with DataParallel
state_dict = torch.load('myfile.pth.tar')
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
model.load_state_dict(new_state_dict)```

I've found this code on the forum but the object is saved on .tar, 
how does this code modify in order to be applied to a .pt file?
Thanks in advance</code></pre>

username: ptrblck
cooked: <p>Try to pass <code>map_location='cpu'</code> to <code>torch.load</code> as it should then directly load the data to the CPU.</p>
IndexError: index 30 is out of bounds for dimension 0 with size 30

username: westye
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48f7366a2a05259a132a1d4ce94b4086b287fa9c.png" data-download-href="https://discuss.pytorch.org/uploads/default/48f7366a2a05259a132a1d4ce94b4086b287fa9c" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48f7366a2a05259a132a1d4ce94b4086b287fa9c_2_690x379.png" alt="4" data-base62-sha1="apu1KULtPiFadqzFQakyz60gMdS" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48f7366a2a05259a132a1d4ce94b4086b287fa9c_2_690x379.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48f7366a2a05259a132a1d4ce94b4086b287fa9c_2_1035x568.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48f7366a2a05259a132a1d4ce94b4086b287fa9c_2_1380x758.png 2x" data-dominant-color="313132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">2560×1408 196 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used the NMT tutorial of Pytorch.<br>
When I am training a larger dataset, this error happened.  Can anyone help me figure this out?</p>
Conditional activation function causes weird behaviour

username: A_Bdeir
cooked: <p>I have a scaled_attention function as follows:</p>
<pre><code class="lang-auto">def scaled_attention(query, key, value, mask=None, attention_type=0):

    qk = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(query.shape[-1])
    if mask is not None: qk = qk.masked_fill(mask == 1, -1e9)

#section 1 for Pytorch forum 

    #qk = F.softmax(qk, dim=-1)
    qk = entmax15(qk, dim=-1)
    #qk = sparsemax(qk, dim=-1)

#section 2 for pytorch forum

    # if attention_type == 0:
    #     qk = F.softmax(qk, dim=-1)
    # else:
    #     if attention_type == 1:
    #         qk = entmax15(qk, dim=-1)
    #     else:
    #         if attention_type == 2:
    #             qk = sparsemax(qk, dim=-1)

    return torch.matmul(qk, value)

</code></pre>
<p>When i manually set the activation function through section 1 (ignore the argument and uncomment desired activation), the model trains as expected. However when I use the if/else statements, the training results become stagnant and the model doesn’t learn anything.</p>
<p>I’ve tried putting code breaks at the activation statements to make sure they aren’t being called but everything appears to run normally.</p>
<p>Is there any specific reason for this behaviour?</p>

username: ptrblck
cooked: <p>If I understand the issue correctly, you are seeing a different training behavior of the model if you compare section 1 vs. section 2 where both sections are running exactly the same operations?<br>
If so, then no this behavior is not expected.<br>
Could you post a minimal, executable code snippet reproducing this behavior, please?</p>
Weird unexpected performance variability

username: rmd
cooked: <p>Hi, so I’ve been working on a model that uses a few different components and when benchmarking found some bizarre inconsistencies with the performance of the various models.</p>
<p>All models are just basic ConvNets with BatchNorm and ReLU layers.</p>
<p>Specs:</p>
<ul>
<li>RTX 3090</li>
<li>CUDA 11.4</li>
<li>PyTorch 1.10.0</li>
<li>cuDNN 8.2.0</li>
</ul>
<p>The initial setup looks somewhat like this:</p>
<pre><code class="lang-auto">model_1 = Model_1().cuda()
model_2 = Model_2().cuda()

x = torch.rand(1, 5, 3, 416, 416, device='cuda')
</code></pre>
<p>Now, the weird part is the performance issue that I experience with different combinations of the models.</p>
<p><strong>Example 1:</strong></p>
<pre><code class="lang-python">for _ in range(100):
    model_2(model_1(x))

for _ in range(100):
    model_2(x)
# &gt;&gt; Second loop takes 0.053 seconds
</code></pre>
<p><strong>Example 2:</strong></p>
<pre><code class="lang-python">for _ in range(100):
    model_2(x)
# &gt;&gt; This loop takes 0.47 seconds
</code></pre>
<p><strong>Example 3:</strong></p>
<pre><code class="lang-python">for _ in range(100):
    model_2(x)

for _ in range(100):
    model_2(x)
# &gt;&gt; Second loop takes 2.5 seconds
</code></pre>
<p><strong>Example 4:</strong></p>
<pre><code class="lang-python">for _ in range(100):
    model_1(x)
# &gt;&gt; This loop takes 1.72 seconds

for _ in range(100):
   model_2(x)
# &gt;&gt; This loop takes 0.01 seconds

for _ in range(100):
    model_1(x)
    model_2(x)
# &gt;&gt; This loop takes 8 seconds
</code></pre>
<p><strong>Example 5:</strong></p>
<pre><code class="lang-python">for _ in range(100):
    model_1(x)
    model_2(x)
# &gt;&gt; This loop takes 5 seconds
</code></pre>
<p>I have no idea what is going on. Why am I experiencing an ~1.6x reduction in performance between <strong>Example 4</strong> and <strong>Example 5</strong> by just removing the two initial loops.</p>
<p>Why does <em>model_2</em> take ~0.47 seconds to loop 100 times alone (<strong>Example 2</strong>) but when I add a loop before with <code>model_2(model_1(x))</code> (<strong>Example 1</strong>) it takes ~0.05 seconds to loop 100 times alone. But then adding a loop with <code>model_2(x)</code> (<strong>Example 3</strong>) reduces the performance by ~5x.</p>
<p>What is going on here? How do I mitigate the issue?</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously so to you would need to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timers.<br>
Otherwise you would profile the kernel launches, will see accumulated times in blocking operations etc., which seems to be the case here.</p>
Torch reshape order

username: tom-programming
cooked: <p>Hello,<br>
When you do <code>torch.reshape()</code> or <code>.view()</code> and, say, you flatten a matrix into an array, is there a guaranteed order the elements will be in?</p>
<p>In numpy you have an <code>order</code> argument that can specify a C/F/A order?</p>
<p>I am looking for something that preserves compatibility with numpy</p>
<p>Thanks in advance</p>

username: tom
cooked: <p>CPU tensors are compatible with numpy, you can use .numpy() to get a numpy array view on the storage.<br>
You can inspect the memory layout using <code>.stride()</code>. C means highest stride first, F means lowest stride first.<br>
For example</p>
<pre><code class="lang-auto">c = torch.randn(5,5)
print (c.stride())
</code></pre>
<p>returns <code>(5,1)</code> for me, so it would be in C order. Similarly, <code>.continguous()</code> makes C order.<br>
A sightly convoluted way to get to F order would then be</p>
<pre><code class="lang-auto">cp = c.permute(list(numpy.argsort(c.stride()))).contiguous()
d = cp.permute(list(numpy.argsort(cp.stride())))
</code></pre>
<p>Now <code>d.stride()</code> will be <code>(1,5)</code> and <code>(c==d).all().item()</code> will be one.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Jaideep_Valani
cooked: <p><a class="mention" href="/u/tom">@tom</a><br>
torch.Size([4, 272, 352])  my tensor shape is this…  0,1 is one row in actual image , 2,3 is second row. But when i do reshape to get whole image , i get incorrect results compared to looping that gives correct image.</p>
<p>test_prd.rehape(520,704). <span class="hashtag">#incorrect</span> results compared  row/column loop</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/reshape-selected-axis-with-torch-reshape/102244/9">here</a>.</p>
Equal method in Pytorch

username: Rami_Ismael
cooked: <p>I wonder whether a quick method that checks if the same model will have the same weight.</p>

username: ptrblck
cooked: <p>You could iterate the parameters of both models and compare them:</p>
<pre><code class="lang-python">for a, b in zip(modelA.parameters(), modelB.parameters()):
    if not (a == b).all():
        print('False')
        break
</code></pre>
<p>Of course you can add more checks (e.g. for the shape etc.).<br>
Also, the loop assumes the parameters are initialized in the same order, but I assume that’s also a requirement for your “equal” method.</p>
Will PyTorch do kernel fusion automatically?

username: 111550
cooked: <p>Consider the following code:</p>
<pre><code class="lang-auto">import torch


# Model definition
linear1 = torch.nn.Linear(1024,1024, bias=False).cuda()
print(torch.cuda.memory_allocated())  # memory = 4194304 (linear1 parameters)
linear2 = torch.nn.Linear(1024, 1, bias=False).cuda()
print(torch.cuda.memory_allocated()) # memory + 4096 (linear2 parameters)

# Input tensor
inputs = torch.tensor([[1.0]*1024]*1024).cuda() # shape = (1024,1024)
print(torch.cuda.memory_allocated()) # momery + 4194304 (input tensor)

# Forward calculation
out = sum(linear2(linear1(inputs))) # shape = (1)
print(torch.cuda.memory_allocated()) # memory + 4194304 (linear1 output) + 512 (out, PyTorch allocates 512B memory even to a 4-byte tensor)
</code></pre>
<p>My question is, where is the GPU memory allocated for linear2 output, which should be 4096 bytes?</p>
<p>Theoretically, this can be fused with the subsequent <code>sum</code> operation (because d<code>loss</code>/d<code>out</code> = d<code>loss</code>/d<code>linear2_out</code>, which means <code>linear2_out</code> is not needed for backward calculation), but is this what actually happened? If so, is there any docs about this? I can only find kernel fusion in JIT docs, but I’m not using JIT here.</p>

username: ptrblck
cooked: <p>I don’t think this operation is fused, as you’ve already mentioned that you are not scripting the model.<br>
<code>torch.cuda.memory_allocated()</code> will return the currently allocated memory, not its peak (use <code>torch.cuda.max_memory_allocated()</code> for is or <code>torch.cuda.memory_summary()</code>).<br>
Based on your description, I would guess that <code>linear2_out</code> is already freed as it’s not needed for the backward calculation and will thus not show up in the next <code>memory_allocated()</code> print statement.</p>

username: 111550
cooked: <p>Thanks for your reply! Yes, this is more reasonable. PyTorch is a dynamic computation graph framework, so it’s possible to free memory when it is not needed.</p>
Pytorch Profiler Kineto Error

username: CHOONG_HONG_LEE
cooked: <p>Hello there. I am exploring the Pytorch Profiler with Tensorboard. I tried to run this tutorial on Jetson AGX Xavier and when I try to run it prompts "Assertion Error: Requested Kineto profiling but Kineto is not available, make sure Pytorch is built with USE_KINETO = 1. I am wondering if I am getting this error because of the platform that I use?</p>

username: ptrblck
cooked: <p>Your current PyTorch build doesn’t seem to come with the kineto profiler, so you could rebuild it with <code>USE_KINETO=1</code> and make sure the install log shows that it’s indeed installed.</p>
Import error - "module functions cannot set METH_CLASS or METH_STATIC"

username: adamdayan
cooked: <p>Hi!</p>
<p>When I try to import torch, I get the error “module functions cannot set METH_CLASS or METH_STATIC”, the full error is below.</p>
<p>This seems to be an issue a few people encountered last year (<a href="https://discuss.pytorch.org/t/valueerror-and-importerror-occurred-when-import-torch/5818/2" class="inline-onebox">ValueError and ImportError occurred when import torch</a> ,   <a href="https://github.com/pytorch/pytorch/issues/2731" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2731</a>) and the suggested solution involved reinstalling numpy with some install flags set (although I don’t understand how this fixed the problem). I’ve tried this solution to no avail. I have the latest stable builds of torch and numpy and I’m running python 3.6. Would be very grateful for some assistance, thanks!</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-17-6c0c4fe4210e&gt; in &lt;module&gt;
      1 import numpy as np
----&gt; 2 import torch
      3 import pandas as pd
      4 import matplotlib.pyplot as plt
      5 from sklearn.preprocessing import OneHotEncoder

~/Dev/indiv_project/indiv_project/lib/python3.6/site-packages/torch/__init__.py in &lt;module&gt;
     77 del _dl_flags
     78 
---&gt; 79 from torch._C import *
     80 
     81 __all__ += [name for name in dir(_C)

ValueError: module functions cannot set METH_CLASS or METH_STATIC
</code></pre>

username: Amirhossin_Rahimi
cooked: <p>I have the same problem on google colab</p>

username: ptrblck
cooked: <p>Did you try to reinstall <code>numpy</code> or check for files in your current working directory using common names (e.g. <code>random.py</code>)?</p>
What are the alternatives for torch.save, torch.Tensor in tensorflow?

username: Jared_shad
cooked: <p>Does anyone know alternative for <code>torch.save, torch.Tensor([])</code> in tensorflow? I assume alternative for <code>torch.save()</code> could be <code>tensorflow.saved_models.save()</code> and alternative for <code>torch.Tensor([])</code> could be <code>tensorflow.Tensor(numpy.empty())</code>, but I am not really sure. Can anyone clear out my doubt?</p>
<p>especially, I am not relly sure what would be correct function in tensorflow for<br>
<code> myVars = torch.Tensor([]).to(self.device)</code></p>

username: ptrblck
cooked: <p>I would recommend to post TF-specific question in their discussion board, as you’ll most likely find the TF experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
BatchNorm-like module that divides batch data by std but not subtract by mean

username: zfzhang
cooked: <p>Hi,<br>
Is there a Pytorch function/module to only divide a batch data by <strong>std</strong>, but not subtract by <strong>mean</strong>?<br>
Also, what is this operation called? <em>Standardization</em>?<br>
Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure if you would like to apply this normalization of each sample or somewhere inside the model.<br>
In the former case, use <code>torchvision.transforms.Normalize</code> and set the <code>mean</code> to zeros.<br>
In the latter case, just divide by <code>torch.std(x)</code> (and specify the dimensions if needed).</p>

username: zfzhang
cooked: <p>Hi, thanks for the prompt reply!<br>
I’m using this to replace <code>BatchNorm2D</code>, so would need a running std estimation. Your mentioned approaches don’t quite fit.  I’ve edited the title accordingly.<br>
I can adapt your manual implementation given <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py" rel="noopener nofollow ugc">here</a>, but there’s a performance (compute/memory) degradation compared to native BatchNorm2D.</p>
Convolution Kernel and input tensor is int(though it is float64 type ), why the convolution output is a float number?

username: crissallan
cooked: <pre><code class="lang-python">class ConvNet(torch.nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.cv1 = torch.nn.Conv2d(1, 2, (3, 3), padding=(0, 0))
        self.init_edge_detect_weight()
    
    def forward(self, x):
        return self.cv1(x)
    
    def init_edge_detect_weight(self):
        sob_kernel1 = (torch.tensor([[-1, 0, 1],
                                    [-2, 0, 2],
                                    [-1, 0, 1]], dtype=torch.float64)).unsqueeze(0).unsqueeze(0)
        
        sob_kernel2 = torch.tensor([[1, 2, 1],
                                    [0, 0, 0],
                                    [-1, -2, 1]], dtype=torch.float64).unsqueeze(0).unsqueeze(0)
        
        self.cv1.weight.data = torch.cat([sob_kernel1, sob_kernel2], dim=0).requires_grad_(False)

net = ConvNet()

raster_ts = torch.ones(1, 1, 128, 128, dtype=torch.float64)

print("unique value weight: ", torch.unique(net.cv1.weight.data))
print("unique value input: ", torch.unique(raster_ts))

out_ts = net(raster_ts)

print(out_ts.shape)
print("unique value output: ", torch.unique(out_ts))
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt;

unique value weight:  tensor([-2., -1.,  0.,  1.,  2.], dtype=torch.float64)
unique value input:  tensor([1.], dtype=torch.float64)
torch.Size([1, 2, 126, 126])
unique value output:  tensor([-0.0541,  2.2535], dtype=torch.float64, grad_fn=&lt;Unique2Backward&gt;)
</code></pre>
<p>Is this caused by the precision of the float number?</p>
<p>Because I used 2 sobel kernel to convolve a image with all values equals to 1.0, there should not be any edge in the image, so I expected the ouput feature map should with all values equals to 0</p>

username: ptrblck
cooked: <p>Your conv layer has also a <code>bias</code>, which you would see in the output.<br>
Use <code>self.cv1 = torch.nn.Conv2d(1, 2, (3, 3), padding=(0, 0), bias=False)</code> and you would get closer to the expected results.<br>
Once this is  solved, your output would still not contain all zeros, as you have a typo here:</p>
<pre><code class="lang-python">        sob_kernel2 = torch.tensor([[1, 2, 1],
                                    [0, 0, 0],
                                    [-1, -2, 1]], dtype=torch.float64).unsqueeze(0).unsqueeze(0)
</code></pre>
<p>and would need to fix the sign of the last element.</p>
URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

username: 111555
cooked: <blockquote>
<p>class CardiacDetectionModel(pl.LightningModule):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.model = torchvision.models.resnet18(pretrained=True)
    self.model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    self.model.fc = torch.nn.Linear(in_features=512 ,out_features=4)
    
    self.optimizer = torch.optim.Adam(self.model.parameters(), lr=1e-4)
    self.loss_fn = torch.nn.MSELoss()
    
def forward(self, data):
    return self.model(data)

def training_step(self, batch, batch_idx):
    x_ray, label = batch
    label = label.float()
    pred = self(x_ray)
    loss = self.loss_fn(pred, label)
    
    self.log("Train Loss", loss)
    
    if batch_idx % 50 == 0:
        self.log_images(x_ray.cpu(), pred.cpu(), label.cpu(), "Train")
    return loss

def validation_step(self, batch, batch_idx):
    x_ray, label = batch
    label = label.float()
    pred = self(x_ray)
    loss = self.loss_fn(pred, label)
    
    self.log("Val Loss", loss)
    
    if batch_idx % 50 == 0:
        self.log_images(x_ray.cpu(), pred.cpu(), label.cpu(), "Val")
    return loss

def log_images(self, x_ray, pred, label, name):
    results = []
    
    for i in range(4):
        coords_labels = label[i]
        coords_pred = pred[i]
        
        img = ((x_ray[i] * 0.252)+0.494).numpy()[0]
        
        x0, y0 = coords_labels[0].int().item(), coords_labels[1].int().item()
        x1, y1 = coords_labels[2].int().item(), coords_labels[3].int().item()
        img = cv2.rectangle(img, (x0, y0), (x1, y1), (0, 0, 0), 2)
        
        x0, y0 = coords_pred[0].int().item(), coords_pred[1].int().item()
        x1, y1 = coords_pred[2].int().item(), coords_pred[3].int().item()
        img = cv2.rectangle(img, (x0, y0), (x1, y1), (1, 1, 1), 2)
        
        results.append(torch.tensor(img).unsqueeze(0))
    
    grid = torchvision.utils.make_grid(results, 2)
    self.logger.experiment.add_image(name, grid, self.global_step)
    
def configure_optimizers(self):
    #注意してください ここでは必ずリストを返す必要があります（オプティマイザーを1つにまとめるだけです :)

    return [self.optimizer]
</code></pre>
<h1>モデルオブジェクトの作成</h1>
<p>model = CardiacDetectionModel() # モデルをインスタンシエートする</p>
</blockquote>
<p>When instantiating the model, if I use google colab, there is no error, but if I use kaggle’s kernel, there is an error.</p>

username: ptrblck
cooked: <p>It seems as if the Kaggle notebook has trouble downloading the pretrained <code>state_dict</code>.<br>
Could you try to manually download it via:</p>
<pre><code class="lang-auto">wget https://download.pytorch.org/models/resnet18-f37072fd.pth
</code></pre>
<p>and load it afterwards via <code>model.load_state_dict(PATH)</code>?</p>
How to clip grad norm grads from torch.autograd.grad

username: zilong
cooked: <p><code>grads = torch.autograd.grad(loss, self.model.parameters(), create_graph=False)</code></p>
<p>Is there a function like <code>torch.nn.utils.clip_grad_norm_</code> for clipping the gradients in this case?</p>

username: ptrblck
cooked: <p>You could reuse the internal implementation of <code>clip_grad_norm_</code> found <a href="https://github.com/pytorch/pytorch/blob/6ae34ea6f8b65d2ad36e417c3d525507901cfd7f/torch/nn/utils/clip_grad.py#L9-L56">here</a>.<br>
E.g. this should work:</p>
<pre><code class="lang-python"># setup
x = torch.randn(2, 3, 224, 224)
model = models.resnet18().eval()
out = model(x)
loss = (out**2).mean()
loss.backward()

# create reference
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

grad_ref = [p.grad.detach().clone() for p in model.parameters()]

# autgrad.grad approach
model.zero_grad()
out = model(x)
loss = (out**2).mean()

grads = torch.autograd.grad(loss, model.parameters(), create_graph=False)

device = grads[0].device
norm_type = 2.0
max_norm = 1.0
total_norm = torch.norm(torch.stack([torch.norm(grad.detach(), norm_type).to(device) for grad in grads]), norm_type)
clip_coef = max_norm / (total_norm + 1e-6)
clip_coef_clamped = torch.clamp(clip_coef, max=1.0)
for grad in grads:
    grad.detach().mul_(clip_coef_clamped.to(grad.device))

# compare
assert len(grad_ref) == len(grads)
for g_ref, g in zip(grad_ref, grads):
    print((g_ref - g).abs().max())
</code></pre>
Errors removing __init__.py when using Arcface?

username: karap
cooked: <p>Hello,</p>
<p>I have a test.py file that uses “class ArcMarginModel(torch.nn.Module)” from an insightface_models.py in a folder named “models” in my project. This is imported in an <strong>init</strong>.py file as follows:</p>
<p>from .insightface_models import ArcMarginModel</p>
<p>When I remove the <strong>init</strong>.py I get the following error (even if I move everything from insightface_models.py into my test.py):</p>
<p>Message=Can’t get attribute ‘ArcMarginModel’ on &lt;module ‘models’ (namespace)&gt;<br>
Source=B:\test_project\models\test.py</p>
<p>My code is as follows:</p>
<pre><code>    arcface_checkpoint = torch.load(arcface_model, map_location=device)
    arcface = arcface_checkpoint['model'].module
    arcface = arcface.to(device)
    self.arcface.eval()
</code></pre>
<p>How can I make this work without the <strong>init</strong>.py? Why am I getting this error?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I guess you are trying to directly store and load the model via:</p>
<pre data-code-wrap="python"><code class="lang-python">model = ArcMarginModel()
torch.save(model, ...)
model = torch.load(path)
</code></pre>
<p>As described in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">this tutorial</a> you would have to make sure to restore the exact file structure and thus using the <code>state_dict</code> instead is the recommended way:</p>
<blockquote>
<p>This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s <a href="https://docs.python.org/3/library/pickle.html">pickle</a> module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.</p>
</blockquote>
Difference between torch.nn.CosineSimilarity and torch.nn.functional.cosine_similarity?

username: cramraj8
cooked: <p>What is the difference between cosine similarity functions <strong>torch.nn.CosineSimilarity</strong> and <strong>torch.nn.functional.cosine_similarity</strong> ? Where can we use and where we shouldn’t ? If we are using it inside a model with back-propagation flow, which one can we use ?</p>

username: KFrank
cooked: <p>Hi Ramraj!</p>
<aside class="quote no-group" data-username="cramraj8" data-post="1" data-topic="138036" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cramraj8/48/24970_2.png" class="avatar"> cramraj8:</div>
<blockquote>
<p>What is the difference between cosine similarity functions <strong>torch.nn.CosineSimilarity</strong> and <strong>torch.nn.functional.cosine_similarity</strong> ?</p>
</blockquote>
</aside>
<p>The two are effectively the same and they can be used essentially<br>
interchangeably.  In particular, they both support backpropagation<br>
in the same way.</p>
<p><code>CosineSimilarity</code> is the class / function-object version (and almost<br>
certainly merely wraps a call to <code>cosine_similarity()</code>), while<br>
<code>cosine_similarity()</code> is the function version.</p>
<p>It’s primarily a matter of style which you choose to use.  However,<br>
<code>CosineSimilarity</code> is a <code>Module</code>, so you could, for example, place<br>
it in a <code>Sequential</code>, if that were to make sense for your use case.</p>
<p>Best.</p>
<p>K. Frank</p>
How could we use torchrun to start multi node training?

username: coincheung
cooked: <p>Hi,</p>
<p>Firstly, I set my code as <a href="https://pytorch.org/docs/stable/elastic/run.html" rel="noopener nofollow ugc">link</a>. And I can use <code>torchrun --nproc_per_node=8 train.py</code> to train on single node. However, if I want to use multi-node, I run the following command for 4 times on 4 nodes separately:</p>
<pre><code class="lang-auto">IP=10.128.11.6
PORT=33221
NNODES=4
NRANK=0
EXE=train.py
JOBID="01"

ENDPORT=${IP}:${PORT}
echo ${ENDPORT}

# python -m torch.distributed.launch --nproc_per_node=8 --master_addr=${IP} --nnodes=${NNODES} --node_rank ${NRANK} ${EXE}

# this work on single node, which shows that training script is not wrong
# torchrun \
#     --nproc_per_node=8 \
#     --rdzv_id=$JOB_ID \
#     --rdzv_backend=c10d \
#     --max_restarts=1 \
#     ${EXE}

# run this on 4 nodes separately, each set different node_rank
torchrun \
    --nnodes=$NNODES \
    --node_rank=$NRANK \
    --nproc_per_node=8 \
    --rdzv_id=$JOB_ID \
    --rdzv_backend=c10d \
    --rdzv_endpoint=${ENDPORT} \
    --max_restarts=1 \
    ${EXE}
</code></pre>
<p>The training cannot start. I just set different <code>--node_rank</code> on each node, and other parameters are same. How could I make it work please?</p>
Loss of generator in GAN increasing

username: msqiqi10
cooked: <p>The GAN loss is defined:</p>
<blockquote>
<p>Prob_1 = D(real_data)<br>
Prob_2 = D(G(z))<br>
D_loss = log(prob_1) + (1 - log(prob_2))<br>
G_loss = -log(D(G(z))</p>
</blockquote>
<p>The loss are plotted here:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1" title="GAN_not_converge"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1_2_666x500.png" alt="GAN_not_converge" data-base62-sha1="vJtxaahdB6GqGFMoqhc0p1Q8o5r" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de6766c1b261d9f3bb74dbab2fdfa45abbc180d1.png 2x" data-dominant-color="FCFBF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">GAN_not_converge</span><span class="informations">1280×960 48.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And as expected, the Prob_1 increases and Prob_2 decreases as the training goes, which means D is learning the right thing as expected.<br>
Does this mean that the training of G and D are not balanced? G cannot keep up with D?<br>
So intuitively, I think it would be better if G is trained more than D, but according to <a href="https://github.com/soumith/ganhacks" rel="noopener nofollow ugc">tips for training GAN</a>, it would be a bad practice. (I tried other tips in this article like optimizers, activation functions, batchNorms etc.)</p>
<p>Would it be a good idea to train G more than D?<br>
Or maybe there are some tips I should try?</p>

username: Rituraj_19_Dutta_100
cooked: <p>GANs are a highly active topic for research. Generally GANs don’t converge well. A typical GAN loss should be something where G loss <code>log(D(G(z))</code>  maximizes and D loss <code>log(D(x))+log(1-D(G(z))</code> minimizes. But that’s not the scenario all the time. Most of the time the discriminator gets fooled easily by the generator. To avoid this:</p>
<ol>
<li>Update the discriminator more often than the generator.</li>
<li>Use WGAN with Gradient Penalty.</li>
</ol>
<p>In some cases the discriminator gets an upperhand. To avoid this:</p>
<ol>
<li>Add dropout and leaky relu in the discriminator layers.</li>
<li>Use label smoothing.</li>
<li>Add noise to both generated and real input before feeding them to the discriminator.</li>
</ol>
<p>**<strong>Note: Don’t expect GAN to converge they may converge and may not. But usually if your generator and discriminator loss converges together then your GAN is good.</strong></p>
[solved] Getting batches of gradients

username: Giuseppe_Castiglione
cooked: <p>I’m trying to figure out how one can compute the gradient for individual samples in a batched fashion.  Specifically, given an input batch, and the score outputs (ex mse for each sample), I want to compute what the gradients are for each item in the batch.  I can think of a simple way to do this, which is just running forward/backward passes on individual items in a for loop, but that seems like it would be slow (since you’re looping in python instead of efficiently doing the forward prop on the whole batch).  Is there a better way to do this?</p>

username: Giuseppe_Castiglione
cooked: <p>Found my own answer!  This function does everything I want:<br>
<a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad</a></p>

username: tor
cooked: <p><a class="mention" href="/u/giuseppe_castiglione">@Giuseppe_Castiglione</a>: could you please elaborate your answer more?<br>
How did you use torch.autograd.grad() to get the gradient of each individual input vector from a batch?<br>
Some code snippet will be appreciated.</p>
<p>One relevant discussion is</p><aside class="quote quote-modified" data-post="1" data-topic="18587">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/71e660/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/efficient-computation-of-per-sample-examples/18587">Efficient computation of per-sample examples</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Additional discussion of <a href="https://github.com/pytorch/pytorch/issues/7786" rel="noopener nofollow ugc">[feature request] Simple and Efficient way to get gradients of each element of a sum</a> 
<a class="mention" href="/u/tom">@tom</a> I hope I interpreted your last post correctly by posting here. 

The code to reproduce the numbers I see is available <a href="https://github.com/fKunstner/fast-individual-gradients-with-autodiff" rel="noopener nofollow ugc">here</a>. 
The main.py file runs different ways of computing individual gradients and checks/times them. 
Your recommendation seems to be along the line of <a href="https://arxiv.org/abs/1510.01799" rel="noopener nofollow ugc">Goodfellow’s derivation</a>, implemented as goodfellow, timing output under goodf. 
I don’t think it is a bug per-se, …
  </blockquote>
</aside>


username: ecolss
cooked: <p>Does this one really solve your problem?</p>

username: DhruvaBansal00
cooked: <p>I was able to use this function for the same task by summing over the “output” tensor and passing it into the function along with the input tensor as-is. The first tensor in the result tuple contains the same number of batches as the input tensor.</p>
When working with nn.TripletMarginLoss - do we need to configure the optimizer to search for max values (instead of min values)?

username: laro
cooked: <p>According to my understanding we are searching the minimum values (wights) for loss function. i.e:</p>
<pre><code> loss = Loss(...) # calc loss value according to loss function metric
 loss.backward()  # calc the gradient
 optim.step()     # according to the gradient descent algorithm - go one step to the minimum
</code></pre>
<p>When dealing with <a href="https://pytorch.org/docs/stable/generated/torch.nn.TripletMarginLoss.html" rel="noopener nofollow ugc">TRIPLETMARGINLOSS</a> (<code>nn.TripletMarginLoss</code>) we are searching  the wights which will maximize this loss function.</p>
<p>So, How can we configure the optimizer (<code>optim.Adam</code> or <code>optim.SGD</code>) to find the maximum value and not the minimum value for the loss function ?</p>

username: InnovArul
cooked: <p>If I understand correctly, you want to optimize the network to maximize a loss function?</p>
<p>If so, you can just <code>minimize</code> the <code>negative of that loss</code> function, which in turn will maximize the loss.<br>
i.e.,</p>
<pre><code class="lang-auto">loss = -Loss(...)
loss.backward()
optim.step() 
</code></pre>
Python builtin <built-in method apply of FunctionMeta object at 0x561db6e84610> is currently not supported in Torchscript:

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I just wanted to ask a brief follow-up question on <code>torch.jit.script</code> being applied to <code>torch.custom.autograd</code> functions. This has been asked <a href="https://discuss.pytorch.org/t/runtimeerror-python-builtin-built-in-method-apply-of-functionmeta-object-at-0x55dad1b31680-is-currently-not-supported-in-torchscript/72089">before</a>, and the answer’s been it’s not currently supported.</p>
<p>Is there any update on this? I did see this issue here, <a href="https://github.com/pytorch/pytorch/issues/22329" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/22329</a>.</p>
<p>I did find this artcile <a href="https://lernapparat.de/pytorch-traceable-differentiable/" rel="noopener nofollow ugc">here</a> as a potential solution. Is this the only way to script custom function within PyTorch as of 1.10?</p>
<p>Thank you!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.autograd import Function

class Square(Function):
  @staticmethod
  def forward(ctx, x):
    ctx.save_for_backward(x)
    return x ** 2
    
  @staticmethod
  def backward(ctx, grad_y):
    x, = ctx.saved_tensors
    return SquareBackward.apply(x, grad_y)
    
class SquareBackward(Function):
  @staticmethod
  def forward(ctx, x, grad_y):
    ctx.save_for_backward(x, grad_y)
    return grad_y * 2 * x

  @staticmethod
  def backward(ctx, grad_grad_x):
    x, grad_y = ctx.saved_tensors
    return 2 * grad_y * grad_grad_x, 2 * x * grad_grad_x
    

class CustomFunc(nn.Module):

  def __init__(self):
    super(CustomFunc, self).__init__()
    
  def forward(self, x):
    return Square.apply(x)
    
func = CustomFunc()

x = torch.randn(10)
y=func(x)

print("x: ",x)
print("y: ",y)
print("x^2: ",x.pow(2)) #check it all works...

jit_func = torch.jit.script(func) #jit function, fails here
</code></pre>
LSTM Model Predicting Horizontal Straight Line Constantly

username: Ryan_2k
cooked: <p>Hello,</p>
<p>I used this tutorial when developing my LSTM model to predict Bitcoin prices and changed it with using my data: <a href="https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/" rel="noopener nofollow ugc">https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/</a></p>
<p>Whenever I use my test data for it to predict it just predicts a horizontal straight line for the whole thing like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0b41972f430e3416129940f81d0953a79901fca.png" alt="download (1)" data-base62-sha1="pdbTyfklE2aSBvDyOjjdqHmgCzU" width="401" height="264"></p>
<p>I don’t know why this is happening because when I trained the model it had a final loss of 0.0000000075. I’m not sure if this is an error in my code or a problem in training my model.</p>
<p>I would appreciate it a lot if someone could help me with this</p>
Ask for help! I want to know a best practice about torch.Tensor

username: redocxz
cooked: <p>I have a Tensor A(256,512,1024) and another index Tensor Index(256,960,2). For example, [1,2] in Index means I want to retrive the 1st and 2nd tensor of A in dimension 1. And finally I can get a Tensor Res(256,960,2,1024).</p>
<p>I can implement only by iteratate all elements. Please help me if you have any good practice.</p>
<p>to simplify my model:</p>
<pre><code class="lang-python">import torch

if __name__ == '__main__':
    src = torch.FloatTensor([
        [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    ])
    idx = torch.LongTensor([
        [[0, 1], [1, 2], [0, 2], [1, 0], [2, 1]]
    ])
    res = torch.zeros((1, 5, 2, 4))
    # I want (1,5,2,4)
    # [ [[1,2,3,4],[5,6,7,8]],[[5,6,7,8],[9,10,11,12]],[...],[...],[....] ]

</code></pre>

username: googlebot
cooked: <p>I believe that is done by:</p>
<p>A.gather(-2, I.view(256,960*2,1).expand(-1,-1,1024)).view(256,960,2,1024)</p>
Default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object

username: NullPointer
cooked: <p>When I run my training loop, I get the following error while trying to iterate over my test DataLoader. However, it does not occur when using my train DataLoader (they are 2 different files).</p>
<p><code>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object</code><br>
This is how I am iterating over the DataLoader</p>
<pre><code class="lang-auto">    for batch in data_loader:
        optimizer.zero_grad()
</code></pre>

username: NullPointer
cooked: <p>I had mixed datatypes in my csv</p>
Why does it think that it's a non-leaf tensor?

username: blu_head
cooked: <p>Hi everyone,<br>
I’ve been trying to optimize images from a specific class and I created the tensor training_data_new myself (serves as a training dataloader). However, it still gives me the error “can’t optimize a non-leaf Tensor” on the line where I create optimizer2. I have tried casting it to float and just multiplying the image by 1.0. Still won’t budge. Does anyone have any ideas?</p>
<p>Here’s a snippet:</p>
<pre><code class="lang-auto">it = 0
for batch, (image, label) in enumerate(zip(training_data_new, training_data_new_label)):
    if label == 0:
        image.requires_grad_(True).float()
        image = image * 1.0
        for k in range(0, 5):
            optimizer2 = torch.optim.SGD([image], lr=0.1)
            loss2 = (((image) - (item_to_transform)) ** 2.0).sum()
            optimizer2.zero_grad()
            loss2.backward(retain_graph=True)
            optimizer2.step()
        training_data_new[it] = image
    it += 1
</code></pre>
<p>I’m a beginner but I’m trying to do my best.<br>
Thank you in advance!</p>

username: KFrank
cooked: <p>Hi Blu!</p>
<aside class="quote no-group" data-username="blu_head" data-post="1" data-topic="137911" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/3bc359/48.png" class="avatar"> blu_head:</div>
<blockquote>
<pre><code class="lang-auto">        image.requires_grad_(True).float()
        image = image * 1.0
</code></pre>
</blockquote>
</aside>
<p>The seemingly innocuous <code>image = image * 1.0</code> creates a new<br>
image (that is then assigned to the python reference variable <code>image</code>)<br>
that, being the result of a tensor computation, is no longer a leaf<br>
variable.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; image = torch.zeros (3, 5, requires_grad = True)
&gt;&gt;&gt; image.is_leaf
True
&gt;&gt;&gt; image = image * 1.0
&gt;&gt;&gt; image.is_leaf
False
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: blu_head
cooked: <p>thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> After reading more about leaf tensors I understood that, so I wasn’t sure why I was told to multiply it by 1.0 by my supervisor as a solution (I had the same error before writing the line where I multiply with 1.0). But I have resolved it.</p>