akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model trains on 1080Ti but quickly diverges on 2080Ti

username: Jacob_Lambert
cooked: <p>Dear PyTorch community,</p>
<p>I’m facing a rather weird problem I’m not sure how to debug anymore - my only guess is a hardware related issue.</p>
<p>I’ve been successfully training a not-so-complex CNN on a 1080Ti (<a href="https://github.com/maudzung/Complex-YOLOv4-Pytorch" rel="noopener nofollow ugc">complex-yolo</a>), trying to improve it.<br>
On a new PC with a 2080Ti, I got crashes in training. After a lot of debugging, I noticed my model suddenly blows up (outputing Infs and NaNs) early in training for no apparent reason - there were no issues with the input data, or targets. Reverting code to states I <em>know</em> trained properly, lowering learning rate, simplifying things, nothing helped. I ran with a fixed seed and deterministic mode to debug the issue.</p>
<p>In desperation I swapped out the 2080Ti for the old 1080Ti, ran the code without any other changes and could successfully train. I’ve had no other issues with the 2080Ti. I can do inference on trained models without issues, I can resume training.</p>
<p>I’m not even sure what info is helpful for debugging this, or where to start. I used the exact same system with both cards:</p>
<ul>
<li>Nvidia Driver Version: 515.43.04</li>
<li>System CUDA 11.4,</li>
<li>Python Environment:
<ul>
<li>pytorch=1.5.0=py3.6_cuda10.2.89_cudnn7.6.5_0</li>
<li>cudatoolkit=10.2.89=hfd86e86_1</li>
</ul>
</li>
</ul>
<p>What can I look into to figure out if there’s something wrong with my hardware - or if there’s some code compatibility issue that leads divergence in training?</p>
<p>Thank you.</p>

username: eqy
cooked: <p>This certainly seems unexpected—I would check if this is still visible on a more current version of PyTorch (e.g., &gt;= 1.11) on the off chance that it’s a bug that has been already fixed. If it’s still visible I would see if there’s a specific layer or part of the model where the outputs diverge substantially.</p>
Set requires_grad according to a binary mask

username: Megh_Bhalerao
cooked: <p>Hi all,<br>
I have a <code>conv2d</code> layer parameters of shape say <code>(x,y,z,w)</code>, and I have a binary mask of the same shape <code>(x,y,z,w)</code>, I want to set the <code>requires_grad</code> attribute to every <strong>single</strong> weight parameter in this <code>conv2d</code> layer, or to be more clear I want to set <code>x*y*z*w</code> <code>requires_grad</code> values, i.e. for every weight value inside this conv filter acoording to the binary mask given. At places which have 1 in the binary mask, the <code>requires_grad = True</code> and vice versa. Is there a neat way of doing this other than looping over the entire filter?<br>
Thanks for your help!<br>
– Megh</p>

username: KFrank
cooked: <p>Hi Megh!</p>
<aside class="quote no-group" data-username="Megh_Bhalerao" data-post="1" data-topic="153040" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/megh_bhalerao/48/13547_2.png" class="avatar"> Megh_Bhalerao:</div>
<blockquote>
<p>to be more clear I want to set <code>x*y*z*w</code> <code>requires_grad</code> values, i.e. for every weight value inside this conv filter acoording to the binary mask given.</p>
</blockquote>
</aside>
<p>There is no way to do what you want – with or without using loops.</p>
<p>The <code>requires_grad</code> property applies to the entire tensor, not to elements<br>
of the tensor individually.</p>
<p>If your use case for this is to “freeze” just certain elements of the <code>Conv2d</code><br>
<code>weight</code>, that is, to only update some of the elements of <code>weight</code> when<br>
training, the safest way is to store a copy of the elements prior to calling<br>
the update, and then restore them after the update.  E.g., something like<br>
this:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    saved_weights = conv_layer.weight.clone()

# run training loop
...
    # inside of training loop
    opt.zero_grad()
    loss.backward()
    opt.step()
    with torch.no_grad():
        conv_layer.weight[mask.logical_not()] = saved_weights[mask.logical_not()]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Forward hook access variables other than inputs and outputs

username: venkatasg
cooked: <p>I’m using <code>register_forward_hook</code> to perform a controlled intervention on a layer’s outputs of a transformer model. For instance:</p>
<pre><code class="lang-auto">hook = model.roberta.encoder.layer[args.layer].register_forward_hook(lambda model, inputs, outputs: intervention(outputs=outputs))
</code></pre>
<p>The <code>intervention()</code> function performs some operations on the hidden state, and I want to observe changes in model behavior when I perform these changes. However, I would like to make the interventions dynamic - I want to perform it only on certain tokens in the hidden state, whereas I’m doing it to all tokens currently. I have a variable listing which positions need to be intervened separately - how do I access this element within a forward hook if it doesn’t get passed through the forward pass? My current idea is to make a global or static variable that keeps track of what batch I am at currently, and to access these from inside the hook function, but this seems quite hacky. Any suggestions would be appreciated.</p>
How to get thread ID in nn.DataParallel

username: Mahzad_Khoshlessan
cooked: <p>Hi all,</p>
<p>How can I get the thread ID in nn.DataParallel? I am running my code on a single node with 4 GPUs and I am not sure how I can get the value of a tensor for each specific thread?</p>
<p>Any thoughts?<br>
Thanks,</p>

username: AlphaBetaGamma96
cooked: <p>I haven’t used <code>nn.DataParallel</code> so default to someone who knows more, but surely couldn’t you just print <code>.device</code> and that would return one of the following: <code>cuda:0</code>, <code>cuda:1</code>, <code>cuda:2</code>, <code>cuda:3</code>?</p>

username: Mahzad_Khoshlessan
cooked: <p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> Yes that makes sense. I am using this and it shows the device which is basically different for each thread.</p>
DataLoader(shuffle=True) creates mall variance in testloss

username: JanoschMenke
cooked: <p>Hi Guys,</p>
<p>so I noticed that when I repeatedly evaluate the my test set the loss calculation changes slightly after each evaluation. But only the Loss the other metric like AUC remain the same. I also use <code>model.eval()</code></p>
<p>My Loss Function looks roughly like this, where <code>mask</code> is a binary Matrix where 0 indicate positions for which no loss should be computed.  I feel like this function should not be an issue as random batches should still yield the same average loss across the whole epoch.</p>
<pre><code class="lang-auto">self.criterion = nn.BCEWithLogitsLoss(reduction="none")
def missingBCEWithLogitsLoss(self, logits, target, mask):
    unreduced_loss = self.criterion(logits, target) * mask
    reduced_loss = torch.div(torch.sum(unreduced_loss), torch.sum(mask))
    return reduced_loss
</code></pre>

username: Andrei_Cristea
cooked: <p>Hi Janosch,</p>
<p>How large is the variation you get when computing the loss over the entire test set i.e. is the difference small enough to be due to floating point issues?</p>
<p>Are you definitely iterating over the entire test set?</p>
<p>Perhaps you could share the snippet of code that does the testing, by iterating over the test dataloader / dataset and whatnot.</p>
<p>Best,<br>
Andrei</p>

username: JanoschMenke
cooked: <p>So the average loss sits at around ~0.250<br>
But the variation is +/- 0.005, which is too big to come from floating points:</p>
<p>What I noticed is when I switch from average across only the “not masked” values, this problems is present. When I instead average across all values it does not happen.</p>
<pre><code class="lang-auto">self.criterion = nn.BCEWithLogitsLoss(reduction="none")
def missingBCEWithLogitsLoss(self, logits, target, mask):
    unreduced_loss = self.criterion(logits, target) * mask
    reduced_loss =torch.mean(unreduced_loss)
    return reduced_loss
</code></pre>

username: JanoschMenke
cooked: <p>Yes the issue is the taking the mean across only the number of “non-masked” values.<br>
That can cause the loss to depend items in the batch.</p>
<p>So for each loss per batch we divide the sum of the loss by the number of non-masked tokens:<br>
Here is an example of two batches</p>
<pre><code class="lang-auto">(10/7) + (5/4) 
</code></pre>
<p>The total loss across the tow batches is : <code>10 + 5 = 15</code><br>
The total number of non-masked values is <code>7 + 4 = 11</code><br>
So the true loss should be <code>15 / 11 = 1.36..</code></p>
<p>However, the loss obtained from the batched computation is</p>
<pre><code class="lang-auto">((10/7) + (5/4) ) = 1.339
</code></pre>
<p>Now if I shuffle my batches:</p>
<pre><code class="lang-auto">(6/5) + (9/6) ) = 1.35
</code></pre>
<p>The Numerators still sum to 15 and the  Denominator still sums to 15. But the loss changed again slightly</p>

username: Andrei_Cristea
cooked: <p>Makes sense! If you replace <code>torch.mean(unreduced_loss)</code> with <code>torch.sum(unreduced_loss)</code>, I think that will fix the issue.</p>
Why torch.Tensor.cuda() utilizes GPU?

username: ihchoi12
cooked: <p>Hello!</p>
<p>I’m trying to understand what the torch.Tensor.cuda() is doing in detail.<br>
I thought its to move the tensor from CPU memory to GPU memory, and CPU performs store operations to the GPU memory without involving the GPU (please correct me if its wrong).</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torchvision.models as models
from torch.profiler import profile, record_function, ProfilerActivity

inputs = torch.randn(1000)
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    inputs = inputs.cuda()
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
</code></pre>
<p>This is the code that I’m profiling. It’s simply create a tensor with 1000 float numbers, and move it to the GPU.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27a67cbad2273921c523fb1c84cc95296fda189c.png" data-download-href="https://discuss.pytorch.org/uploads/default/27a67cbad2273921c523fb1c84cc95296fda189c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a67cbad2273921c523fb1c84cc95296fda189c_2_690x144.png" alt="image" data-base62-sha1="5ELjKDlG9whjemMjZgCvq4urlbm" width="690" height="144" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a67cbad2273921c523fb1c84cc95296fda189c_2_690x144.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a67cbad2273921c523fb1c84cc95296fda189c_2_1035x216.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a67cbad2273921c523fb1c84cc95296fda189c_2_1380x288.png 2x" data-dominant-color="353741"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2154×450 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And here is the profile result.</p>
<p>I actually observe some CUDA kernels are running, and GPU utilization is not 0% (different from my expectation). Could anyone let me know what I’m misunderstanding here?</p>
Is inheriting from Dataset necessary for the creation of a custom dataset?

username: kuzand
cooked: <p>I am creating my own dataset class which takes a path to a csv file containing image paths and labels:</p>
<pre><code class="lang-auto">class MyDataset():
    def __init__(self, csv_path):
        ...

    def __len__(self):
        return len(...)

    def __getitem__(self, index):
        ...
        return (img, label)
</code></pre>
<p>As you can see it contains the methods <code>__init__</code>, <code>__getitem__</code> and <code>__len__</code>.</p>
<p>Now, this is the source code for the Pytorch’s Dataset:</p>
<pre><code class="lang-auto">class Dataset(object):
    def __getitem__(self, index):
        raise NotImplementedError

    def __add__(self, other):
        return ConcatDataset([self, other])
</code></pre>
<p>and in docs it says that:</p>
<blockquote>
<p>All datasets that represent a map from keys to data samples should subclass it</p>
</blockquote>
<p>But I don’t see anything special except the <code>__add__</code> method which I think in my case is not needed (otherwise I could write my own).  Is it still necessary to inherit from Dataset after having implemented my own <code>__getitem__</code>  and  <code>__len__</code>, to be able to create a dataloader later on?  What advantage is there from subclassing it?</p>

username: tom
cooked: <aside class="quote no-group" data-username="kuzand" data-post="1" data-topic="71098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kuzand/48/8268_2.png" class="avatar"> kuzand:</div>
<blockquote>
<p>Is it still necessary to inherit from Dataset after having implemented my own <code>__getitem__</code> and <code>__len__</code> , to be able to create a dataloader later on? What advantage is there from subclassing it?</p>
</blockquote>
</aside>
<p>Will it work without subclassing Dataset? Probably in most cases.<br>
Do you get anything from skipping subclassing Dataset? Probably not.<br>
Will it bite you when third party code tests <code>isinstance(ds, Dataset)</code> to see if it got a dataset or if someone use type hinting with your code? Yes.</p>
<p>Best regards</p>
<p>Thomas</p>

username: vedal
cooked: <p>Does subclassing from Dataset give any performance gains compared to skipping subclassing?</p>

username: tom
cooked: <p>The performance difference should be negligible if at all measurable.</p>
<p>Best regards</p>
<p>Thomas</p>

username: vedal
cooked: <p>Thank you for the swift and clear answer <a class="mention" href="/u/tom">@tom</a></p>
How to handle pruning dependencies?

username: SofiaCP
cooked: <p>Hi,</p>
<p>If I were to prune a pre-trained model (like a resnet18 for example) using a pruning option like <code>torch.nn.utils.prune.random_structured()</code>, how can I handle dependencies on those layers such as skip connections and concatenations?</p>
<p>Thank you</p>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad()

username: animesh22
cooked: <p>I am trying to train SRGAN from scratch. I have read solutions for this type of problem, but it would be great if someone could help me debug my code.</p>
<pre><code class="lang-auto">gen_model = Generator().to(device, non_blocking=True)
disc_model  = Discriminator().to(device, non_blocking=True)
opt_gen = optim.Adam(gen_model.parameters(), lr=0.01)
opt_disc = optim.Adam(disc_model.parameters(), lr=0.01)
from torch.nn.modules.loss import BCELoss

def train_model(gen, disc):
  for epoch in range(20):
    run_loss_disc = 0
    run_loss_gen = 0
    for data in train:
      low_res, high_res = data[0].to(device, non_blocking=True, dtype=torch.float).permute(0, 3, 1, 2),data[1].to(device, non_blocking=True, dtype=torch.float).permute(0, 3, 1, 2)
      #--------Discriminator-----------------
     
      gen_image = gen(low_res)
      gen_image = gen_image.detach()
      disc_gen = disc(gen_image)
      disc_real = disc(high_res)
      p=nn.BCEWithLogitsLoss()
      loss_gen = p(disc_real, torch.ones_like(disc_real))
      
      loss_real = p(disc_gen, torch.zeros_like(disc_gen))
      loss_disc = loss_gen + loss_real
      opt_disc.zero_grad()
      loss_disc.backward()
      
      run_loss_disc+=loss_disc
      #---------Generator--------------------
      cont_loss = vgg_loss(high_res, gen_image)
      adv_loss = 1e-3*p(disc_gen, torch.ones_like(disc_gen))
      gen_loss = cont_loss+(10^-3)*adv_loss
      opt_gen.zero_grad()
      gen_loss.backward()
      opt_disc.step()
      opt_gen.step()
      run_loss_gen+=gen_loss
    print("Run Loss Discriminator: %d", run_loss_disc)
    print("Run Loss Generator: %d", run_loss_gen)

train_model(gen_model, disc_model)
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>This line of code:</p>
<pre><code class="lang-python">run_loss_disc+=loss_disc
</code></pre>
<p>looks wrong as you are accumulating the current discriminator loss including the entire computation graph.<br>
If you want to accumulate the loss value only for printing purposes, use <code>run_loss_dict += loss_dict.item()</code> or <code>.detach()</code> the tensor.</p>

username: animesh22
cooked: <p>Hello. Tried this. I still face the same error.</p>
Faster RCNN Box detection per image

username: RLA
cooked: <p>Hi,</p>
<p>For the Faster RCNN model and e.g. MaskRCNN there is a parameter called box_detections_per_image and the default value is 100. When I set it to one however I still get more than one prediction, so what does the parameter actually do?</p>
<p>Is it the number of boxes inside the region proposal?</p>
<p>What should the default be if you are only looking for one object in the image?</p>

username: RLA
cooked: <p>This has now been solved</p>
Torch.cuda.amp.autocast memory leak

username: Pascal_Niville
cooked: <p>We experienced a memory leak issue, when using the autocast functionality.</p>
<p>The below code resulted in the memory leak, but without the autocast it worked as expected.<br>
Replacing the loss.item() with loss.float() solved the issue for us. Removing the memory leak when using the autocast</p>
<pre><code class="lang-auto">epoch_loss = 0
for input, output in iter(dataloader):
            with autocast():
                   output = model(input)
            loss = loss_fn(output, target)
           epoch_loss += loss.item()
</code></pre>
<p>Pytorch version 1.10.0<br>
We spend some days on finding the issue, so hopefully, this will help others solve their problem faster.</p>

username: ptrblck
cooked: <p>Could you provide a minimal, executable code snippet to reproduce the issue, please?</p>
How to plot the loss function

username: Fathima
cooked: <p>Can someone show me how to plot the train and valid loss? what’s the best way to visualize it? perhaps, some explanations for your visualization, please.</p>
<pre><code class="lang-auto">import numpy as np
batch_size = 32
epochs = 3
min_valid_loss = np.inf

for e in range(epochs):
    train_loss = 0.0
    model.train()     # Optional when not using Model Specific layer
    for data, target in train_loader:
         # Transfer Data to GPU if available
        if torch.cuda.is_available():
            data, target = data.cuda(), target.cuda()
            print("targets.data", targets.item())
        
        
         # Clear the gradients
        optimizer.zero_grad()
        
        # Forward Pass
        output = (model(data))
        squeezed_output = torch.squeeze(output)

        
        # Find the Loss
        loss = criterion(squeezed_output.view(-1,1),target.view(-1,1))
        
        # Calculate gradients 
        loss.backward()
        # Update Weights
        optimizer.step()
        # Calculate Loss
        train_loss += loss.item()
    
    valid_loss = 0.0
    model.eval()     # Optional when not using Model Specific layer
    for data, target in valid_loader:
        if torch.cuda.is_available():
            data, target = data.cuda(), target.cuda()
        
        output = model(data)
        squeezed_output = torch.squeeze(output)
        
        loss = criterion(squeezed_output.view(-1,1),target.view(-1,1))
        valid_loss = loss.item() * data.size(0)
        

    print(f'Epoch {e+1} \t\t Training Loss: {train_loss / len(train_loader)} \t\t Validation Loss: {valid_loss / len(valid_loader)}')
   

    if min_valid_loss &gt; valid_loss:
        print(f'Validation Loss Decreased({min_valid_loss:.6f}---&gt;{valid_loss:.6f}) \t Saving The Model')
        min_valid_loss = valid_loss
</code></pre>

username: jayz
cooked: <p>I think you should calculate the mean train and val losses (mean over the batch) for each epoch, append them to a list, and plot the training and validation error curves over the number of epochs. That’s a starting point.</p>

username: Fathima
cooked: <p>Could you share with me a piece of code that I can use to calculate it?</p>
Building Torchvision C++ API from source

username: ntm0110
cooked: <p>I have been trying to build torchvision from source (master) on Windows 10. I cannot get past the cmake step. I see there is a dependency on the Torch package and cmake specifically asks for TorchConfig.cmake or torch-config.cmake. I have gotten this to work in the past by explicitly passing in --DCMAKE_PREFIX_PATH=“C:\Path\To\Torch” but most recently this has not worked for me. Can the prebuilt windows binary for Torch satisfy this cmake requirement? Can the Python package for Torch satisfy it? What is the actual path I need to supply to make this work? Is it looking for the torch.dll or torch.lib or something?</p>

username: peterjc123
cooked: <p>If you want to invoke the cmake script <a href="https://github.com/pytorch/vision/blob/master/CMakeLists.txt" rel="nofollow noopener">here</a>, I think LibTorch is sufficient. Here is what I have done to make it work:</p>
<ol>
<li>Download <a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.4.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.4.0.zip</a> and extract to <code>D:\libtorch</code>
</li>
<li>Clone pybind11 and install to <code>D:\pybind11</code>
</li>
</ol>
<pre><code class="lang-auto">git clone https://github.com/pybind/pybind11.git
cd pybind11
mkdir build
cd build
cmake -DPYBIND11_TEST=OFF -DCMAKE_INSTALL_PREFIX="D:\pybind11" ..
</code></pre>
<ol start="3">
<li>Clone torchvision and build</li>
</ol>
<pre><code class="lang-auto">git clone https://github.com/pytorch/vision.git
cd vision
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="D:\libtorch;D:\pybind11" ..
</code></pre>

username: ntm0110
cooked: <p>Thanks Pu Jiachen. I had to modify that a little for my particular case to work. What ended up working for me was</p>
<p>cmake -DCMAKE_PREFIX_PATH=“D:/path/to/libtorch” -Dpybind11_DIR=“D:/path/to/pybind11/share/cmake/pybind11” … -G “Visual Studio 15 2017 Win64”</p>

username: cv22
cooked: <p>Hi,</p>
<p>have tried to build Torchvision using cmake in windows10.</p>
<p>Torch 1.11, cuda 11.3, cuDNN v8.3.3</p>
<p>able to go through cmake process but after I click on torchvision.sln and using visual studio 2019 to build, the following error appears</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Severity</th>
<th>Code</th>
<th>Description</th>
<th>Project</th>
<th>File</th>
<th>Line</th>
<th>Suppression State</th>
</tr>
</thead>
<tbody>
<tr>
<td>Error</td>
<td>C2664</td>
<td>‘void jpeg_mem_dest(j_compress_ptr,unsigned char **,unsigned long *)’: cannot convert argument 3 from ‘vision::image::JpegSizeType *’ to ‘unsigned long *’</td>
<td>torchvision</td>
<td>C:\vision\torchvision\csrc\io\image\cpu\encode_jpeg.cpp</td>
<td>87</td>
<td></td>
</tr>
</tbody>
</table>
</div><p>I have installed jpeg using<br>
.\vcpkg install --triplet x64-windows libjpeg-turbo</p>
<p>the following files are created<br>
o	Jpeg.lib in C:\vcpkg\installed\x64-windows\lib<br>
o	Jpeg62.dll in C:\vcpkg\installed\x64-windows\bin</p>
<p>possible to help?</p>
Building torchvision c++ API from source in windows 10

username: cv22
cooked: <p><strong>strong text</strong> Hi,</p>
<p>have tried to build Torchvision using cmake in windows10.</p>
<p>Torch 1.11, cuda 11.3, cuDNN 8.3.3</p>
<p>able to go through cmake process but after I click on torchvision.sln and using visual studio 2019 to build, the following error appears</p>
<p>Expand Table</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Severity</th>
<th>Code</th>
<th>Description</th>
<th>Project</th>
<th>File</th>
<th>Line</th>
<th>Suppression State</th>
</tr>
</thead>
<tbody>
<tr>
<td>Error</td>
<td>C2664</td>
<td>‘void jpeg_mem_dest(j_compress_ptr,unsigned char **,unsigned long *)’: cannot convert argument 3 from ‘vision::image::JpegSizeType *’ to ‘unsigned long *’</td>
<td>torchvision</td>
<td>C:\vision\torchvision\csrc\io\image\cpu\encode_jpeg.cpp</td>
<td>87</td>
<td></td>
</tr>
</tbody>
</table>
</div><p>I have installed jpeg using<br>
.\vcpkg install --triplet x64-windows libjpeg-turbo</p>
<p>the following files are created<br>
o Jpeg.lib in C:\vcpkg\installed\x64-windows\lib<br>
o Jpeg62.dll in C:\vcpkg\installed\x64-windows\bin</p>
<p>possible to help?</p>
How to use stacked blstm with Luong attention

username: Rafi
cooked: <p>I try to implement a seq2seq with attention model. I want to use nn.LSTM with bidirectional=True and n_layers &gt;1.</p>
<p>I am confused about n_layers. Here is the documentation:<br>
<code>* **num_layers** – Number of recurrent layers. E.g., setting </code>num_layers=2<code> would mean stacking two LSTMs together to form a stacked LSTM, with the second LSTM taking in outputs of the first LSTM and computing the final results. Default: 1</code></p>
<p>Because of the input of the second layer is the output of the first layer, I expect that the hidden_state shape should corporate n_layers.</p>
<p>I understand that for the bidirectional=True, I can add both direction for the hidden state, How can we do if we use n_layers&gt;1. Most of the code online use only 1 layer for encoder and decoder!</p>

username: vdw
cooked: <p>You can have a look at my implementation of mine; it’s a RNN-based classifier with attention, i.e., only the encoder part of a Seq2Seq model. But that’s all you need. The code is a bit verbose as it classifier is configurable using GRU or LSTM, uni or bidirectional, different number of RNN layers, different numbers of linear layers at the end. In a nutshell:</p>
<ul>
<li>
<p>Yes, you can add the 2 directions. In my implementation, I concatenate them: notice the <code> self.params.rnn_hidden_dim * self.num_directions</code> when creating the Attention layer. You would need to this if you add the 2 directions, of course. Don’t ask me which solution is better/preferred :).</p>
</li>
<li>
<p>Regarding the number of RNN layers, just attend over the last layer. In this way, it doesn’t matter at all how many layers do you have. Check out the <code>forward()</code> method, <code>rnn_output</code> is all you need as it gives you all states of the sequence with respect to last RNN layer.</p>
</li>
</ul>
<p>I hope that helps a bit.</p>
Pytorch CNN for Regression

username: dzdang
cooked: <p>Does anyone know of any Pytorch CNN examples for regression? I can’t seem to find any regression examples (everything I’ve seen is for classification).</p>

username: vmirly1
cooked: <p>I think the tasks related to images are mostly classification tasks. However, there are some applications for regression but more specifically ordinal-regression, such as age estimation. You can see this paper for an example of ordinal-regression with CNN: <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/app/S21-20.pdf" rel="nofollow noopener">https://www.cv-foundation.org/openaccess/content_cvpr_2016/app/S21-20.pdf</a></p>

username: dzdang
cooked: <p><a class="mention" href="/u/vmirly1">@vmirly1</a> I’ve definitely seen papers implementing CNNs for regression. I was actually trying to see if there are any Pytorch examples using CNNs on regression problems. I suspect that the only thing I need to do different in a regression problem in Pytorch is change the cost function to MSE.</p>

username: ptrblck
cooked: <p>Basically yes. Probably you would also change the last layer to give the desired number of outputs as well as remove some non-linearity on the last layer such as <code>F.log_softmax</code> (if used before).</p>

username: dzdang
cooked: <p>What if it was nonlinear regression, would you still want to remove non-linearity?<br>
Also, are the activation functions in the layers before the output layer typically the same for regression and classification?</p>

username: ptrblck
cooked: <p>I just meant the last non-linearity. The activation functions between the layers should still be used. I would try to use pretty much the same architecture besides the small changes necessary for regression.</p>

username: dzdang
cooked: <p>Got it, thanks! I suspected the same, however, I do find it somewhat ironic and intriguing that pretty much the same architecture can be used for both regression and classification except for the loss function and some minor details in the output layer.</p>

username: sgbaird
cooked: <p>Would be nice to see an example of a <code>pytorch</code> module that someone recommends or a basic MWE.</p>
Layernorm doesn't result in std=0 per data point

username: milan_kalkenings
cooked: <p>layernorm doesn’t seem to calculate what it should. I compared the results of nn.LayerNorm and a manual calculation:</p>
<p>code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

batch = torch.tensor([[4, 3, 1],
                      [0, 2, 0]]).float()

layernorm = nn.LayerNorm(normalized_shape=3, eps=0, elementwise_affine=False)

mean0 = torch.mean(batch[0])
var0 = torch.var(batch[0])
result = (batch[0] - mean0) / torch.sqrt(var0)
</code></pre>
<p>for layernorm:</p>
<pre><code class="lang-auto">tensor([[ 1.0690,  0.2673, -1.3363],
        [-0.7071,  1.4142, -0.7071]])
std: tensor(1.2247)
</code></pre>
<p>for my manual version:</p>
<pre><code class="lang-auto">tensor([ 0.8729,  0.2182, -1.0911])
std: tensor(1.)
</code></pre>

username: ptrblck
cooked: <p>In your manual approach you are using the unbiased variance.<br>
From the docs:</p>
<blockquote>
<p>The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False)</p>
</blockquote>
<p>I also get the same results using the <code>nn.LayerNorm</code> module and a manual approach:</p>
<pre data-code-wrap="python"><code class="lang-python">batch = torch.tensor([[4, 3, 1],
                      [0, 2, 0]]).float()

layernorm = nn.LayerNorm(normalized_shape=3, eps=0, elementwise_affine=False)

out = layernorm(batch)

mean0 = torch.mean(batch, dim=-1)
var0 = torch.var(batch, unbiased=False, dim=-1)
result = (batch - mean0.unsqueeze(1)) / torch.sqrt(var0.unsqueeze(1))

print(out)
# tensor([[ 1.0690,  0.2673, -1.3363],
#        [-0.7071,  1.4142, -0.7071]], dtype=torch.float32)

print(result)
# tensor([[ 1.0690,  0.2673, -1.3363],
#         [-0.7071,  1.4142, -0.7071]], dtype=torch.float32)

print(out.std())
# tensor(1.0954, dtype=torch.float32)
print(out[0].std())
# tensor(1.2247, dtype=torch.float32)


# single sample
mean0 = torch.mean(batch[0])
var0 = torch.var(batch[0], unbiased=False)
result = (batch[0] - mean0) / torch.sqrt(var0)

print(result)
# tensor([ 1.0690,  0.2673, -1.3363], dtype=torch.float32)

print(result.std())
#tensor(1.2247, dtype=torch.float32)
</code></pre>
Taking weighted average of tensors

username: jc_dex
cooked: <p>I have a network that spits out 5 tensors of equal dimensions. Each tensor represents a segmented output of the same image.</p>
<p>I want to also train part of the network to take the weighted average of these tensors. How would I take the average of these tensors? Would I used an <code>nn.Conv2d</code> after concatenating them?</p>
<p>The idea I have in my head is something like:<br>
Output of network: <code>t1, t2, t3, t4, t5</code><br>
Then <code>(x1 * t1 + x2 * t2 + x3 * t3 + x4 * t4 + x5 * t5) / (x1 + x2 + x3 + x4 + x5)</code>, where <code>xi</code> are the weights, and <code>ti</code> are the tensor outputs from the network.</p>
<p>Any advice appreciated.</p>

username: Andrei_Cristea
cooked: <p>What about something as simple as this:</p>
<pre><code class="lang-auto">tensor_shape = (3, 224, 224)  # shape of each tensor
five_tensors = torch.randn(5, *tensor_shape, requires_grad=True)
weights = torch.rand(5, requires_grad=True)
weighted_avg = (weights.view(5, 1, 1, 1) * five_tensors).sum(dim=0) / weights.sum()

# now to check that it does what we want, by looking at a random element
print((five_tensors[:, 0, 100, 100] * weights).sum() / weights.sum())
print(weighted_avg[0, 100, 100])

Output:
tensor(-0.5245, grad_fn=&lt;DivBackward0&gt;)
tensor(-0.5245, grad_fn=&lt;SelectBackward&gt;)
</code></pre>
CUDA 11.6 or 11.7 support

username: Bergen_Truckung
cooked: <p>Can PyTorch work with CUDA 11.6 or 11.7?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Yes, you could install the binaries with the CUDA 11.6 runtime by swapping the version from the provided install commands and you can of course build from source using both CUDA versions.<br>
We are currently working on the bringup of the 11.7 binaries.</p>
How to achieve fast masking?

username: 111414
cooked: <p>How can I achieve fast masking as follows:</p>
<pre><code class="lang-python">import torch
q = torch.Tensor([2, 3, 6, 2])
x = torch.randn(4, 10, 3, 3)
for i in range(4):
    x[i, int(q[i]):] = 0
print(x)
</code></pre>

username: KFrank
cooked: <p>Hi Bin!</p>
<aside class="quote no-group" data-username="111414" data-post="1" data-topic="153006" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111414/48/31166_2.png" class="avatar"> 111414:</div>
<blockquote>
<p>How can I achieve fast masking as follows:</p>
</blockquote>
</aside>
<p>You can get  rid of the loop by multiplying <code>x</code> inplace with a boolean<br>
mask tensor:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; q = torch.Tensor([2, 3, 6, 2])
&gt;&gt;&gt; x = torch.randn(4, 10, 3, 3)
&gt;&gt;&gt; xB = x.clone()
&gt;&gt;&gt; for i in range(4):
...     x[i, int(q[i]):] = 0
...
&gt;&gt;&gt; xB *= (torch.arange (10).unsqueeze (0).expand (4, 10) &lt; q.unsqueeze (1)).unsqueeze (-1).unsqueeze (-1)
&gt;&gt;&gt; torch.equal (x, xB)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Toggle @pytorch.jit.script decorator on / off

username: jayz
cooked: <p>Hello,</p>
<p>is there a way to toggle the toggle <span class="mention">@pytorch.jit.script</span> decorator on / off, like:</p>
<pre><code class="lang-python">use_jit = True / False
@torch.jit.script(enable=use_jit)
def function(x)
    pass
</code></pre>
<p>Thanks!</p>
Trouble in iterating dataset

username: Viswanadham_jyothi_k
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98edb5c624f51d300664c06eb2679f5cc8b516fc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/98edb5c624f51d300664c06eb2679f5cc8b516fc" title="Screenshot 2022-05-31 at 1.51.25 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98edb5c624f51d300664c06eb2679f5cc8b516fc_2_651x500.jpeg" alt="Screenshot 2022-05-31 at 1.51.25 PM" data-base62-sha1="lORRKyQgLKRYHsW1W041yByNkMk" width="651" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98edb5c624f51d300664c06eb2679f5cc8b516fc_2_651x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98edb5c624f51d300664c06eb2679f5cc8b516fc_2_976x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98edb5c624f51d300664c06eb2679f5cc8b516fc_2_1302x1000.jpeg 2x" data-dominant-color="F7F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-05-31 at 1.51.25 PM</span><span class="informations">1404×1078 97.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
_MapDatasetFetcher.fetch(self, possibly_batched_index)</p>

username: nivek
cooked: <p>The issue seems to be within the <code>collate_fn</code>. What is the <code>collate_fn</code> that you are passing into DataLoader? Within that function, is <code>F</code> defined?</p>
Loss function error: element 0 of tensors does not require grad and does not have a grad_fn

username: Aigerim_Bazarkhanova
cooked: <p>Hi all! How to solve following issue:  element 0 of tensors does not require grad and does not have a grad_fn?<br>
I want to find the solution to 2nd order ODE?<br>
My code as follows:</p>
<pre><code class="lang-auto">def loss_fun(inputs,var):
    n = inputs.size()[0]
    loss = torch.tensor(0.0, requires_grad=True, dtype=torch.float64)    
    loss_fun.outputs = torch.zeros(inputs.size())
    
    
    for i in range(n):
        x = Variable(inputs[i], requires_grad=True)
        output = sum_wrapper(x, var)
        loss_fun.outputs[i] = output
        dydx_1 = torch.autograd.grad(output, x, grad_outputs=torch.ones_like(output),
                                            create_graph=True)[0]
        dydx_2 = torch.autograd.grad(dydx_1, x, grad_outputs=torch.ones_like(dydx_1),
                                            create_graph=True)[0]

        
        eq1 = dydx_2 + output# y' = x^2+y^2-1
        loss = loss.add(torch.mean(eq1**2))
    
    # add the initial condition point
    ic = sum_wrapper(torch.tensor([0.],dtype=torch.float64), var)     # y(x=0) = 0.0
    loss = loss.add(ic**2)
    return loss

var = Variable(inits, requires_grad=True)
opt = torch.optim.Adam([var], lr=lr, eps=0.001)
losses = []
varss = []
for it in range(steps):
    varss.append(var.detach().clone())
    # Feedforward and backpropagation + step
    
    opt.zero_grad()
    loss = loss_fun(x_data, var)
    losses.append(loss.item())
    loss.backward()
    opt.step() 
    print('it {}, loss {}'.format(it, loss.item()))
</code></pre>
How to reduce test loss with high test accuracy?

username: Laya1
cooked: <p>I have a 27-class classification problem with a time series of 1600 samples as the input data. I have a model made up of simple feedforward network with 4 hidden layers that has very good training loss (0.14). On the test data, it has very high accuracy (98%), but a a very high loss (2.45). It appears that the model makes very few mistakes, but those mistakes are very severe.</p>
<p>This happens regardless of the optimiser, the number of epochs and different initialisations. I also observe similar behavior with a 1-D convolutional model.</p>
<p>Is there a way to correct for this behavior and reduce the loss?</p>
Stack image tensors for video processing?

username: Mariusmarten
cooked: <p>My ConvLSTM network requires the following input shape: (64, 10, 3, 32, 32) [batch size, length of the seq., color channel, img. size). Previously I have only worked with the Conv part of the network which required input shape: (64, 3, 32, 32). Can I now simply stack my tensors or is another procedure better suited?</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        
        self.conv1 = nn.Conv2d(3, 10, 5)
        self.conv2 = nn.Conv2d(10, 20, 5)
        self.conv3 = nn.Conv2d(20, 30, 5)
        self.pool = nn.MaxPool2d(2, 2)

    def forward(self, i):
        x = i.view(-1, i.shape[2], i.shape[3], i.shape[4])
        x = F.relu(self.conv1(x))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(i.shape[0], i.shape[1], -1)
        return x
    
class LSTM(nn.Module):
    def __init__(self):
        super(LSTM, self).__init__()
        self.lstm = nn.LSTM(480, 100)
        self.fc = nn.Linear(100*10, 2)
        
    def forward(self, x):
        x, _ = self.lstm(x)
        x = x.view(x.shape[0], -1)
        print('here', x.shape)
        x = self.fc(x)
        return x  
</code></pre>
<p>EDIT: I think what I’m looking for is a dataloader for video that takes as input a list of images.</p>
Use one “freezed” backbone with different heads

username: Bruno_Patricio
cooked: <p>Hi,</p>
<p>I have one question: With distinct trained models with a “freezed” backbone, is it possible to use just one instance of the backbone and the resulting features on all the heads from all the models, like shown in the diagram?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ed8f942579b104c4ae6b373590a00763adcc7cc.png" alt="Capturar" data-base62-sha1="8XYsXz6l82vmWG9V8DzENgC3us4" width="584" height="312"></p>
<p>Thanks for the help!</p>
What is the correct way of copying weights of one model into another?

username: Shahid_Siddiqui
cooked: <p>Hi!</p>
<p>I am trying to copy weights of a pretrained model into another model of exactly similar structure. The original model gives an accuracy of 94% on a binary image classification problem but the target model is unable to predict, results in predicting only one class for entire test set.</p>
<p>I used this piece of code to manually copy weights from pretrained model to target:</p>
<p>modelmix.stem[0].weight = modelSep.stem[0].weight<br>
modelmix.stem[1].weight = modelSep.stem[1].weight<br>
modelmix.stem[1].bias = modelSep.stem[1].bias</p>
<p>where modelmix is the target and modelSep is the pretrained model. Used the same snippet for all the layers. The target model is not working. Using pytorch 1.1. Thank you</p>

username: Vishak_Raj
cooked: <p><a class="mention" href="/u/shahid_siddiqui">@Shahid_Siddiqui</a>, did you find the reason, why the target model isnt working?</p>
Normalize ellipses based on rotation

username: Roua_Rouatbi
cooked: <p>I know this is not a deep learning problem but I thought someone might know how.<br>
I fitted an ellipse around multiple different objects and I’m trying to retrieve the x and y of the small objects inside each ellipse. Those coordinates however depend on the rotation of the ellipse I want them to be normalized based on the axes for example.</p>

username: Roua_Rouatbi
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a37062200f70add240f8ac6aaac55375196cb73a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a37062200f70add240f8ac6aaac55375196cb73a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a37062200f70add240f8ac6aaac55375196cb73a_2_690x234.png" alt="image" data-base62-sha1="njQBGmX9yj4vrG6lmYPwSTSBNzA" width="690" height="234" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a37062200f70add240f8ac6aaac55375196cb73a_2_690x234.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a37062200f70add240f8ac6aaac55375196cb73a_2_1035x351.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a37062200f70add240f8ac6aaac55375196cb73a_2_1380x468.png 2x" data-dominant-color="B3BAB8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2292×780 237 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These are the ellipses I want to find the coordinates of the objects inside of each one and normalize them</p>
Does gradients change in Model.eval()?

username: gaza_palestine
cooked: <p>I am confused between with torch.no_grad() and model.eval(). Does gradients change in model.eval() mode, or anyway to check that?</p>

username: ptrblck
cooked: <p>The gradient calculation is independent from the training mode in the model, which is changed via <code>model.train()</code> and <code>model.eval()</code>.<br>
To disable the gradient calculation, set the <code>.requires_grad</code> attribute of all parameters so <code>False</code> or wrap the forward pass into <code>with torch.no_grad()</code>.<br>
Calling <code>model.eval()</code> will change the behavior of some layers, such as <code>nn.Dropout</code>, which will be disabled, and <code>nn.BatchNormXd</code>, which will use the running stats during evaluation.</p>

username: Varal7
cooked: <p>I have found this note useful in understanding the different ways one can disable gradients:</p>
<p><a href="https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation</a></p>

username: is_Juncheng
cooked: <p>Do you mean that in model.eval(), the parameters of the BN layer will continue to be updated, just using different statistics?</p>

username: ptrblck
cooked: <p>No, after calling <code>model.eval()</code> the running stats will be used to normalize the input activations and these running stats will not be updated in batchnorm layers using the default setup.</p>
Error in losstrain

username: linda125542
cooked: <pre><code class="lang-auto">def train(epoch):
  
    model.train()
    tr_loss = 0
    # getting the training set
    x_train, y_train = Variable(train_x), Variable(train_y)
    # getting the validation set
    x_val, y_val = Variable(val_x), Variable(val_y)
    # converting the data into GPU format
    if torch.cuda.is_available():
        #model.cuda()
        x_train = x_train.cuda()
        y_train = y_train.cuda()
        x_val = x_val.cuda()
        y_val = y_val.cuda()

    # clearing the Gradients of the model parameters
    optimizer.zero_grad()

    # prediction for training and validation set
    output_train = model(x_train.float())
    output_val = model(x_val.float())

    # computing the training and validation loss
    loss_train = criterion(output_train, y_train.long())
    loss_val = criterion(output_val, y_val.long())
    train_losses.append(loss_train)
    val_losses.append(loss_val)
    
    # computing the updated weights of all the model parameters
    ###loss_train.backward()
    optimizer.step()
    tr_loss = loss_train.item()
    
    if epoch % 2 == 0:
        # printing the validation loss
        print('Epoch : ', epoch + 1, '\t', 'loss :', loss_train, loss_val)
</code></pre>
<pre><code class="lang-auto">   train_losses = []
    # empty list to store validation losses
    val_losses = []
    correct = 0
    total = 0
    predictions =[]
    actuals=[]
    predictiontest=[]
    # training the model
    m1=0
    m2=0
    for epoch in range(n_epochs):
        
      train(epoch)
     
    # plotting the training and validation loss
    plt.plot(train_losses, label='Training loss')
    plt.plot(val_losses, label='Validation loss')
</code></pre>
<p>RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.</p>
<p>help me plzz</p>

username: ptrblck
cooked: <p>Did you try to use the suggested approach from the error message?</p>
<aside class="quote no-group" data-username="linda125542" data-post="1" data-topic="152942">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/90ced4/48.png" class="avatar"> linda125542:</div>
<blockquote>
<p>RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.</p>
</blockquote>
</aside>
<p>It seems you are trying to call <code>numpy()</code> on a tensor which is attached to the computation graph so would need to detach it first.</p>

username: linda125542
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I did not understand how I solve</p>

username: ptrblck
cooked: <p>Try to call <code>tensor.detach().cpu().numpy()</code> instead of <code>tensor.numpy()</code>.</p>

username: linda125542
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> in my coded</p>
<pre><code class="lang-auto">with torch.no_grad():
        output1 = model((train_x.float()))     

        softmax1 = torch.exp(output1).cpu()    
        prob1 = list(softmax1.numpy())
</code></pre>
<p>I replace this numpy()  by tensor.detach().cpu().numpy() ??</p>