akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Predicting a combination of targets with a regression

username: cosmicnet
cooked: <p>Hello,</p>
<p>I have around 40 features and 4 target descriptors. I’ve trained models against each target individually, and I know there is a relationship. Even a simple linear regression does reasonably well.</p>
<p>I’m aware that each target descriptor has an effect on the features, such that there is function of all 4 targets that would have their cumulative effect and be more predictable than each one is individually.</p>
<p>It seems to me that I could discover this with a simple linear regression, but I’m not clear how I would train.</p>
<p>I can only imagine a training loop within a training loop. Such that the outer loop has the 4 target linear regression producing a single target, then the inner loop trains a model with the 40 features against that to see how well it can predict, with a loss produced from that prediction and used in the outer training loop.</p>
<p>I’m not sure I’ve explained that very well, and regardless, there may well be a much simpler solution.</p>
<p>Advice appreciated.</p>

username: googlebot
cooked: <p>if you use a single model with 4 outputs, an output layer implicitly models covariance (ie produces correlated predictions)</p>
<p>but that’s a model with fully shared parameters/features in hidden layers, that may be good or bad depending on how this matches with the true process. when this is undesirable, it is possible to have model designs with some non-shared parameters and/or layers, aka multi-task learning</p>
<p>what you’ve described is an extreme non-sharing design, its primary downside is O(num_outputs) time&amp;memory costs</p>
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of size: : [4]

username: Lohith_Kumaar
cooked: <p>def train_model(model, loss_fn, optimizer, lr_scheduler, num_epochs=25):<br>
since= time.time()</p>
<pre><code>best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0

for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs -1))
    print('-' * 10)
    
    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()
        else:
            model.eval()
            
        running_loss = 0
        running_corrects = 0
        
        for inputs, outputs in dataloaders[phase]:
            inputs = inputs.to(device)
            outputs = outputs.to(device)
            
            optimizer.zero_grad()
            
            &gt; with torch.set_grad_enabled(phase == 'train'):
</code></pre>
<blockquote>
<pre><code>                probabilities = inputs
                _,predicions = torch.max(probabilities, 1)
                loss = loss_fn(probabilities, outputs)
</code></pre>
</blockquote>
<pre><code>                if phase == 'train':
                    loss.backward()
                    optimizer.step()
                  
            running_loss += loss.items()*input.size(0)
            running_corrects += torch.sum(predictions == output.data)
            if phase == 'train':
                lr_scheduler.step()
            epoch_loss = running_loss / data_size[phase]
            epoch_acc = running_corrects.double() / data_size[phase]
            
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))
            
            if phase == 'val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
        print()
        
    time_elapsed = time.time() -since
    print('Training complete in {:.Of}m {:.Of}s'.format(time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    
    model.load_state_dict(best_model_wts)
    return model
</code></pre>
<p>I was working on this code and got that error I am unable to fine how to debug it.<br>
The marked lines are where I am getting error.</p>
Misunderstanding on CrossEntropyLoss

username: sardanian
cooked: <p>I am a bit confused on how CrossEntropyLoss works or I am not using it properly.</p>
<p>I understand the mathematical formula CrossEntropyLoss and am trying to implement it for AlphaZero.</p>
<p>My issue is with the policy part of the loss function which uses CrossEntropyLoss portion. I am new to this, but when I use CrossEntropyLoss on the predicted policy’s probabilities with the labeled policy’s probabilities it doesn’t seem to give me a number near 0 when the model is close. Even when I give it an identical model tensor for input and target, it still gives me a non-zero larger number) so I am confused as to how this is helping me as a loss function.</p>
<p>Here is an example set of code where I am trying out CrossEntropyLoss and manually running CrossEntropyLoss through the sum of the multiplication of the policy with the log of the predictions</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>input = torch.FloatTensor([[.2,.3,.3,.2],[.3,.2,.3,.2],[0,0,.8,.2],[0.1,0.1,.6,.2]])<br>
logInput = input.log()<br>
cross_entropy_loss = nn.CrossEntropyLoss(reduce=False, reduction=“mean”)<br>
output = cross_entropy_loss(input, input)<br>
output2 = torch.sum(-input * torch.nan_to_num(input.log(), neginf=0.0), dim=1)</p>
<p>print('output: ', output)<br>
print('output2: ', output2)</p>
<p>output:  tensor([1.3775, 1.3775, 1.0151, 1.2390])<br>
output2:  tensor([1.3662, 1.3662, 0.5004, 1.0889])</p>
<p>When I use it on a larger tensor (for example I am using a tensor of length 4672) then it gets very wonky and the outputs end up around 8.4</p>
<p>Three questions:</p>
<ol>
<li>Am I doing something wrong in trying to implement cross entropy loss? Of course instead of running it against itself I would run it against predictions, but this was just to test the loss function</li>
<li>If this is the correct output of CrossEntropyLoss, how will this help the AlphaGo algorithm learn?</li>
<li>Why are the above two outputs different? I thought this was what the CrossEntropyLoss calculation was.</li>
</ol>
<p>Thank you</p>

username: mMagmer
cooked: <p><code>corssentropy</code> gets model score  (-∞,+∞) not probabilitis.<br>
it combines logsoftmax and NLL.</p>
Convert tesnor to array problem

username: prog_asker
cooked: <p>I need to save the tensor in a file after converting it into a list of arrays. I tried that</p>
<pre><code class="lang-auto">results = []
vec=model(preprocess(image).unsqueeze(0))
vec1=vec.numpy()
for ind in range(1):     
    results.append(vec1[ind])
return results
</code></pre>
<p>and got</p>
<blockquote>
<p>RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.</p>
</blockquote>
<p>then tried to convert this line <code>vec1=vec.numpy()</code> to <code>vec1=vec.detach().cpu().numpy()</code><br>
but got <code>killed</code> after some time</p>

username: ptrblck
cooked: <p>The process was most likely <code>killed</code> if you were running out of host memory, so try to reduce the memory requirement of your application.</p>

username: Avinash_20
cooked: <p>I am having the same issue, How did you solve this bro?</p>
Change Tensor shape

username: asura
cooked: <p>Hi,</p>
<p>I had a very noob question:<br>
I want to change my tensor shape from <code>[5, 3, 84, 84]</code> to <code>[5, 1, 28, 28]</code><br>
That is, change image size from 84x84 to 28x28, and convert RGB to GrayScale.</p>
<p>P.S. I don’t want to make use of transforms, as I want to keep the original tensor <code>[5, 3, 84, 84]</code> for a different operation.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="asura" data-post="1" data-topic="143064">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asura/48/28647_2.png" class="avatar"> asura:</div>
<blockquote>
<p>P.S. I don’t want to make use of transforms, as I want to keep the original tensor <code>[5, 3, 84, 84]</code> for a different operation.</p>
</blockquote>
</aside>
<p>You could still use transformations and keep the original tensor alive as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">trans = transforms.Compose([
    transforms.Grayscale(),
    transforms.Resize((28, 28)),
])

x = torch.randn([5, 3, 84, 84])
out = trans(x)
print(x.shape)
# &gt; torch.Size([5, 3, 84, 84])
print(out.shape)
# &gt; torch.Size([5, 1, 28, 28])
</code></pre>
Determine inter-layer channel size dependency

username: ralphmao
cooked: <p>I want to implement a function that modifies the output channel of a layer on the fly. To ensure that the model is still valid after the modification, I will need to find out all layers that depends on that layer (and there can be trivial layers/functions like relu between them). Is there any existing infra/function in PyTorch that can help me with this functionality?</p>
How to change the code to use feature extractor from torchvision

username: ramdhan1989
cooked: <p>I would like to change the feature extractor in the github code below</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bbrattoli/JigsawPuzzlePytorch/blob/master/JigsawNetwork.py">
  <header class="source">

      <a href="https://github.com/bbrattoli/JigsawPuzzlePytorch/blob/master/JigsawNetwork.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bbrattoli/JigsawPuzzlePytorch/blob/master/JigsawNetwork.py" target="_blank" rel="noopener nofollow ugc">bbrattoli/JigsawPuzzlePytorch/blob/master/JigsawNetwork.py</a></h4>


      <pre><code class="lang-py"># -*- coding: utf-8 -*-
"""
Created on Wed Sep 13 15:57:01 2017

@author: Biagio Brattoli
"""
import torch
import torch.nn as nn
from torch import cat
import torch.nn.init as init

import sys
sys.path.append('Utils')
from Layers import LRN

class Network(nn.Module):

    def __init__(self, classes=1000):
        super(Network, self).__init__()

</code></pre>



  This file has been truncated. <a href="https://github.com/bbrattoli/JigsawPuzzlePytorch/blob/master/JigsawNetwork.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
How to change self.conv to use something from the torchvision models directly ?</p>
<p>thank you</p>
Training epochs are so slow

username: OBouldjedri
cooked: <p>I am working on a dataset that I extracted from JSON files, but the training is so slow I was wondering what could be wrong with my dataset definition:</p>
<p>class JsonDataset(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self, files):<br>
super(JsonDataset).<strong>init</strong>()<br>
self.files = files</p>
<pre><code>    self.data_full_list=[]
    for i in range(len(self.files)):
       
        liste_files_current_act=self.files[i] 

        for j in range(len(liste_files_current_act)):

             with open(liste_files_current_act[j]) as file:
                 for line in file:
                   
                    line_tmp = file.readline()
                  
                    if line_tmp == None or line_tmp == '':
                        print('I got a null or empty string value for data in a file')
                        if line_tmp== None:
                            print('it is NONE')
                        if line_tmp== '':
                            print('it is a blank')
                        
                    else:
                        res_line= json.loads(str(line_tmp))
                    
                    data_line=res_line['data']
      
                    self.data_full_list.append([data_line, i])
          
def __len__(self):
    return(len(self.data_full_list))    

               
def __getitem__(self, idx):
    
    start = time.time()
    
    data,label=self.data_full_list[idx]
   
    data=data[0:164]
    
    data=torch.FloatTensor(data)
   
    if timestep=='5sec':
        data=data.repeat(250,1)
    
    if timestep=='10sec':
        
        data=data.repeat(500,1)    
    
    if timestep=='15sec':
        data=data.repeat(750,1)  
        
        
    end1 = time.time()
    print('time point 1',end1 - start)    
    
    data_uwb1=data
    data_uwb2=data
    data_uwb3=data

    np_UWB3 = data_uwb3.cpu().detach().numpy()
    np_UWB2 = data_uwb2.cpu().detach().numpy()
    np_UWB1 = data_uwb1.cpu().detach().numpy()
    
    np_fusion=np.stack((np_UWB1,np_UWB2,np_UWB3), axis=0)
    
    
    
    data=torch.from_numpy(np_fusion)
    end2 = time.time()
    print('time point 2',end2 - start)
     
    return(data,label)
</code></pre>
<p>it seems to me that once the training enter the training loop it takes much time to start the first iteration of this loop:<br>
for data, target in train_loader:</p>
Embedding from batch of images

username: Mordashkina
cooked: <p>Hello!<br>
I have a dataset with PET-CT scans, it looks like directories for every patient with about 300 (from 250 to 350) slices per patient. I need to get 1 embedding for 1 person, I’ve tried to use 3D resnet, but it wasn’t enoght memory in my server. How can I make it?<br>
I have the idea to make a loop in my forward function and for every mini-batch get an embedding, but after it i need to concatenate all of embeddings. For example, in output I’ll have 10 embeddings size [1, 300] for every mini-batch and in batch output I need 1 embedding size [1, 300] for this patient. How can I make it?<br>
I’m not sure, that it’s the best idea at all and I’ll really appreciate any help</p>

username: ptrblck
cooked: <p>I don’t know what “embedding” refers to in your use case, but if you want to store e.g. the penultimate activation of the model you could either return this activation directly in the <code>forward</code> method or you could use forward hooks.<br>
Iterating the <code>DataLoader</code> would be the standard approach and you could store the outputs in e.g. a <code>list</code> and create the tensor afterwards.</p>
GPU-enabled training and testing in Windows 10

username: hjung
cooked: <p><strong>In Windows 10</strong>, my computer has NVIDIA driver 456.71 and I installed PyTorch using<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</code></p>
<p>Then, I checked whether PyTorch used CUDA by typing the followings:</p>
<pre><code class="lang-auto">import torch
torch.cuda.is_available()
</code></pre>
<p>And it returned <code>True</code>.</p>
<p>It’s pretty weird to me because I’ve heard that we need to install both NVIDIA drivers and CUDA to use GPUs.</p>
<p>Thus, my question is: Can I use PyTorch and GPU-enabled training and testing without manually installing CUDA for Windows?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/hjung">@hjung</a> although you are getting <code>True</code> for <code>torch.cuda.is_available()</code>, without underlying nvidia drivers you will be unable to use it for tensor operations</p>
<p>There is an older thread discussing the same thing</p><aside class="quote" data-post="1" data-topic="60181">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n0obcoder/48/16606_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-it-required-to-set-up-cuda-on-pc-before-installing-cuda-enabled-pytorch/60181">Is it required to set-up CUDA on PC before installing CUDA enabled pytorch?</a> 
  </div>
  <blockquote>
    Can I simply go to pytorch website and use the following link to download a CUDA enabled pytorch library ? Or do i have to set up the CUDA on my device first, before installing the  CUDA enabled pytorch ? 
pip3 install torch===1.3.0 torchvision===0.4.1 -f https://download.pytorch.org/whl/torch_stable.html
  </blockquote>
</aside>


username: hjung
cooked: <p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> From the link, it seems that the PyTorch installation command includes the CUDA <em><strong>runtime</strong></em>, which  I think is sufficient to use use GPUs for training and testing. Thus, we do not need to install manually CUDA for Windows.</p>
<p>Could you (or anyone) please correct me if I am wrong?</p>
<aside class="quote" data-post="19" data-topic="60181">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-it-required-to-set-up-cuda-on-pc-before-installing-cuda-enabled-pytorch/60181/19">Is it required to set-up CUDA on PC before installing CUDA enabled pytorch?</a> 
  </div>
  <blockquote>
    No CUDA toolkit will be installed using the current binaries, but the CUDA runtime, which explains why you could execute GPU workloads, but not build anything from source.
  </blockquote>
</aside>


username: ptrblck
cooked: <p>Yes, you are right and don’t need to install a local CUDA toolkit (just the drivers) unless you want to build PyTorch from source or a custom CUDA extension.</p>
RuntimeError: stride should contain 2 elements for 2D convolution

username: fardinadii
cooked: <p>Hi<br>
im trying to quantize efficinetdet model but i faced with this error:</p>
<pre><code>400         if self.padding_mode == 'zeros':
401             self._packed_params = torch.ops.quantized.conv2d_prepack(
</code></pre>
<p> → 402                 w, b, self.stride, self.padding, self.dilation, self.groups)<br>
403         else:<br>
404             self._packed_params = torch.ops.quantized.conv2d_prepack(</p>
<p>RuntimeError: stride should contain 2 elements for 2D convolution.</p>
<p>git repo:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/f1d1993a342aea8f1a8ed0a92309cc2c25749a2a88ad2f4735b407f0abee0dca/zylo117/Yet-Another-EfficientDet-Pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - zylo117/Yet-Another-EfficientDet-Pytorch: The pytorch re-implement...</a></h3>

  <p>The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights. - GitHub - zylo117/Yet-Another-EfficientDet-Pytorch: The pytorch re-implement of the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: L-Reichardt
cooked: <p>Haven’t found much for conv2d_prepack, but have you tried passing a tuple as stride?</p>
<pre><code class="lang-auto">self._packed_params = torch.ops.quantized.conv2d_prepack(w, b, (self.stride, self.stride), self.padding, self.dilation, self.groups)
</code></pre>
<p>If that is the issue, you might need to pass padding as a tuple as well.</p>
Is this implementation for GradCam correct?

username: GustavoVargasHakim
cooked: <p>It shows no errors but what I get is simply a red square. If the model was wrong, I would expect at least an incorrect gradcam but I’m not sure. Also, I just modified the class to receive a custom model (basically ResNet18) and the same last 4th layer is chosen.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://medium.com/the-owl/gradcam-in-pytorch-7b700caa79e5">
  <header class="source">

      <a href="https://medium.com/the-owl/gradcam-in-pytorch-7b700caa79e5" target="_blank" rel="noopener nofollow ugc" title="11:57AM - 31 July 2021">Medium – 31 Jul 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/244;"><img src="https://miro.medium.com/max/1200/1*CdsOIHzwbidbr9Yzf6qbAw.png" class="thumbnail" width="690" height="244"></div>

<h3><a href="https://medium.com/the-owl/gradcam-in-pytorch-7b700caa79e5" target="_blank" rel="noopener nofollow ugc">GradCAM in PyTorch</a></h3>

  <p>Implementing GradCAM in PyTorch</p>

  <p>
    <span class="label1">Reading time: 3 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you!</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/gustavovargashakim">@GustavoVargasHakim</a> the code looks alright<br>
Can you share your code on how you are using it</p>
Using pretrained models in convolutional autoencoders

username: sparshgarg23
cooked: <p>Is it possible to use a pretrained model like resnet/vgg in a convolutional autoencoder?If so then how can we use it.I know that for standard classification purposes,we ignore the last classification layer,should the same apply in convolutional autoencoders?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/sparshgarg23">@sparshgarg23</a> For an autoencoder setup, the last layer should represent the learned embedding. So you can just remove the activation layer and then make it flow to your decoder setup</p>
<p>Some pointers though</p>
<ul>
<li>
<p>Generally the decoder setup is exactly same as the encoder setup</p>
</li>
<li>
<p>You might end up training a huge decoder since your encoder is vgg/resnet. You will have to come up with a transpose of the pretrained model and use that as the decoder, allowing only certain layers of the encoder and decoder to get updated</p>
</li>
</ul>
<p>Following is an article that will help you come up with the model architecture</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="256" height="256">

      <a href="https://towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763" target="_blank" rel="noopener nofollow ugc" title="03:42AM - 17 November 2021">Medium – 17 Nov 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/255;"><img src="https://miro.medium.com/max/739/1*a1Yy2bG_zMnqRov93SWs4w.png" class="thumbnail" width="690" height="255"></div>

<h3><a href="https://towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763" target="_blank" rel="noopener nofollow ugc">Convolutional Autoencoders for Image Noise Reduction</a></h3>

  <p>In “Anomaly Detection with Autoencoders Made Easy” I mentioned that the Autoencoders have been widely applied in dimension reduction and…</p>

  <p>
    <span class="label1">Reading time: 10 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to multiply all model's parameters by some constant value?

username: Noam_Salomonski
cooked: <p>I want to implement the following algorithm, <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="noopener nofollow ugc">from this book, section 13.6</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/56f01cea379b2efd41f2f836973fbb623f5f9214.png" data-download-href="https://discuss.pytorch.org/uploads/default/56f01cea379b2efd41f2f836973fbb623f5f9214" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/56f01cea379b2efd41f2f836973fbb623f5f9214_2_690x425.png" alt="Untitled" data-base62-sha1="cp5v7Zu4qca4KlBYcq4ymSaHuDy" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/56f01cea379b2efd41f2f836973fbb623f5f9214_2_690x425.png, https://discuss.pytorch.org/uploads/default/original/2X/5/56f01cea379b2efd41f2f836973fbb623f5f9214.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/56f01cea379b2efd41f2f836973fbb623f5f9214.png 2x" data-dominant-color="E0E0E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">954×588 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Therefore, I need to be able to update all the network’s parameters by a constant factor.</p>
<p>I found the following code to acomplish that</p>
<pre><code class="lang-auto">state_dict = net.state_dict()

for name, param in state_dict.items():
    # Transform the parameter as required.
    transformed_param = param * 0.9

    # Update the parameter.
    state_dict[name].copy_(transformed_param)
</code></pre>
<p>but it doesn’t seem right, that all parameters have to pass through my code, rather than have some vector operation (which is implemented in a low level language, and is cuda supported).</p>
<p>I’m sure there is a better way of doing this.</p>

username: anantguptadbl
cooked: <pre><code class="lang-auto">state_dict = net.state_dict() # This is no longer required

for cur_param in net.parameters():
    cur_param.data =  cur_param.data*0.1
</code></pre>
Data2Vec: Implement Teacher Update

username: blueeagle
cooked: <p>Hi everyone,</p>
<p>I just stumbled upon the new <a href="https://scontent-frt3-1.xx.fbcdn.net/v/t39.8562-6/271974914_483120576492438_4239522333319653600_n.pdf?_nc_cat=107&amp;ccb=1-5&amp;_nc_sid=ae5e01&amp;_nc_ohc=9HtSivaeiYUAX-amhxX&amp;_nc_ht=scontent-frt3-1.xx&amp;oh=00_AT-1jBBW9e7d7HWTBxi66_9uJPzswVogQbVxnEpDftJK2A&amp;oe=61FBE0D1" rel="noopener nofollow ugc">Data2Vec Paper</a> of Meta AI.</p>
<p>In section 3.3 the authors describe how the parameter updates for the so called teacher model is done. If I got it right, the parameters get updated according to<br>
W_t = (1-k) * W_t + k * W_s,<br>
with W_t being the teacher model’s parameters, W_s being the student model’s parameters and k being a constant factor.</p>
<p>I tried to reimplement this parameter update. In order to do this, I have to multiply all parameters of the teacher Model (W_t) with the constant factor (1-k). The only solution I found to this, was the following <a href="https://discuss.pytorch.org/t/how-to-multiply-all-models-parameters-by-some-constant-value/37464">(from this post here)</a>:</p>
<pre><code class="lang-auto">for name, param in state_dict.items():
    # Transform the parameter as required.
    transformed_param = param * 0.9

    # Update the parameter.
    state_dict[name].copy_(transformed_param)
</code></pre>
<p>However, this looks highly inefficient to me, as this is no vectorized operation. Is there a faster way to do so?</p>
<p>Thanks for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: anantguptadbl
cooked: <p>Assuming your model is net</p>
<pre><code class="lang-auto">state_dict = net.state_dict() # This is no longer required

for cur_param in net.parameters():
    cur_param.data =  cur_param.data*0.1
</code></pre>
Freezing only one layer of a lstm

username: PRamoneda
cooked: <p>I have a pretrained model with this layer:</p>
<pre><code class="lang-auto">self.rnn1 = nn.LSTM(..., num_layers=3, ....)
</code></pre>
<p>Previously, I have frozen all the model…</p>
<p>How can I freeze only the last layer of the LSTM in a clear way?</p>
<p>In other words, Is it possible to convert a lstm with 3 layers pretrained in three layers of LSTMs (1 layer each one) in a easy way?</p>

username: anantguptadbl
cooked: <pre><code class="lang-auto">rnn1 = nn.LSTM(input_size=10, hidden_size=5, num_layers=3)
relevant_parameters = [i for i,(param_name,param_value) in enumerate(list(rnn1.named_parameters())) if 'l2' in param_name]
for i,cur_parameter in enumerate(rnn1.parameters()):
    if i in relevant_parameters:
        print("Setting for {0}".format(i))
        cur_parameter.requires_grad=False
</code></pre>
Getting the number of training examples from the loader

username: Bixqu
cooked: <p>I am training a very simple net on CIFAR10 and I want to get the number of examples from the loader,</p>
<pre><code>trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
</code></pre>
<p>Is there a way to use <code>trainloader.number_of_examples</code> or something similar?</p>

username: fmassa
cooked: <p>The number of examples is present from the dataset, not the dataloader.<br>
So you can get the number of examples by using <code>len(trainset)</code></p>

username: stash-196
cooked: <p>how about telling dataloader how many samples to load? Is that possible?</p>

username: ptrblck
cooked: <p>You could use a <code>SubsetSampler</code>, pass the desired indices to it, and pass the sampler to the <code>DataLoader</code> in order to define the number of returned samples.</p>

username: Juans
cooked: <p>If anyone is looking to get the number of data points in the training set from the DataLoader object:</p>
<pre><code class="lang-auto">len(trainloader.dataset)
</code></pre>
<p>In contrast <code>len(trainloader)</code> which gives the “number of batches”, see however second warning in the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">docs</a>.</p>
Undefined symbol, cannot figure out why this time

username: lelouedec
cooked: <p>Good afternoon,</p>
<p>I have been trying to compile some CPP extension I coded 2 years ago but on the newest version of pytorch for the past 2 days.<br>
I keep receiving the error : undefined symbole :  _ZNK2at10TensorBase8data_ptrIPfEEPT_v<br>
when importing the extension compiled.<br>
I checked cuda toolkit and pytorch and spent some time matching them between system and conda and pytorch, there is no error during compilation and I even reduced all the code to one function which is as follow to try to debug that :</p>
<pre><code class="lang-auto">
void compute_distances(int b, int n, int m,  at::Tensor  xyz1p, at::Tensor  xyz2p){

    auto  xyz1   = xyz1p.contiguous().data&lt;float*&gt;();
    auto  xyz2   = xyz2p.contiguous().data&lt;float*&gt;();

    for (int i=0;i&lt;b;++i) {
        float x1 = xyz1[i][0];
        float y1 = xyz1[i][1];
        float z1 = xyz1[i][2];
        for (int j=0;j&lt;b;++j) {
            float x2 = xyz1[j][0];
            float y2 = xyz1[j][1];
            float z2 = xyz1[j][2];
            auto distance = sqrtf((x2-x1)*(x2-x1)+(y2-y1)*(y2-y1)+(z2-z1)*(z2-z1));
            xyz2[i][j] = distance;
        }
    }


}
</code></pre>
<p>Any idea what might be the cause this time ?<br>
My guess is that it is coming from the arrays and how I am handling the C++ float **, but I am really not sure.</p>
<p>Yours faithfully</p>
<p>Justin L</p>
<p>PS : I am importing torch before the extension of course</p>

username: lelouedec
cooked: <p>I am just stupid.<br>
If anyone passes here, doing :</p>
<pre><code class="lang-auto">xyz2p.contiguous().data&lt;float&gt;();
</code></pre>
<p>returns a 1D array of the data, so you need to do your indexing as following :</p>
<pre><code class="lang-auto">void compute_distances(int n, int m,  at::Tensor  xyz1p, at::Tensor  xyz2p){

    auto  xyz1   = xyz1p.contiguous().data_ptr&lt;float&gt;();
    auto  start = xyz1;
    auto  xyz2   = xyz2p.contiguous().data_ptr&lt;float&gt;();
    auto start2 = xyz2;
    for (int i=0;i&lt;n;i++) {
        float x1 = *xyz1;
        float y1 = *(1+xyz1);
        float z1 = *(2+xyz1);
        xyz1+=3;
        for (int j=0;j&lt;n;j++) {
            float x2 = *((j*3)+start);
            float y2 = *((j*3)+1+start);
            float z2 = *((j*3)+2+start);
            float distance = sqrtf((x2-x1)*(x2-x1)+(y2-y1)*(y2-y1)+(z2-z1)*(z2-z1));
            if(j==i){
                *xyz2 = 1000000.0;
            }else{
                *xyz2 = distance;
            }
            xyz2++;
        }
    }



}
</code></pre>
<p>And it should work</p>
Defining class weights for BCE with multi-channel gaussian masks

username: YASJAY
cooked: <p>How to define class weights for multiclass BINARY_CROSS_ENTROPY_WITH_LOGITS with gaussian masks as targets?</p>

username: YASJAY
cooked: <p>anyone? like to know tips if this is possible.</p>

username: L-Reichardt
cooked: <p>Have a look at <a href="https://medium.com/gumgum-tech/handling-class-imbalance-by-introducing-sample-weighting-in-the-loss-function-3bdebd8203b4" rel="noopener nofollow ugc">this article</a>, I hope it will help and can be adapted to your use-case.</p>
Reducing the size of Pytorch datasets created with IamgeFolder in a Balanced fashion

username: samin_hamidi
cooked: <p>I have a data_set module created from TinyImageNet and I want to reduce the size of the training set in a balanced fashion so that each class has 100 data points instead of 500. I tried using</p>
<blockquote>
<p>torch.utils.data.Subset()</p>
</blockquote>
<p>but it does not change the dataset size in my setting. This is my code:</p>
<pre><code class="lang-auto">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import os.path
import torch.utils.data as data
from torchvision import transforms
from torchvision import datasets
from sklearn.model_selection import train_test_split


meta = {'rgb_mean':[0.485, 0.456, 0.406], 'rgb_std': [0.229, 0.224, 0.225]}



class TinyImageNet99199Base(data.Dataset):
    def __init__(self, inet):
        super(TinyImageNet99199Base, self).__init__()
        self._inet = inet

    def __getitem__(self, index):
        data, target = self._inet[index]

        example_dict = {
            "input1": data,
            "target1": target,
            "index": index,
        }

        return example_dict

    def __len__(self):
        return len(self._inet)


class TinyImageNet99199Train(TinyImageNet99199Base):
    def __init__(self, args, root):
        d = os.path.dirname(root)


        inet_0 = datasets.ImageFolder(
                        root,
                        transforms.Compose([

                            transforms.RandomHorizontalFlip(),
                            transforms.ToTensor(),
                            transforms.Normalize(mean=meta['rgb_mean'], std=meta['rgb_std']),
                        ])
                     )
        super(TinyImageNet99199Train, self).__init__(inet_0)

        
        X_train_indices, X_valid_indices, Y_train_indices, _ = train_test_split(range(len(inet_0)), inet_0.targets, stratify = inet_0.targets, 
        test_size=0.2, random_state=100, shuffle=True)
       
        inet = data.Subset(inet_0,  X_valid_indices)
      

class TinyImageNet99199Valid(TinyImageNet99199Base):
    def __init__(self, args, root):
        d = os.path.dirname(root)


        inet = datasets.ImageFolder(
        root,
        transforms.Compose([

            transforms.ToTensor(),
            transforms.Normalize(mean=meta['rgb_mean'], std=meta['rgb_std']),
        ]))
        super(TinyImageNet99199Valid, self).__init__(inet)
</code></pre>
<p>Any alternative solutions that I can try? intuitively this should work but maybe there is a bug here that I haven’t been able to find. Any help is very much appreciated!</p>

username: ptrblck
cooked: <p>In your example you are initializing <code>TinyImageNet99199Base</code> with <code>uper(TinyImageNet99199Train, self).__init__(inet_0)</code> before wrapping <code>inet_0</code> into <code>Subset</code> so <code>inet = data.Subset(inet_0,  X_valid_indices)</code> won’t be used at all.<br>
Apply the <code>Subset</code> first and pass it to the base class.</p>

username: samin_hamidi
cooked: <p>thank you. It is solved!</p>
Infiniband bandwith needed to scale with DDP

username: maxlacour
cooked: <p>Can anyone share experiences in terms of needed bandwith to scale with DDP. Is 25GBe enough or do we need 40GBe, 50GBe, 100GBe, 200 GBe? Our target is a small cluster of 4 nodes with 8 GPUs.</p>
<p>Is GPUDirect RDMA a must have for our setup?</p>
<p>What about GPUDirect storage? Is this needed?</p>
<p>Any other considerations we should make to scale well with pytorch DDP?</p>

username: H-Huang
cooked: <p>For such a small cluster I think 25 GBe should be enough, but it depends on the size of your model and the parameter gradients that need to be synchronized. GPUDirect RDMA and GPUDirect storage are not absolutely necessary to use DDP. It would be best to consider measuring the DDP forward, backward, and optimizer step delay, then find the bottleneck and see if you can optimize from there.</p>
ResNet50 + multi input model + torch concate

username: alib022
cooked: <p>Hello everyone,</p>
<p>I am working on a model to convert mono frontal image to a bird eye view.<br>
My plan is to use ResNet50 as a features extractor for each object and combine the features with coordinates of the bounding box and the output would be the bounding box coordinate in the bird eye view. I’ve built my model using following code:</p>
<pre><code class="lang-auto">class DriverBEV(nn.Module):
    """
    This model is designed to estimate the location of objects in the bird eye view
    using images and coordinates of objects in the frontal view
    """
    def __init__(self):
        
        super(DriverBEV, self).__init__()
        
        # we are using ResNet50 to extract object features
        self.resnet50 = models.resnet50(pretrained=True)
        
        
        # get the second-to-last layer to extract the features
        self.resnet50 = nn.Sequential(*list(self.resnet50.children())[:-1])
        
        # disable gradient computation
        for param in self.resnet50.parameters():
            param.requires_grad = False
            
        # starting to build a fully connected net to intake the coordinates
        self.fc1 = nn.Linear(4, 64)
        self.fc2 = nn.Linear(64, 256)
        self.fc3 = nn.Linear(256,500)
        
        
        self.fc4 = nn.Linear(2548, 1024)
        self.fc5 = nn.Linear(1024, 512)
        self.fc6 = nn.Linear(512, 256)
        self.fc7 = nn.Linear(256, 128)
        
        self.fc8 = nn.Linear(128, 4)
        
      
        self.dropout = nn.Dropout(0.25)
  

    def forward(self, image, coordinates):
        # extract img's features
        features = self.resnet50(image)
        
        coord_enco = self.dropout(F.relu(self.fc1(coordinates)))
        coord_enco = self.dropout(F.relu(self.fc2(coord_enco)))
        coord_enco = self.dropout(F.relu(self.fc3(coord_enco)))
        
        merge = torch.cat((features, coord_enco), 1)
        
        cood_dec = self.dropout(F.relu(self.fc4(merge)))
        cood_dec = self.dropout(F.relu(self.fc5(cood_dec)))
        cood_dec = self.dropout(F.relu(self.fc6(cood_dec)))
        cood_dec = self.dropout(F.relu(self.fc7(cood_dec)))
        
        cood_dec = self.fc8(cood_dec)
        
        return cood_dec
</code></pre>
<p>I am not sure if my model is correct, any suggestions/corrections would be appreciated.</p>
Caculating Hessian matrix and eigenvalues

username: Mari
cooked: <p>Hi,</p>
<p>I am trying to calculate Hessian matrix and eigenvalues/vectors for below neural network:</p>
<p>dataset = TensorDataset(x,y)</p>
<p>dataloader = DataLoader(dataset, batch_size=batchsize,shuffle=True,num_workers = 0,drop_last = False )<br>
class Net2_u(nn.Module):</p>
<pre><code>    #The __init__ function stack the layers of the 
    #network Sequentially 
    def __init__(self):
        super(Net2_u, self).__init__()
        self.main = nn.Sequential(
            nn.Flatten(),
            nn.Linear(input_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            
            nn.Linear(h_n,1),
        )
    def forward(self,x):
        output = self.main(x)
        return  output
</code></pre>
<p>net2_u = Net2_u().to(device)<br>
optimizer_u = optim.Adam(net2_u.parameters(), lr=learning_rate, betas = (0.9,0.99),eps = 10**-15)</p>
<p>for inputs, targets in dataloader:<br>
break<br>
inputs, targets = inputs.cuda(), targets.cuda()</p>
<p>for epoch in range (epochs):<br>
for batch_idx, (x_in,y_in) in enumerate(dataloader):<br>
net2_u.zero_grad()<br>
loss = criterion(x,y)<br>
loss.backward()<br>
optimizer_u.step()<br>
hessian_comp = hessian(net2_u, loss, data= (inputs, targets), cuda=True)<br>
top_eigenvalues, top_eigenvector = hessian_comp.eigenvalues()</p>
<p>I am getting this error:<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (256x1 and 2x70)</p>
<p>What is this error for?<br>
Can someone tell me how i am able to calculate the Hessian matrix for this kind of network?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html?highlight=hessian#torch.autograd.functional.hessian"><code>torch.autograd.functional.hessian</code></a> doesn’t accept a <code>data</code> argument so I guess you are using a custom implementation?<br>
Does your code work without this custom implementation or with the built-in one?</p>

username: Mari
cooked: <p>I imported hessian from PyHessian library.</p>
<p>Without this two lines, hessian_comp = hessian(net2_u, loss, data= (inputs, targets), cuda=True)<br>
top_eigenvalues, top_eigenvector = hessian_comp.eigenvalues(), my code works very well but when i want to calculate hessian matrix and add hessian line it gives the error i mentioned.</p>
<p>I tried torch.autograd.functional.hessian a few weeks ago but i can remember that i got error for this one too.</p>
Torch.nn.functional.one_hot not working as expected

username: aditya003
cooked: <p>Hi,</p>
<p>I was trying to convert a numpy array, as shown in the cell below, to a one-hot encoded vector. The shape of the input gt_arr is [N, H, W] and I wanted it to convert to [N,3,H,W] using the command below:</p>
<p><code>gt_arr_hot = nn.functional.one_hot(torch.from_numpy(gt_arr).to(torch.int64), num_classes=3).permute(0,3,1,2)</code></p>
<p>I get some weird output as show below:</p>
<p><strong>Input:</strong><br>
<code>np.unique(gt_arr) ---&gt; array([0, 1, 2], dtype=uint8)</code></p>
<p><strong>Output:</strong><br>
<code>gt_arr_hot.unique() ---&gt; tensor([ 0, 1, 281474976710656, 2251799813685248, 4503599627370496, 4503599627370497, 9007199254740992, 9007199254740993, 18014398509481984, 18014398509481985, 22517998136852480, 36028797018963968]</code></p>
<p>It doesn’t happen often but it happens randomly on some arrays. Am I doing something wrong here? Any help is much appreciated. Thank you in advance! and excuse me for my formatting skills.</p>
<p>Pytorch version 1.7.1</p>

username: ptrblck
cooked: <p>The output points towards an uninitialized tensor and looks like a bug.<br>
I’m not able to reproduce it using this code snippet:</p>
<pre><code class="lang-python">for _ in range(10000):
    N, H, W = 8, 24, 24
    gt_arr = np.random.randint(0, 3, (N, H, W))
    gt_arr_hot = nn.functional.one_hot(torch.from_numpy(gt_arr).to(torch.int64), num_classes=3).permute(0,3,1,2)
    
    if gt_arr_hot.unique().nelement() != 2:
        print('Error')
        print(np.unique(gt_arr))
        print(gt_arr_hot.unique())
</code></pre>
<p>Could you run it and check, if you are able to get these invalid outputs using it?</p>

username: aditya003
cooked: <p>Even I’m unable to reproduce the above results now. I’m not sure what caused this to happen but it was a strange result. I’ll post it here if I find something more or if I’m able to consistently reproduce the bad results. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: Yuhongze_Zhou
cooked: <p>I encountered the same issue. pytorch 1.7.1…</p>

username: aditya003
cooked: <p>I apologize for shooting this question here before thorough analysis. I ran memtest on my RAM and found out that several memory locations on one of my RAMs were corrupt and were returning wrong results during read operation, hence the unexpected output.</p>
Training loss decrease slowly

username: cbd
cooked: <p>Training loss decrease slowly with different learning rate. Optimizer used is adam. I tried with different scheduling scheme but it follow the same. I started with small dataset. If i need to converge loss i have to go with larger number of epochs but its time consuming.</p>
<p>One important point is, for all learning rates(0.01,0.001,0.0001,0.0008,0.008) loss follow the same pattern. On which parameter should i need to work to reduce loss faster?</p>

username: my3bikaht
cooked: <p>Adam is tricky sometimes and in my experience it likes smaller lr. Have you tried good old SGD?</p>

username: cbd
cooked: <p>Yes but loss value is too much high.</p>
Value_error: only one element tensors can be converted to Python scalars

username: Hasan_Khan
cooked: <p>While generating predictions for my trained RNN model, I was getting the</p>
<blockquote>
<p>too many values to unpack (expected 2)</p>
</blockquote>
<p>error. Which got resolved using <code>.item()</code> function. However, after resolving the above error now I am getting this  error.</p>
<blockquote>
<p>only one element tensors can be converted to Python scalars</p>
</blockquote>
<p>My <strong>prediction_loader</strong> instance consist of training data having the <strong>labels</strong> and <strong>training set</strong>.<br>
<strong>labels</strong> are denoted by <code>batch_y</code> and <strong>training set</strong> as <code>batch_x</code></p>
<p>Code and error snippets:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83d62f675d9338b7db43a058bbb6b85dd78c444a.png" data-download-href="https://discuss.pytorch.org/uploads/default/83d62f675d9338b7db43a058bbb6b85dd78c444a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83d62f675d9338b7db43a058bbb6b85dd78c444a.png" alt="image" data-base62-sha1="iOhraBapgVtnv1rZjn6E3ABdkbE" width="643" height="500" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">971×754 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: my3bikaht
cooked: <p><code>for batch in loader:</code><br>
if your loader returns both inputs <strong>and</strong> labels, batch will contain tuple (x,y), replace it with<br>
<code>for batch_x, batch_y in loader:</code><br>
also item() does not give you single tensor. It extracts value of that tensor in Python format (like simple float64). Backprop will not work on item()</p>
Convert Pytorch model into Keras

username: ShivaniA
cooked: <p>I am trying to convert YOLOV5 Pytorch model into Keras. While loading the pretrained model using torch.load getting the following error: ModuleNotFoundError: No module named ‘models’<br>
In pytorch hub YOLO is not available so converting or loading Yolo model is supported or not?</p>
<pre><code class="lang-auto">model_path = '/content/yolov5s.pt' 
# Load the weights 
state_dict = torch.load(model_path)
</code></pre>
<p>Thanks!</p>

username: creativeguitar
cooked: <p>this worked for me</p>
<pre><code class="lang-auto">model = torch.load(model_path, map_location="cpu").eval()
model.state_dict()
</code></pre>
<p>I am also looking on how to convert a pytorch pt file model into Keras.</p>