akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch 2-D and 3-D Table Interpolation

darthbark: <p>I am trying to get 2-D and 3-D interpolation table lookup running in pytorch, but I don’t believe <a href="https://pytorch.org/docs/stable/generated/torch.lerp.html" rel="noopener nofollow ugc">torch.lerp</a> supports it and haven’t been able to find any other pytorch native solution. Has anyone done a neural network approximation of a 2-D or 3-D linear interpolation table in pytorch? I am wondering if it is a worthwhile endeavor for speed at inference time (especially if the required size of NN to approximate a table is large) or if there is a simpler/more standard way that I missed.</p>
<p>For some background: I am trying to encode a known physical relationships that is in the form a linear interpolation table into a LSTM time series forecast system I am designing. The interpolation table queries would be done during training, so this isn’t simply a pre-processing step.</p>
Multiple Input Densenet

AllyBird: <p>Hi there,<br>
I’m trying to build a model that takes 2 images as inputs. I’ve tried out some simple examples to do this on the forum and those have worked well.</p>
<p>But I’m keen to scale this up to a more complex architecture like Densenet but I can’t find how to do this. Is anyone able to provide example or explain how one could make a Densenet that takes two inputs that are maybe concatenated a few layers in?</p>
<p>Thanks very much,<br>
Ally</p>
Strange time Consumption after forward with c++

hfengzhi: <p>i try to run this  operation after  the forward ,it  spend more than 100ms.it’s strange as the forward only need 2ms.<br>
i thought it’s because the gpu free memory take lot of time. I try to  do that  let it sleep one second after the forward ,it run fast.</p>
<pre><code class="lang-auto">boundingBoxesOfOne = torch::masked_select(boundingBoxesOfOne, mask).detach().cpu();
</code></pre>
<p>Is there any way to get him to run fast?<br>
I will be very grateful for any help.</p>

yf225: <p>When you measure the runtime of forward, did you use <code>cudaStreamSynchronize(...)</code> (which is equivalent to <code>torch.cuda.synchronize()</code> in Python)? This can affect runtime measurement because CUDA is by default asynchronous.</p>

narurajun: <p>Hello, I am experiencing the same issue, the operations takes 45ms for me, were you able to resolve it?</p>
Increase speed of loading RAW images

quangduyhcmut: <p>Hi,</p>
<p>I’m trying to reproduce results from paper Learning to see in the dark <a href="https://github.com/cchen156/Learning-to-See-in-the-Dark" rel="noopener nofollow ugc">this repo</a> using Pytorch. I have referred to implementations from <a href="https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch" rel="noopener nofollow ugc">this repo</a> and <a href="https://github.com/huyvnphan/Learning_To_See_In_The_Dark" rel="noopener nofollow ugc">this repo</a> but I had some issues.<br>
As far as I know, these repos load RAW images in two ways:</p>
<ul>
<li>Use PyTorch’s dataloader, read RAW image using rawpy package in custom <strong>getitem</strong>. In this way, I can use all the image’s pairs for training and validation but speed is extremely low because of these file are quite heavy. 1 epoch took me about 4200 secs to finish. (cydonia999’s implementation)</li>
<li>Load all images into a dictionary/array variable. Basically images are stored in RAM so this way require lots of RAM memory. Images only need to be loaded once. After that, traning phase and validation phase is done very quickly (about 30s-45s/1 epoch on V100 GPU).</li>
</ul>
<p>I prefer the second method than the first one because 2000 epochs with 1 epoch takes more than 1 hour could be a really long time. But due to the limited RAM I have using Colab Pro (25GB), I can not load all the images. Now, I’m trying to load half of the images, and random loading again after 100 epoch. Could this be the most suitable way for me? Or is there any way to speed up loading process using dataloader?</p>
<p>Best regards,</p>

JuanFMontesinos: <p>Some ideas here:<br>
What’s the image sizes? Can you resample them before hand?<br>
Have you considered using numpy arrays saving the array directly?¿<br>
Have u tried to use numpy memory map?</p>

quangduyhcmut: <p>Image’s shape is (1424, 2128, 4) with np.uint16.<br>
Currently I’m using numpy arrays to hold the images by allocating like this:</p>
<pre><code class="lang-auto">X = np.zeros((m_x, 1424, 2128, 4), dtype=np.uint16) 
Y = np.zeros((m_y, 1424, 2128, 3), dtype=np.uint16)
</code></pre>
<p>And then replace the zero value with the image’s value.</p>
<p>What is the benefit of using memory map?</p>
<p>Thank you for replying me!</p>

JuanFMontesinos: <p>Hmm you can create a big numpy array with all the images stacked together and then just read those images/fragments that you need. You save the required time to instantiate the array and so.</p>
<p>Another option you have is going for uint8 (would be reading half of the data)</p>
<p>Apart of that there is not much you can do.<br>
You may explore DALI library <a href="https://developer.nvidia.com/dali" class="inline-onebox" rel="noopener nofollow ugc">NVIDIA Developer Data Loading Library (DALI)</a><br>
which has a dataloader faster than pytorch’s but it’s not as simple to use.</p>

quangduyhcmut: <p>Thank you for suggestion!</p>
How should I finetune a pretrain model?

S_M: <p>Hi, I use a pre-train model like below</p>
<pre><code class="lang-auto">    model= Net().to(device)  
    model.load_state_dict(torch.load("cnn_alex.pkl"))
</code></pre>
<p>and I set <code>model. train()</code> before training but It doesn’t converge. Can anyone help me?</p>

hsubramani42: <p>Though you are using pretrained model, you have to like any other model like initializing optimizers with the model parameters and train model over a mini batches or dataset. Also, you can freeze the model parameters by iterating over batches using</p>
<pre><code class="lang-auto">for param in model.parameters: 
    param.requires_grad=False
</code></pre>
<p>and fine tune the last layers by enabling <code>requires_grad = true</code> or add few layers , to get better results.</p>

S_M: <aside class="quote no-group" data-username="hsubramani42" data-post="2" data-topic="119443">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hsubramani42/48/37263_2.png" class="avatar"> hsubramani42:</div>
<blockquote>
<pre><code class="lang-auto">for param in model.parameters: 
    param.requires_grad=False
</code></pre>
</blockquote>
</aside>
<p>Hi, I set the first layers with <code>requires_grad = False</code> using</p>
<pre><code class="lang-auto">    for name, param in model.named_parameters():
        if name == 'fc2.weight':
            break
        else:
            param.requires_grad_(False)
</code></pre>
<p>and leave the two last fully connected layers with <code>requires_grad = True</code>. Then I trained and tested the model in a normal way. It did not converge <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

zeros: <p>1.checkout your device whether on gpu<br>
2.if your traindata isn’t converged,checkout the tensor’type of <code>longint</code> or <code>float</code>,sometimes loss or acc will calcuated zero when labels set float type.</p>

Tejan_Mehndiratta: <p>See if this link is of any help, otherwise I can write a code and forward it to you.</p>
<p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
Is it possible to quantize less than 8-bit?

Ricky_96: <p>Hey all. I’ve taken a look at quantization recently for my final university project. I’ve seen that apparently PyTorch support at most 8-bit quantization. But is there any way to quantize my neural network to a lower precision (e.g. 4-bit or 2-bit)? Is it impossible instead? Please respond me.</p>

jerryzh168: <p>it’s not supported yet, responded here: <a href="https://discuss.pytorch.org/t/quantization-aware-training-lower-than-8-bits/118845" class="inline-onebox">Quantization aware training lower than 8-bits?</a></p>
Gradient norm of the last layer

Hiqmat_Nisa: <p>Hi,<br>
I have 6161 images of handwritten text lines. I am using batch size of 20 images. I am doing augmentation on every batch on the fly.<br>
Below is the image of the last fc layer gradient norm. I am also clipping the gradient norm at value 2.<br>
The question is, the stochastic gradient should decrease during training. right?  The spikes in the gradients is normal or there something wrong with the training?<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6c59b149f499a04bde4bed7277cabb47f7b17ed.png" alt="decoder.fc_out.weight-hybrideattention" data-base62-sha1="zd2PdNb75194ypsTIwxr1PXMncp" width="640" height="480"></p>
Torch.cuda.is_available() gives false for cuda11.3

Nathan_Martin: <p>Hi there,<br>
I am on WSL 2 Ubuntu 20.04<br>
Here is my torch cuda output, NVCC output, and nvidia-smi output</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2631ff07816875fe50f3ab5aaea6c92062bb3fd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a2631ff07816875fe50f3ab5aaea6c92062bb3fd" title="combine_images"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2631ff07816875fe50f3ab5aaea6c92062bb3fd_2_532x500.jpeg" alt="combine_images" data-base62-sha1="naxITcxyeYz6SPBhzi1ACn2HBsV" width="532" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2631ff07816875fe50f3ab5aaea6c92062bb3fd_2_532x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2631ff07816875fe50f3ab5aaea6c92062bb3fd_2_798x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2631ff07816875fe50f3ab5aaea6c92062bb3fd.jpeg 2x" data-dominant-color="454545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">combine_images</span><span class="informations">902×847 222 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have a 2070 super in my computer, any tips appreciated</p>
U-Net Segmentation - Dice Loss fluctuating

aswinshriramt: <p>Hi,</p>
<p>I am trying to build a U-Net Multi-Class Segmentation model for the brain tumor dataset. I implemented the dice loss using nn.module and some guidance from other implementations on the internet. But during my training, my loss is fluctuating and not converging. If I train my model using CrossEntropyLoss it is converging well. When I was debugging with the required_gradient it seems to be False for the output from the loss. I am unable to find out the issue here.</p>
<p>Thanks in Advance.</p>
<p>The loss implementation:</p>
<pre><code class="lang-auto">class DiceLoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceLoss, self).__init__()
        self.weights = weight

def forward(self, inputs, targets, eps=0.001):
        
        inputs = torch.argmax(F.log_softmax(inputs, dim=1), dim=1)
        inputs = F.one_hot(inputs, 5).float()
        targets = F.one_hot(targets, 5).float()
        intersection = (inputs.view(4,-1,5) * targets.view(4,-1,5)).sum(1)
        total = (inputs.view(4,-1,5) + targets.view(4,-1,5)).sum(1)
        dice = (2 * intersection + eps)/(total + eps)
        diceLoss = 1 - (dice.mean(0) * self.weights).sum()/self.weights.sum()

        return diceLoss
</code></pre>

ptrblck: <p><code>torch.argmax</code> is not differentiable and would thus detach the output from the computation graph.<br>
This should also yield an error such as:</p>
<pre><code class="lang-python">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>so I’m not sure if you are rewrapping the loss in a new tensor or why it’s not raised.<br>
I’m not sure where this implementation comes from, but note that <a href="https://kornia.readthedocs.io/en/v0.1.2/_modules/torchgeometry/losses/dice.html">other implementations</a> are using <code>softmax</code> to calculate the probabilities for each class.</p>

aswinshriramt: <p>Yes, I got the same error you posted here.</p>
<p>Ok, yeah the argmax seems to detach the node from the graph. Let me try it with F.Softmax.</p>
<p>Thank you!</p>

aswinshriramt: <pre><code class="lang-auto">class IoULoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(IoULoss, self).__init__()
        self.weights = weight

    def forward(self, inputs, targets, eps=0.001):
        inputs = nn.Softmax(dim=1)(inputs)
        targets = F.one_hot(targets.long(), 5).float()

        intersection = torch.sum((inputs.permute(0,2,3,1)*targets), (1,2))

        total = torch.sum((inputs.permute(0,2,3,1) + targets), (1,2))

        union = total - intersection

        IoU = (intersection + eps)/(union + eps)

        iouLoss = 1 - (IoU.mean(0) * self.weights).sum()/self.weights.sum()

        return iouLoss
</code></pre>
<p>Thank you. It worked now. But unfortunately, the model doesn’t seem to be learning from this loss.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea7d814b7f3ea9122a8e07705a852dad6c1e7a98.jpeg" alt="image" data-base62-sha1="xsoCwMHC6ZZckKtpMfOB2s7OZdu" width="665" height="426"></p>
Torch.lstsq output size incorrect

mlizhardy: <p>I have a A, 10-by-6  source tensor, and a b, 10-by-1 destination tensor.<br>
I want to solve for x such that Ax = b.  x should be a 6-by-1 tensor (or a 1-by-6 since documentation says the transform is returned)</p>
<pre><code class="lang-auto">import torch

A = torch.ones((10,6))
b = torch.ones((10,1))
M, _ = torch.lstsq(b, A)
print(M.shape)
</code></pre>
<p>output: torch.Size([10, 1])</p>
<p>What’s the problem here and how do I get my proper 6-by-1 tensor output??</p>

mlizhardy: <p>answered here: <a href="https://github.com/pytorch/pytorch/issues/56833" class="inline-onebox" rel="noopener nofollow ugc">torch.lstsq returns wrong tensor size · Issue #56833 · pytorch/pytorch · GitHub</a></p>
What are the groups in distributedDataParalel and why do we need it

dang-qi: <p>I checked the <a href="https://github.com/facebookresearch/detectron2/blob/c1689c7d7b1c1af0157cd90b84f296b7b2ce2124/detectron2/engine/launch.py#L121" rel="noopener nofollow ugc">code</a> in detectron2 and I found that they build new group for each machine. I haven’t learned much about the distributed system and I am just curious about why do they do that. I tried to search for it but I couldn’t find it. Can anyone explain this?</p>

pritamdamania87: <p>I would recommend reading the following documentation: <a href="https://pytorch.org/docs/stable/distributed.html#groups" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.8.1 documentation</a></p>
Loss doesn't decrease if model is switched to use GroupNorm instead of BatchNorm

Victory: <p>I have a network that trains fine while using BatchNorm. If I keep everything else same and just change the normalization from BatchNorm to GroupNorm, I am noticing that the training fails. That is, the loss pretty much stagnates at a high value and doesn’t go down at all. But, if I switch to BatchNorm, the loss keeps steadily going down. Any idea what might be the reason for such a big difference between BatchNorm and GroupNorm behaviors?</p>
<p>I am using PyTorch 1.7.1.</p>
<p>Searching the forum, I found the following where someone else also reports loss not going down with GroupNorm. But, in that case, the issue could potentially be related to the way the model is being changed. In my case though, I am just changing the code to use GN and start training afresh.</p><aside class="quote quote-modified" data-post="1" data-topic="108865">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/53a042/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/gradients-not-computed-after-replacing-batchnorm-with-groupnorm/108865">Gradients not computed after replacing BatchNorm with GroupNorm</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I’m fine-tuning Resnet18, and wanted to try replacing BatchNorms with GroupNorms to see if I can do smaller batches, but somehow, none of the weights in the network are updating when I replace BatchNorms with GroupNorms. 
Here is my code to replace BatchNorms: 
def convert_bn_model_to_gn(module, num_groups=16):
    mod = module
    if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
        num_groups = 1
        mod = torch.nn.GroupNorm(num_groups, module.num_features,
               …
  </blockquote>
</aside>
Which One is the Best Way to Create Training Sequences for LSTM-based Class Prediction on Time-series Data?

bbasaran: <p>Let’s say I have time-series data in the following way. I need to create training sequences of a fix length as an input to my LSTM model on PyTorch.</p>
<pre><code class="lang-auto"># Time, Bitrate, Class
  0.2,  312,     1
  0.3,  319      1
  0.5,  227      0
  0.6,  229      0   
  0.7,  219      0    
  0.8,  341      1    
  1.0,  401      2      
</code></pre>
<p>Once trained, the model should be able to predict classes of test data without classes. I am not sure how to create the training sequences. Let’s say I define a sequence_length = 3 for the data snippet above. I have two approaches:</p>
<ol>
<li>By shifting a window of sequence_length. In this case, the sequences:</li>
</ol>
<pre><code class="lang-auto"># Sequence,                    Label of the Sequence
  [(312,1), (319,1), (227,0)]  0
  [(319,1), (227,0), (229,0)]  0
  [(227,0), (229,0), (219,0)]  0
  [(229,0), (219,0), (341,1)]  1
  [(219,0), (341,0), (401,2)]  2
</code></pre>
<p>The label of the first sequence is 0 (since the last element’s class is 0). However, this seems not okay, because even if the last label is 0, the former two are not.</p>
<ol start="2">
<li>Fetching class by class up to sequence_length. if any of them is less than the sequence_length, I pad with the last element (since all tensors should have the same shape):</li>
</ol>
<pre><code class="lang-auto"># Sequence,                   Label of the Sequence
 [(312,1), (319,1), (319,1)]  1   &lt;- padded with the last element (319,1)
 [(227,0), (229,0), (219,0)]  0
 [(341,1), (341,1), (341,1)]  1   &lt;- padded with the last element (341,1)
 [(401,2), (401,2), (401,2)]  2   &lt;- padded with the last element (401,2)
</code></pre>
<p>Instead of right-padding with the last element, would zero padding from the left make more sense? If so, do I need to do something to make LSTM ignore those padded zeros?</p>
<p>Thanks!</p>
How to set a limit to gpu usage

Jean-Porte: <p>Hi,<br>
with tensorflow I can set a limit to gpu usage, so that I can use 50% of gpu and my co-workers (or myself on another notebook) can use 50%</p>
<p>I just have to do this:</p>
<pre><code class="lang-auto">config = tf.ConfigProto(gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=0.7))
sess = tf.InteractiveSession(config=config)
</code></pre>
<p>Do you know how to do this with pytorch ?</p>
<p>Thanks</p>

smth: <p>there is no way to do this in pytorch. However, PyTorch doesn’t pre-occupy the GPU’s entire memory, so if your computation only uses 50% of GPU, only that much is locked by PyTorch</p>

Bruno_Barbarioli: <p>Is this still the case?</p>

simonp: <p>In case someone still stumbles upon this question: With PyTorch 1.8, there is <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.set_per_process_memory_fraction" rel="noopener nofollow ugc">set_per_process_memory_fraction</a></p>
Transfer Learning with ResNet50 Out of Memory Error Attempting to get 2048-element outputs

nicaushaz: <p>Hello,</p>
<p>I am trying to extract 2048-element representations of fMNIST from a pre-trained resnet50. I need to store these vectors, presumably in a [60000,2048] data structure(currently trying tensors) in order to use them in subsequent experiments. My workflow is as below:</p>
<ol>
<li>Load fMNIST from torchvision</li>
<li>Create dataloaders</li>
<li>Loop through batches (for loop on a dataloder) extracting representations:<br>
- Inside the loop I call repeat_interleave to convert the gresycale channel to 3D (essentially copying the channel 3 times not doing any conversion)<br>
- create a temp variable storing those representations (batch_size, 2048)<br>
- send temporary output to cpu (because apparently it won’t even do 1 iteration on GPU)<br>
- store out in the final tensor.</li>
</ol>
<p>After 4 iterations of batch size 64, a Tesla K4 from colab with 15GB crashes with out of memory error. Trying 1 iteration and checking memory with nvidia-smi shows 6.7 GB of GPU memory reserved. It might be naive but I would assume that this would be the memory needed at each iteration, but apparently it isn’t so?<br>
How can I resolve this issue? What is the pytorch-way of generating such outputs from a pretrained model and storing them?</p>
<p>PS I have tried smaller batches and it works for a couple of extra iterations but it still crashes. I am assuming that intermediate temporary variables are created and not discarded at each loop. Also, tried to tackle this by calling:</p>
<pre><code class="lang-auto">del out
gc.collect()
torch.cuda.empty_cache()
</code></pre>
<p>but to no avail…any help would be greatly appreciated. Thank you.</p>

ptrblck: <p>Based on the description it seems that you are storing tensors in e.g. a <code>list</code>, which are still attached to the computation graph. Thus you wouldn’t only store the tensors, but the entire graph with all intermediate tensors in each iteration.<br>
If you don’t need to call <code>backward()</code> on the stored tensor or any tensor created from it, you could <code>detach()</code> the tensor before storing it in the <code>list</code> (or any other container).</p>

nicaushaz: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> . I solved this by calling .detach().cpu().numpy() on the output of the model.</p>
Writing my own derivative formula

se3: <p>Hi,</p>
<p>Is there any way to supply a derivative formula to the Autograd engine? I want to build a network and compute gradients using a custom formula. Any help are greatly appreciated!</p>

JuanFMontesinos: <p><a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a><br>
You just have to overwrite the backward function in the <code>nn.Module</code></p>

albanD: <p>Note that the backward function is not on the <code>nn.Module</code> but on an <code>autograd.Function</code>!<br>
You have more details here as well: <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox">Extending PyTorch — PyTorch 1.8.1 documentation</a></p>

JuanFMontesinos: <p>Hups! Have to say never tried it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  thanks for the clarification.</p>
Vgg16 model validation accuracy is stuck

P0t4to042: <p>Hi, I am working on a CNN model for MRI brain images classification (Alzheimer disease), I use transfer learning method for image classification - vgg16 model trained on ImageNet (1000 classes). I’ve modified last layer so the size of each output sample is set to 2.</p>
<p>I am using SGD optimizer, weighted CrossEntropyLoss as loss function because dataset is not balanced and learning rate scheduler - StepLR.</p>
<p>I encountered a problem with validation accuracy that stays (almost) the same or is decreasing (approximately 70% whereas state of the art in this topic is around 95-99%), I thought that the problem is overfitting but then I’ve tried to use L2 regularization, data augmentation and dropout but nothing helped. MRI images are normalized.</p>
<p>Here is a code of my training function</p>
<pre><code class="lang-auto">def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  
            else:
                model.eval()   

            running_loss = 0.0
            running_corrects = 0

            for inputs, masks, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs.float())
                    _, preds = torch.max(outputs, 1)

                    loss = criterion(outputs, labels)

                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            if phase == 'val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    model.load_state_dict(best_model_wts)
    return model
</code></pre>
<p>I am training the model for 100 epochs.<br>
Learning rate is 0.001</p>
<p>I would appreciate any help. Thanks in advance</p>
How to update a Pytorch variable and module parameters at the same time?

Jiansong_Zhang: <p>This is my Network structure:</p>
<pre><code class="lang-auto">
class Generator(nn.Module):
    def __init__(self, latent_size=128): # 128 &lt;&lt; 784 = 28*28
        super().__init__()
        self.latent_size = latent_size
        self.inputlayer = nn.Sequential(
            nn.Linear(latent_size, 25),
            nn.Tanh(),
        )
        self.hidden1 = nn.Sequential(
            nn.Linear(25, 100),
            nn.Tanh(),
        )
        self.hidden2 = nn.Sequential(
            nn.Linear(100, 200),
            nn.Tanh(),
        )
        self.deconv_out = nn.Linear(200, 784) # 784 = 28 * 28 for mnist

    def forward(self,input):
        net = self.inputlayer(input)
        net = self.hidden1(net)
        net = self.hidden2(net)
        net = self.deconv_out(net)
        return self.transform(net).view(-1,1,28, 28)


class DataGenerator(Generator):
    def __init__(self, latent_size=128):
        super().__init__(latent_size=latent_size)
        self.transform = lambda x: torch.sigmoid(x)
</code></pre>
<p>I was trying to pass in a latent variable Z from uniform distribution, and update Z and the network parameters during the training time.</p>
<pre><code class="lang-auto">nz = 128   # dimensionality of the latent code
beta = 0.1

data_gen = DataGenerator().to(device)
batch_size = 64
Z = torch.empty(batch_size, nz, device=device)
Z = Z.uniform_()
Z = Variable(Z,requires_grad=True)

lrate = 1e-4
data_gen_optimizer = optim.Adam(
    data_gen.parameters(), lr=lrate, betas=(0.5, 0.9))
</code></pre>
<p>I was trying to pass [Z,data_gen.parameters()] together to the optimizer at the same time, it gives the following error:</p>
<pre><code class="lang-auto">data_gen_optimizer = optim.Adam(
    [data_gen.parameters(),Z], lr=lrate, betas=(0.5, 0.9))
optimizer can only optimize Tensors, but one of the params is Module.parameters
</code></pre>
<p>So can anybody gives advice and help on how to let the optimizer to update the module parameters and Z at the same time?</p>

albanD: <p>Hi,</p>
<p><code>data_gen.parameters()</code> returns a generator. So you most likely want to do something like: <code>list(dat_gen.paramters()) + [Z,]</code></p>
How to deactivate at convenience the effects of forward pre hooks during optimization?

miguelalba96: <p>Hello, I have a model which has attached to its weights something pretty similar to (weight normalization) <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/weight_norm.py at master · pytorch/pytorch · GitHub</a> . In weight normalization the parameters are transformed into <code>w = g/||v||*v</code> and you update g and v during training.</p>
<p>suppose my parameters are a linear combination: <code>w* = aw + b</code> (where some pre-rained weights w are frozen,  <code>a</code> and <code>b</code> are initialized in ones and zeros respectively; a and b require grads), When I do a forward/backward pass with the model using the hooks I get the gradients → <code>grad(loss(aw + b)))</code>. I would like to get <code>grad(loss(b))</code> → (the gradients of the model containing only the parameters b).</p>
<p>Currently to do that I make a copy of the network before adding the hooks with <code>register_forward_pre_hook</code> and initialize its weights as 0. then I copy the parameters b in every update from the big model (w* = aw + b) and paste it in that initial copy without hooks, then do a forward/backward pass to get <code>grad(loss(b))</code>. My problem is related to the efficiency, I would like to know if there is a way to suppress the effect of a and w during a new forward/backward computation so I could get the gradients of a network only using b during the optimization.</p>
<p>This moreover what I do:</p>
<p>model 1 : <code>w* = aw + b</code>   (w pre-trained, a as ones, b as zeros)<br>
model 2 : b  (b as zeros)</p>
<ol>
<li>forward on model 1</li>
<li>Take the state dict from model 1 and paste only the parameters b in model 2</li>
<li>forward/backward on model 2 and get the gradients <code>grad(loss(b))</code>
</li>
<li>Update model 1 parameters</li>
</ol>
Problem with newest version of libtorch

Willem: <p>I followed the tutorial here to make a c++ front-end application for pytorch:</p>
<pre><code>https://pytorch.org/tutorials/advanced/cpp_frontend.html
</code></pre>
<p>I used GPU version. I could succesfully cmake and make.</p>
<p>Now I updated to the latest version of libtorch. I can still cmake, but on make, I get the following error:</p>
<pre><code>[willem@int2 build]$ make
make[2]: *** No rule to make target `/sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/cuDNN/7.6.5.32-CUDA-10.1.243/include/lib64', needed by `polIter'.  Stop.
make[1]: *** [CMakeFiles/polIter.dir/all] Error 2
make: *** [all] Error 2
</code></pre>
<p>Any ideas on what is causing this?</p>
Adjusting parameters in the forward pass

shreyd: <p>Given some arbitrary model, I would like to manipulate each parameter p before it is involved in a computation. I’m interested in multiplying each p by some learnable constant c. I can’t manually adjust each parameter after the forward pass is completed because then the learnable constant would not be a part of the computation graph. Ideally, the operation p*c would happen before p is used in the forward pass. Then, during the backwards pass, both p and c would receive a gradient wrt the loss.</p>
<p>I looked into hooks to achieve this, but all the posts I have seen so far just use hooks to print out the underlying tensor or gradient information. Is there a way to use hooks in the forward pass to adjust parameters or is this not possible? I <em>could</em> manually define each parameter in the network, and then have control over it; however, my current model has a lot of convolutional layers, so I would prefer not to do this manual work.</p>

MariosOreo: <aside class="quote no-group" data-username="shreyd" data-post="1" data-topic="40326">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shreyd/48/10179_2.png" class="avatar"> shreyd:</div>
<blockquote>
<p>Ideally, the operation p*c would happen before p is used in the forward pass. Then, during the backwards pass, both p and c would receive a gradient wrt the loss.</p>
</blockquote>
</aside>
<p>Hello,</p>
<p>I tried to define hooks to achieve it but encountered some problem.</p>
<pre><code class="lang-auto">c = nn.Parameter(torch.randn(1), requires_grad=True)
opt = torch.optim.SGD(model.parameters(), lr=0.001)
opt.add_param_group({'params': c})

def hooks(module, input):
    module.weight = nn.Parameter(module.weight * c)

for module in model.children():
    if isinstance(module, nn.Conv2d):
        module.register_forward_pre_hook(hooks)
</code></pre>
<p><code>c</code> is defined as <code>nn.Parameter</code> and if we want to modify <code>module.weight</code> we must pass <code>nn.Parameter</code> or <code>None</code> to it, then registered it to each child_module.<br>
<strong>But</strong>, c is still not a learnable param, and what I found is:</p>
<ul>
<li>module.weight is leaf variable, it is okay since it used to update.</li>
<li>c is also leaf variable, it means that gradient flow will not pass back to c? and c is not included in the computation graph?</li>
</ul>

miguelalba96: <p>you can do something similar to weight normalization <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/weight_norm.py at master · pytorch/pytorch · GitHub</a> , it will involve in every forward/backward pass those learnable constants</p>
AttributeError: module 'torch.utils' has no attribute 'data'

zuoxingdong: <p>This runtime error is resulted from</p>
<pre><code class="lang-auto">import torch
torch.utils.data.DataLoader

-&gt;

AttributeError: module 'torch.utils' has no attribute 'data'
</code></pre>

smth: <pre><code class="lang-python">import torch
import torch.utils.data
</code></pre>

cakeeatingpolarbear: <p>Might be a bit off topic, but do we ned to do this?</p>

davidavdav: <p>Thanks, I ran into the same problem.</p>
<p>I suppose <code>torch/utils/__init__.py</code> does not contain an <code>import data</code> or something along these lines.</p>

tengerye: <p><a class="mention" href="/u/smth">@smth</a> May I know the reason why do we have to <code>import torch.utils.data</code> instead of using <code>torch.utils.data.DataLoader</code> directly please?</p>

theGuyWithBlackTie: <p>You can do that as well. In that case just do like this:</p>
<pre><code class="lang-auto">import torch.utils.data.DataLoader as &lt;something&gt;
</code></pre>
<p>You have to fill <code>something</code> with whatever word you like</p>
Try to implement Adaptive Batch Norm for GAN

RamSh: <p>My task has the following description:</p>
<p>We condition on the noise vector multiple times throughout the network. One of the most popular ways of doing that is via adaptive batch normalization:</p>
<p>𝑥=𝑥−𝜇𝜎𝛾+𝛽,𝛾=𝑓(𝑧), 𝛽=𝑔(𝑧)</p>
<p>The first part of this operation is a standard batch normalization, but instead of optimizing  𝛾  and  𝛽  as a vector, we optimize functions  𝑓  and 𝑔 , which predict affine parameters from a noise vector 𝑧 . Typically these functions are simple linear mappings.</p>
<p>I don’t know how to do that. I am pretty new to PyTorch. Please, help me.</p>
PyTorch Evaluation function with Mask R-CNN does not work

aliceinland: <p>Hi, I’m facing a problem with the evaluation method.<br>
I’m following the tutorial available in PyTorch for finetuning a Mask-RCNN model with my own dataset (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.8.1+cu102 documentation</a>).<br>
The training does not have any problem but when the evaluation process starts it gets stuck. There are no errors, nothing that can suggest the reason for the one this is not working.<br>
As code, I have used the one available in the PyTorch Github repository, also used in the tutorial mentioned above (<a href="https://github.com/pytorch/vision/blob/dc42f933f3343c76727dbfba6e4242f1bcb8e1a0/references/detection/engine.py" class="inline-onebox" rel="noopener nofollow ugc">vision/engine.py at dc42f933f3343c76727dbfba6e4242f1bcb8e1a0 · pytorch/vision · GitHub</a>), the evaluation function one. I modified it by removing the CPU’s use and using only the GPU, a Tesla T4. Any suggestion on how to fix the problem or which could be the reason for this behaviour? Considering that there are no bottlenecks that could interfere with the model’s behaviour by following the GPU usage.</p>
How to change the CMakefile if I call python from c++ code?

Lily_Liu: <p>Hi, I am trying to modify the “torch/csrc/autograd/saved_variable.cpp” file, I call some python function in the file. I made the following modification:</p>
<p><span class="hashtag">#include</span> &lt;torch/csrc/python_headers.h&gt;<br>
<span class="hashtag">#include</span> &lt;pybind11/pybind11.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/csrc/utils/object_ptr.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/csrc/autograd/python_variable.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/csrc/Exceptions.h&gt;<br>
// original include<br>
SavedVariable::SavedVariable(const Variable&amp; variable, bool is_output, bool is_inplace_view)  {<br>
auto mymodule = THPObjectPtr(PyImport_ImportModule(“mymodule”));<br>
// prepare inputs<br>
// call corresponding python functions<br>
// get c++ object from python results<br>
}</p>
<p>Which cmake file should I change to make it compile? If I do not change anything, it will throw the error “/home/ec2-user/pytorch/build/lib/libtorch_cpu.so: undefined reference to `PyInstanceMethod_Type’”<br>
I think I need to link the python target and include the python lib somewhere, but where should I put it? Thanks a lot!</p>
Pic should be Tensor or ndarray. Got <class ‘dict’>

moogliecake: <p>**I tried in previous discussions but I need to transform both the keypoint values ​​and the photo in same size</p>
<p>like the tutorial, I cannot extract the data I have given as a sample-dict to examine it after I have converted it and I cannot feed it into the model. Where exactly is the problem?</p>
<pre><code class="lang-auto">My transform code :
transform = transforms.Compose([transforms.ToPILImage(),transforms.Resize(256),transforms.RandomCrop(224),transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])])
</code></pre>
<p>and</p>
<pre><code class="lang-auto">transformedDataset = keypointData(csv_path=keypoint_path_training,image_path=image_train,transform=transform)
print('Number of transformed images: ', len(transformedDataset))
valid = int(2*len(transformedDataset)/10)
train_set, val_set = torch.utils.data.random_split(transformedDataset, [len(transformedDataset)-valid, valid])
train_set = torch.utils.data.DataLoader(train_set,batch_size = 16,shuffle = True,num_workers=0)
val_set = torch.utils.data.DataLoader(val_set,batch_size = 16,shuffle = True,num_workers=0)

inputsTrain= next(iter(train_set))
inputsValid = next(iter(val_set))
</code></pre>
<blockquote>
<p>pic should be Tensor or ndarray. Got &lt;class ‘dict’&gt;.</p>
</blockquote>
<p>i try to convert numpy array but its not running</p>

ptrblck: <p>I’m not sure, if you are following <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">this tutorial</a>, but if so, note that the transformations are taken from <a href="https://github.com/pytorch/vision/blob/d4195587166134c3806ae81458d08b06f5e00295/references/detection/transforms.py"><code>references/detection/transforms.py</code></a> as described in the tutorial, so you might need to use them as well.<br>
The difference to <code>torchvision.transforms</code> is that these transformation accept the <code>image</code> and <code>target</code> tensors.</p>
WeightedRandomSampler in PyTorch_Geometric Dataloader

Jop_Verbeek: <p>Hello everyone,</p>
<p>I am writing a program to perform graph classification using PyTorch Geometric, however I have a very imbalanced dataset. I was wondering if I can easily implement a method like WeightedRandomSampler in normal PyTorch?</p>
<p>Thanks in advance!</p>

ptrblck: <p>I’m not familiar with PyTorch Geometric, but in case you can precompute the targets and each sample has one corresponding target class, you could take a look at <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this code snippet</a>, which explains the usage of the <code>WeightedRandomSampler</code>.</p>
Code strangely sometimes works and sometimes doesn't

Ricky_96: <p>Hi guys. My code is simply a “variation” of this code here: <a href="https://github.com/leimao/PyTorch-Quantization-Aware-Training/blob/main/cifar.py" rel="noopener nofollow ugc">https://github.com/leimao/PyTorch-Quantization-Aware-Training/blob/main/cifar.py</a> with the difference that I’m using MNIST dataset instead of CIFAR-10 and a different model (AlexNet). I don’t know why but sometimes accuracy is very high (starting from almost 91%) while sometimes it’s simply just very low (11%). If you’re asking i’ve changed the train_transform and test_transform according to the MNIST dataset and set everything accordingly to this. Please help me i’m freaking out</p>

ptrblck: <p>Depending on your use case, model, etc. the training might be sensitive to e.g. the initialization of the model as well as other hyperparameters.<br>
You could try to use other init methods from <code>torch.nn.init</code> or play around with other hyperparameters such as the learning rate (maybe it’s too high and lets the model diverge in a lot of cases).</p>
Get topk results from segmentation one-hot masks, but keep dims

CBGxd: <p>I’m not sure someone asked it before…<br>
Says we have a one-hot segmentation mask with BxNxHxW, where B=batch, N=class-categories, H=height, W=width of the mask. I’m trying to find the top 3 maximize results from each one-hot <em>element-wise</em> vector of the map, and let other unselected elements be 0 (which means we want the dimension not to change).<br>
It is like a keepdim version of the torch.topk.</p>
<p>I didn’t find a way to solve this…   I appreciate any advice in here, thanks.</p>

KFrank: <p>Hi CBGxd!</p>
<aside class="quote no-group quote-modified" data-username="CBGxd" data-post="1" data-topic="119352" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbgxd/48/14153_2.png" class="avatar"> CBGxd:</div>
<blockquote>
<p>I’m trying to find the top 3 maximize results from each one-hot vector, and let others be 0 (which means we want the dimension not to change).</p>
</blockquote>
</aside>
<p>I’m not at all sure what you are asking here.  A <em>one-hot</em> vector is a<br>
vector that consists of all zeroes except for one one.  So it doesn’t<br>
make sense to ask for the three largest values (or <code>topk()</code> in general)<br>
in a one-hot vector.</p>
<p>Could you illustrate what you are asking with explicit numerical tensors?</p>
<p>Best.</p>
<p>K. Frank</p>

CBGxd: <p>Thanks for your feedback.<br>
Let’s just ignore the batch issue in here.<br>
Says we have a segmentation map, which is 320x320. And the model’s prediction could be in a one-hot fashion, which is 21x320x320 (the 21 is the class category).<br>
At the moment, I want to find the top 3 maximize elements along the first dimension (i.e., 21), for the entire map. Meanwhile, I want all the other’s non-selected elements filled to zero.  Therefore, the result remains the same architecture, as 21x320x320.<br>
It will be more like a topk mask for the original one-hot result.</p>

CBGxd: <p>I think I find the solution. To anyone who might encounter similar issues, it works like this:<br>
1). find the topk index of the one-hot segmentation mask:<br>
_, index = torch.topk(one-hot-result, k= 3, dim=1)<br>
2). expand to the desired class:<br>
expand = torch.nn.functional.one_hot(index.squeeze())<br>
topk_mask = expand.sum(dim=1)  # note: due to we have k maximize value, I deal it as multi-label<br>
3). multiply with original one-hot result:<br>
topk_result = torch.mul(topk_mask, one-hot-result)</p>
KL Divergence for Multi-Label Classification

arjung: <p>I need to use KL Divergence as my loss for a multi-label classification problem with 5 classes (Eqn. 6 of this <a href="https://arxiv.org/pdf/2005.02231.pdf" rel="noopener nofollow ugc">paper</a>). I have soft ground truth targets from a teacher network of the form [0.99, 0.01, 0.99, 0.1, 0.1] for each sample (since its a multi-label problem, a sample can belong to multiple classes), and predictions which are five probabilities that <em>don’t</em> sum up to 1.</p>
<p>How can the KL divergence be computed? It seems like nn.KLDivLoss() expects the probabilities to add up to 1 – otherwise, I get a negative loss.</p>

KFrank: <p>Hi Arjung!</p>
<aside class="quote no-group" data-username="arjung" data-post="1" data-topic="118884" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arjung/48/13035_2.png" class="avatar"> arjung:</div>
<blockquote>
<p>I need to use KL Divergence as my loss for a multi-label classification problem with 5 classes</p>
</blockquote>
</aside>
<p>The short answer is use <code>BCEWithLogitsLoss</code>.</p>
<p>Several comments:</p>
<p>KL divergence and cross entropy are closely related.  For fixed<br>
targets, KL divergence and cross entropy differ by a constant that<br>
is independent of your predictions (so it doesn’t affect training).</p>
<p>You should understand your multi-label, 5-class task to be a set of<br>
5 binary classification tasks (that share the same network). That is,<br>
class-0 can be “yes” or “no”, class-1 can be “yes” or “no” (independent<br>
of the value for class-0), and so on.</p>
<p>So, whether you call it KL divergence or cross entropy, you want to<br>
calculate the loss for each of your 5 binary problems separately, and<br>
then add (or average) them together.  You could certainly do this<br>
with <code>KLDivLoss</code> (with a loop), but <code>KLDivLoss</code> isn’t set up to do this<br>
automatically.  <code>BCEWithLogitsLoss</code> is.</p>
<p>For <code>BCEWithLogitsLoss</code> you want your predictions (the output of your<br>
model) to have shape <code>[nBatch, nClass = 5]</code> and be raw-score <em>logits</em>,<br>
rather than probabilities.  (So don’t pass your logit predictions through<br>
<code>softmax()</code> to convert them to probabilities; <code>softmax()</code> is, in effect,<br>
built into <code>BCEWithLogitsLoss</code>.)  Your targets should also have shape<br>
<code>[nBatch, nClass = 5]</code> and should be the probabilities of each of<br>
your samples being (independently) in each of your 5 classes.</p>
<p>(And to confirm, <code>BCEWithLogitsLoss</code> does accept “soft” targets that<br>
are probabilities between 0.0 and 1.0, rather than requiring “hard”<br>
0-1 targets.)</p>
<p>Best.</p>
<p>K. Frank</p>

arjung: <p>Thanks a lot for such a detailed response! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>