akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Functorch's per-sample gradients and Opacus

tholop: <p>Hi, PyTorch recently released the first version of <a href="https://pytorch.org/functorch/stable/" rel="noopener nofollow ugc">functorch</a>. This is exciting because functorch makes it easy to compute per-sample gradients, like in JAX. They even have a <a href="https://pytorch.org/functorch/0.1.0/notebooks/per_sample_grads.html" rel="noopener nofollow ugc">dedicated tutorial about per-sample gradients with functorch</a>.</p>
<p>Although functorch is still in beta, I am curious about the implications for Opacus. Do you have any plans to use functorch? Do you think it can improve the efficiency of the GradSampler compared to the existing hand-crafted gradient hooks? (in addition to making it easier to support different types of layers, of course)</p>
<p>Thanks!</p>

ffuuugor: <p>Very good question! We were waiting for functorch release to assess the current state, so will investigate now.<br>
If you have any insights or thoughts - they’d be very welcome</p>

tholop: <p>Thanks, that’s good to hear!</p>

alexandresablayrolle: <p>As an aside, we are looking for cases where Opacus is much slower than JAX. In the benchmarks we know, Opacus 1.0 often comes very close to JAX in terms of speed (at most 20% slower) despite not enjoying the power of vmap.</p>
<p>If you know of cases where that’s not true, we would be happy to take a look <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">.</p>
Cpuinfo: CMake `find_package` support

mtrofin: <p>Hello,</p>
<p>there’s an open PR, <a href="https://github.com/pytorch/cpuinfo/pull/69" class="inline-onebox" rel="noopener nofollow ugc">Support CMake find_package by luncliff · Pull Request #69 · pytorch/cpuinfo · GitHub</a>, that would add support for <code>find_package</code> from other cmake projects consuming <code>cpuinfo</code>. Could this patch be merged (basically, the author did exactly what we were looking for, for our scenario) - or is supporting <code>find_package</code> not something desirable?</p>
<p>Thanks!</p>

ptrblck: <p>Feel free to add your use case to the PR directly so that the code owners can see it.</p>

mtrofin: <p>Thanks - I think my colleague Petr <a href="https://github.com/pytorch/cpuinfo/pull/69#issuecomment-1056066417" rel="noopener nofollow ugc">did that</a> - would that be what they’d be looking for?</p>
<p>Thanks!</p>
Computing the gradient of a function f2 inside the backward() call of a different function f1

lihenryhfl: <p>I would like to implement Eq. 6 in <a href="https://arxiv.org/pdf/1805.08498.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1805.08498.pdf</a>:</p>
<p>d/dphi z = - (d/dz S_phi(z))^{-1} d/dphi S_phi(z),</p>
<p>where z = S_phi^{-1}(eps) for some eps. In essence, letting <code>f1</code> = S_phi^{-1} and <code>f2</code> = S_phi, I’d like to compute</p>
<p>d/dphi <code>f2</code></p>
<p>inside of the <code>backward()</code> call of a <code>torch.autograd.Function</code> that implements <code>f1</code>. My current strategy is to create a new computation graph by cloning phi, i.e.</p>
<pre><code class="lang-auto">phi2 = phi.detach().clone().requires_grad_()
F = f2(phi2, z)
F.backward(grad_output) # RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>inside of the <code>backward()</code> call of <code>f1</code>. However, it appears to me that all of <code>backward()</code> is wrapped in a <code>with torch.no_grad()</code> statement, so I get an error when I call <code>F.backward()</code>. How do I compute gradients inside <code>backward()</code> calls?</p>
<p>Edit: Added a sentence to the post for clarity</p>

soulitzer: <p>You can you this API to reenable grad <a href="https://pytorch.org/docs/stable/generated/torch.enable_grad.html" class="inline-onebox" rel="noopener nofollow ugc">enable_grad — PyTorch 1.11.0 documentation</a></p>

lihenryhfl: <p>Yes! This was exactly what I was looking for. Thank you very much!</p>
Training loss nan is produced on GPU but not on CPU for CNN

luiejunior: <p>I’m trying to run the <code>train.py</code> file from <a href="https://github.com/gtamba/pytorch-slim-cnn" class="inline-onebox" rel="noopener nofollow ugc">GitHub - gtamba/pytorch-slim-cnn: A pytorch implementation of SlimCNN for Facial Attribute Classification : https://arxiv.org/abs/1907.02157</a>. When I run it on my GPU, I get nan for all of the loss values in the training values. However, when I run it on CPU, I don’t get any nan values for the loss.</p>
<p>When I add the line torch.autograd.set_detect_anomaly(True), it comes up with the error message ‘RuntimeError: Function ‘BinaryCrossEntropyWithLogitsBackward0’ returned nan values in its 0th output.’</p>
<p>What seems to be the issue here, and how can I fix it? I have a NVIDIA RTX 3070 GPU and am on CUDA version 11.4. I’m using torch version 1.10.0. Let me know if there’s any other information I can provide to diagnose the problem.</p>
<p>Thanks!</p>

illtellyoulater: <p>Hello,<br>
did you understand what was causing this problem?<br>
I’m seeing the same issue on a GTX 1660 TI gpu, but it automagically disappears using a GTX 1050.<br>
Any help would be appreciated.<br>
Thank you.</p>
Transformer Loss Going Down but Greedy Decoding Gives Empty String

Rajesh_Karmakar: <p>Hi,</p>
<p>I am training a transformer model to translate english into german.</p>
<p>The loss is going down per epoch.</p>
<p>But when I do the decoding I get  an empty string always:</p>
<pre><code class="lang-auto">            # x is the source sentence indices, batchsize x sequence_len
            x = x.transpose(0,1)
            encoder_emb = self.encoder_emb(x)
            encoder_emb_pos = self.pos_encoder(encoder_emb) #.transpose(0, 1)
            src_mask, src_padding_mask = self.create_mask_inference(x)
            encoder_output = self.encoder(encoder_emb_pos, mask=src_mask, src_key_padding_mask=src_padding_mask)

            output = torch.ones(1, x.size()[1]).fill_(self.vocab.bos_id()).long().cuda()

            for t in range(1, self.output_len):
                tgt_emb = self.decoder_emb(output) # [:, :t]
                tgt_emb_pos = self.pos_encoder(tgt_emb) # .transpose(0, 1)
                tgt_mask = torch.nn.Transformer().generate_square_subsequent_mask(t).cuda()
                decoder_output = self.decoder(tgt=tgt_emb_pos, memory=encoder_output, tgt_mask=tgt_mask)
                pred_proba_t = self.predictor(decoder_output)[-1, :, :]
                _, next_token = torch.max(pred_proba_t, dim=1)
                output = torch.cat([output, torch.unsqueeze(next_token, 0)], dim=0)
 
            return output
</code></pre>
<p>Is there something wrong with the decoding code in the block above? Thank you for your help.</p>
CTCLoss: get alpha and beta

cerisara: <p>Hi,</p>
<p>Any chance to be able to easily retrieve the alpha and beta (or gamma) tensors after computing the CTCLoss ?<br>
The objective is to exploit them in another loss function that would force the alignment to be balanced across the sequence of output states (the number of input timesteps assigned to each output state should be nearly the same for all states).</p>
<p>Thank you !</p>

Vivek_Tyagi1: <p>Hi Christophe,</p>
<p>I don’t have the answer on retrieving alpha and beta prob tensors computed in the CTC loss which in turn uses the famous forward_backward algorithm  used in the HMM-GMM model parameter estimation. However, reading your final objective of forcing same number of time steps in each state, you don’t need a HMM and hence CTC like loss for that as it is designed to do precisely the opposite of that i.e dynamic time warping. If you still want to force equal number of time-steps  in each state, you can use simpler model which does uniform segmentation of your input time series to your target sequences states and you woulnd’t need CTC loss and the alpha beta prob tensors.</p>

cerisara: <p>Right, in fact, I want to use the “uniform segmentation” criterion as a kind of regularizer for the CTC loss. I still want to be able to train the model with flexible alignments with CTC, but I basically want to avoid degenerated cases where “one state takes all”. Of course, this may be achieved through careful initialization, but a regularizer would be more powerful than simple uniform initialization, which influence decreases with more training epochs.<br>
Thanks for your reply !</p>
Why is the previous input symbol's tensor detached in seq2seq decoder in NMT

Vivek_Tyagi1: <p>Consider the tutorial on seq2seq NMT on PyTorch documentation.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>On this page in the method train (towards the end of the page) , we have <code>decoder_input = topi.squeeze().detach()  # detach from history as input</code></p>
<p>My question is why do we have to detach this tensor from history. In fact this is the decoded word/char with the highest probability in the decoder at the previous time step and it will be input to the next time steps’ GRU decoding along with the attention weighted encoder_outputs. So theoretically we want the gradients of this tensor i.e. <code>decoder_input</code> to be calculated w.r.t all the time steps it directly or indirectly (thru GRU) affected. So that means we should not detach its history to allow the gradients to flow to previous time steps too.</p>
<p>In fact this is what is happening in the RNN or LSTM cells too where we allow the memory cell, forget gate, input’s gate etc’s gradients to flow across all time steps and we don’t explicitly detach any tensor. So why is the  <code>decoder_input</code> tensor detached in this case. To me this seems like a major bug which can affect the theoretical basis of the encoder-decoder with attention NMT model and thereby it accuracy.</p>
<pre><code class="lang-auto">def train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):
    encoder_hidden = encoder.initHidden()
    ...
    ...
    
    else:
        # Without teacher forcing: use its own predictions as the next input
        for di in range(target_length):
            decoder_output, decoder_hidden, decoder_attention = decoder(
                decoder_input, decoder_hidden, encoder_outputs)
            topv, topi = decoder_output.topk(1)
            decoder_input = topi.squeeze().detach()  # detach from history as input
            
</code></pre>
Where to add DataParallel

KanZa: <p>I am trying to run the following code with multiple GPUs but I am not getting where to add the following?</p>
<p>net = torch.nn.DataParallel(model, device_ids=[0, 1, 2])</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py">
  <header class="source">

      <a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py" target="_blank" rel="noopener nofollow ugc">razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py</a></h4>


      <pre><code class="lang-py">'''
GCN Active Learning
'''

# Python
import os
import random
import torch
import numpy as np
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
import torch.optim.lr_scheduler as lr_scheduler
from torch.utils.data.sampler import SubsetRandomSampler
import torchvision.transforms as T
import torchvision.models as models
import argparse
# Custom
import models.resnet as resnet
from models.resnet import vgg11
</code></pre>



  This file has been truncated. <a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">with torch.cuda.device(CUDA_VISIBLE_DEVICES):
                if args.dataset == "fashionmnist":
                    resnet18    = resnet.ResNet18fm(num_classes=NO_CLASSES).cuda()
                else:
                    #resnet18    = vgg11().cuda() 
                    resnet18    = resnet.ResNet18(num_classes=NO_CLASSES).cuda()
                if method == 'lloss':
                    #loss_module = LossNet(feature_sizes=[16,8,4,2], num_channels=[128,128,256,512]).cuda()
                    loss_module = LossNet().cuda()

            models      = {'backbone': resnet18}
            if method =='lloss':
                models = {'backbone': resnet18, 'module': loss_module}
            
            torch.backends.cudnn.benchmark = True
            models = torch.nn.DataParallel(models, device_ids=[0])
</code></pre>
Randomly select and train parameters

Abhinav_Singhal: <p>For any semantic segmentation model, I wish to randomly select a few parameters (say 30%) and train only those parameters in the backward pass, but the forward function should use all the parameters. Also, I wish to get some id or tag for these parameters because I wish to do so multiple times on the same model for different categories of images (possibly with different loss functions). Kindly guide me as to how this can be done?</p>
BatchNorm in eval without running var and mean buffer

samsja: <p>Hey guys:</p>
<p>I want to find a way to run batch norm in eval mode for inference without using the running mean and var compute during training.</p>
<p>Indeed in the model I am currently working with the pretrained weights contains unstable batch norm statistics that basically break the model by outputting completely wrong result, and I can’t retrain the model atm. See here for more detail about the issue <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561/3" class="inline-onebox">Model.eval() gives incorrect loss for model with batchnorm layers - #3 by smth</a>.</p>
<p>So my question is how do I forget the running var/mean in eval mode for the batch norm and be able to process batch of size one (for inference)?</p>
<pre><code class="lang-python">class EvilBatchNorm2d(torch.nn.BatchNorm2d):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.running_var = None
        self.running_mean = None
</code></pre>
<p>Right now I am using this custom batch norm, however putting both running var and mean to None makes the forward to work like in training and thus it need at least of batch of size two. So I added this trick right now. But I am not satisfy with it.</p>
<pre><code class="lang-python">class EvilBatchNorm2d(torch.nn.BatchNorm2d):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.running_var = None
        self.running_mean = None

    def forward(self, input: Tensor) -&gt; Tensor:
        if input.shape[0] == 1:
            input = input.expand(2, -1, -1, -1)

        return super().forward(input)
</code></pre>
<p>Why is the best way to do it ? Alternatively can someone point me to the c++ code for the batch norm forward function so that I can reproduce the correct behavior by myself ?</p>
<p>Thanks in advance</p>

ptrblck: <p>I’m unsure what the desired behavior would be in your use case.<br>
If you don’t want to use the running stats at all, you could use <code>track_running_stats=False</code> such that the batch stats will be used to normalize the input activation during training and evaluation.<br>
However, based on your explanation it also seems that you are using a single sample and thus the stats calculation would fail.<br>
In this case you could reset the running stats to zero for the mean and ones for the stddev, but I would expect the achieved performance would be bad since you are not normalizing the inputs at all and are thus potentially changing the activation range completely.</p>
Clarification: random resize / crop / resized_crop

Innat: <p>Hello. I was reading the doc of the following three transformations.</p>
<ul>
<li><a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomCrop.html#torchvision.transforms.RandomCrop" rel="noopener nofollow ugc"><code>torchvision.transforms.</code> <code>RandomCrop</code></a></li>
<li>
<code>torchvision.transforms.</code> <code>Resize</code>
</li>
<li><a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomResizedCrop.html#torchvision.transforms.RandomResizedCrop" rel="noopener nofollow ugc"><code>torchvision.transforms.</code> <code>RandomResizedCrop</code></a></li>
</ul>
<p>Is the operation of <code>RandomCrop</code> + <code>Resize</code> EQUIVALENT EXACTLY to <code>RandomResizedCrop</code>?</p>
<p>Are there any differences? Is there any reason other than conveniences to have <code>RandomResizedCrop</code> than the combination of <code>RandomCrop + Resize</code> operations?</p>

thecho7: <p>They operate exactly the same.<br>
However, <code>RandomResizedCrop</code> runs faster than <code>Crop + Resize</code></p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as transforms
from PIL import Image
import glob
import time

image_path = glob.glob('*.png')
image = Image.open(image_path[0])

c_r = transforms.Compose([
    transforms.RandomCrop(size=(100, 100)),
    transforms.Resize(size=(100, 100))
    ])

r_c = transforms.Compose([
    transforms.RandomResizedCrop(size=(100, 100))
    ])

start = time.time()
out = c_r(image)
print(f'{time.time() - start}') # 0.012

start = time.time()
out = r_c(image)
print(f'{time.time() - start}') # 0.003
</code></pre>

Innat: <p>Thanks for the information. What could be the cause of the execution gap? The implementation details should be very similar! <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
[W NNPACK.cpp:79] Could not initialize NNPACK! Reason: Unsupported hardware

zouqi: <p>Mar 11. This bug occurs when I run the code in my env.<br>
And then I run on my terminal  ：</p>
<pre><code>     print(torch.cuda.is_available())     ##False
</code></pre>
<p>I don’t know why.</p>
KL-divergence for Multivariate Gaussian VAE

Khadijeh_Arabi: <p>How to calculate KL-divergence in multivariate Gaussian distribution is the following code correct?</p>
<pre><code class="lang-auto">        mu, logvar = self.encode(x.view(-1, 2))
        z = self.reparameterize(mu, logvar)
        #q0 = torch.distributions.normal.Normal(mu, (0.5 * logvar).exp())
        q0 = MultivariateNormal(mu, (0.5 * logvar).exp())
       # prior = torch.distributions.normal.Normal(0., 1.)
        prior = MultivariateNormal(mu,logvar)
        log_prior_z = prior.log_prob(z).sum(-1)
        log_q_z = q0.log_prob(z).sum(-1)
        KL = log_q_z - log_prior_z
</code></pre>
How to locate data location:global memory or shared memory?

Ziyu_Huang: <p>Just like torch.mm(a, b), we have a and b on GPU, on global memory now(?), and we use deeper implementation to use shared memory or something by cublas, and then we output a result to global memory, is that correct? Not sure…So I am wondering is there a function like a.device() in python to show, where the data currently is? Thank you!!!</p>

ptrblck: <p>Yes, your data lives in the global memory on the GPU. Shared memory, L1, L2 etc. is used in the CUDA kernels.</p>
Predict labels and return percentage

Roua_Rouatbi: <p>I trained a model on 0 and 1 labels now I’m testing it on unlabeled data</p>
<pre><code class="lang-auto">labels=[0,1]
for i, images in enumerate(imgset_loader):
    images = images.to(device)
    net = net.double()
    outputs = net(images)
    _, predicted = torch.max(outputs.data, 1)
    print(labels[predicted])
</code></pre>
<p>The last line returns an error : TypeError: only integer tensors of a single element can be converted to an index</p>
<p>I want the prediction to show a percentage of how many 0 labels it found on the data, how many 1s, and how many others.</p>

ptrblck: <p>Transform <code>labels</code> to a tensor via <code>labels = torch.tensor(labels)</code> and it should work.</p>

mularo: <p>def validation(model,device,data_loader_val):<br>
correct = 0<br>
total = 0<br>
accuracy = 0.0<br>
val_loss = 0.0<br>
with torch.no_grad():<br>
for data in data_loader_val:<br>
images, true_labels = data<br>
images,true_labels = images.to(device), true_labels(device)<br>
<span class="hashtag-raw">#测试数据</span><br>
output = model(images)<br>
val_loss += neural_funcs.cross_entropy(output,true_labels).item()<br>
pred = output.max(1,keepdim=True)[1] <span class="hashtag-raw">#值</span>，索引<br>
correct += pred.eq(true_labels.view_as(pred)).sum().item()<br>
val_loss /= len(data_loader_val.dataset)<br>
total = len(data_loader_val.dataset)<br>
accuracy = correct/total</p>
<p>我在运行代码是，显示这里出现错误，报错信息是<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b05afb5892e8fb49cbcb8ec3c129eed00f598d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b05afb5892e8fb49cbcb8ec3c129eed00f598d3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b05afb5892e8fb49cbcb8ec3c129eed00f598d3.png" alt="image" data-base62-sha1="aHFZA671XqpYitEpCyjW04Rj9Pt" width="690" height="125" data-dominant-color="333033"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">956×174 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
请问是为什么？怎么可以解决呢？</p>

ptrblck: <p><code>true_labels</code> seems to be a tensor, so your code seems to be missing the <code>.to()</code> operation.</p>
CUDA Streams make code slower

Tushar_Gautam: <p>Hi</p>
<p>I’m using CUDA streams to perform two operations concurrently on a GPU. However, my code with multiple streams is running slower than the one with single (default) stream. Here’s my code for both cases and <code>streams</code> function runs twice as slow as the <code>normal</code>.</p>
<pre><code class="lang-python">def streams(X, dX, W1):
    s = torch.cuda.Stream()  # Create a new stream.
    with torch.cuda.stream(s):
        dZ1 = torch.matmul(dX, W1.T)
    Z1 = torch.matmul(X, W1.T)
        
    return Z1, dZ1
</code></pre>
<pre><code class="lang-python">def normal(X, dX, W1):    
    Z1 = torch.matmul(X, W1.T)
    dZ1 = torch.matmul(dX, W1.T)
    return Z1, dZ1
</code></pre>
<p>In the above example code, I want to compute <code>Z1</code> and <code>dZ1</code> concurrently as they’re independent. I can’t understand why this is happening?</p>

ptrblck: <p>Could you create a profile of this workload (e.g. via the PyTorch profiler or Nsight Systems) and share it here, please?</p>
How do I let DDP quit if one rank has OOM error

Jiaji_Huang: <p>In a distributed data parallel job, due to the special format of data batches, some ranks may receive more data samples than the others. When these “heavy” ranks encounter out of memory issue, the job just hangs there. I think the “light” ranks are waiting for the “heavy” ones to finish. Is there a way to signal all the ranks so they can all quit?</p>
Dynamic Quantization accuracy loss

ori: <p>Hi everyone,</p>
<p>I am trying Dynamic Quantization and my model is mostly composed of linear layers. Hence is cant do any other quantization techniques. But with dynamic quantization of qint8 model size was reduced but the accuracy of model too. Is there any ways i can improve my accuracy ???</p>

suraj.pt: <p>Static quantization supports Linear modules, if you’d like to try it out. Accuracy will inevitably drop, but it can be controlled by selectively quantizing some layers in your model (as opposed to all)</p>

ori: <p>when i try with static quantization,</p>
<pre><code class="lang-auto">backend = "fbgemm"
model_baseline.qconfig = t.quantization.get_default_qconfig(backend)
t.backends.quantized.engine = backend
model_static_quantized = t.quantization.prepare(model_baseline, inplace=False)
model_static_quantized = t.quantization.convert(model_static_quantized, inplace=False)
</code></pre>
<p>i get error as</p>
<pre><code class="lang-auto">AssertionError: The only supported dtype for nnq.Embedding is torch.quint8
</code></pre>
<p>is there any way to set dtype before</p>

suraj.pt: <p>Does this post help you? <a href="https://discuss.pytorch.org/t/is-it-planned-to-support-nn-embeddings-quantization/89154/15" class="inline-onebox">Is it planned to support nn.Embeddings quantization? - #15 by supriyar</a></p>

jerryzh168: <aside class="quote no-group" data-username="ori" data-post="3" data-topic="146082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ori/48/47422_2.png" class="avatar"> ori:</div>
<blockquote>
<pre><code class="lang-auto">AssertionError: The only supported dtype for nnq.Embedding is torch.quint8
</code></pre>
</blockquote>
</aside>
<p>here are the supported qconfigs for embedding: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/fx/test_quantize_fx.py#L5645" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/test/quantization/fx/test_quantize_fx.py#L5645</a>, you can try setting the qconfig for the embedding ops to these qconfigs I think</p>
Migrating from torch.distributed.launch to torchrun

haowxu: <p>Hi guys I am working rewriting some previous training code using multiple nodes, on Azure ML.<br>
The previous code uses the command<br>
<code>python -m torch.distributed.launch --nnodes $NUM_NODES --nproc_per_node $NUM_TRAINERS --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT $TRAINING_COMMAND</code><br>
and works well now.<br>
When I followed <a href="https://pytorch.org/docs/stable/elastic/run.html" rel="noopener nofollow ugc">this</a> to migrate the code to<br>
<code>torchrun --nnodes=$NUM_NODES --nproc_per_node $NUM_TRAINERS --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT $TRAINING_COMMAND</code><br>
I got <code>RendezvousTimeoutError</code> on the master node and <code>RendezvousConnectionError</code> on the other node.<br>
Could anyone help find solution or figure what I did wrong? Thanks!</p>

pbelevich: <p><a class="mention" href="/u/aivanou">@aivanou</a> <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> ?</p>

Kiuk_Chung: <p>Could you provide us with the actual command (with the real values for nnodes, nprocs_per_node, etc)? We’re you running across multiple hosts for both commands? torchrun and torch.distributed.launch both use the same underlying main entrypoint (torch.distributed.run), so I’m wondering if the two commands were invoked in exactly the same setup.</p>

haowxu: <p>Sure. The command for <code>torch.distributed.launch</code> is<br>
<code>python3 -m torch.distributed.launch --nnodes 2 --nproc_per_node 4 --node_rank 0 --master_addr 10.0.0.8 --master_port 6000 $TRAINING_COMMAND</code>, where the node rank is changed to 1, 2 or 3 for other processes.<br>
For <code>torchrun</code> the command is<br>
<code>torchrun --nodes=2 --nproc_per_node=4 --rdzv_id=0 --rdzv_backend=c10d --rdzv_endpoint=10.0.0.8:6000 $TRAINING_COMMAND</code>. I just use the node rank as rendezvous ID.<br>
Actually I’ve tried passing other rendezvous backend like <code>etcd</code> but also failed.</p>

haowxu: <p>Please check the reply above. Sorry for not replying your post.</p>

Kiuk_Chung: <p>Could you paste the full log output of both nodes? I believe what is happening is that the fully-qualified-domain-name (<code>socket.getfqdn()</code>) on node0 returns the <code>hostname.domain</code> that has no DNS route from node1 (hence, the timeout). There’s two ways around this:</p>
<ol>
<li>Make sure that the 2 nodes have a public ip</li>
<li>Run with “static” rendezvous (just skip the <code>--rdzv_backend</code> option, it will default to “static”), which has the same options as <code>torch.distributed.launch</code>:<pre><code class="lang-auto">$ torchrun --node_rank=0 # or 1 
                 --nnodes=2 
                 --nproc_per_node=4 
                 --master_addr=10.0.0.8 
                 --master_port=6000
                 $TRAINING_CMD
</code></pre>
</li>
</ol>
Build libtorch1.10.2(c++) from source doesn't generate libtorch_cuda.so

111392: <p>I build the libtorch1.10.2 by following command</p>
<pre><code class="lang-auto">git clone --recurse-submodules -j6 https://github.com/pytorch/pytorch.git -b v1.10.2
cd pytorch/tools/
mkdir build
cd build
 python3 ../tools/build_libtorch.py
</code></pre>
<p>after finishing the build successfully, so file related cuda was missing<br>
ex)</p>
<pre><code class="lang-auto">  libtorch_cuda.so
  libc10_cuda.so
  libcaffe2_detectron_ops_gpu.so
</code></pre>
<p>btw I could generate those so file with libtorch1.7 and libtorch1.9 with no problem</p>
<p>Do you have any idea why it didn’t generate libtorch_cuda.so file with libtorch1.10.2</p>
<p>I use</p>
<pre><code class="lang-auto">GPU RTX2060 
NVIDIA Driver 470.57.02
CUDA11.0
Cudnn8
</code></pre>
Reinitializing the weights after each cross validation fold

cakeeatingpolarbear: <p>I was wondering how I would re-initialize the weights of my model without having to re-instantiate the model?</p>

smth: <p>This is not a robust solution and wont work for anything except core torch.nn layers, but this works:</p>
<pre><code class="lang-auto">for layer in model.children():
   if hasattr(layer, 'reset_parameters'):
       layer.reset_parameters()
</code></pre>

cakeeatingpolarbear: <p>I ended up saving the initial model parameters into a temporary file and then reloading it at the start of each CV fold.</p>

Brando_Miranda: <p>two questions:</p>
<ol>
<li>for it to work for <strong>any</strong> layer, the user has to implement their own layer with that method I assume, right?</li>
<li>why didn’t you use <code>.modules()</code>?</li>
</ol>

Brando_Miranda: <p>based on this we probably need <code>.modules()</code> to properly recursively go into all modules in the model:</p>
<aside class="quote" data-post="3" data-topic="4551">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ea5d25/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/module-children-vs-module-modules/4551/3">Module.children() vs Module.modules()</a> 
  </div>
  <blockquote>
    Ah, Makes sense. 
Sorry for such a basic question but what is the intended use for .modules()?
  </blockquote>
</aside>


rilesdg3: <p>The C++ way is below</p>
<pre><code class="lang-auto">auto net = std::make_shared&lt;Net&gt;();

for(auto pair :net-&gt;children())
    if(pair-&gt;name() == "torch::nn::LinearImpl")
        pair-&gt;as&lt;torch::nn::LinearImpl&gt;()-&gt;reset_parameters();
}

were Net is

struct Net : torch::nn::Module {
......
};
</code></pre>
Custom loss function which counts the number of true element in a boolean array

mehedi_hasan_eee: <p>These are some lines from my loss function. <code>output</code> is the output of a multiclass classification network.</p>
<pre><code class="lang-auto">bin_count=torch.bincount(torch.where(output&gt;.1)[0], minlength=output.shape[0])

dr_output = (bin_count == 1) &amp; (torch.argmax(output, dim=1)==labels)
</code></pre>
<p>I want <code>dr_output.sum()</code> to be part of my loss function. But there are many limitations in my implementation. Some functions are non-differentiable in pytorch, and also <code>dr_output</code> may be zero which is also not allowed if I only use <code>dr_output</code> as my loss. Can anyone please suggest to me a way around these problems?</p>

soulitzer: <p>You could use soft argmax instead of argmax to get around the non-differentiablity of argmax. Other non-differentible functions that you are using may also have soft analogues.</p>
Pytorch convolution neural network tune class weights

ni_tempe: <p>I have a neural network as below for binary prediction. My classes are heavily imbalanced and class 1 occurs only 2% of times. Showing last few layers only</p>
<pre><code>self.batch_norm2 = nn.BatchNorm1d(num_filters)

self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)

self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)

self.fc3 = nn.Linear(fc2_neurons, 1)
</code></pre>
<p>My loss is as below.</p>
<pre><code>BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_wts[1]/class_wts[0])
</code></pre>
<p>Now if I would like to try various <code>pos_weight</code> values, is there a way to do it? something like hyper parameter tuning for <code>pos_weight</code>? Ideally I would like to do it using Bayesian Optimization  (<code>from bayes_opt import BayesianOptimization</code>) but if there is any other way let me know.</p>
<p>As of now only thought that I have is to perform exhaustive search. But it wont be very efficient</p>
How do AI researchers find the right hyperparameters for the models they create?

sofiane: <p>How do AI researchers find the right hyperparameters for the models they create ? In their white papers they often provide the hyperparameter configurations, but how do they find them? is there a procedure to know which hyperparameter is problematic (number of layers or learning rate or both for example), Or how can you know wether you network isn’t correct at all or you only have to use a different activation function.</p>

KFrank: <p>Hi So!</p>
<aside class="quote no-group" data-username="sofiane" data-post="1" data-topic="146034" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e19adc/48.png" class="avatar"> sofiane:</div>
<blockquote>
<p>How do AI researchers find the right hyperparameters for the models they create ?</p>
</blockquote>
</aside>
<p>Most commonly, folks just try a bunch of different things until they get<br>
tired of trying more things, and then use the best combination they found.</p>
<p>Sometimes they try things based on experience or intuition or ideas they’ve<br>
seen in the literature or suggestions friends make.  But a lot of the time<br>
they just try things at random.</p>
<p>To be a little less flippant, people will systematically study  various<br>
hyperparameters.  The <a href="https://arxiv.org/abs/2201.03545" rel="noopener nofollow ugc">ConvNeXt</a> paper is one example of this, although<br>
it is more about systematically adjusting architectural features, rather<br>
than hyperparameters.</p>
<p>Best.</p>
<p>K. Frank</p>
Number of processes per GPU while running distributed data parallel

Jiaji_Huang: <p>I have a question on basic concept. When we launch a distributed data parallel job over n GPUs, how many processes should each GPU have?</p>
<p>I ask this because I notice different patterns while typing nvidia-smi. Sometimes, each GPU has one single PID. And in total, there are n different PIDs. That’s what I would expect.</p>
<p>However, sometimes each GPU has n PIDs, with one PID using actual memory and all other (n-1) using 0 memory. Moreover, all GPUs shares the same PIDs. That’s hard for me to understand. Are the PIDs taking zero memory due to communication?</p>

rvarm1: <p>You’re right that DDP training should use one GPU per process.</p>
<p>Could you give an example nvidia-smi output that indicates the second case that you’ve mentioned? In general, this might happen if memory is allocated or operations are done on a GPU device that is not “assigned” to that process, although if DDP itself, and not the application, is doing this it is likely an issue we should fix. A script to reproduce would also be helpful.</p>

Jiaji_Huang: <p>This is a screenshot of nvidia-smi’s output, for the 2nd case. Let me explain a bit. There are 10 GPUs on this node. Someone else is using GPU 1 and 2. GPU 6-9 is not used. I’m using GPU 0, 3, 4, 5, running a DDP job with world_size=4, i.e., n=4.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77826feac93a9bfe28792197471861c0ad5dbf0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b77826feac93a9bfe28792197471861c0ad5dbf0" title="Screen Shot 2022-03-10 at 8.59.12 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77826feac93a9bfe28792197471861c0ad5dbf0.png" alt="Screen Shot 2022-03-10 at 8.59.12 AM" data-base62-sha1="qb2NJSz06ztvlZ6fVq0b3XccYIo" width="283" height="499" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-03-10 at 8.59.12 AM</span><span class="informations">557×981 16.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Detect theft on live video

jasg: <p>hello can anyone guide me or share resources for live camera theft detection, I want to get up to speed with this solution.</p>

suraj.pt: <p>There are a number of object/pose detection libraries built on top of pytorch you could use for this. Processing live video gets trickier, but check out d2go that I think works well in these cases  <a href="https://github.com/facebookresearch/d2go" class="inline-onebox">GitHub - facebookresearch/d2go: D2Go is a toolkit for efficient deep learning</a>.</p>
How can the outputs from torchvisions IoU be used to determine True Positives, False Positives and False Negatives?

SU801T: <p>Below is an example of IoU being applied to bounding boxes from one image (one image per batch).</p>
<pre><code class="lang-auto">iou = torchvision.ops.box_iou(predictions['boxes'], targets[0]['boxes'])
print(iou)

tensor([[0.0000, 0.0000],
        [0.0000, 0.0000],
        [0.0000, 0.0000],
        [0.0000, 0.0000],
        [0.0000, 0.9322],
        [0.8021, 0.0000],
        [0.0000, 0.0000],
        [0.0000, 0.0000]])

</code></pre>
<p>I understand that the tensors above show pairwise outputs related to the IoU overlap per predicted bounding box. So, in this case there are two bounding boxes with an overlap determined for the first bounding box in the 6th tensor and for the second bounding box, in the 5th tensor.</p>
<p>However, I would like to know how I can use this information to determine true positives, false positives and false negatives. It’s unclear to me ATM how I can relate these boxes to the previous info such as predictions, targets and image id name…</p>
<p>I would eventually like to save the coordinates of these true positive and false positives bounding boxes and extract them from the image later.</p>
Can you make it simpler than this?

lattice: <p>I saw a paper and implemented a similar coding. Can it be made simpler than this? For example, not use the for loop. The model looks like this, and this is what I made.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2eae2e8d23fb64d1b91db554284a30fd343c7c5e.png" alt="image" data-base62-sha1="6EX95cqPdpHotDHQDKl0K9fHbKm" width="390" height="333"></p>
<pre><code class="lang-auto">class FeatureSelectionNetwork(nn.Module):
    def __init__(self, input_size):
        super().__init__()
        
        self.input_size = input_size
        self.params = nn.ParameterList([nn.Parameter(torch.randn(())) for i in range(input_size)])
            
            
    def forward(self, x):
        if self.input_size != x.shape[1]:
            raise Exception('input size must be same')
        
        wx = []
        for i, param in enumerate(self.params):
            wx.append(x[:, i] * param)
        
        return F.relu(torch.stack(wx, dim=1))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/1/7108f68517c9435201d16f6c029c66816ce397af.png" alt="image" data-base62-sha1="g7X9KoKvyO95sGHu7rVgKT0mi1V" width="688" height="229"></p>
<p>could you make it faster? And do you think this layer can do feature selection in neural networks?</p>
Proper way to implement autoregressive decoder in Seq2Seq tasks

AfonsoSalgadoSousa: <p>Hi. I am trying to implement an LSTM-based Encoder-Decoder model for sequence-to-sequence. I took inspiration from <code>fairseq</code> and built a decoder with conventional embedding-dropouts-recurrent-linear layers. The forward method is like this:</p>
<pre><code class="lang-auto">hidden = state.hidden
x = self.embed_tokens(prev_output_tokens)
emb = self.dropout_in_module(x)
x, hidden_t = self.rnn(emb, hidden)
x = self.dropout_out_module(x)
return x, State(hidden=hidden_t)
</code></pre>
<p>The <code>prev_output_tokens</code> is a batch with the target sentences shifted one character to the right and starting with eos_token.<br>
Currently, my generations (I am using greedy search) are very poor. When I started looking into other implementations, I found that there are implementations that more explicitly target the generation step, just like <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">this one</a> in the following excerpt:</p>
<pre><code class="lang-auto">for di in range(max_length):
     decoder_output, decoder_hidden, decoder_attention = decoder(
        decoder_input, decoder_hidden, encoder_outputs)
     decoder_attentions[di] = decoder_attention.data
     topv, topi = decoder_output.data.topk(1)
     if topi.item() == EOS_token:
          decoded_words.append('&lt;EOS&gt;')
          break
     else:
          decoded_words.append(output_lang.index2word[topi.item()])

     decoder_input = topi.squeeze().detach()
</code></pre>
<p>Can someone tell me the best strategy and why mine (the same as <code>fairseq</code>, I suppose) does not work – or does, but I am doing something wrong. Thanks in advance for any help you can provide.</p>