akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Use tensorcore explicitly on non-DL code

jtang10: <p>I’m using pytorch basically as a GPU-accelerated numpy due to its simplicity to switch between CPU and GPU.<br>
Now I’m trying to use tensorcore to accelerate some parts of the code that is mostly matmul. Though I have wrapped it with <code>torch.autocast()</code>, the tensorcore is not actually used by the profiling result.</p>
<p>The involved data is casted to float32 first (actually complex64 mostly) and then transferred to pytorch and CUDA, so I don’t think I’m violating the part described in the doc:</p>
<blockquote>
<p>Ops called with an explicit <code>dtype=...</code> argument are not eligible, and will produce output that respects the <code>dtype</code> argument.</p>
</blockquote>
<p>I have the following questions:</p>
<ol>
<li>Would casting <code>b</code> and <code>c</code> into float16 and call <code>b @ c</code> automatically lead to tensorcore usage?</li>
<li>Are there any ways that I can explicitly use tensorcore without doing autocast? e.g. call something explicitly to use tensorcore?</li>
<li>Does tensorcore work for complex data?</li>
</ol>

ptrblck: <p><code>Autocast</code> should use mixed-precision if the input data and parameters are in <code>float32</code>. Could you verify that your GPU actually has TensorCores?</p>

jtang10: <p>Thanks for replying. I have a 2080 Ti so I think there are Tensorcore on it. I can confirm that the tensorcore is used if tensors are in float32 but not in complex64, so maybe complex format is not supported for tft operations?</p>
<pre><code class="lang-auto">device = torch.device("cuda")
# a = (torch.randn((40, 80)) + 1j * torch.randn((40, 80))).to(device)
# b = (torch.randn((80, 40)) + 1j * torch.randn((80, 40))).to(device)
a = torch.randn((40, 80)).to(device)
b = torch.randn((80, 40)).to(device)

iters = 10
prof = profiler.profile(
    schedule=profiler.schedule(wait=0, warmup=2, active=iters-2, repeat=1),
    with_stack=True,
    on_trace_ready=profiler.tensorboard_trace_handler("./tft", worker_name="tft_real"),
    profile_memory=True,
    record_shapes=True,
)

times = []
for _ in range(iters):
    prof.start()
    with torch.autocast("cuda"):
        c = a @ b
    prof.step()
    prof.stop()
print(a.dtype, c.dtype)
</code></pre>
Fourier transform in latest pytorch

meshghi: <p>Hi, I have a code that’s written for older versions of PyTorch, and I get the following error when I run it on the latest version of PyTorch. Is there any way to fix the issue for the latest version of PyTorch?</p>
<p>The code:</p>
<pre><code class="lang-auto">fft_src = torch.rfft(src_img.clone(), signal_ndim=2, onesided=False)
</code></pre>
<p>The error:</p>
<pre><code class="lang-auto">AttributeError: module 'torch' has no attribute 'rfft'
</code></pre>

tom: <p>Hi Mohammad,</p>
<p>for some releases, PyTorch is trying to move closer to the NumPy API where it provides equivalent functions.  The <a href="https://pytorch.org/docs/1.7.0/generated/torch.rfft.html" rel="noopener nofollow ugc">deprecation notice in PyTorch 1.7</a> (which I had to look up myself, so don’t feel bad about not seeing it), your use with <code>onesided=False</code> should be replaced by <a href="https://pytorch.org/docs/stable/generated/torch.fft.fft.html#torch.fft.fft" rel="noopener nofollow ugc"><code>torch.fft.fft</code></a>, but for 2d I think <a href="https://pytorch.org/docs/stable/generated/torch.fft.fft2.html#torch.fft.fft2" rel="noopener nofollow ugc"><code>torch.fft.fft2</code></a> might be a better fit, though I did not try.</p>
<p>If you want to verify that it is working as expected and you have access to an old version of PyTorch, you could try to replicate the old PyTorch behaviour using NumPy’s numpy.fft.fft2 and once you have this install the new version and use the compatible NumPy command to replicate in new PyTorch (and the “compatible API” idea is that this is very easy). If you want to know at which version to split if you need to support both, I would use the new for 1.7+ and the old for up to 1.6. Though these versions seem pretty old, so if you don’t have to support them…</p>
<p>Best regards</p>
<p>Thomas</p>

meshghi: <p>Hi <a class="mention" href="/u/tom">@tom</a>, thank you very much for your detailed reply. I actually ended up creating a new venv with an older version of PyTorch that supports <code>torch.rfft</code>. And, I realized there is no function in the newer version of PyTorch that can reproduce the results that I want. So, I’m just using an older version of PyTorch for this specific run. Thanks again!</p>
Memory usage during backpropagation

Marco_Conati: <p>I’m hoping to use a very large, pretrained, 80 layer transformer-based language model for a new task. My plan is to finetune the language model on my dataset and then use activations from the 40th layer of the language model to train simple linear probes on my new task.</p>
<p>For finetuning the model, I don’t have the memory capacity to train the whole model. I only have the ability to store gradients for one layer at a time on my GPU. So, I’m planning to set param.requires_grad as true for just the 40th layer of the model, and false for the rest. If I did this, would Pytorch still hold the gradients for the 41st-80th layers in memory for backpropagation? Or, would it be able to delete gradients as it backpropagates. I’m thinking it would be possible to calculate gradients for the 80th layer, backpropagate gradients for the 79th, delete gradients for the 80th, etc… so that gradients for only one transformer are stored at a time.</p>

suraj.pt: <p>Hi <a class="mention" href="/u/marco_conati">@Marco_Conati</a></p>
<p>Setting <code>requires_grad(False)</code> detaches the weights from the computational graph, so it will not be used in backpropagation.</p>
<blockquote>
<p>gradients for only one transformer are stored at a time</p>
</blockquote>
<p>See <a href="https://pytorch.org/docs/stable/checkpoint.html?highlight=checkpoint#torch.utils.checkpoint.checkpoint">torch.utils.checkpoint</a> for this. Hope this helps!</p>

Marco_Conati: <p>Oh checkpoint looks very helpful. Thanks!</p>
<p>How does Pytorch calculate gradients for parameters earlier in the model if requires_grad is set to False at later layers?</p>
<p>For example, if I had a sequential model of linear layers and set requires_grad(False) at the final layer, how can earlier layers backpropogate gradients? Doesn’t it break the chain rule?</p>

suraj.pt: <p>The gradients of earlier parameters do not depend on the gradients of later parameters.</p>
<p>To calculate gradients of earlier parameters, autograd only requires the gradients w.r.t inputs or intermediary values in the network. OTOH the values in <code>.grad</code> (which are not computed when <code>requires_grad(False)</code>) are the gradients w.r.t. the parameter; this is only used for updating the parameter.</p>

Marco_Conati: <p>Okay I see thanks. I think I understand how early gradients should still be calculable with requires_grad(False) down the line</p>
<p>I just tried to freeze all but a middle layer of my model, and I am getting a “One of the differentiated Tensors does not require grad” error. Do you have any insight into how to debug this? My initial searches online were very cryptic</p>

suraj.pt: <p>I cannot reproduce this, can you share an example?</p>
Is there a tutorial for DoubleBackward for functions written in C++?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I saw the tutorial for writing a custom function with double backward support: <a href="https://pytorch.org/tutorials/intermediate/custom_function_double_backward_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Double Backward with Custom Functions — PyTorch Tutorials 1.10.1+cu102 documentation</a> .However, is there a tutorial for writing a function in C++ that is later bound into python?</p>
<p>Within this <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">tutorial</a> it shows how to define a <code>torch.autograd.Function</code> function with C++ forward and backward methods that are binded in python from C++ but there’s no tutorial on defining the entire function within C++? Is there a tutorial I’ve missed? (With a way to define a C++ function with a DoubleBackward)</p>
<p>Thank you!</p>
Gradient manipulation loading and setting

manthanthakker40: <p>Hi,</p>
<p>I am trying to modify the gradients for Multi-task learning. I have a model with multiple modules/layers and I want a list of all gradients in each layer/module. I want to manipulate gradients, let’s say add a similar dimension list to the gradient list computed for simplicity and then want to assign each module/layer the modified gradients. Is there a function to do this, so far I could find I can create a list with the following snippet, how can I assign these gradients back?</p>
<pre><code class="lang-auto">def gradient2vec(): 
    gradient_list=[]
    for name, param in model.named_parameters():
    if param.requires_grad:
        print name, param.data
        gradient_list.append(param.grad)
    return gradient_list
def vec2gradient(model, gradient_list):
?????
</code></pre>

albanD: <p>Hi,</p>
<p>This looks pretty ok, you can add them back by either overriding the existing gradient by doing param.grad = saved_value. Or accumulating by doing <code>param.grad += saved_value</code>.</p>
How do I turn a float tensor into one hot tensor with the first non zero dimension on?

dg_torch: <p>I have a tensor of floats and want to mark the first non-zero entry.<br>
So using single dimension examples:<br>
[0., 0.5, 0.3] would turn to [0,1,0] and [-2.,1.,1.] to [1,0,0]</p>
<p>In my case, I have a 3 dim tensor and want to do it along the first dim.<br>
I couldn’t find a simple way of doing it using tensor operations. Right now I am using for loops which is quite slow.</p>
<p>Here is my code using for loops:</p>
<pre><code class="lang-auto">num_stacks,N1,N2 = tensor.shape
      
flatten_tensor_with_distance_dim_last = tensor.permute(1, 2, 0).view(-1, num_stacks)
for distance_tensor in flatten_tensor_with_distance_dim_last:
  non_zero_entry_visited = False
  for i in range(len(distance_tensor)):
      if non_zero_entry_visited:
          distance_tensor[i] = 0.
      elif distance_tensor[i] &gt; 0.:
          non_zero_entry_visited = True
          distance_tensor[i] = 1.
# back to original shape
return flatten_tensor_with_distance_dim_last.view(N1, N2, num_stacks).permute(2, 0, 1)
</code></pre>
<p>Thanks!</p>

KFrank: <p>Hi Dg!</p>
<aside class="quote no-group" data-username="dg_torch" data-post="1" data-topic="145889" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dg_torch/48/45800_2.png" class="avatar"> dg_torch:</div>
<blockquote>
<p>[0., 0.5, 0.3] would turn to [0,1,0] and [-2.,1.,1.] to [1,0,0]</p>
</blockquote>
</aside>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; t = torch.tensor ([[0.0, -2.0], [0.5, 1.0], [0.3, 1.0]])
&gt;&gt;&gt; (((t != 0).cumsum (0).cumsum (0)) == 1).long()
tensor([[0, 1],
        [1, 0],
        [0, 0]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Strange behaviour with torch.clamp()

pyhh: <p>This behavior seems strange:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.tensor([1,2,3,4,5])
&gt;&gt;&gt; torch.clamp(x, 2, 1)
tensor([2, 1, 1, 1, 1])
</code></pre>
<p>Based on the note in the docs, I would have expected all values to be set to the max of 1: <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html#torch.clamp" class="inline-onebox" rel="noopener nofollow ugc">torch.clamp — PyTorch 1.10 documentation</a></p>
<p>Python 3.7.12<br>
Torch 1.5.0</p>
<p>Any thoughts?</p>

AlphaBetaGamma96: <p>So, I’ve just run this I got the expected behaviour.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.tensor([1,2,3,4,5])
&gt;&gt;&gt; torch.clamp(x, 2, 1)
tensor([1, 1, 1, 1, 1])
</code></pre>
<p>My torch version is <code>1.11.0.dev20220201+cu111</code>, so this bug has been patched but I’m not sure which version resolved it.</p>
<p>Also, do you think there could be an issue given that the <code>min</code> is larger than the <code>max</code> value? Perhaps repeat it with <code>torch.clamp(x,0,1)</code> and see if that works?</p>
Dynamic input channels for layers

dr.interned: <p>Hello everyone.<br>
I want to create a transpose upsampling layer like this</p>
<pre><code class="lang-python">class TransposeX2(nn.Sequential):
    def __init__(self, in_channels, out_channels):
        layers=[
            nn.ConvTranspose2d(in_channels, out_channels, kernel_size=4, stride=2, padding=1, groups=out_channels),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(),
        ]
        super().__init__(*layers)
</code></pre>
<p>but i also want to use this layer multiple times. What should I do to avoid creating this layer on every upsample operation like</p>
<pre><code class="lang-python">self.upsample1=TransposeX2(4,8)
self.upsample2=TransposeX2(8,8)
def forward(self, x):
    a = self.upsample1(x)
    b = self.upsample2(a)
</code></pre>
<p>I want to create upsample layer once and then use it everytime like this:</p>
<pre><code class="lang-python">self.upsample=TransposeX2()
def forward(self, x):
    a = self.upsample(x)
    b = self.upsample(a)
</code></pre>
<p>Is it possible to create layer without setting input_channels, but make layer accept any input channels.</p>
<p>I’ve tried to do this as shown below, but i think thats not a proper way to do this, and also onnx convertion show errors, when I create layers like this</p>
<pre><code class="lang-python">def forward(self, x):
    ic = x.shape[1]
    oc = (x.shape[1]//2)
    x = nn.ConvTranspose2d(ic, oc, kernel_size=4, stride=2, padding=1, groups=oc).to(x.device)(x)
    x = nn.BatchNorm2d(oc).to(x.device)(x)
    x= nn.ReLU().to(x.device)(x)
    return x
</code></pre>
<p>I’ve also tried functional api <code>F.conv_transpose2d</code>, but it does wierd stuff (or I did smth wrong) and onnx conversion does not accept this too.</p>
<p>Thank you for help!</p>

eqy: <p>Fundamentally I don’t think it is possible for a <em>single</em> conv2d transpose layer to accept any number of input channels, as the size of the weights depends on this parameter! What does it mean when we need more weights than we have, or when we want to ignore some existing weights due to the variation in input channels?</p>
<p>Instead, a workaround you might try is to simply define some number of conv2d_transpose layers ahead of time and to dispatch to the correct one depending on current shape.<br>
For example:</p>
<pre><code class="lang-auto">def forward(self, x):
    if x.size(0) == size1:
        x = layer1(x)
    elif x.size(0) == size2:
        x = layer2(x)
...
</code></pre>

FeHng: <p>Thanks, it solved my problem perfectly</p>
DataParallel and custom gpu numbers

KanZa: <p>I want to use data-parallel for the following code, but I am confused about how to implement it CUDA_VISIBLE_DEVICES is set as 0 for 1 GPU in a separate file.<br>
I will give the number of GPUs according to availability --gpu 4 5 6</p>
<p>facing the following error<br>
self.module.to(self.src_device_obj)<br>
AttributeError: module ‘torchvision.models’ has no attribute ‘to’</p>
<pre><code class="lang-auto">parser = argparse.ArgumentParser()
#list of parser
if __name__ == '__main__':
     .  
     .
     train_loader = DataLoader(data_train, batch_size=BATCH, 
                                    sampler=SubsetRandomSampler(labeled_set), 
                                    pin_memory=True, drop_last=True)
     test_loader  = DataLoader(data_test, batch_size=BATCH)
     dataloaders  = {'train': train_loader, 'test': test_loader}
     
     #Data Parallel here I will give gpus 
     models = torch.nn.DataParallel(models , args.gpus).cuda()
 
     for cycle in range(CYCLES):
            
            # Randomly sample 10000 unlabeled data points
            if not args.total:
                random.shuffle(unlabeled_set)
                subset = unlabeled_set[:SUBSET]   #SubSet=10,000
                #num = sys.getsizeof(subset)
                #print(num)   #80056

            # Model - create new instance for every cycle so that it resets
            with torch.cuda.device(CUDA_VISIBLE_DEVICES):
                if args.dataset == "fashionmnist":
                    resnet18    = resnet.ResNet18fm(num_classes=NO_CLASSES).cuda()
                else:
                    #resnet18    = vgg11().cuda() 
                    resnet18    = resnet.ResNet18(num_classes=NO_CLASSES).cuda()
                if method == 'lloss':
                    #loss_module = LossNet(feature_sizes=[16,8,4,2], num_channels=[128,128,256,512]).cuda()
                    loss_module = LossNet().cuda()

            models      = {'backbone': resnet18}
            if method =='lloss':
                models = {'backbone': resnet18, 'module': loss_module}
            
            if method =='lloss':
                    models = {'backbone': resnet18, 'module': loss_module}

            torch.backends.cudnn.benchmark = True
</code></pre>
How can I set my batchsize as a function that in every batch one sample from each class presented

S_M: <p>Hi all<br>
I want my batch in each iteration of epoch covers all claasses from my dataset. For example if MNIST is selected, the first batch will be like (0, 1, 2, 3, 4, 5, 6, 7, 8, 9). It covers the samples from all classes (i.e. 10 classes with labels 0 to 9).<br>
Any example code in pytorch would be very appreciated.</p>

thecho7: <p>I know what you’re trying but what if your dataset is imbalanced?<br>
I mean you can obtain (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) at the beginning but it will become to (4, 4, 4, 4, 4, 4, 4, 4, 4, 4) at the end of training (assuming it has a lot of 4 more than other numbers).</p>
<p>Consequently, I suggest you to use <code>WeightedRandomSampler</code>.<br>
I give you an example <a href="https://github.com/ptrblck/pytorch_misc/blob/master/weighted_sampling.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch_misc/weighted_sampling.py at master · ptrblck/pytorch_misc · GitHub</a></p>
<p>Thanks</p>

S_M: <p>Hi thanks for your time to answer me. I am confused about how to put all samples together in a batch and for simplicity, I assumed my dataset is balanced. I mean as you know the default method for batch collecting is random for example in the first batch you may have<br>
(1, 2, 0, 4, 8, 4, 6, 3, 0, 4)<br>
we don’t have any samples from classes 3, 5, 7, and 9.<br>
I need to observe all labels in every batch.</p>
What and why use inputs (image tensors).cpu().data[j]?

Haizhuolaojisite: <p>Why we need to firstly convert inputs tensors to cpu, and then get tensor values by calling data()?<br>
What is the data attribute used for and what will it return? If I call this visualize_model on GPU, do I need to convert to cpu tensors, or I can directly imshow(inputs.data[j])?</p>
<p>Can anyone help explain please? Thanks in advance!!</p>
<pre><code class="lang-auto">def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)
</code></pre>

thecho7: <ol>
<li>
<code>pyplot</code> doesn’t support the functions on GPU.<br>
This is why you should copy the tensor by <code>.cpu()</code>.</li>
<li>As I know, <code>.data</code> is deprecated. You don’t need to use that. But <code>pyplot</code> doesn’t support the plot function with <code>torch.Tensor</code> so it looks good to convert the tensor into numpy array by <code>.numpy()</code>.</li>
<li>You can directly use <code>imshow(input.cpu().numpy())</code>
</li>
</ol>
How to choose "dim =0/1" for softmax or logsoftmax

Rahul_Yadav: <p>Hi,</p>
<p>What are criteria for choosing “dim=0 or 1” for nn.Softmax and nn.LogSoftmax</p>

ptrblck: <p>Usually you would like to normalize the probabilities (log probabilities) in the feature dimension (dim1) and treat the samples in the batch independently (dim0).<br>
If you apply <code>F.softmax(logits, dim=1)</code>, the probabilities for each sample will sum to 1:</p>
<pre><code class="lang-python"># 4 samples, 2 output classes
logits = torch.randn(4, 2)
print(F.softmax(logits, dim=1))
&gt; tensor([[0.7869, 0.2131],
        [0.4869, 0.5131],
        [0.2928, 0.7072],
        [0.2506, 0.7494]])
</code></pre>

Haizhuolaojisite: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I found PyTorch official example use dim=0 for muiticlass classification. Why it use dim=0 here?<br>
<a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a><br>
The function is as below:</p>
<h1>
<a name="helper-functions-1" class="anchor" href="#helper-functions-1"></a>helper functions</h1>
<pre><code class="lang-auto">def images_to_probs(net, images):
    '''
    Generates predictions and corresponding probabilities from a trained
    network and a list of images
    '''
    output = net(images)
    # convert output probabilities to predicted class
    _, preds_tensor = torch.max(output, 1)
    preds = np.squeeze(preds_tensor.numpy())
    return preds, [F.softmax(el, dim=0)[i].item() for i, el in zip(preds, output)]
</code></pre>

ptrblck: <p>The posted code snippet iterates <code>output</code> so <code>el</code> won’t have the batch dimension anymore and <code>dim=0</code> will thus use the “class dimension”.</p>
DICOM images in Unet segmentation model

Giulia_Carvalhal: <p>Hi everyone!</p>
<p>I have a U-net segmentation model that I used to deal with tiff images. Now I am working with DICOM. And the results of my model it has become really bad and weird. I did make some changes in my code to work with DICOM, but maybe he is not correct yet?</p>
<pre><code class="lang-auto">#pydicom to array, resize, normalize
        image=pydicom.read_file(image_path)
        image=image.pixel_array
        image = (image - np.min(image)) / (np.max(image) - np.min(image))
        image= np.array(image, dtype=np.float32)
        image = Image.fromarray(image)
        image = image.resize([int(image.width*0.1), int(image.height*0.1)], Image.LANCZOS) 
        image = np.array(image, dtype=np.float32)


</code></pre>
<p>Am I doing something wrong?</p>
<p>Best,<br>
Giulia</p>

ptrblck: <p>Your code looks generally OK, but I would verify what exactly:</p>
<pre><code class="lang-python">image = Image.fromarray(image)
</code></pre>
<p>is doing.<br>
Its input should be a normalized (in <code>[0, 1]</code>) <code>np.array</code> in <code>float32</code>, but assuming your DICOM has multiple slices I would expect <code>Image.fromarray</code> to fail.</p>
MSELoss - issues with reduction

lazypanda: <p>Hello.<br>
I am trying to train a basic autoencoder(seq2seq) using LSTMs.<br>
I’m using the MSELoss for loss calculation. The issue is that I am getting NaN values when I set reduction to mean or none. However,  when reduction is set to sum, I’m getting expected large values.<br>
I’m fairly new to pytorch, so could someone please explain this ? Is this an expected action or a bug ?<br>
Please note:<br>
 → reduction=none/mean gives proper loss values for a small dummy dataset. However, my real dataset(quite large-librispeech) gives NaN values.<br>
–&gt;I have normalized my dataset.<br>
 → The data is padded with zeros, but I have used pack_padded_sequence () before feeding into the encoder.</p>
<p>I would like to understand more about this. Thank you.</p>

suraj.pt: <p>Hi <a class="mention" href="/u/lazypanda">@lazypanda</a> , can you share a minimal example to reproduce this?</p>

lazypanda: <p>Hi, thanks for your reply.<br>
I am actually trying to run the following code for Librispeech:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/sidwa/Audio2Vec">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/sidwa/Audio2Vec" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/d3d67ab43ceeb91625ec1f329246b9e58ae0ceeb137e066cef5c28d412be5ea1/sidwa/Audio2Vec" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/sidwa/Audio2Vec" target="_blank" rel="noopener nofollow ugc">GitHub - sidwa/Audio2Vec: Complete Pytorch code for model with data pipeline</a></h3>

  <p>Complete Pytorch code for model with data pipeline - GitHub - sidwa/Audio2Vec: Complete Pytorch code for model with data pipeline</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have written a small script to create a MFCC dataset from audio samples of Librispeech. I can share that with you if you need it. Please let me know. Hope this helps. Thanks.</p>

suraj.pt: <p>Yes, please share the <strong>minimal working example (MWE)</strong> i.e. only the code required to reproduce this error on our end. Also, generally reducing your query to a MWE helps you implicitly debug your issue <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Forward hook and input of type generator

Hatem: <p>I want to attach a forward hook to a module with an Input of type Generator and do some operation using the number of elements in the generator and the shape of the elements, how could it be done?<br>
Note:<br>
The generator is finite.<br>
Example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class Sum(nn.Module):
    def __init__(self):
        super(Sum, self).__init__()
    def forward(self, x):
        return sum(x)
def sum_hook(mod, input, output):
    print(type(input[0]))
    print(len(list(input[0]))) # 0 
s = Sum()
h = s.register_forward_hook(sum_hook)
i = torch.randn((1,2,3,4))
x = s((i for _ in range(3))) # generator
x = s([i for _ in range(3)]) # list
h.remove()
</code></pre>
<p>My explanation is that the generator is consumed and that’s why the input length is zero. Is there a workaround? (I know that one can use a list, and it works, but using the generator is faster)</p>
<p>Thank you in advance.</p>
How to read an image created with the legacy torchtext?

DodoI: <p>Hi,<br>
I build plenty of models before I forced myself to upgrade the pytorch/torchtext. All the model images include also vocabs. I am saving them within torch.save() Problem is that the old Vocab is now in torchtext.legacy. When I read an image with pytorch.load() then the vocab type is: &lt;class ‘torchtext.vocab.Vocab’&gt;. But the image is actually legacy Vocab and of course it does not work.<br>
How can I get the legacy Vocab from the image so I can convert it to the new one?</p>
<p>Thanks for any hint</p>
<p>Dodo</p>
Pytorch's AUC Meter

aleemsidra: <p>I am using PyTorch’s <code>AUCMeter</code> to calculate the AUC value. In the instances when target is all 0’s,<code> tpr</code> is coming to be <code>NaN</code>, as its formula is: <code>tpr /= (self.targets.sum() * 1.0)</code>. While <code>fpr</code> comes to <code>NaN</code> when all targets are 1 as its formula is <code>fpr /= ((self.targets - 1.0).sum() * -1.0)</code>. How to calculate <code>tpr</code> and <code>fpr</code> in such instances and avoid <code>NaN</code>? Below is the complete code of <code>AUCMeter</code></p>
<pre><code>def __init__(self):
        super(AUCMeter, self).__init__()
        self.reset()

def reset(self):
        self.scores = torch.DoubleTensor(torch.DoubleStorage()).numpy()
        self.targets = torch.LongTensor(torch.LongStorage()).numpy()

def add(self, output, target):
        if torch.is_tensor(output):
            output = output.cpu().squeeze().numpy()
        if torch.is_tensor(target):
            target = target.cpu().squeeze().numpy()
        elif isinstance(target, numbers.Number):
            target = np.asarray([target])
        assert np.ndim(output) == 1, \
            'wrong output size (1D expected)'
        assert np.ndim(target) == 1, \
            'wrong target size (1D expected)'
        assert output.shape[0] == target.shape[0], \
            'number of outputs and targets does not match'
        assert np.all(np.add(np.equal(target, 1), np.equal(target, 0))), \
            'targets should be binary (0, 1)'

        self.scores = np.append(self.scores, output)
        self.targets = np.append(self.targets, target)
        print("scores", self.scores)
        print("targets", self.targets )

def value(self):
        # case when number of elements added are 0
        if self.scores.shape[0] == 0:
            return (0.5, 0.0, 0.0)

        # sorting the arrays
        scores, sortind = torch.sort(torch.from_numpy(
            self.scores), dim=0, descending=True)
        scores = scores.numpy()
        sortind = sortind.numpy()

        # creating the roc curve
        tpr = np.zeros(shape=(scores.size + 1), dtype=np.float64)
        fpr = np.zeros(shape=(scores.size + 1), dtype=np.float64)

        for i in range(1, scores.size + 1):
            if self.targets[sortind[i - 1]] == 1:
                tpr[i] = tpr[i - 1] + 1
                fpr[i] = fpr[i - 1]
            else:
                tpr[i] = tpr[i - 1]
                fpr[i] = fpr[i - 1] + 1

        tpr /= (self.targets.sum() * 1.0)
        fpr /= ((self.targets - 1.0).sum() * -1.0)
        print('tpr', tpr )
        print("fpr", fpr)
        # calculating area under curve using trapezoidal rule
        n = tpr.shape[0]
        h = fpr[1:n] - fpr[0:n - 1]
        sum_h = np.zeros(fpr.shape)
        sum_h[0:n - 1] = h
        sum_h[1:n] += h
        area = (sum_h * tpr).sum() / 2.0
        print('area')
        return (area, tpr, fpr)</code></pre>

suraj.pt: <p>Hi,<br>
You can add an <code>epsilon</code> to make the denominator non-zero like</p>
<pre><code class="lang-auto">eps = 1e-7
tpr /= ((self.targets.sum() + eps) * 1.0)
fpr /= (((self.targets - 1.0).sum() + eps)  * -1.0)
</code></pre>
nn.Transformer not learning, generates repeated tokens

memduh: <p>Hello, I am getting repeated tokens and the training is not converging while using nn.Transformer. My training loop looks like this:</p>
<pre><code class="lang-auto">tgt_output = tgt_input[1:, :]
decoder_input = tgt_input[:-1, :]
tgt_mask = tgt_mask[:, :-1]
output = model.forward(src_input, src_key_mask=src_mask, 
                       tgt=decoder_input, tgt_key_mask=tgt_mask)
output_flat = output.view(-1, vocab_size)
tgt_flat = tgt_output.flatten()
loss = criterion(output_flat, tgt_flat)
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 0.5)
optimizer.step()
</code></pre>
<p>Where the source and target batches are sequences of token indices. The architecture just learns to predict the most frequent token of them all giving stuff like this when I try to evaluate:</p>
<p><code>[1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]</code></p>

JohnCwok: <p>I am having the same issue.<br>
If you figured it out, may I ask what the problem was?</p>

SenSai_Onegai: <p>Same issue too, exactly same … Have you figured out the problem? I am stuck for quite a long time …</p>
Modifying backward for all modules in the model without rewriting it for all

aynmeme: <p>So lets say I want to mess with the computed gradients at each node in the computational graph before passing it to the previous node. for example say I want to square all computed gradients before passing them back further. yes, I want to mess with the chain rule and backpropagation. But I’d like to know how can i do it without having to write a new backward for every element in my model. what would be the shortest path to accomplish this?</p>

suraj.pt: <p>Hi <a class="mention" href="/u/aynmeme">@aynmeme</a> you might find backward hooks useful - docs: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook" class="inline-onebox">Module — PyTorch 1.10 documentation</a></p>
How to use torchvision cpp api?

Byunghun_Lee: <p>Hi dev.<br>
Now I’m trying to use torchvision nms cpp module in my project.<br>
But, with document in github, I keep failing to build torchvision.</p>
<p>my cmake version is 3.10.2.<br>
and I’m using torchvision release/0.8.0 branch.</p>
<p>with those env, I keep having error as below</p>
<pre><code class="lang-auto">  By not providing "FindPython3.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Python3", but
  CMake did not find one.

  Could not find a package configuration file provided by "Python3" with any
  of the following names:

    Python3Config.cmake
    python3-config.cmake

  Add the installation prefix of "Python3" to CMAKE_PREFIX_PATH or set
  "Python3_DIR" to a directory containing one of the above files.  If
  "Python3" provides a separate development package or SDK, be sure it has
  been installed.
</code></pre>
<pre><code class="lang-auto">CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) 
CMake Warning at /usr/local/lib/libtorch/share/cmake/Caffe2/public/cuda.cmake:31 (message):
  Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  /usr/local/lib/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /usr/local/lib/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:17 (find_package)


CMake Error at /usr/local/lib/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries.  Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
  /usr/local/lib/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:17 (find_package)
</code></pre>
<p>So I tried to downgrade torchvision to version v0.4.0.<br>
But cuda error keep comming up.</p>
<p>If you have any advice about using torchvision cpp api, then pls tell me.</p>
<p>thanks for reading</p>
<p>regards.</p>
File_system sharing strategy in multiprocessing gets collective ops stuck

christianperone: <p>If I use the <code>file_system</code> strategy in multiprocessing with a distributed training, my collective calls through NCCL get stuck and then timeouts, this happens randomly in different times at every training session. Is there anything known that would make <code>file_system</code> strategy fail when doing distributed training ?</p>

pritamdamania87: <p>My first guess is that this is probably not related to <code>file_system</code> strategy. The reason being that the <code>file_system</code> is only used at initialization time and after that there is no dependency on it. I would suggest running your code with environment variables <code>NCCL_DEBUG=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL</code> and share the output of that. This would give a lot more insight into what might be going on here.</p>

christianperone: <p>Thank you <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a>, with the default strategy it works fine but fails due to many file descriptors open, and with the <code>file_system</code> it fails during collective calls. I will try to debug further, but distributed debugging can be pretty difficult.</p>
What is the best way to initialize weighted parameters in a custom loss function?

mark_eu: <p>Hi everyone,</p>
<p>relating to the question <a href="https://discuss.pytorch.org/t/custom-loss-function-with-trainable-parameters/145706">Custom loss with trainable parameters</a>, I have two questions:</p>
<ol>
<li>Range for A and B?</li>
</ol>
<p>Is there a way to constraint A and B to positive values? If yes, is it possible to ensure their values range between 0 and 1?</p>
<ol start="2">
<li>Initialization for A and B</li>
</ol>
<p>What would be the best way to initialize these parameters?</p>
<p>Thank you</p>

mMagmer: <p>I don’t think you will get anything out of it.<br>
Suppose that you have two loss which are adverse to each other in a way.<br>
Then your setup may stuck at a = 0, b =1, then you’re only minimizing your second loss.<br>
But if you want, you can set up it as</p>
<pre><code class="lang-auto">f = #parameter
a = sigmoid(f)
b = 1-a
</code></pre>
<p>This way, weights are positive and will add up to 1.<br>
If you don’t force them to have at least positive sum, they both will converge to zero, and you’re minimizing nothing.<br>
either way you should see these weights as <code>hyper-parameter</code> and grid sreach combine with cross validation is common way for tuning them.</p>
What is the max seed you can set up?

deepfailure: <p>What is the maximum seed one can use with:</p>
<p>torch.manual_seed(seed)</p>
<p>Is it the same maximum seed as with CUDA set seed operations?</p>
<p>Thanks for the answer.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.manual_seed.html?highlight=manual_seed#torch.manual_seed">docs</a> of <code>torch.manual_seed</code>:</p>
<blockquote>
<p><strong>seed</strong> (<a href="https://docs.python.org/3/library/functions.html#int"><em>int</em></a>) – The desired seed. Value must be within the inclusive range [-0x8000_0000_0000_0000, 0xffff_ffff_ffff_ffff]. Otherwise, a RuntimeError is raised. Negative inputs are remapped to positive values with the formula 0xffff_ffff_ffff_ffff + seed.</p>
</blockquote>
<p>So the largest valid value would be <code>2**64-1</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.manual_seed(18446744073709551615) # works
torch.manual_seed(2**64-1) # works

torch.manual_seed(18446744073709551615+1) # fails
# &gt; RuntimeError: Overflow when unpacking long
torch.manual_seed(2**64) # fails
# &gt; RuntimeError: Overflow when unpacking long
</code></pre>
Performance_check

hull11: <p>How do I calculate time spent by DataLoader vs time spent in training each epoch?</p>

ptrblck: <p>You could check the ImageNet example and reuse the <a href="https://github.com/pytorch/examples/blob/e5f075f2ece6700c28fa730f8f23b4b0736a4074/imagenet/main.py#L268"><code>data_time</code></a> counter to check how long the next iteration has to wait for the next batch. Note that this approach is not profiling the <code>DataLoader</code> directly, since it’s preloading the next batches in the background, but could still be useful to check for a data loading bottleneck in your application.</p>
Pytorch parameter initialization

CCanxue: <p>When I initialize the model parameters, I find that the methods of the tensor object do not work, I hope the experts will help me<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a228d734e411b3dad0463807840ce2cbf4f508.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a4a228d734e411b3dad0463807840ce2cbf4f508" title="屏幕截图 2022-03-06 124020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4a228d734e411b3dad0463807840ce2cbf4f508_2_495x500.jpeg" alt="屏幕截图 2022-03-06 124020" data-base62-sha1="nupJhtrJVfx0GxrN1sWi6kS44nC" width="495" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4a228d734e411b3dad0463807840ce2cbf4f508_2_495x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a228d734e411b3dad0463807840ce2cbf4f508.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a228d734e411b3dad0463807840ce2cbf4f508.jpeg 2x" data-dominant-color="EEEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2022-03-06 124020</span><span class="informations">717×723 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t know if your IDE is just claiming this is an error, but in any case you shouldn’t use the <code>.data</code> attribute anymore as it can have unwanted side effects.<br>
Wrap the code into a <code>with torch.no_grad()</code> guard and it should work:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
print(model.weight.mean(), model.weight.std())
# &gt; tensor(0.0017, grad_fn=&lt;MeanBackward0&gt;) tensor(0.1861, grad_fn=&lt;StdBackward0&gt;)

with torch.no_grad():
    model.weight.normal_(0, 1)

print(model.weight.mean(), model.weight.std())
# &gt; tensor(0.0941, grad_fn=&lt;MeanBackward0&gt;) tensor(1.0250, grad_fn=&lt;StdBackward0&gt;)
</code></pre>
He size of tensor a (16) must match the size of tensor b (53) at non-singleton dimension 1

Tornike: <p>Hi, trying to calculate Accuracy over the whole data.</p>
<p>The error is because output of the model is 53 and target tensor size is 16(the batch size)</p>
<p>This is the code:</p>
<pre><code class="lang-auto">for step,batch in enumerate(val_loader):
    ids = batch['input_ids'].to(device, dtype = torch.long)
    mask = batch['attention_mask'].to(device, dtype = torch.long)
    token_type_ids = batch['token_type_ids'].to(device, dtype = torch.long)
    targets = batch['labels'].to(device)

    with torch.no_grad():
      logits=model(ids,mask,token_type_ids)
    loss=loss_f(logits,targets)
    val_loss.append(loss.item())
    preds=torch.argmax(logits,dim=1).flatten()
    **val_acc_=(preds==targets).cpu().numpy().mean()*100** # this is the error
    val_accuracy.append(val_acc_.item())
  val_loss=np.mean(val_loss)
  val_acc=np.mean(val_accuracy)
  
  return val_loss,val_acc
</code></pre>
<p>How do i solve that? or is there any other way to calculate accuracy?</p>

ptrblck: <aside class="quote no-group" data-username="Tornike" data-post="1" data-topic="145636">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tornike/48/36047_2.png" class="avatar"> Tornike:</div>
<blockquote>
<p>The error is because output of the model is 53 and target tensor size is 16(the batch size)</p>
</blockquote>
</aside>
<p>Based on this statement it seems your model changes the batch size of the activations, so check the <code>forward</code> method for <code>view</code> or <code>reshape</code> operations and make sure they are not flattening the batch dimension but use something like <code>x = x.view(x.size(0), -1)</code>.</p>
Graph convolutional network: please can someone help me with this error

Contoss: <p>IndexError                                Traceback (most recent call last)<br>
 in ()<br>
22     for batch in train_loader:<br>
23         optimizer.zero_grad()<br>
—&gt; 24         y_train_prediction = model(batch)<br>
25         loss = MSE(denormalize_output(y_train_prediction, y_val_mean, y_val_std), denormalize_output(batch.y, y_val_mean, y_val_std))<br>
26         loss.backward()</p>
<p>3 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p> in forward(self, data)<br>
15         # x: Node feature matrix of shape [num_nodes, in_channels]<br>
16         # edge_index: Graph connectivity matrix of shape [2, num_edges]<br>
—&gt; 17         x = self.conv1(x, edge_index).relu()<br>
18         x = self.conv2(x, edge_index)<br>
19         return x</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/torch_geometric/nn/conv/gcn_conv.py in forward(self, x, edge_index, edge_weight)<br>
162                 if cache is None:<br>
163                     edge_index, edge_weight = gcn_norm(  # yapf: disable<br>
 → 164                         edge_index, edge_weight, x.size(self.node_dim),<br>
165                         self.improved, self.add_self_loops)<br>
166                     if self.cached:</p>
<p>IndexError: Dimension out of range (expected to be in range of [-1, 0], but got -2)</p>

ptrblck: <p>I guess the <code>x.size</code> operation fails and raises the error as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10)
node_dim = -2
x.size(node_dim)
# &gt; IndexError: Dimension out of range (expected to be in range of [-1, 0], but got -2)
</code></pre>
<p>If so, check the dimensions of <code>x</code> as well as the value of <code>self.node_dim</code> and make sure the latter can be used to get the size of <code>x</code>.</p>
How to install lightning-flash for specific cuda version

Eddous: <p>Hello,<br>
I want to install an older version of lightning-flash, which would run with cuda 10.1. I know there is an option for torch described at <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">https://pytorch.org/get-started/previous-versions/</a>. But I couldn’t find instructions anywhere for lightning-flash.<br>
Thank you very much!</p>
<p>PS: I hope that this forum is also for lightning and flash. If it’s not the case, I apologize.</p>

ptrblck: <p>I’m not deeply familiar with <code>flash</code>, but based on the <a href="https://github.com/PyTorchLightning/lightning-flash">repository</a> I don’t think there is any CUDA dependency, as only Python seems to be used (I also couldn’t find any <code>.cu</code> files there).</p>

Eddous: <p>Thank you for your response. The problem is that I cannot train on GPU because when installing flash, it installs the newest version of the torch, which is not compatible with old cuda =&gt; it’s not possible to train on GPU. So to clarify the question: Is it possible to install an older version of lightning-flash with a specific version of the torch?</p>

ptrblck: <p>I don’t know which dependencies <code>lightning-flash</code> uses, but based on your description it seems a CPU-only PyTorch version is installed. Maybe <code>pip install --no-dependencies ...</code> would help to skip replacing PyTorch with the CPU binary.</p>