akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Where is c++ source code for torch.opt.torchaudio._lfilter?

HyukJin: <p>Hello. I would like to access torch.opt.torchaudio._lfilter, but it seems to be written in C++. Could you possibly inform me of the location of the C++ source code for torch.opt.torchaudio._lfilter?<br>
Thank you.</p>

ptrblck: <p>The host code can be found <a href="https://github.com/pytorch/audio/blob/d5a94336c2f32a31629d49d2b9d417e66e0e4e6c/src/libtorchaudio/lfilter.cpp#L11">here</a> and the CUDA code can be found <a href="https://github.com/pytorch/audio/blob/d5a94336c2f32a31629d49d2b9d417e66e0e4e6c/src/libtorchaudio/iir_cuda.cu#L36">here</a>.</p>
Replace a specific attribute in RecursiveScriptModule

GreyZzzzzzXh: <pre><code class="lang-auto">import torch
from torch import nn

class ADD1(nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        return x + 1

class ADD2(nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        return x + 2

class TEST(nn.Module):
    def __init__(self):
        super().__init__()
        self.add = ADD1()
    def forward(self, x):
        return self.add(x)

x = torch.zeros(1)

test = torch.jit.script(TEST().eval())
print(test.graph)
print(test(x))
# tensor([1.])

new_add = torch.jit.script(ADD2().eval())

# test.add = new_add

# Traceback (most recent call last):
#   File "test2.py", line 41, in &lt;module&gt;
#     test.add = new_add
#   File "/usr/local/lib/python3.8/site-packages/torch/jit/_script.py", line 791, in __setattr__
#     self._modules[attr] = value
#   File "/usr/local/lib/python3.8/site-packages/torch/jit/_script.py", line 244, in __setitem__
#     self._c.setattr(k, v)
# RuntimeError: Expected a value of type '__torch__.ADD1 (of Python compilation unit at: 0x55b702ec2cb0)' for field 'add', but found '__torch__.ADD2 (of Python compilation unit at: 0x55b702ec2cb0)'

test.add._reconstruct(new_add._c)
print(test.graph)
print(test(x))
# tensor([1.])
</code></pre>
<p>How to replace <code>test.add</code> with <code>new_add</code> in torchscript? I tried <code>test.add = new_add</code> and <code>test.add._reconstruct(new_add._c)</code> , but neither of them works.<br>
Thanks</p>
Batch-wise image augmentation with dynamic parameters in PyTorch

teaegg26: <p>I’m currently working on a code for automated data augmentation in PyTorch and I’m wondering if there’s a method to apply a set of augmentations with varying parameters to an entire batch at once.<br>
Here’s how I’m currently implementing it using a loop:</p>
<pre><code class="lang-auto"># Assuming we sample a list of parameters using a parameter search algorithm
theta0 = [1, 0, 1, ..., 0]
theta1 = [58, 75, 44, ..., 14]

# Apply augmentation for each image in each batch
...
for i, (inputs, targets) in enumerate(data_loader):
    transformed_imgs = []
    for j in range(len(inputs)):
        transform = transforms.Compose([
            transforms.RandomHorizontalFlip(theta0[j]),
            lambda x: transforms.functional.rotate(x, theta1[j])
        ])
        im = inputs[j]
        transformed_imgs.append(transform(im))
</code></pre>
<p>To the best of my knowledge, PyTorch enables the application of augmentations with either entirely fixed or entirely random parameters through the <code>torchvision.transforms</code> module.<br>
I came across <a href="https://discuss.pytorch.org/t/applying-different-data-augmentation-per-image-in-a-mini-batch/139136/3">this post</a> from a few years ago, where the response suggests applying augmentations per sample.<br>
Is there an efficient way to achieve this kind of functionality?</p>
<p>Thanks in advance!</p>
Transform Scale

btnbrd: <p>Hello,<br>
transforms ToTensor scales all values from [0, 255] to [0,1], but after Normalize has values from random interval and I have warning from matplotlib. Is it possible to scale values manually?</p>

ptrblck: <aside class="quote no-group" data-username="btnbrd" data-post="1" data-topic="196076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btnbrd/48/62834_2.png" class="avatar"> btnbrd:</div>
<blockquote>
<p>but after Normalize has values from random interval</p>
</blockquote>
</aside>
<p>This is not the case as <code>Normalize</code> will subtract the provided <code>mean</code> from the input and divide by the <code>stddev</code> to standardize the output. These values are not random.</p>
<aside class="quote no-group" data-username="btnbrd" data-post="1" data-topic="196076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btnbrd/48/62834_2.png" class="avatar"> btnbrd:</div>
<blockquote>
<p>Is it possible to scale values manually?</p>
</blockquote>
</aside>
<p>Yes, you can write a custom transformation.</p>
Generating images at each step during DDPM training

Arda: <p>I am working to implement CycleGAN setup for image to text translation. Regarding this, I am trying to use DDPM model as an image generator. But since ddpm learns from the noise, I need to have inference step to produce generated images. But if I incorporate the inference step of the diffusion model during the training phase of the CycleGAN, it could lead to backpropagation errors, effectively breaking the computational graph since there will be no gradients. Do you have any suggestions about this ?</p>
LSTM time-series forecasting with datasets as multiple independent CSV files

CrocodileA: <p>Hello. I am trying to train an LSTM model that can predict/forecast one target using 5 features as network input. My datasets are in CSV files; each file represents an independent scenario that starts from t = 0 s to t = 100 s with a time step of 1 s; which means I cannot stack them together sequentially. For example, let’s say I have 50 CSV files, then each file will have 100 rows (representing the time steps) with a total of 6 columns (5 features + 1 target). In other words, each file is a 2-D matrix of size (100 * 6).</p>
<p>My goal is to use an LSTM to receive a sequence of the 5 features (e.g., seq_length=5) as input and predict/forecast one future value of the target at a time starting from t = 0 to t = 100. I am going to split the 50 datasets into training, validation, and testing sets. My questions are:</p>
<ol>
<li>
<p>How can I efficiently create Pytorch DataSet and DataLoader for the training sets?</p>
</li>
<li>
<p>Is it reasonable to create a Pytorch DataSet and DataLoader for each training file and then loop<br>
through all train files one at a time? Accumulating the loss after each training file, then taking the<br>
average once all training files have been used?</p>
</li>
<li>
<p>Since all my data are time-series data (functions of time), should I or should I not use<br>
shuffle=True when using DataLoader?</p>
</li>
</ol>
<p>I am a beginner in ML and have a basic background in such topics. If my questions are unclear or need clarification, please let me know. Any feedback is much appreciated, thanks in advance!</p>
Copying Graph between Instances

Johannes: <p>Hi all,</p>
<p>I want to copy the computational graph between <code>nn.Parameter</code> tensors. I think the MWE below illustrates my use case in its simplest form.</p>
<pre><code class="lang-auto">class Test(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.p = nn.Parameter(torch.tensor([1.]))

t = Test()
t1 = Test()

t1.p = ( * )

(t1.p ** 2).backward()
assert t1.p.grad == t.p.grad
</code></pre>
<p>What should we put in <code>( * )</code> to get the same gradients for <code>t1.p</code> and <code>t.p</code>, i.e. to have <code>t1.p.grad == t.p.grad</code>?<br>
Thank you very much in advance for your help!</p>

albanD: <p>Hey!</p>
<p>I think the simplest thing to do would be:</p>
<pre><code class="lang-auto">t1.p.grad = torch.zeros_like(t1.p)
t.p.grad = t1.p.grad
</code></pre>
Gradient update query

Salman_Abbasi: <p>Hello</p>
<p>I am trying to update gradeint. Would the following script update the gradient</p>
<p>for grad, param in zip(ss, dnn_Linear.parameters()):<br>
pp=param*Total_loss_model<br>
param=pp<br>
Does the varaiable param, updates the gradient? If yes, do i just need to use optimizer.step() ?</p>
<p>Thanks</p>
CUDNN_STATUS_NOT_INITIALIZED with ditributed training

Darshan_Gera: <p>Hi. I am running the code on four  L4 GPUs  on gcp with distributed training using torch 1.8 and cuda 11.8.<br>
Following code gives<br>
self.proj = nn.Conv3d(<br>
in_channels=in_chans,<br>
out_channels=embed_dim,<br>
kernel_size=(self.tubelet_size, patch_size[0], patch_size[1]),<br>
stride=(self.tubelet_size, patch_size[0], patch_size[1]))<br>
x = self.proj(x).flatten(2).transpose(1, 2)</p>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</p>
<p>Same code works fine if I don’t use distributed training. Please let me know what could be the issue.</p>

ptrblck: <p>Could you update PyTorch to the latest stable release <code>2.1.2+cu121</code> and check if you would still run into this issue?</p>

Darshan_Gera: <p>Thanks. It worked with latest version. But I am not sure why it should not work with older version.</p>
How to implement the transpose of an interpolation

Kai_Lan: <p>I hope to have a function for transpose of a bilinear / trilinear interpolation. For example,</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

x = torch.arange(16).reshape(1, 1, 4, 4).float()
y = torch.arange(16).reshape(1, 1, 2, 2).float()
</code></pre>
<p>I hope to have some downsamping function <code>f</code> such that</p>
<pre><code class="lang-auto">(F.interpolate(y, size=(4, 4), mode='bilinear') * x).sum(dim=(-2,-1)) = (y * f(x, output_size=(2,2)) ).sum(dim=(-2,-1))
</code></pre>
<p>Does such function exist in PyTorch?</p>

ptrblck: <p>Could you describe what the transpose of an interpolation should be?</p>
F1 score - binary vs multiclass with 2 classes

Gillian: <p>Dear all,<br>
For once it’s not really a question that I ask but more something i want to share with you all.<br>
I’ve spent way too much time in the field without knowing the following details. Maybe it was clear for most of you, but i think it’s an important point to raise and the documentation could be clearer about that.</p>
<p>Let’s check the following code.</p>
<pre><code class="lang-auto">import torch
from torcheval.metrics.functional import binary_f1_score, multiclass_f1_score
pred = torch.tensor([0, 1, 1, 0, 1, 0, 1, 0, 1])
target = torch.tensor([0, 1, 1, 0, 0, 0, 0, 0, 0])
print(binary_f1_score(pred, target))   # 0.5714
print(multiclass_f1_score(pred, target, num_classes=2, average='micro'))  # 0.6667
print(multiclass_f1_score(pred, target, num_classes=2, average='macro'))  # 0.6494

from sklearn.metrics import f1_score
print(f1_score(target, pred))  # 0.5714
print(f1_score(target, pred, average='macro'))  # 0.6494
</code></pre>
<p>You can see that the f1_score of scikit learn gives the same results than the binary_f1_score of pytorch, because scikit learn use a default ‘binary’ mode not existing in multiclass_f1_score.</p>
<p>Mr Erwan gives a great explanation here: <a href="https://datascience.stackexchange.com/questions/84953/macro-averaged-in-binary-classification" rel="noopener nofollow ugc">machine learning - Macro averaged in binary classification - Data Science Stack Exchange</a><br>
<em>Commonly there is a majority and a minority class, and naturally the majority class is easier to predict for the classifier. That’s why the minority class is usually chosen as the positive class: by choosing the most difficult class to predict, the performance value represents more precisely the real ability of the classifier. As a consequence, the macro-average performance is often better than the performance on the positive class, since the former includes the “easy” class.</em></p>
<p>In conclusion, when you work with 2 class-classification, be careful if you compare your result with a 2-class averaged f1 score or with a binary f1-score. In my case I had a great difference of 0.5 vs 0.72.</p>
Code for wav2vec2 model WAV2VEC2_ASR_BASE_960H in pytorch

mi_gupta: <p>Hi. I wanted to analyse the layer by layer output of the WAV2VEC2_ASR_BASE_960H model and analyse the whole layer by layer output.<br>
but not able to get the model code anywhere. and<br>
not able to print the torch summary for the same.|<br>
can anyone help me reg this</p>
<p>iam using this code</p>
<p>torchsummary.summary(model,(1,54400)</p>
<p>getting error like expecting 2d input getting (2,1,54400) size</p>

ptrblck: <p><code>torchsummary</code> is deprecated and replaced by <code>torchinfo</code> so try to use it instead.</p>

mi_gupta: <p>Hi. got it. thanks.<br>
can you please tell me where can i get full model code in pytorch for WAV2VEC2_ASR_BASE_960H? thanks in advance</p>

ptrblck: <p>I assume the model can be created using the util. functions from <a href="https://github.com/pytorch/audio/blob/b2d9c3e315405f2b5cfdfa5b93f849d5b27a4109/src/torchaudio/pipelines/_wav2vec2/impl.py#L310">here</a>.</p>

mi_gupta: <p>Thanks. got the code .<br>
still if anywhere i can get the code for WAV2VEC2_ASR_BASE_960H,<br>
i have to write whole thing in C-code./? code from scratch i have to write. it</p>
Nearest elements

prachiPawade: <p>Hi,</p>
<p>How can i find nearest elements for all values in a tensor from another tensor?<br>
For example,<br>
a = [2,8,4,1] and b = [3,12,5,7]<br>
then result should be [3,7,5,3]</p>
<p>Thank you</p>

smth: <p>To find the nearest elements for all values in a tensor <code>a</code> from another tensor <code>b</code> in PyTorch, you can use a technique that involves calculating the absolute difference between each element in <code>a</code> and all elements in <code>b</code>, and then finding the element in <code>b</code> that has the minimum difference for each element in <code>a</code>.</p>
<pre><code class="lang-auto">a = torch.tensor([2, 8, 4, 1])
b = torch.tensor([3, 12, 5, 7])

# Calculate the absolute differences. 
# the unsqueeze here broadcasts a to each row in b
diff = torch.abs(a.unsqueeze(1) - b)

# Find the indices of the nearest elements
nearest_indices = torch.argmin(diff, dim=1)

# Get the nearest elements from b
nearest_elements = b[nearest_indices]
</code></pre>

prachiPawade: <p>Hi Smth,</p>
<p>Thank yo for the response. The answer is very clear.</p>
<p>If in a case, a and b are mutidimentinal (say [2,2]), how to go about it?</p>

prachiPawade: <p>I have tried it in the following manner,</p>
<p>a = torch.tensor([[2,8],[16,25]])</p>
<p>b = torch.tensor([[23,9],[1,13]])</p>
<h1><a name="calculate-the-absolute-differences-1" class="anchor" href="#calculate-the-absolute-differences-1"></a>Calculate the absolute differences.</h1>
<h1><a name="the-unsqueeze-here-broadcasts-a-to-each-row-in-b-2" class="anchor" href="#the-unsqueeze-here-broadcasts-a-to-each-row-in-b-2"></a>the unsqueeze here broadcasts a to each row in b</h1>
<p>diff = torch.abs(b.reshape(4,1) - a.flatten())</p>
<h1><a name="find-the-indices-of-the-nearest-elements-3" class="anchor" href="#find-the-indices-of-the-nearest-elements-3"></a>Find the indices of the nearest elements</h1>
<p>nearest_indices = torch.argmin(diff, dim=1)</p>
<h1><a name="get-the-nearest-elements-from-b-4" class="anchor" href="#get-the-nearest-elements-from-b-4"></a>Get the nearest elements from b</h1>
<p>nearest_elements = a.flatten()[nearest_indices]</p>
<p>nearest_elements = nearest_elements.reshape(a.shape)</p>
<p>But this is working only for small tensors. I want to apply it on big tensors for which the memory requirement is high.</p>
<p>Could you please suggets any feasible way…</p>
<p>Thank you,<br>
Prachi</p>
Faster rcnn with resnet and sift

ricardo_assaad: <p>Hi everyone,</p>
<p>I have a pytorch implementation of faster rcnn with resnet50 and fpn. I was doing some research and learned about feature descriptors and i found out about SIFT and i believe it can be a great addition to my model.<br>
At this point in time i m not sure how to implement it into my architecture to get the model to learn these additional features. I would really appreciate the help and guidance on this matter</p>
<p>Thanks in advance</p>
Gradient Accumulationin pytorch

External_happy: <p>Hello, I’m implementing gradient accumulation in my code to address memory limitations with large batch sizes. The choice of my loss function depends on the overall probability values across the entire batch. Here’s my current code:</p>
<pre><code class="lang-auto">#----------------------- for grad accumulation --------------------------------------
global_step = 0 
# -----------------------------------------------------------------------------------
for step, (x, y, idx) in enumerate(metric_logger.log_every(data_loader_u, print_freq, header)):
    # ----------------------- unlabelled --------------------------------------------
    x = x.to(device, non_blocking=True)
    y = y.to(device, non_blocking=True)
    with torch.cuda.amp.autocast():
        logit = model(x)
        probs_all = F.softmax(logits,dim=-1) 
        probs_batch_avg = probs_all.mean(0) # average prediction probability across all gpus
        # moving average          
        if global_step==0:
            probs_avg = probs_batch_avg
        else:    
            probs_avg = 0.5*(probs_avg.detach()+probs_batch_avg)
        loss_x = -(torch.log(probs_avg)).mean() / 0.5
        
        #----------------------------------------------------------
        loss = (loss_x) / accumulation_steps
    ## ---------------------- for grad accumulation ------------------------
    loss.backward()
    if (data_iter_step + 1) % accumulation_steps == 0:
        global_step+=1
        optimizer.zero_grad()
    torch.cuda.synchronize()
</code></pre>
<p>However, I can’t be sure if the probs_avg  is still computed for the accumulated batch, not the minibatch ?</p>
Regarding using different GPUs, the model effect drops off a cliff.

guanshanmengyin: <p>My result on the training set on Nvidia 1060 was 97%, and the result on the test set was 94%. But after I switched to Nvidia V100, the result of my training set was 100%, and the result of the test set was only 30%. I was very confused about what was causing it.</p>

ptrblck: <p>I would recommend checking the datasets and actual training script as we have seen a few issues in this discussion board claiming a different GPU causes convergence issues while it was related to something else. Just recently <a href="https://discuss.pytorch.org/t/i-got-different-results-from-different-gpus/194661/2">this post</a> claimed the same and it turned out some dataset folders were empty.</p>
How to adjust training hypter-parameters when training on more large dataset?

JohnHerry: <p>Hi, all<br>
I have a question about training strategy optiming when training dataset get over large.<br>
I am using AdamW optimizer, and  ExponentialLR scheduler.  we are training an Text-To-Speech model with 10 speakers and 40000 samples, in a single-card environment.<br>
Initially we call scheduler.step() after each epoch, because the value of steps-per-epoch is not very big. that is fine.<br>
Afterward, We should train another instance on a dataset of over 2000 speakers and millions of sampels, we found that the training became not so stable. It is more likely to fall into suboptimal result, and some times the training process will break down.<br>
Because our GPU resouce can not afford a bigger batch_size,  so this means that the value of steps-per-epoch is very big.<br>
I think there are two ways to adjust the training hyper-parameters:  1、keep the mode that scheduler.step() after each epoch, but give a  bigger lr_decay value of AdamW.  2、make scheduler.step() every N training steps instead of one-epoch.<br>
I do not know which one is better, can anyone provide some experience? Thanks.</p>
Transformer model for series classification

ptrch_c_m: <p>Hello,</p>
<p>the title more or less says it all. I want to use the transformers architecture, i.e. a transformer model, for series classification. I don’t want to predict time series, just to classify them.</p>
<p>The series are not very long, the average length is 220 elements, and I will try to modify it, if there is constraint on the model input. Is there a suitable model for that?</p>
The fastest way to calculate 5mil*5mil dot products? I have two A100 and 96 core CPU with 500gb memory

vprokopev: <p>I need to multiply a 5mil*768 matrix by it’s transpose. So, approximately 5mil**2 dot products of vectors vectors of 768 elements. As mentioned in title, my resources are:</p>
<ul>
<li>96 core CPU.</li>
<li>500gb of memory</li>
<li>Two Nvidia A100 GPUs.</li>
</ul>
<p>Please, help me, how do I do it the fastest, considering my memory (and GPU memory) constraints.</p>