akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataset from variable shaped arrays

username: jambato
cooked: <p>I have a directory with 5000 numpy files, each containing an array of shape [n, m] where <em>n</em> is the number of sequences of length <em>m</em>. The number of sequences <em>n</em> is different for each file and each sequence constitutes a training sample. So far my solution has been to iterate over every file and create a list containing all the sequences, the list is then fed to<code> __getitem__</code>. This solutions consumes too much memory and slows down training. Is there a more efficient way to build the Dataset?</p>
<pre><code class="lang-auto">class SequenceDataset(Data.Dataset):
    def __init__(self, signal_dir):
        file_list = os.listdir(signal_dir)
        self.file_count = len(file_list)
        print('file number:', self.file_count)
        self.file_list = file_list
        self.signal_path = signal_dir

        self.idx = 0
        self.signal_len = self.get_signal_length()

        self.signals_pool,  self.len_signals =self.getpool() 


    def __len__(self):
        return self.len_signals

    def get_signal_length(self):
        signal = np.load(self.signal_path+'/'+self.file_list[0])
        signal_length = signal.shape[1]
        return signal_length
  
    def getpool(self):
        signals = []
        counter = 0
        for f in self.file_list:
            signal = np.load(self.signal_path+'/'+f)
            signals.append(signal)

        signal_array = torch.from_numpy(np.concatenate(signals))
        
        len_signals = signal_array.shape[0]

        return signal_array.unsqueeze(2), len_signals

    def __getitem__(self, item):
        signal = self.signals_pool[item]
        
        return signal
</code></pre>

username: ptrblck
cooked: <p>Instead of preloading the data in the <code>__init__</code> method you could store the paths only and load and process each file lazily in the <code>__getitem__</code> method which would reduce the memory usage.</p>

username: jambato
cooked: <p>Thank you! I will try that!</p>
Monkey-patching the `forward()` pass of an `nn.Module`

username: Sayak_Paul
cooked: <p>I am trying to monkey-patch the <code>forward()</code> method of an <code>nn.Module</code>. Here’s my <code>nn.Module</code>:</p>
<pre><code class="lang-py">import torch.nn as nn 

class GPT5(nn.Module):
    embed_dim = 768
    num_heads = 12
    q_proj = nn.Linear(embed_dim, embed_dim)
    head_dim = embed_dim // num_heads
    scale = head_dim**-0.5

    def forward(self, hidden_states):
        return self.q_proj(hidden_states) * self.scale
</code></pre>
<p>The following works as usual:</p>
<pre><code class="lang-py">import torch 

gpt5 = GPT5()
gpt5(torch.randn(1, 10, 768)).size()
</code></pre>
<p>Do monkey-patching:</p>
<pre><code class="lang-py">gpt5 = GPT5()
new_forward = lambda x: l.forward(x) + 1
gpt5.forward = new_forward
</code></pre>
<p>The following then raises an error:</p>
<pre><code class="lang-py">gpt5(torch.randn(1, 10, 768)).size()
</code></pre>
<pre><code class="lang-bash">─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in &lt;module&gt;:1                                                                                    │
│                                                                                                  │
│ /usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py:1194 in _call_impl             │
│                                                                                                  │
│   1191 │   │   # this function, and just call forward.                                           │
│   1192 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1193 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1194 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1195 │   │   # Do not call functions when jit is used                                          │
│   1196 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1197 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│ in &lt;lambda&gt;:2                                                                                    │
│ in &lt;lambda&gt;:2                                                                                    │
</code></pre>

username: soulitzer
cooked: <p>Does the following work for you?</p>
<pre><code class="lang-python">class GPT5(nn.Module):
    embed_dim = 768
    num_heads = 12
    q_proj = nn.Linear(embed_dim, embed_dim)
    head_dim = embed_dim // num_heads
    scale = head_dim**-0.5

    def forward(self, hidden_states):
        return self.q_proj(hidden_states) * self.scale

gpt5 = GPT5()
old_forward = gpt5.forward

new_forward = lambda x: old_forward(x) + 1
gpt5.forward = new_forward

gpt5(torch.randn(1, 10, 768))

</code></pre>

username: Sayak_Paul
cooked: <p>Thanks! It works. Couldn’t realize the fix would be about separately storing the forward in a variable.</p>
Matrix multiplying single tensor along dimension

username: Givani_Boekestijn
cooked: <p>Hi,<br>
I am having trouble with calculating a specific matrix product.<br>
Lets say I have a single tensor A of size (N,m,m)  and i want to caclulate the matrix product of the N tensors sized (m,m).<br>
So for example if N=3, i want A[0,:,:]<span class="mention">@A</span>[1,:,:]<span class="mention">@A</span>[2,:,:]. Is there a way to do this without using a for loop?</p>

username: soulitzer
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/generated/torch.linalg.multi_dot.html#torch.linalg.multi_dot:" class="inline-onebox" rel="noopener nofollow ugc">torch.linalg.multi_dot — PyTorch 2.0 documentation</a></p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.stack([torch.eye(3) * 2, torch.eye(3) * 3, torch.eye(3) * 5])
&gt;&gt;&gt; a
tensor([[[2., 0., 0.],
         [0., 2., 0.],
         [0., 0., 2.]],

        [[3., 0., 0.],
         [0., 3., 0.],
         [0., 0., 3.]],

        [[5., 0., 0.],
         [0., 5., 0.],
         [0., 0., 5.]]])
&gt;&gt;&gt; a.shape
torch.Size([3, 3, 3])
&gt;&gt;&gt; torch.linalg.multi_dot(a.unbind(0))
tensor([[30.,  0.,  0.],
        [ 0., 30.,  0.],
        [ 0.,  0., 30.]])
&gt;&gt;&gt;
</code></pre>

username: Givani_Boekestijn
cooked: <p>Thanks a lot! It seems the thing I was missing was the .unbind() operation</p>
Why is my loss graph heavily fluctatuing?

username: Zelreedy
cooked: <p>I am working on a CNN project on an image dataset. I am applying Early Stopping technique in order to train the model. However, after training the model and obtaining the loss graph, it is heavily fluctuating, as shown in the image below. I have tried training the model more than once and I get a similar loss graph: <a href="https://i.stack.imgur.com/Jpxh8.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/Jpxh8.png" alt="enter image description here" width="668" height="440"></a></p>
<p>This is my model structure:</p>
<p><a href="https://i.stack.imgur.com/ud9q8.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/ud9q8.png" alt="enter image description here" width="504" height="208"></a></p>
<p>Therefore, is this a normal loss graph? If not, what may be the underlying issue and how can I solve it? I have read that regularization can solve this problem, but I am already applying Dropout. Is there anything else worth investigating?</p>
<p>Note that this is not really affecting my validation accuracy (accuracy on unseen data). It performs decently on it, with 93.3% accuracy.</p>
Corrupt JPEG data: 24 extraneous bytes before marker 0xd9

username: Soori
cooked: <p>Hi,</p>
<p>When I check the folder of my images by below code, there is no corrupted images.<br>
but when I use a custom dataset and start to train the network this error happens during training (it does not stop the training and it will go on after that, but I do not know if it affect the parameters being trained and I wonder how I can handle this).</p>
<p>Thanks,</p>
<p>Epoch: 0<br>
Corrupt JPEG data: 24 extraneous bytes before marker 0xd9<br>
[===========================================================&gt;…]  Step: 147ms | Tot: 11s720ms | Loss: 1.285 | Acc: 57.554% (640/111 12/12 2<br>
[====================================================&gt;…]  Step: 883ms | Tot: 4s311ms | Loss: 0.665 | Acc: 60.587% (289/47 5/5 5<br>
Saving…</p>
<p>annotations = pd.read_csv(datapath, sep=’\t’)<br>
root_dir=’/home/ubuntu/files’<br>
for index in range(len(annotations)):<br>
img_path = os.path.join(root_dir, annotations.loc[index, ‘image_path’])<br>
try:<br>
img = Image.open(img_path)<br>
img.verify()<br>
except:<br>
print(‘Bad file:’, index) # print out the names of corrupt files</p>

username: ptrblck
cooked: <p>I would try to track down the image file, which is raising this error and either remove it or load and save it again to hopefully get rid of the JPEG corruption.</p>

username: Soori
cooked: <p>Thanks. I did the same thing. Using mogrify command line tool (provided by ImageMagick), I could detect which image cause this error.</p>

username: Ko_R
cooked: <p>can you share the command line of mogrify for detection, thanks a lot</p>

username: Tihomir_Teixeira
cooked: <p>Thank you for the help, i had a similar problem whilst trying to send images through an api</p>
What am I missing when using the new pretrained weight initialization

username: Csaba_Botos
cooked: <p>I have attempted at least 10x to put up with the bad habit of using <code>pretrained=True</code> in the past but every time I tried to pass the string <code>weights="IMAGENET1K_V2"</code> I get the</p>
<pre><code class="lang-auto">return cls._member_map_[name]
KeyError: 'IMAGENET1K_V2'
</code></pre>
<p>I must be missing something, should I expect <a href="https://github.com/pytorch/vision/releases/tag/v0.13.0" rel="noopener nofollow ugc">this syntax</a> to work?</p>

username: ptrblck
cooked: <p><code>resnet50(weights="IMAGENET1K_V2")</code> works for me using <code>torchvision==0.15.1+cu118</code>.</p>

username: Csaba_Botos
cooked: <p>I found the bug, for some reason in the slurm instance the right pytorch but the wrong torchvision was linked! my bad <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> thanks!</p>
Composite loss from multiple forward passes with DDP

username: EachOneChew
cooked: <p>I’d just like to confirm that DDP would behave correctly in the following use case. If I’m not mistaken, upon <code>loss.backward()</code> each parameter should accumulate the <strong>combined</strong> gradient from <code>loss_1</code> and <code>loss_2</code>, so the gradient synchronization should only happen once? If I’m correct, Does this extend to composite losses with an arbitrary number of components?</p>
<pre><code class="lang-auto">model = DDP(some_model)

training loop:
    p1 = model(x1)
    p2 = model(x2)

    loss = loss_1(p1) + loss_2(p2)
    loss.backward()
    optimizer.step()
</code></pre>
<p>Quick update, I’ve read <a href="https://discuss.pytorch.org/t/multi-model-and-multi-forward-in-distirbuted-data-parallel/84503/2">this thread</a> and it seems this will not work. Please let me know if anything has changed since then.</p>

username: rvarm1
cooked: <p>Gradient synchronization should indeed happen only once.</p>
<p>In general, DDP’s usability should be the same as that of local training, so if this approach isn’t giving correct results, please file a github issue with a minimal repro.</p>

username: EachOneChew
cooked: <p>I see, thank you. Unfortunately it does seem like running forward twice does not work under DDP; I got around it by wrapping both forward calls into a single parent module’s forward, but I might have made a mistake somewhere else.</p>
Applying weiner filter and pytorch iffrt error

username: asifkhan
cooked: <p>Hi!</p>
<p>I need help and kind guidance. I am trying to implement and apply a wiener filter on the input image of the shape:<br>
input = [1,3,128,128]<br>
<strong>1)</strong> When I try to perform iffrt:</p>
<p>return torch.irfft(Y.div(G),2,signal_sizes = input.shape[-2:]), c</p>
<p>the following error occurred :</p>
<p>AttributeError: module ‘torch’ has no attribute ‘irfft’</p>
<p><strong>2)</strong> Following the above error I also tried with fft</p>
<p>return torch.fft.irfft(Y.div(G),2,signal_sizes = input.shape[-2:]), c</p>
<p>the following error occurred:</p>
<p>TypeError: fft_irfft() got an unexpected keyword argument ‘signal_sizes’</p>

username: ptrblck
cooked: <p>The second approach is correct and you should use <code>torch.fft.irfft</code>.<br>
This call does not accept a <code>signal_sizes</code> argument as seen in <a href="https://pytorch.org/docs/stable/generated/torch.fft.irfft.html#torch.fft.irfft">the docs</a>.</p>

username: asifkhan
cooked: <aside class="quote no-group" data-username="asifkhan" data-post="1" data-topic="175985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d2c977/48.png" class="avatar"> asifkhan:</div>
<blockquote>
<p>return torch.fft.irfft(Y.div(G),2,signal_sizes = input.shape[-2:]), c</p>
</blockquote>
</aside>
<p>Thanks a lot for your kind response.</p>
<p>I followed the doc you mentioned in your reply and pass n and dim parameters but got errors :</p>
<p>Y = [1,1,3,128,128,2]</p>
<p>G = [1,4,3,128,128,1]</p>
<ol>
<li>return torch.fft.irfft(Y.div(G),2, n = input.shape[-2:])</li>
</ol>
<p>TypeError: fft_irfft() got multiple values for argument ‘n’</p>
<ol start="2">
<li>return torch.fft.irfft(Y.div(G),2, dim = input.shape[-2:])</li>
</ol>
<p>TypeError: fft_irfft(): argument ‘dim’ must be int, not torch.Size</p>
<p>I want to use “input.shape[-2:]” or something equivalent to it but no idea how to do it in torch.fft.irfft.</p>
<p><em>Output should be like ==&gt; batch x N x channels x height x width</em> =&gt; 1x4x3x128x128</p>
Change labels of Subset

username: Xiaa
cooked: <p>Hi,</p>
<p>I need help to find a way to change the labels of my data (ex: change all 5 into 7).</p>
<p>I am using the MNIST dataset, and split it in several subsets that I save in files and load later. For my experiments I cannot directly use the Subsets, I have to save them in a file and load them after. And I cannot change the label before saving the data.</p>
<p>This is how I load my data:</p>
<pre><code class="lang-auto">class SubsetLoader(Dataset):
    def __init__(self, filename):
        self.data = torch.load(filename)

    def __getitem__(self, index):
        return self.data[index]
    
    def __len__(self):
        return len(self.data)
</code></pre>
<p>I tried this among other solutions: <code>dataset.targets[dataset.targets == 5] = 7</code> after loading the data.<br>
But subset has no attribute ‘targets’ and ‘SubsetLoader’ object has no attribute ‘targets’.</p>
<p>Is there a way to change the labels of my subsets then ?</p>

username: thecho7
cooked: <p>Please give me a code snippet that how you saved those subsets</p>

username: Xiaa
cooked: <p>This is how I save the data:</p>
<pre><code class="lang-auto">mnist_dataset = datasets.MNIST(root='../data', train=True, download=True, transform=transform)

def random_split_strat(clients):
    nb_clients = clients
    num_classes = 10 

    r = []

    a = [ran.random() for _ in range(nb_clients)]
    s = sum(a)
    a = [ i/s for i in a ]

    for i in range(len(a)):
        r.append(a[i])

    mnist_subsets = random_split(mnist_dataset, r, generator=torch.Generator().manual_seed(42))

    for i, subset in enumerate(mnist_subsets):
        filename = f'mnist_subset_{i}.pt'
        torch.save(subset, path+filename)    

</code></pre>

username: thecho7
cooked: <aside class="quote no-group" data-username="Xiaa" data-post="1" data-topic="175958">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xiaa/48/58846_2.png" class="avatar"> Xiaa:</div>
<blockquote>
<p>dataset.targets[dataset.targets == 5] = 7</p>
</blockquote>
</aside>
<p>Have you tried the above before saving the subsets?</p>
<pre><code class="lang-auto">mnist_dataset.targets[mnist_dataset.targets == 5] = 7
</code></pre>

username: Xiaa
cooked: <p>Unfortunately I have to save the data without modifying the labels, and then change the labels when I load them</p>

username: thecho7
cooked: <p>Here’s a sample code</p>
<pre><code class="lang-auto">f = torch.load("mnist_subset_0.pt")
f.dataset.targets[f.dataset.targets == 5] = 7
</code></pre>
<p>should work</p>

username: Xiaa
cooked: <p>This works ! Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
SystemExit: 2 error when calling parse_args() while using MTCNN

username: Niraja
cooked: <p>I am using: from facenet_pytorch import MTCNN<br>
I am getting this error while running following code.<br>
“”"<br>
parser = argparse.ArgumentParser()<br>
parser.add_argument(“-i”, “–input”, required = True,<br>
help = ‘path to the input data’)<br>
<span class="hashtag">#parser</span>.add_argument(“-f”, required=False)</p>
<p>args = vars(parser.parse_args(“-i /–input”))<br>
“”"<br>
This error has been written while running this code<br>
usage: ipykernel_launcher.py [-h] -i INPUT<br>
ipykernel_launcher.py: error: the following arguments are required: -i/–input<br>
An exception has occurred, use %tb to see the full traceback.</p>
<p>I don’t understand how to solve this error. I am first time using MTCNN and argparse. Please help me to solve this error. Thank you in advance</p>
RAM gradullay increase during training and the priocess is killed by OS

username: two_Two
cooked: <p>I met the situation that the usage of RAM gradullay increasing when the <code>num_worker</code> is set too large, I would like to know why this happen, <a href="https://www.kaggle.com/code/shoheiazuma/stable-diffusion-vit-baseline-train" rel="noopener nofollow ugc">the code that I use</a> and the dataset and data collector part is shown below:</p>
<pre><code class="lang-auto">class DiffusionDataset(Dataset):
    def __init__(self, df, transform):
        self.df = df
        self.transform = transform
        
    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        row = self.df.iloc[idx]
        image = Image.open(row['filepath'])
        image = self.transform(image)
        prompt = row['prompt']
        return image, prompt


class DiffusionCollator:
    def __init__(self):
        self.st_model = SentenceTransformer(
            '/kaggle/input/sentence-transformers-222/all-MiniLM-L6-v2',
            device='cpu'
        )
    
    def __call__(self, batch):
        images, prompts = zip(*batch)
        images = torch.stack(images)
        prompt_embeddings = self.st_model.encode(
            prompts, 
            show_progress_bar=False, 
            convert_to_tensor=True
        )
        return images, prompt_embeddings
</code></pre>
Torch.save() OSError: [Errno 22] Invalid argument:

username: Pxstorxs
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a4c95397c802459af3c5efb9b71873ab93a133.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9a4c95397c802459af3c5efb9b71873ab93a133" title="Screenshot_4"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a4c95397c802459af3c5efb9b71873ab93a133.png" alt="Screenshot_4" data-base62-sha1="xkUiQfNwci9FdVH8iNkPVopcq7F" width="690" height="183" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_4</span><span class="informations">839×223 8.26 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Any ideas on why does this happen?</p>

username: ptrblck
cooked: <p>I guess Windows doesn’t allow <code>:</code> in a filename as it’s used for their drive letters, so remove it and try to save the file again.</p>

username: Peter_Wayne
cooked: <p>Thank you very much! Hope you all the best!</p>
Image resolution for U-Net with pretrained ResNet34

username: Anastasia_Berlianna
cooked: <p>hi, i’m using pretrained resnet34 for my unet encoder, for input images resolution, does it have to be 224x224 or is it okay to use 256x256? thanks!</p>

username: ptrblck
cooked: <p>Classification models provided by <code>torchvision.models</code> accept inputs with variable input shapes as long as the spatial size is not too small.<br>
Assuming you don’t see any functionality issues, I would guess your approach should work.</p>

username: Anastasia_Berlianna
cooked: <p>Yes, but i have this issue<br>
<a href="https://discuss.pytorch.org/t/u-net-cant-perfectly-segment-image/175974">U-Net segmentation result</a></p>
Fork() is significantly slower than spawn() in PyTorch DataLoader

username: francescocarzaniga
cooked: <p>I am loading an HDF5 file in a Dataset (I am making sure that everything is picklable, so that is not a problem) and using DataLoader with multiprocessing to read multiple chunks at a time.</p>
<p>When I leave the fork context as default there is no performance improvement in passing from 0 workers to 10, i.e. it takes more time to load a 32-item batch with multiprocessing than without. When I use the spawn (or forkserver) context, it takes a while to spawn the workers but then the data loading is significantly faster.</p>
<p>I am wondering if this tradeoff is expected or if there is something wrong with fork multiprocessing.</p>
<p>Thanks a lot!</p>

username: soulitzer
cooked: <p>Does using fork take up all your memory? Maybe its resorting to swap?</p>

username: francescocarzaniga
cooked: <p>Doesn’t look like it, I still have tons of free RAM during execution.</p>
<p>It seems that the slowdown is related to the reading operation inside <code>__getitem__</code>. When I get rid of the <code>sample = h5_file[a:b]</code> line (just as example of what reading looks like) forking gives the expected speedup.</p>
Cant get GPU support working on Windows 10

username: Frizz
cooked: <p>I cant get my GPU working with Pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=12" title=":worried:" class="emoji" alt=":worried:" loading="lazy" width="20" height="20"></p>
<ul>
<li>clean Notebook, fresh Windows 10 installation.</li>
<li>newest Nvidia drivers, CUDA is installed and the demos are working fine.</li>
<li>Pytorch is installed as per instruction from the official webpage: conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia</li>
</ul>
<p>Still:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.version.cuda)
None
</code></pre>
<p>Do I have to specifically tell Python to use the CUDA version of Pytorch? What am I missing?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51f28e9ae9c2264aef5f855714962b464c0f962b.png" data-download-href="https://discuss.pytorch.org/uploads/default/51f28e9ae9c2264aef5f855714962b464c0f962b" title="image2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51f28e9ae9c2264aef5f855714962b464c0f962b.png" alt="image2" data-base62-sha1="bGWlLYTv5q5rGPVZe7NbHacbytR" width="690" height="405" data-dominant-color="121111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image2</span><span class="informations">1045×614 35.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>What does <code>print(torch.version.cuda)</code> return in your environment?</p>
Include torchvision in singularity image

username: Carsten_Ditzel
cooked: <p>I have been trying to set up a singularity container containing PyTorch via this <a href="https://hub.docker.com/r/pytorch/pytorch/tags" rel="nofollow noopener">image</a> and it seems that it works out fine. I have, however, some trouble including torchvision in my setup. I tried the following lines without much success</p>
<pre><code class="lang-auto">Bootstrap: docker
From: pytorch/pytorch:latest

%runscript
    exec echo "Building singularity image..."

%post
apt update &amp;&amp; apt install -y python3-pip
python3 -m pip install --upgrade pip
python3 -m pip install torchvision
pip --no-cache-dir install --upgrade torchvision
pip install torchvision
python3 -m pip install numpy
python3 -m pip install matplotlib
python3 -m pip install h5py
</code></pre>
<p>but when I try to queue in a job with slurm I get the error that torchvision cannot be found. Since there is no separate image for torchvision what is the most suitable way to include torchvision as a dependency?</p>

username: Carsten_Ditzel
cooked: <p>noone here working with singularity images?</p>

username: Jonas_Muller
cooked: <p>I have the same problem</p>
Compile seems targting cuda:0 rather than the device the model is on

username: heihei
cooked: <p>The pytorch compile works great when I only have one GPU (gtx 1660 super, no tensor cores) installed on my computer. However, when I installed a new GPU card RTX 3060 (has tensor cores), which becomes the new “cuda:0”, even though I still chose gtx 1660 (“cuda:1” now) to run the model, the pytorch compile still choose rtx 3060, which has tensor cores as the targeted archecture. And I got the following warning:</p>
<pre><code class="lang-auto">
compile_fx.py:90: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.

</code></pre>
<p>and error:</p>
<pre><code class="lang-auto">
compiler.py", line 1671, in _init_handles

mod, func, n_regs, n_spills = cuda_utils.load_binary(self.metadata["name"], self.asm["cubin"], self.shared, device)

RuntimeError: Triton Error [CUDA]: device kernel image is invalid

</code></pre>
<p>Is there a way I can specify the device to the torch.compile() function?</p>

username: marksaroufim
cooked: <p>Hmm we might need to clean up the warning to work better in multi GPU setup and as far as not respecting the device that seems like a bug so please open an issue here <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a></p>

username: heihei
cooked: <p>I have created an issue: <a href="https://github.com/pytorch/pytorch/issues/97693" class="inline-onebox" rel="noopener nofollow ugc">Compile targts cuda:0 rather than the device the model is on · Issue #97693 · pytorch/pytorch · GitHub</a></p>
Compare weights of models ddp after backward

username: Richard_Jenkins
cooked: <p>I am using DDP with 4 GPUs on a single node. I want to print and compare the weights of the 4 models after the syncing backward() call to prove to myself that they are all equal after the backward().  FYI I only do a backward() that syncs to the other model ranks in the last training iteration of a batch. (All other backward() calls are using the no_sync() context).</p>
<p>My idea is to print this at the top of the validation function, not after the backward() statement in the training function in case there is a timing issue with the 4 processes doing their updates asynchronously.</p>
<p>At the top of the validation function print the sum of the model weights and the GPU rank ID.</p>
<p>I am borrowing a function from here…</p>
<aside class="quote" data-post="10" data-topic="77672">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/access-all-weights-of-a-model/77672/10">Access all weights of a model</a> 
  </div>
  <blockquote>
    You could flatten each parameter, append it in a list, and create a tensor: 
model = models.resnet18()

params = []
for param in model.parameters():
    params.append(param.view(-1))
params = torch.cat(params)
print(params.shape)
&gt; torch.Size([11689512])
  </blockquote>
</aside>

<p>model = models.resnet18()</p>
<p>params = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for param in model.parameters():<br>
params.append(param.view(-1))<br>
params = torch.cat(params)<br>
print(params.shape)</p>
<blockquote>
<p>torch.Size([11689512])</p>
</blockquote>
<p>I only want the Params where it’s a weight param, not a bias param.<br>
model.parameters() will return all trainable parameters so also the bias params, if available.<br>
To filter the weights I could use model.named_parameters() and filter for “weight” in each parameter’s name.</p>
<p>Do you suspect this is the correct method to check that the 4 rank models all have the same weights after doing a backward syncing update at the end of the epoch?</p>
torchaudio.io.StreamReader doesn't throw error when seeking to time stamp more than the duration of audio file

username: lokesh_v
cooked: <p>I am trying to get the audio chunk of audio file between specific start time and end time</p>
<p>Consider a audio of duration 10 seconds. Now i need to get chunk from 4 sec to 7 sec</p>
<p>torchaudio.info doesn’t give correct <code>num_frames</code> for io.BytesIO flac audio file. So there is no way to find the total number of frames in the given audio to check for out of bounds start offset Ref: <a href="https://github.com/pytorch/audio/issues/2524" class="inline-onebox" rel="noopener nofollow ugc">[v0.12] torchaudio.info reports num_frames=0 for MP3 · Issue #2524 · pytorch/audio · GitHub</a></p>
<p>What I did to get the chunk of audio with start and end offsets.</p>
<pre><code class="lang-auto">def read_audio(audio_file, start, end):
    audio_file.seek(0)
    reader = StreamReader(audio_file)
    sample_rate = int(reader.get_src_stream_info(reader.default_audio_stream).sample_rate)

    reader.seek(start * sample_rate)
    reader.add_basic_audio_stream(frames_per_chunk=(end - start) * sample_rate)

    return list(reader.stream())[0].pop()
</code></pre>
<p>This is working as intended for start time less than the duration of audio file. But when we give the start time more than the duration of audio file, It doesn’t throw error or return empty tensor. Is there any way to know the given offsets are out of bounds.</p>
Requires_grad = False after first batch

username: Mohammed_Zidane
cooked: <p>Hi,<br>
I am implementing a model such that<br>
1- Set requires_grad = False under two conditions<br>
ONLY for the two layers of the model<br>
After feeding the FIRST batch of the data, from the second batch freeze and do not update the<br>
weights.<br>
2- after finishing the whole training process, I collect all the parameters’ grads ‘including the ones in the first two layers that were obtained only once with the first data batch’</p>
<p>I implemented it around 2 months ago and it worked but I set a new conda environment with the latest pytorch version and I am getting error that I have nontype gradients.<br>
Are there new pytorch releases causing this error? if yes, any suggestions?<br>
Thanks!</p>

username: ptrblck
cooked: <p>The latest PyTorch <code>2.0.0</code> release sets the <code>set_to_none</code> argument to <code>True</code> by default in the <code>zero_grad</code> call, which will delete the <code>.grad</code> attribute of referenced parameters and will thus save memory.<br>
I don’t know what exactly your code is doing, but maybe setting this attribute to <code>False</code> again allows your code to run?</p>
Fine tune VGG-Face

username: youb
cooked: <p>I would like to  fine-tune the pre-trained VGG-Face network as described below:</p>
<p>min_{W,θ}  ∑ {i=1 to N} L(sigmoid(Wη(a_i; θ)), yi)</p>
<p>where where η(a_i; θ) represents the output of the last full connected layer in the VGG-Face network.<br>
θ and W separately denotes the network parameters of the VGG-Face network and the weights of the sigmoid layer.<br>
L is the cross entropy.<br>
Please some one help me.<br>
Thank you</p>

username: thecho7
cooked: <p>What do you want to do?<br>
Write a loss function?</p>

username: youb
cooked: <p>Want to minimize to loss function as described below:<br>
Given i-th input video clip a i (i = 1, 2, · · · N) and its corresponding Big-Five personality score y_i , we fine-tune the pre-trained VGG-Face network to obtain deep segment-level audio feature representations, as described below:<br>
min W_VG,θ_VG ∑ {i=1 to N } L(sigmoid(W_VG * η_VG(a_i; θ_VG)), y_i)<br>
where η_VG(a_i; θ_VG) represents the output of the last full connected layer in the VGG-Face network. θ_VG and W_VG separately denotes the network parameters of the VGGish network and the weights of the sigmoid layer.</p>
ParameterDict keys *must* be strings?

username: rasmus
cooked: <p>Is there a good reason for this? I would like to use general hashable objects as keys in my ParameterDict, just as I would in a normal Python dict.<br>
Currently, the Pytorch documentation lies by claiming “ParameterDict can be indexed like a regular Python dictionary”, since the indexing must be done only with strings.</p>

username: ptrblck
cooked: <p>There is an <a href="https://github.com/pytorch/pytorch/issues/11714">issue</a> about a similar topic on github (<code>ModuleDict</code> instead of <code>ParameterDict</code>).<br>
If you like, you could describe your use case there.</p>

username: rasmus
cooked: <p>Thanks, I’m also just going with the str(int) solution for now, it’s probably not that much of a performance-loss. Still a bit ugly is all.</p>

username: ptrblck
cooked: <p>Sure, I get it, and I think it’s a valid discussion, since there seem to be a few users noting this behavior. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: SimonW
cooked: <p>The main reason is that we need to get a string key for the registered submodule, for purpose like saving state dict. Many things are hashable, but not all of them hash to same value when you run the same code in a new process.</p>

username: Alon_Rozental
cooked: <p>can we reopen this issue a dict with integer only keys will be very useful:<br>
this is the use case:</p>
<p>Pytorch ParameterDict can only accept string type keys.</p>
<p>My model takes inputs that are large integers such as -18272849 and 12345678. These integers represent the murmur hash function of categorical features with low cardinality so the number of different values for the integers is small but the integers themselves are very large.</p>
<p>I want to use a ParameterDict OR an embedding layer for them without converting them to strings and having some standard embedding solution after that. How do I do that?</p>
Is there a way to convert Proxy object to its original object 

username: shy_the
cooked: <p>import torch.nn as nn<br>
import copy<br>
from torch.quantization.quantize_fx import prepare_fx, convert_fx<br>
from torch.quantization import (<br>
get_default_qconfig,<br>
)</p>
<p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.l1 = nn.Linear(4, 4)</p>
<pre><code>def forward(self, x):
    l = int(x.shape[0] / 2)   
    return self.l1(x.view(l, -1))
</code></pre>
<p>model = MyModel()<br>
model.eval()<br>
qconfig = get_default_qconfig(“fbgemm”)<br>
qconfig_dict = {<br>
“”: qconfig,<br>
# ‘object_type’: []<br>
}<br>
model_to_quantize = copy.deepcopy(model)<br>
prepared_model = prepare_fx(model_to_quantize, qconfig_dict)<br>
print("prepared model: ", prepared_model)</p>
<p>quantized_model = convert_fx(prepared_model)</p>
<p>File “/mnt/e/mm/FoundationModel-Agriculture/classification/mini_swin.py”, line 19, in forward<br>
l = int(x.shape[0] / 2)<br>
TypeError: int() argument must be a string, a bytes-like object or a number, not ‘Proxy’</p>
How to implement class weights on only one loss function

username: Shorya_Singhal
cooked: <p>I am using transfer learning on a faster rcnn model (resnet 50) on a dataset with 19 classes with imbalanced class data ranging from 600 images for one class to 60 images for another class. I wanted to use class weights to combat this issue of class imbalance, but i am not able to understand how to implement class weights on only one loss function which is the classification loss out of the 4 loss functions in faster rcnn and return the total loss consisting of all 4 loss functions but class weights only applied on classification loss for backprop. Please help me to understand how to achieve this.</p>
Invalid chunk size

username: sanipanwala
cooked: <p>Hello,</p>
<p>I’m getting an invalid chunk size issue during training with multiple GPUs with a single node.</p>
<p><strong>Error Message: malloc_consolidate() invalid chunk size</strong></p>
<p>Can someone please help me?</p>
<p>Thanks,<br>
Sani</p>

username: rvarm1
cooked: <p>Hi, please file an issue with an example of how to reproduce the crash, thanks!</p>
FSDP with self-customize optimizer

username: chexiangying
cooked: <p>I want to set different learning rates based on parameter names, but I found that the parameter names of fsdp have changed. If I want to set different learning rates based on parameter names in fsdp, how can I achieve this?</p>

username: rvarm1
cooked: <p>To enable this, you can use the <code>use_orig_params=True</code> flag with FSDP.</p>
Confusion in prepare_qat method

username: Anirudh_Alameluvari
cooked: <p>Here’s a recipe from <a href="https://pytorch.org/blog/quantization-in-practice/" class="inline-onebox" rel="noopener nofollow ugc">Practical Quantization in PyTorch | PyTorch</a></p>
<pre><code class="lang-auto"># QAT follows the same steps as PTQ, with the exception of the training loop before you actually convert the model to its quantized version

import torch
from torch import nn

backend = "fbgemm"  # running on a x86 CPU. Use "qnnpack" if running on ARM.

m = nn.Sequential(
     nn.Conv2d(2,64,8),
     nn.ReLU(),
     nn.Conv2d(64, 128, 8),
     nn.ReLU()
)

"""Fuse"""
torch.quantization.fuse_modules(m, ['0','1'], inplace=True) # fuse first Conv-ReLU pair
torch.quantization.fuse_modules(m, ['2','3'], inplace=True) # fuse second Conv-ReLU pair

"""Insert stubs"""
m = nn.Sequential(torch.quantization.QuantStub(), 
                  *m, 
                  torch.quantization.DeQuantStub())

"""Prepare"""
m.train()
m.qconfig = torch.quantization.get_default_qconfig(backend)
torch.quantization.prepare_qat(m, inplace=True)
</code></pre>
<p>The prepare_qat method propogates qconfig to all the leaf nodes, runs convert which swaps modules and then calls the prepare method which add observers to the modules.</p>
<p><strong>In the code above, quant and dequant stubs are added to the model. How are they not altered by the convert call in prepare_qat?</strong></p>
<p>See below for prepare_qat implementation from <a href="https://github.com/pytorch/pytorch/blob/748d9d24940cd17938df963456c90fa1a13f3932/torch/ao/quantization/quantize.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/quantize.py at 748d9d24940cd17938df963456c90fa1a13f3932 · pytorch/pytorch · GitHub</a></p>
<pre><code class="lang-auto">def prepare_qat(model, mapping=None, inplace=False, allow_list=None):
    r"""
    Prepares a copy of the model for quantization calibration or
    quantization-aware training and converts it to quantized version.
    Quantization configuration should be assigned preemptively
    to individual submodules in `.qconfig` attribute.
    Args:
        model: input model to be modified in-place
        mapping: dictionary that maps float modules to quantized modules to be
                 replaced.
        inplace: carry out model transformations in-place, the original module
                 is mutated
        allow_list: a set that lists out allowable modules to be propagated with qconfig
    """
    torch._C._log_api_usage_once("quantization_api.quantize.prepare_qat")
    if mapping is None:
        mapping = get_default_qat_module_mappings()

    if not inplace:
        model = copy.deepcopy(model)

    propagate_qconfig_(model, qconfig_dict=None, allow_list=allow_list)
    convert(model, mapping=mapping, inplace=True, remove_qconfig=False)
    prepare(model, observer_non_leaf_module_list=set(mapping.values()), inplace=True)
    return model
</code></pre>
<p>Thank you!</p>
Torch matmul give false results sometimes

username: 3yuan
cooked: <p>I use the libtorch in my c++ project on Orin jetson platform.<br>
When using matmul, I found it gave false result somtime.</p>
<p>the first matrix size is [1600, 1216, 4].<br>
the second matrix size is [B, 4, 3], and the B stands for batch size<br>
the dtype of both matrixes is float32.<br>
I use the matmul as the following:</p>
<p>torch.matmul(m1.view({-1, 4}), m2);</p>
<p>I found that when B is small(such as &lt;= 2), the result is right. But when B is large(such as 30), the result<br>
is false.</p>
<p>When I tried to change the dtype to be float64, it can give right result.</p>
<p>In my opinion, float32 is enough for the computaion. An example data is like this:<br>
M1 = “”"<br>
089.4820 042.1113 022.1533 001.0000<br>
089.4870 042.1113 022.1523 001.0000<br>
089.4921 042.1113 022.1523 001.0000<br>
089.4970 042.1113 022.1523 001.0000<br>
089.5020 042.1113 022.1523 001.0000<br>
089.5070 042.1113 022.1523 001.0000<br>
089.5120 042.1113 022.1514 001.0000<br>
089.5170 042.1113 022.1514 001.0000<br>
089.5220 042.1113 022.1514 001.0000<br>
089.5270 042.1113 022.1514 001.0000<br>
“”"<br>
M2 = “”"<br>
-1.6649 0001.3089 -0.2094<br>
0001.1314 0001.5086 -0.5520<br>
-0.3370 -1.4009 -0.8071<br>
0107.2991 -150.7653 0062.4391<br>
“”"</p>
<p>Also, when using the matmul implemented by myself, the result is right.<br>
So I think it may be (not sure) a bug of libtorch. And is there any idea to sovle it?<br>
Thanks</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet to reproduce the issue, the results you are seeing, as well as more information about your setup (i.e. which JetPack version are you using, how did you install PyTorch etc.)?</p>
Torchscript error - Module 'GCNConv' has no attribute 'inspector'

username: dvlp-r
cooked: <p>Hi, I am writing here because when trying to export my model and save it using torchscript, the following error appears and I was not able to find any solution on the web.</p>
<p>RuntimeError: Module ‘GCNConv’ has no attribute ‘inspector’ (This attribute exists on the Python module, but we failed to convert Python type: ‘torch_geometric.nn.conv.utils.inspector.Inspector’ to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type Inspector. Its type was inferred; try adding a type annotation for the attribute.):</p>
<pre><code class="lang-auto">File "/Users/---/miniconda3/envs/---/lib/python3.7/site-packages/torch_geometric/nn/conv/message_passing.py", line 534
    
        self._explain = explain
        self.inspector.inspect(self.explain_message, pop_first=True)
        ~~~~~~~~~~~~~~ &lt;--- HERE
        self.__user_args__ = self.inspector.keys(methods).difference(
            self.special_args)
</code></pre>
<p>The model I am trying to export is the one provided by Open Graph Benchmark (OGB) for graph prediction. It can be fount at the following link: <a href="https://github.com/snap-stanford/ogb/tree/master/examples/graphproppred/mol" class="inline-onebox" rel="noopener nofollow ugc">ogb/examples/graphproppred/mol at master · snap-stanford/ogb · GitHub</a></p>
<p>I am new to PyTorch, thank you in advance for your help.</p>
Stuck as .backward() using DataParallel with GAN

username: kakahingwong
cooked: <p>I planned to use multiple GPUs to train the generator of a GAN model for video prediction.  The training works with one GPU (using nn.parallel.DataParallel with one device_ids or simply without DataParallel), but gets stuck at .backward if there are multiple device_ids.  The relevant part of my code is captured below.  Thanks in advance!</p>
<blockquote>
<p>device_ids = [0,1,2]<br>
device_discriminator = device_ids[0]</p>
<p>G_LR = 1e-4<br>
D_LR = 1e-4</p>
<p>generator = nn.parallel.DataParallel(EF(encoder, forecaster), device_ids=device_ids,dim=1).to(f’cuda:{device_ids[0]}‘)<br>
discriminator = Discriminator().to(f’cuda:{device_discriminator}’)</p>
<p>criterionG = MSE().to(f’cuda:{device_ids[0]}‘)<br>
criterionD = nn.BCELoss(reduction=‘mean’).to(f’cuda:{device_ids[0]}’)</p>
<p>optimizerG = torch.optim.Adam(generator.parameters(), lr=G_LR)<br>
optimizerD = torch.optim.Adam(discriminator.parameters(), lr=D_LR)</p>
<p>true_label = torch.tensor(1, dtype=torch.float, device=device_discriminator) # true and fake_label refer to the sequence being real (1) or fake (0)<br>
fake_label = torch.tensor(0, dtype=torch.float, device=device_discriminator)</p>
<p>prediction = generator(train_data) <span class="hashtag-raw">#with</span> shape [12, 3, 1, 256, 256], 3 being the batch size which is also equal to the number of devices used for training</p>
<p>true_seq = torch.cat((train_data, train_label), 0)<br>
pred_seq = torch.cat((train_data, prediction), 0)</p>
<p>generator.train()</p>
<p>for i in range(pred_seq.shape[1]):</p>
<pre><code>D_output = discriminator(pred_seq[:,[i],...].to(device_discriminator))

mse = criterionG(prediction[:,[i],...], train_label[:,[i],...]).to(device_discriminator)

GAN_loss = criterionD(D_output, fake_label)

errG = -1*GAN_loss * weight + mse

torch.nn.utils.clip_grad_value_(generator.parameters(), clip_value=50.0)

for param in generator.parameters():
  param.grad = None

errG.backward() #STUCK AT HERE

optimizerG.step()
</code></pre>
</blockquote>
MobilenetV2 changes in model during quantization

username: Anirudh_Alameluvari
cooked: <p>Hi all, I need help in understanding some of the changes to the model during quantization.</p>
<p>Pre quantization setup:</p>
<ol>
<li>Relu6 converted to Relu</li>
<li>Quantstub and Dequantstub added to the end of the model (after classifier)<br>
(quant): QuantStub()<br>
(dequant): DeQuantStub()<br>
After QAT and convert</li>
<li>Batch norm layers seem to be removed<br>
<strong>First block of the model after preprocessing:</strong><br>
(features): Sequential(<br>
(0): Conv2dNormActivation(<br>
(0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
)<br>
<strong>First block of model after quantization</strong><br>
(features): Sequential(<br>
(0): Conv2dNormActivation(<br>
(0): QuantizedConvReLU2d(3, 32, kernel_size=(3, 3), stride=(2, 2), scale=0.030749445781111717, zero_point=0, padding=(1, 1))<br>
(1): Identity()<br>
(2): Identity()<br>
)</li>
</ol>
<p>What would the reasoning be behind these changes?</p>

username: eqy
cooked: <p>For the removal of the batchnorm layers, if this is post-training quantization, one reason could be that batchnorm at inference time is doing a scale+shift which could be covered by the quantized layer so adding a separate scale + shift could be redundant.</p>

username: Anirudh_Alameluvari
cooked: <p>That’s what I thought too! Any ideas on why relu6 gets converted to relu?</p>