akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Checkpoint breaks grads

SN4KEBYTE: <p>I’m training my text classification model and I’m using gradient checkpointing to save some memory. It looks like this:</p>
<pre><code class="lang-auto">...
embeddings = self.embedding_layer(
   x,
)
encoded = checkpoint.checkpoint(
    self.encoder,
    embeddings,
)
clf = self.linear(
    encoded,
)
...
</code></pre>
<p>But when I’m examining gradients of my model, I see that all gradients (except the last layer) are set to None:</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
            print(name, param.grad)
</code></pre>
<p>What is the problem and how can I solve it?<br>
The most interesting thing - without checkpoint everything works fine.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> you’re my last hope <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>

basingse: <p>I am kind of new to gradient checkpointing so my questions may be silly</p>
<ul>
<li>Did you check your gradient of parameters after running <code>loss.backward()</code> ?</li>
<li>Can you please share the elaborated code (If possible), so that I can run and try to see what is the issue ?</li>
</ul>
Pad multiple torch tensor over the last dim

Mohamed_Nabih: <p>I have multiple torch tensors with the following shape<br>
‘’’<br>
x1 = torch.Size([1, 512, 177])<br>
x2 = torch.Size([1, 512, 250])<br>
x3 = torch.Size([1, 512, 313])<br>
‘’’<br>
How I can pad all these tensors by 0 over the last dimension, to have a unique shape like ([1, 512, 350]).</p>
<p>Thanks</p>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group" data-username="Mohamed_Nabih" data-post="1" data-topic="134783" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohamed_nabih/48/19887_2.png" class="avatar"> Mohamed_Nabih:</div>
<blockquote>
<p>How I can pad all these tensors by 0 over the last dimension, to have a unique shape like ([1, 512, 350]).</p>
</blockquote>
</aside>
<p>Use <code>torch.cat()</code>.  (Doing so will create new tensors, but you can’t<br>
avoid that – pytorch can’t increase the size of a tensor without creating<br>
a new tensor.)</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; x1 = torch.ones (1, 512, 177)
&gt;&gt;&gt; pad = 350
&gt;&gt;&gt; x1_pad = torch.cat ((x1, torch.zeros (1, 512, pad - x1.shape[2])), 2)
&gt;&gt;&gt; x1_pad.shape
torch.Size([1, 512, 350])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Behaviour of id with detached tensors

egill: <p>I noticed something strange, when you run:</p>
<pre><code class="lang-auto">a = torch.tensor(0)
b = torch.tensor(1)

print(id(a) == id(b))
print(id(a.detach()) == id(b.detach()))
</code></pre>
<p>The output of the first <code>print</code> is <code>False</code> whereas it’s <code>True</code> for the second - the detached tensors get the same python <code>id</code> (i.e. memory address). Is this behavior expected?</p>
<p>What’s worse, this seems to be nonteterministic:</p>
<pre><code class="lang-auto">for i in range(20):
    print(id(a.detach()) == id(b.detach()))
</code></pre>
<p>if you run this loop a few times in a row, you should notice that the detached tensors occasionally get different memory addresses. What’s going on here?</p>

tom: <p>I don’t think this is how it works.</p>
<p>So <code>a.detach()</code> creates a new (Python) Tensor object using the same underlying memory to store values (a “view” in autograd lingo). Now id gets the id (the address of the Python object). But after <code>id</code> returns, the refcount of the Python Tensor returned by <code>a.detach() </code> goes to 0 and Python may decide to deallocate it. Then <code>b.detach()</code> creates a new Tensor that may be at the same memory location if Python collected the unused <code>a.detach()</code>.</p>
<p>So as a summary:</p>
<ul>
<li>
<code>a.detach()</code> always creates a new Tensor, albeit one sharing the underlying memory blob for the values with <code>a</code>.</li>
<li>what you are seeing is more a behaviour of Python w.r.t. anonymous temporary objects you create.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
My model's output tensors dont't change given by diffent images

TigerVersusT: <p>I use a loop instead of data loader to read images  and use model to predict them, however, my model always ouputs the same tensor. I 'm sure the input images are diffrent, and comverted to tensors with proper shape, below are the key codes, can anyone figure out the problem?</p>
<p>Classes = os.listdir(valdir)<br>
for Class in Classes:<br>
imgs = os.listdir(os.path.join(valdir, Class))</p>
<pre><code>    for img in imgs:
        total += 1

        imgPath = os.path.join(valdir, Class, img)
        im = fetchImage(imgPath, trans)
        im = im.cuda(0)
        print(im)
        output = model(im)</code></pre>

TigerVersusT: <p>I’m sorry, I’ve loaded the worry model</p>
"requires_grad" after list addition

cbd: <p>In below code, let say i have l1 and l2 loss which require grad is True and they are in form of list in variable “c”. After addition, requires_grad is “none”. I want to add the losses and want<br>
“Total_L” requires_grad “True” such that it will backpropagate in backward pass.</p>
<pre><code class="lang-auto">import torch
import numpy as np
l1= torch.ones([1],requires_grad=True)
print("l1 is ",l1)

l2= torch.zeros([1],requires_grad=True)
print("l2 is ",l2)

comb_L=[l1,l2]
print("comb_L is ",comb_L)

Total_L=torch.tensor(comb_L).sum(0)
print("Total loss is ",Total_L,Total_L.grad_fn)</code></pre>

AlphaBetaGamma96: <p>you’ll want to replace <code>torch.tensor(comb_L)</code> with <code>torch.stack(comb_L)</code> and that’ll have a <code>grad_fn</code></p>
<pre><code class="lang-auto">import torch
import numpy as np
l1= torch.ones([1],requires_grad=True)
print("l1 is ",l1)

l2= torch.zeros([1],requires_grad=True)
print("l2 is ",l2)

comb_L=[l1,l2]
print("comb_L is ",comb_L)

Total_L=torch.stack(comb_L).sum(0)
print("Total loss is ",Total_L,Total_L.grad_fn)
</code></pre>
Pytorch method to map input for multi-label classification

lima: <p>Hello peeps,</p>
<p>Sorry for the noob question.<br>
I am doing a multi-label classification for audio data. My input data is features from audio frames. The label classes are 34 phonemes, and each class has many labels (descriptors of that particular phoneme/class). The total number of descriptors are 10. The classes and labels look like this:</p>
<pre><code class="lang-auto">  voiced       b, d, g, j, l , v, z, Z,
  unvoiced     f, k, p, s, t 
  labial       p, b, m, f, v 
  fricative    f, s, S, x,
  (...10)
</code></pre>
<p>My understanding is that this is a multi-label classification and I need to map each class/phoneme to its lables/descriptors., so that I have a matrix of (classes, labels) where each class is a row that contain 1 where label is present and 0 where label is not present.</p>
<p>Internet posts suggest this could be done with <code>sklearn.preprocessing.OneHotEncoder</code>,<br>
but I would like to know if there is a more proper pytorch way to create the mapping matrix, and if it’s even the best way to go for this kind of task.<br>
Also, what would the input shape look like.</p>
<p>Thanks</p>
Mismatching stable version

Neuroxic: <p>Ive used <strong>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge</strong><br>
to install the latest stable version of pytorch for windows.<br>
The latest stable version is supposed to the <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">webstite</a> version 1.9.1.</p>
<p>But the torch.__version__ is set to 1.10.0. Also the field torch.version.cuda is not set this is unfavorable for my project.</p>
<p>Is there any workaround to get torch version 1.9.1 with cuda version 1.11?</p>

ptrblck: <p>The install instructions are given as:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
</code></pre>
<p>while you are using the <code>conda-forge</code> channel.</p>

Neuroxic: <p>Thanks for your suggestion, but when I change my OS type to Windows the nvidia channel changes to the conda-forge channel.</p>

ptrblck: <p>Ah, sorry for missing it’s Windows. In that case I won’t be of much help, as I wouldn’t be able to test it.</p>
Customize plots on tensor board (average and errorbar over multiple simulations)

Torcione: <p>Hi everyone,<br>
I’m starting using tensor board; let’s say that for a given problem I run several identical simulations (for each simulation I have a folder where some relevant measures (accuracy, loss, precision…) of the specific sample are collected at each epoch/step).<br>
Is there a simple way to get from tensorboard plots represetnting the average of these measures over the simulations performed and the corresponding errorbar (standard deviation)?</p>
Named tensors Warning

seunghyuck.kim: <p>Hi, please give me some advice.</p>
<p>This warning below is always occur when I run my test code.</p>
<blockquote>
<p>UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  …\c10/core/TensorImpl.h:1156.)<br>
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)</p>
</blockquote>
<p>The recognition results seem not weird, so probably this error is not affecting the processing at this moment, but I am curious if it is 100% ok or not.</p>
<p>I have found a doc related to named_tensor in the link below:<br>
<a href="https://pytorch.org/docs/stable/named_tensor.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/named_tensor.html</a></p>
<p>I am not using any methods related to tensor name in my code, so it’s hard to find why this warning occur.</p>
<p>Below is my code for test:</p>
<pre><code class="lang-auto">test_transforms = transforms.Compose([transforms.Resize((224,224)),
                                      transforms.ToTensor(),
                                      transforms.Lambda(lambda x: x.repeat(3, 1, 1)),
                                      transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
                                      ])
real = Image.open(image)
inputimage = test_transforms(real).unsqueeze_(0)
img_variable = Variable(inputimage)
img_variable = img_variable.to(device)
output = model(img_variable)
stm = nn.Softmax(dim=1)
</code></pre>

ptrblck: <p>I think you can ignore it for now and either install the nightly release or wait for the next stable release as the issue was tracked and fixed <a href="https://github.com/pytorch/pytorch/issues/60053">here</a>.</p>

seunghyuck.kim: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your kind reply!<br>
But the warning message makes me nervous.<br>
Is there anything I can do to check if there actually is any problem related this or not?<br>
What kind of error could be possible do you think if it happended?</p>

Sreenitha_Kasarapu: <p>/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)<br>
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)</p>
<p>I am getting the above error and its effecting my output, I should get plots of model execution time but was unsuccessful. Could someone help me in resolving this issue. Thanks in advance.</p>

ptrblck: <p>You are seeing a warning, not an error, which should not influence any results.<br>
The warning spamming is a known issue and was tracked <a href="https://github.com/pytorch/pytorch/issues/60053">here</a> and already fixed. You could update PyTorch to the latest stable or nightly release to get rid of this warning.</p>
Dataloader from two Datasets

maverick09: <p>Hello everyone.<br>
I have 2 datasets for a classification problem. Each has labels of different sizes. I want to sample from both at the same time for every iteration of training. I could do something like:<br>
<code>train_dl = DataLoader(trainset_1+trainset_2, shuffle = True, batch_size=128, num_workers=4)</code><br>
But the issue is the mismatch in dimensions of labels and the data loader can’t stack tensors of different dimensions.<br>
Is there any workaround for this?</p>

Artlind: <p>Hello ! An easy workaround is to create two dataloaders, one for eatch dataset.</p>
<pre><code class="lang-auto">a, b = torch.ones(100,5,5), torch.zeros(100,4,4)
dla = torch.utils.data.DataLoader(a, batch_size=2)
dlb = torch.utils.data.DataLoader(b, batch_size=2)
for i,j in zip(dla,dlb):
    print(i,j)
</code></pre>

maverick09: <p>Thanks a lot! This works. I was wondering if there is any way to do this with just one data loader.</p>
How to plot segmentation mask from the given annotation?

Kalyan: <p>I have the following annotation</p>
<p>{“segmentation”: [[413.03,131.72,…,132.36]],“area”: 1067.9535499999993,“iscrowd”: 0,“image_id”: 1000,“bbox”: [405.93,120.42,37.13,45.52],“category_id”: 1,“id”: 1245349}.</p>
<p>I want to plot the segmentation and bounding box on my input image using the above segmentation and bbox values.</p>
CrossEntropyLoss and OneHot classes

pumplerod: <p>I’m having some trouble understanding CrossEntropyLoss as it relates to one_hot encoded classes.  The docs use random numbers for the values, so to better understand I created a set of values and targets which I expect to show zero loss…</p>
<p>I have 5 classes, and 5 one_hot encoded vectors (1 for each class), I then provide a target index corresponding to each class.</p>
<p>I’m using <em>reduction=‘none’</em> to show the values for each sample loss.</p>
<pre><code class="lang-auto">loss_fn = torch.nn.CrossEntropyLoss( reduction='none')
x = torch.tensor( [[ 1.0, 0.0, 0.0, 0.0, 0.0], 
                   [ 0.0, 1.0, 0.0, 0.0, 0.0], 
                   [ 0.0, 0.0, 1.0, 0.0, 0.0], 
                   [ 0.0, 0.0, 0.0, 1.0, 0.0], 
                   [ 0.0, 0.0, 0.0, 0.0, 1.0]], requires_grad=False).to( 'cpu')
y = torch.tensor( [ 0, 1, 2, 3, 4], requires_grad=False).type( torch.long).to( 'cpu')
loss = loss_fn( x, y)
print( loss)
</code></pre>
<p>results…</p>
<pre><code class="lang-auto">tensor([0.9048, 0.9048, 0.9048, 0.9048, 0.9048])
</code></pre>
<p>I really can’t follow the logic here.  Could someone please elaborate?</p>

tom: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc"><code>CrossEntropyLoss</code></a> takes unnormalized scores (simetimes called “logits”) as inputs.<br>
So if you want <code>x</code> tensor to represent the “perfect predictions”, you would could take the log of what you have (send the 0s to -inf and use 0 (anything finite woudl do) for the 1s).</p>
<p>Best regards</p>
<p>Thomas</p>

pumplerod: <p>That’s very helpful.  Thank you.  A bit more for me to wrap my head around, but those number make some sense now at least.</p>
Inference on large image using model parallelism?

Innat: <p>Hello.<br>
Let’s say you have a very large image to infer and a single GPU can load the model but throws OOM for large input. In such a situation, what do you think if we split the model’s layers on multiple GPU / machines and run forward pass accordingly.</p>
<p>For example, an image X and model B with layer 1, layer 2, and layer 3. For a single GPU inference with this model on image X, it gives OOM. So, how about we place these 3 model layers (layer 1, layer 2, and layer 3) on 3 different GPUs. And sequentially we pass the data X on layer 1 (GPU:1) and pass the output of GPU:1 to GPU:2 and output of GPU:2 to GPU:3. Now, the Image can’t fit into one GPU itself.</p>

ptrblck: <p>Your approach is known as model sharding and would work.<br>
However, since only a single GPU would be used during this approach, there are more advanced techniques such as pipeline parallelism, which would try to overlap the workloads more.</p>
Looking for Example to compare cpu vs gpu

hgfedcba: <p>When I compare pytorch on cpu and gpu in two use cases of mine the gpu is always a bit slower. I would like to have a code example I can just execute myself where the gpu is supposed to beat the cpu. If it works I want to use it to improve my code. However I wasn’t able to find any pytorch code where a comparsion between cpu and gpu is implemented, except for 1-2 that needed very big image database downloads.</p>
<p>Does someone have a link/code I could use?</p>

ptrblck: <p>You could use a <code>torchvision</code> model (e.g. pick a ResNet) and compare the performance using random input tensors (e.g. <code>torch.randn(64, 3, 224, 224)</code>).</p>
High resolution GAN

Najeh_Nafti: <p>How can I work with high resolution images (256px) with a GAN model that works only on low images (64px)?<br>
Can I add some layers or anything else to support that?</p>

ptrblck: <p>Technically your model might already be able to accept larger input shapes. However, using high-res inputs might be tricky as the model often failed in the past. You could take a look at e.g. the <a href="https://arxiv.org/abs/1812.04948">StyleGAN paper</a> and check its training schedule (also check newer papers, as the scheduling might have been improved already).</p>
AttributeError: module 'torch' has no attribute 'maximum'

Ziyu_Chen: <p>I ran a simple code on my macOS with torch version = 1.6.0 and python version = 3.7.4:<br>
import torch<br>
x = torch.tensor([[1],[2],[3],[5]],dtype = torch.float)<br>
y = torch.tensor([[1],[2],[3],[4]],dtype = torch.float)<br>
print(torch.maximum(x,y))<br>
But I got AttributeError: module ‘torch’ has no attribute ‘maximum’.<br>
Torch does have maximum function which returns the elementwise maximum of two tensors. Where did I get wrong? I’ve been stuck on this for quite a while. Thanks.</p>

tom: <p>I think it is <code>torch.max(x, y)</code>. Not that it’s going to be terribly interesting for your tensors…</p>
<p>Best regards</p>
<p>Thomas</p>

Ziyu_Chen: <p>Hi tom, yes, torch.max can give the right output and it can also give the maximum value of all the elements of a tensor. But I was wondering why torch.maximum would not work, since it’s pretty clear this function is designed for it as in <a href="https://pytorch.org/docs/master/generated/torch.maximum.html#torch.maximum" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.maximum.html#torch.maximum</a></p>

tom: <p>They were added to the dev branch just a month ago (<a href="https://github.com/pytorch/pytorch/pull/42579" rel="noopener nofollow ugc">Aug 26</a>) and you are using the dev branch documentation.</p>

Ziyu_Chen: <p>I see, thank you very much!</p>

ashish_singh: <p>hi<br>
I am getting error torch has no ‘minimum’ attribute<br>
i am doing bigbird model project in pytorch</p>

ptrblck: <p>You might need to update PyTorch to be able to use this method.<br>
<a class="mention" href="/u/tom">@tom</a> mentioned the PR where the <code>minimum</code> and `maximum methods were merged and since it’s ~1 year old, I assume you are also using an older PyTorch release.</p>
Accelerating Dataloading with Prefetching

ziqipang: <p>I understand this could be an old topic, but I really want to know about the recent progress of Pytorch. Recently, I am training a model on Horovod (1 node, 4 gpus), and my dataloader uses 32 workers.</p>
<p>At this time, I am wondering if I can further improve the efficiency of data loading? One potential way is to implement a prefetcher, but on the internet there are some posts saying that Pytorch dataloader automatically uses prefetching.</p>
<p>Therefore, I am wondering:</p>
<ul>
<li>If implementing prefetching is still needed?</li>
<li>If the answer is yes, are there any example implementations for the multi-gpu settings?</li>
</ul>
<p>Thank you so much!</p>
Why did you refactor autograd.Function by using static functions?

wyg1997: <p>I found the warning in <code>Function.__call__()</code> showing <code>Legacy autograd function with non-static forward method is deprecated...</code>. It use static forward method in latest code, and users must use <code>Function.apply()</code> to call custom functions.</p>
<p>I wonder why do this kind of refactoring. Is there any document or discuss to consult for me? Please tell me and thank a lot.</p>

soulitzer: <p>See <a href="https://github.com/pytorch/pytorch/pull/22983#discussion_r304642671" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/22983#discussion_r304642671</a></p>
<p>tl;dr: usage of legacy autograd function is error prone, if you invoke it twice for example, it will actually share the grad_fn, which may cause correctness issues</p>
Net.to('cuda:0') does not put layers in lists to GPU

Haziq: <p>I have the following model where encoder2 is a list of layers (not sequential). However, feeding the data on a GPU through encoder2 results in the data not being in GPU.</p>
<pre><code class="lang-auto">class model(nn.Module):
    def __init__(self, args):
        super(model, self).__init__()
                                           
        self.encoder1 = make_mlp([6, 128], ["relu"])
        self.encoder2 = [make_mlp([6, 128], ["relu"]) for _ in range(2)]

    def forward(self, data, mode):
        
        data = torch.stack([self.encoder2[i](data[i])] for i in range(2))       
        print(data.type())

</code></pre>
<pre><code class="lang-auto">def make_mlp(dim_list, activations, batch_norm=False, dropout=0):

    if len(dim_list) == 0 and len(activations) == 0:
        return nn.Identity()

    assert len(dim_list) == len(activations)+1
    
    layers = []
    for dim_in, dim_out, activation in zip(dim_list[:-1], dim_list[1:], activations):
        layers.append(nn.Linear(dim_in, dim_out))
        if batch_norm:
            layers.append(nn.BatchNorm1d(dim_out))
        if activation == 'relu':
            layers.append(nn.ReLU())
        elif activation == 'leakyrelu':
            layers.append(nn.LeakyReLU())
        if dropout &gt; 0:
            layers.append(nn.Dropout(p=dropout))
    return nn.Sequential(*layers)
</code></pre>
<p>How can I fix this?</p>
<pre><code class="lang-auto">net = model()
net = net.to('cuda:0')
data = data.cuda()
data = net(data)
</code></pre>

tumble-weed: <p>hi,<br>
pytorch has a construct called <code>nn.ModuleList</code> that is exactly for your use case. please <a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html" rel="noopener nofollow ugc">have a look here</a></p>
Reasons and Inspection for NAN loss during Seq2Seq training?

ankitvad: <p>I have a Seq2Seq model that I’ve been training and working on for the past few months. Recently I did a couple of modifications in the data module loading and learning rate warm up etc. I also created a new dataset of 10million training pairs. During training, the model seems to go to NAN suddenly in the 3rd or 2nd epoch. This is really confusing since the  loss doesn’t go high up or down. It just becomes NAN suddenly and in the 3rd epoch which means that it successfully went over the whole dataset Atleast once!</p>
<p>To troubleshoot I tried training my new model with my previous dataset and it seems to be training perfectly on the old dataset! I checked my new(large) dataset and removed all nan and null values from the dataset. What else can I check in the dataset to make sure it does not crash? It’s an NLP dataset of text sentence pairs. Like Machine Translation. I’m using the BERTTokenizerFast for tokenization. Since there are 10million rows, it’s kinda impossible to read through it manually.</p>
<p>I’m using pytorch-lightning with ddp-sharded and 16-bit precision for training in all the experiments.</p>

tumble-weed: <p>16 bit precision has a tendency to truncate values to 0 more often than 32 bit. this might be triggering a inf which might lead to a downwind nan.  you can trigger a breakpoint by</p>
<ol>
<li>checking the output of the network (i.e. end of forward pass) for inf or nan. and stopping if you encounter it</li>
<li>checking the first layer gradients for nan/inf ( i.e. end of the backward pass), and stopping the code (before optimizer.step is actually called and ends up corrupting your weights) if this happens</li>
</ol>
<p>then you can inspect the conditions around which the nans arise ( was it in the forward or backward, had something to do with data/ float 16?)</p>
Not getting desired result from saving model

Md_Shahriar: <p>Inside the train of a DCGAN model, if i save the model like this and load it later. Then i do not get desirable result. Is there any problem saving model like this. But if generate image without saving and loading, i get more good result.</p>
<pre><code class="lang-auto">        if (errD &lt; ls_D):
            torch.save(netD.state_dict(), 'D.pt')
            ls_D = errD

        if (errG &lt; ls_G):
            torch.save(netG.state_dict(), 'G.pt')
            ls_G = errG
</code></pre>
<p>Training Code is this…</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for i, data in enumerate(dataloader):
        
        netD.zero_grad()
        real_cpu = data[0].cuda()
        batch_size = real_cpu.shape[0]
        label = torch.ones(batch_size).cuda()
        
        output = netD(real_cpu).flatten()
        errD_real = criterion(output.float(), label.float())
        
        errD_real.backward()
        D_x = output.mean().item()
        
        ######################
        label.fill_(fake_label)
        noise = torch.randn(batch_size, nz, 1, 1).cuda()
        
        fake = netG(noise)
        output = netD(fake.detach()).flatten()
        errD_fake = criterion(output.float(), label.float())
        
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        
        optimizerD.step()
        #######################################################
        
        netG.zero_grad()
        label.fill_(real_label)
        
        output = netD(fake).flatten()
        errG = criterion(output.float(), label.float())
        
        errG.backward()
        D_G_z2 = output.mean().item()
        optimizerG.step()
        
        
        G_losses.append(errD.item())
        D_losses.append(errG.item())
        
        if (errD &lt; ls_D):
            torch.save(netD.state_dict(), 'D.pt')
            ls_D = errD


        if (errG &lt; ls_G):
            torch.save(netG.state_dict(), 'G.pt')
            ls_G = errG

        


</code></pre>

tumble-weed: <p>can you try this:<br>
where you are saving a model, put in a code that you reload it back into new variables. so if you saved <code>net_G</code>, load it back to <code>net_G1</code>. then pass the same noise through both the models to see if you get the exact same output. if you dont then it is confirmed there is something fishy, more than the qualitative observation of good and not good images.</p>
<p>off the top of my head, this may be due to some stochastic part of the code , or it may be due to .train and .eval behaviour which causes normalization layers to behave differently.</p>
Streaming I/O of potentially large tensors?

ankitvad: <p>If you have an application where you can have a potentially huge Dataset (but it’s generated in a sequential/streaming) manner then is there a way to save the Tensor outputs in a streaming manner?</p>
<p>For example, if we want to preprocess a dataset by tokenizing and running it through a BERT transformer and then save the hidden states Output from each layer is it possible to do in an efficient way?</p>
<p>For instance:</p>
<pre><code class="lang-auto">'''
Let's say BatchSize = 50 and num_layers = 13 (for BERT). Data-input length(T) = 200
If the dataset has 20000 items. Then we have:
'''
a = torch.Tensor(torch.randn(13,50,200,HDim))
#This array 20000/50 times!
# If we concatenate it it becomes Massive! Even in terms of numpy-arrays!
</code></pre>
<p>Is there a way to write out ‘a’ in a streaming way? Using HDF5/Zarr/mmap/PyTables or anything?</p>

Huang587: <p>The specialisations you mention are specific types used to provide a standard interface to a variety of data sources. For example, a FileInputStream and an ObjectInputStream will both implement the InputStream interface, but will operate on Files and Objects respectively<a href="https://www.prepaidgiftbalance.net/" rel="noopener nofollow ugc">.</a></p>

ankitvad: <p>I was able to solve this issue by using - zarr. It is quite fast and convenient depending on the dataset type.</p>
.to(device) slowing the code

s_n: <p>Hi,<br>
Whenever I do a .to(device) for any tensor in my code it takes too much time and makes my code slow. I am not able to understand why is that happening?</p>

AlbanOdot: <p>Hello,</p>
<p>Your tensors might be huge and the bandwidth to your GPU not big enough thus slowing down the process.</p>
<p>You can reduce your batchsize and accumulate the gradient (by not using optimizer.zero_grad() at every backward() call). This will most likely reduce the bottleneck around the bandwidth.</p>
<p>if you reduce your batchsize by let’s say 4 you can use</p>
<pre><code class="lang-auto">if counter % 4 == 0:
    optimizer.zero_grad()
.
have fun with your loss here
.
if counter % 4 == 0:
    optimizer.step()
counter += 1
</code></pre>

albanD: <p>Note as well that the <code>.to()</code> functions are the only blocking operations on the CUDA api that is completely asynchronous otherwise.<br>
So if you try to time your code without using <code>torch.cuda.synchronize()</code> properly, it is expected that the <code>.to()</code> functions are the only ones that will take a significant amount of time ! But this is only because these functions are waiting on the GPU to finish processing all other operations before returning.</p>

s_n: <p>Actually I was using .to(device) in different code (not for calculating loss) where I need to create torch variable and copy my inputs to this variable and do some operations on this variable.</p>

s_n: <p>with torch.cuda.synchronize() as well I get the same timing.<br>
As in still .to(device) consume too much time in my profiling results.</p>
How to conda install OpenCV along with PyTorch in a Miniconda?

vadimkantorov: <p>Does anybody have any working recipes? My failed attempt is at <a href="https://github.com/pytorch/vision/issues/4665" class="inline-onebox" rel="noopener nofollow ugc">Installation with OpenCV triggers install of torchvision 0.2.2 · Issue #4665 · pytorch/vision · GitHub</a></p>
<p>I need to have the shared OpenCV libraries / headers, so <code>pip install opencv-python-headless</code> doesn’t cut it.</p>
<p>Or maybe if using <code>mamba</code> is a solution, I would appreciate any <code>mamba</code> installation scripts as well. I have never used <code>mamba</code> before for now…</p>

vadimkantorov: <p>I think I found one way: installing <code>torchvision</code> from pip instead of conda:</p>
<pre><code class="lang-auto"># install.sh

PREFIX=/home/gamir/lab/vadim/prefix
mkdir -p $PREFIX

curl -L -so $PREFIX/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh &amp;&amp; chmod +x $PREFIX/miniconda.sh &amp;&amp; $PREFIX/miniconda.sh -b -p $PREFIX/miniconda &amp;&amp; rm $PREFIX/miniconda.sh

export PATH=$PREFIX/miniconda/bin:$PATH

conda install -y opencv pytorch cudatoolkit=11.1 -c pytorch -c conda-forge

python -m pip install torchvision 
</code></pre>
<pre><code class="lang-auto"># enter.sh

PREFIX=/home/gamir/lab/vadim/prefix

PATH=$PREFIX/miniconda/bin:$PATH
LD_LIBRARY_PATH=$PREFIX/miniconda/lib:$LD_LIBRARY_PATH

PATH="$PATH" LD_LIBRARY_PATH="$LD_LIBRARY_PATH" bash --init-file &lt;(echo "export TERM=xterm-256color; export MPLBACKEND=agg; export PS1='CONDA [\u@\h \W]\$ '")
</code></pre>
Pytorch for loop slow down

ywcmaike: <p>a = torch.rand(2000,3000).cuda()<br>
time_start = time.time()<br>
for i in range(2048):<br>
max_index = torch.argmax(a)<br>
print(“run %.2f” % (time.time() - time_start))</p>
<p>the top 1024 index in for loop return result quickly， but the latter is slower，<br>
how to fix this problem?</p>

albanD: <p>Hi,</p>
<p>This is because the cuda api is asynchronous. So the first ones just launch job onto the GPU. But once the queue of tasks is full, it has to wait. And so it slows down.<br>
If you want correct timing, you need to do:</p>
<pre><code class="lang-python">a = torch.rand(2000,3000).cuda()
torch.cuda.synchronize()
time_start = time.time()
for i in range(2048):
    max_index = torch.argmax(a)
    torch.cuda.synchronize()
    print(“run %.2f” % (time.time() - time_start))
</code></pre>
Custom Dataset that splits the data and applies Transforms accordingly

jimmiemunyi: <p>I have searched online and the forums and the only solution so far I have found is to use this external library: <a href="https://github.com/szymonmaszke/torchdata" class="inline-onebox" rel="noopener nofollow ugc">GitHub - szymonmaszke/torchdata: PyTorch dataset extended with map, cache etc. (tensorflow.data like)</a></p>
<p>So before I use that I was wondering whether there is a solution native to PyTorch.</p>
<p>Here is my dataset:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
  def __init__(self, fns, transform=None):
    self.fns = fns
    self.transform = transform
  
  def __len__(self):
    return len(self.fns)
  
  def __getitem__(self, idx):
    image = Image.open(self.fns[idx]).convert("RGB")
    label = self.fns[idx].parent.name

    if self.transform is not None:
      # apply transforms here
    
    return image, label
</code></pre>
<p>For context, the images are all in respective folders but not split to train and valid.<br>
Therefore I use this code to split it randomly:</p>
<pre><code class="lang-auto">  dset = CustomDataset(filenames)
  num_items = len(dset)
  num_train = round(num_items * 0.8)
  num_val = num_items - num_train
  train_ds, val_ds = random_split(dset, [num_train, num_val])
</code></pre>
<p>The problem now is now that I have the specific datasets, I need to apply specific transforms depending if they are training data or valid data.</p>
<p>Im guessing I could create a class function like <code>def split_data(self,....):</code> inside the class but I’m not sure how to proceed.<br>
Any leads? Or any better way to construct this custom dataset that does the splitting for us?</p>

ptrblck: <p>Instead of using <code>random_split</code> you could create two <code>CustomDataset</code> instances each one with the different transformation:</p>
<pre><code class="lang-python">train_dataset = CustomDataset(filenames, train_transform)
val_dataset = CustomDataset(filenames, train_transform)
</code></pre>
<p>and then use <code>Subset</code> on both with their corresponding indices:</p>
<pre><code class="lang-python">train_dataset = Subset(train_dataset, train_indices)
val_dataset = Subset(val_dataset, val_indices)
</code></pre>
<p>which will make sure that only the <code>*_indices</code> are used to draw samples from the internal dataset.<br>
Also, since you are lazily loading the data (which is great!) the memory overhead should be small (only the file paths would be duplicated, but that shouldn’t matter).</p>

jimmiemunyi: <p>Thank you. I’ll try it and let you know if it works</p>
LiteModuleLoader equivellent module in android native

yousanke: <p>Hi All,</p>
<p>In libtorch, we use the torch::jit::load to read a torchscript model saved by pytorch. I’d like to know what is the equivalent function in the pytorch_android_lite. In the <a href="https://github.com/pytorch/android-demo-app/tree/master/HelloWorldApp" rel="noopener nofollow ugc">HelloWorld</a> example, <code>LiteModuleLoader</code> is called to restore the saved model, but I am not sure what is the suggested way to do so on the native c++ side.</p>
<p>Thanks for helping.</p>

Linbin: <p>The c++ api is torch::jit::_load_for_mobile. You can find more information in the iOS example: <a href="https://pytorch.org/tutorials/prototype/lite_interpreter.html#ios" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Introduce lite interpreter workflow in Android and iOS — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>

yousanke: <p>Thanks for replying. The including list in the example actually helped me.</p>
<pre><code class="lang-auto">
//#import &lt;LibTorch/LibTorch.h&gt;
#include "ATen/ATen.h"
#include "caffe2/core/timer.h"
#include "caffe2/utils/string_utils.h"
#include "torch/csrc/autograd/grad_mode.h"
#include "torch/script.h"
#include &lt;torch/csrc/jit/mobile/function.h&gt;
#include &lt;torch/csrc/jit/mobile/import.h&gt;
#include &lt;torch/csrc/jit/mobile/interpreter.h&gt;
#include &lt;torch/csrc/jit/mobile/module.h&gt;
#include &lt;torch/csrc/jit/mobile/observer.h&gt;
</code></pre>
Combining deterministic rules with neural network

tobias1: <p>Hello everyone,</p>
<p>I want to set up a classification model that, in a condensed version, has 4 input features [x_1, x_2, x_3, x_4] and predicts 4 classes {y_1, y_2, y_3, y_4}. Now there exits a rule that says:<br>
if x_1 &lt; 0,6 then the the class can only be y_1 or y_2<br>
else the class can only be y_3 or y_4.</p>
<p>They way i thought of, would be to train separate models for the 2 cases and then combine them with a simple if condition. However, given that this solution does not result in a pytorch nn.module at inference it would be super slow, considering the if condition would only be used on a single input.<br>
This could be circumvented by collecting multiple inputs with the same result for the if condition and then feeding them to the model as a vector. However, considering in reality i will have multiple if conditions and i would like to create a solution that is dynamic with respect the these conditions (so i could easily add, remove, adapt them without changing much of the models code) and further more i am working with image data so i have to rearrange the predictions for each pixel afterwards into the same order, it seems rather complicated to keep track of all the positions of the singular inputs.</p>
<p>So long story short - my question is: Is there a way to model a deterministic decision as a pytorch layer?  Or is there a way to build a pytoch neural network representing my desired model, containing deterministic decisions?</p>
<p>Best wishes,<br>
Tobias</p>
Mixed precision with generative adversarial network(GAN)

Hedwig_Huang: <p>Hi guys, I am a new comer for deep learning and I am working on GAN now, I want to ask that if I use <code>torch.cuda.amp.autocast</code> and <code>torch.cuda.amp.GradScaler</code> while training, will it really inference my model appearance??</p>
<p>Could you show some examples that GAN’s performance with fp16?</p>
<p>by the way I didn’t use torch.nn.parallel, I just assign each generator to a GPU, and send the variable from GPU1 to GPU0 and from GPU0 to GPU1 and so on, will I lose some details in the loss backwarding and calculating??</p>
Loop the same image 5 times for predict, the first loop is much slower than others

myccc456: <pre><code class="lang-auto">class model_predict():
    def __init__(self, model_path, use_cuda):
        self.use_cuda = use_cuda  # torch.cuda.is_available()
        self.transform_train = transforms.Compose([
            transforms.Resize((128, 128)),
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
        ])
        nn = torch.load(model_path)
        self.net = nn['net']
        self.net.eval()
        if self.use_cuda:
            self.net.cuda()

    def test(self, img_path):
        #start1 = time.time()
        testset = MyDataset(txt=img_path, transform=self.transform_train)
        testloader = torch.utils.data.DataLoader(testset, batch_size=100, shuffle=False, num_workers=0)
        for batch_idx, (inputs, targets) in enumerate(testloader):
            if self.use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()
            inputs, targets = Variable(inputs), Variable(targets)
            start1 = time.time()
            outputs = self.net(inputs)
            _, predicted = torch.max(outputs.data, 1)
            print("Test time used1:", (time.time() - start1) * 1000)
            return predicted[0]

model = model_predict('../model/model1.pt', torch.cuda.is_available())

array = [
    '../imges/1.jpg',
    '../imges/1.jpg',
    '../imges/1.jpg',
    '../imges/1.jpg',
    '../imges/1.jpg'
]

for img in array:
    model.test(img)

</code></pre>
<p>The result time:<br>
Test time used1: 1501.086711883545<br>
Test time used1: 23.436307907104492<br>
Test time used1: 23.624181747436523<br>
Test time used1: 23.0100154876709<br>
Test time used1: 23.102998733520508</p>
<p>I do not know why the first loop is slower than other?<br>
Any ideas?</p>

ptrblck: <p>CUDA calls are asynchronous, so if you would like to time them, you should call <code>torch.cuda.cynchronize()</code> before starting and stopping the timer.<br>
Besides that, the first loop might be slower, because the <code>DataLoader</code> has to load the initial batch which might take some time. The following batches might be already waiting in the queue.</p>

Arun_Kumar4: <p>in which line we write this code torch.cuda…synchronize</p>

ptrblck: <p>Add them before starting and stopping the timers via:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.perf_counter()
</code></pre>
<p>or use <code>torch.utils.benchmark</code>, which will synchronize and add warmup iterations.</p>

Arun_Kumar4: <p>sir predication time of model takes 30secs i want to descrse predication time any idea for that</p>

ptrblck: <p>Profile the code first and narrow down where the bottleneck is. Without any information about the workload and which part is slow we can only speculate.</p>

Arun_Kumar4: <p>how to profile the code i dont know</p>

ptrblck: <p>You could use the <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/">PyTorch Profiler</a> or e.g. Nsight Systems.</p>

Arun_Kumar4: <p>i predicating image time is slow</p>
Broadcasting issues

milan_kalkenings: <p>Hi all,<br>
attention_logits is of size <code>torch.Size([16, 12, 260, 260])</code></p>
<p>attention_mask is of size <code>torch.Size([16, 260])</code></p>
<p>i want to perform <code>attention_logits = attention_logits.masked_fill(attention_mask == 0, -1e7)</code><br>
but it throws <code>RuntimeError: The size of tensor a (16) must match the size of tensor b (260) at non-singleton dimension 2</code> because of broadcasting issues.</p>
<p>what’s the best way to perform this masked_fill on the rows of the attention_logits tensor?</p>

tumble-weed: <p>you can add singleton dimensions using <code>None</code> in <code>attention_mask</code><br>
<code>attention_logits = attention_logits.masked_fill(attention_mask[:,None,:,None],-1e7)</code></p>

milan_kalkenings: <p>thanks for the fast solution!<br>
it works for me with<br>
<code>attention_logits = attention_logits.masked_fill(attention_mask[:,None,None, :],-1e7)</code></p>
Distributed send deadlock when communication graph is connected

Fangyuan: <p>When the communication graph between threads is connected, the program will get stucked.<br>
It seems like a deadlock.<br>
But I don’t know how to figure it out.</p>
<p>I write a simple demo to reproduce it.</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process


def run(rank, size):
    tensor = torch.zeros(1)
    rec_tensor = torch.zeros(1)

    tensor += rank
    if rank == 0:
        dist.send(tensor=tensor, dst=1)
        dist.recv(tensor=rec_tensor, src=2)
    elif rank == 1:
        dist.send(tensor=tensor, dst=2)
        dist.recv(tensor=rec_tensor, src=0)
    else:
        dist.send(tensor=tensor, dst=0)
        dist.recv(tensor=rec_tensor, src=1)
        pass
    print("Rank ", rank, " has data ", rec_tensor)


def init_processes(rank, size, fn, backend="gloo"):
    """Initialize the distributed environment."""
    os.environ["MASTER_ADDR"] = "127.0.0.1"
    os.environ["MASTER_PORT"] = "29500"
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 3
    processes = []
    torch.multiprocessing.set_start_method("spawn")
    for rank in range(size):
        p = Process(target=init_processes, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>

ptrblck: <p>The docs point out that the <code>send</code> is applied synchronously, so the process would wait until the tensor is received. If you want to use the async API use <code>isend</code> and <code>irecv</code> and make sure the tensors are available before using them.</p>
Taking nth Derivatives of x(t)

George_Stepaniants: <p>Hello,</p>
<p>I have a neural network function x(t) which accepts a scalar t and returns a scalar x. I have 5000 time points t_1, …, t_m and would like to compute the nth order derivatives of x with respect to t at these different time points.</p>
<p>Currently, I am using the naive approach of computing the nth derivative wrt. t at each t_i sample point separately as described here: <a href="https://stackoverflow.com/questions/50322833/higher-order-gradients-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python - Higher order gradients in pytorch - Stack Overflow</a><br>
but this is prohibitively slow!</p>
<p>I have looked into potential ways of making torch.autograd.grad accept batches of input points (t_1, …, t_m) but with no success. It seems like this functionality is not implemented but I was hoping there is a better approach than computing the nth derivative with respect to each sample in a for loop.</p>
<p>Thank you!</p>

albanD: <p>We actually just added that to <code>autograd.grad</code> : <a href="https://pytorch.org/docs/master/generated/torch.autograd.grad.html?highlight=grad#torch.autograd.grad" class="inline-onebox">torch.autograd.grad — PyTorch master documentation</a></p>
<p>You can use the <code>is_grads_batched</code> flag to specify that the grads you give have an extra batch.</p>

George_Stepaniants: <p>I feel like is_grads_batched is not doing exactly what I was expecting. Namely, given a neural network x(t), it is computing the derivative at one fixed point x’(t*) and then for a batch of inputs t_1, …, t_m it is computing the product x’(t*) t_1, …, x’(t*) t_m.</p>
<p>What I am trying to do is compute x’(t_1), …, x’(t_m). In other words, I would hope to compute the derivative of the neural network x(t) at a batch of sample points t_1, …, t_m. Is there a batched way to do this in pytorch?</p>

George_Stepaniants: <p>Turns out all I needed was a function like this:</p>
<pre><code class="lang-auto">def deriv(self, t, n=1):
        dx = self.forward(t)
        for i in range(n):
            dx = torch.autograd.grad(outputs=dx, inputs=t, grad_outputs=torch.ones_like(t), create_graph=True)[0]
        return dx
</code></pre>

albanD: <p>Hi,</p>
<p>You can have your function be able to take a batch of inputs (most provided functions handle that already). And then backward through that.</p>
<p>Not sure what your code sample is doing since <code>i</code> is not used inside the loop.</p>

George_Stepaniants: <p>i is not used inside the loop because it is just computing the nth derivative by differentiating the previous value of dx.</p>

albanD: <p>Ho I didn’t understood your notation sorry… Yes that would be the way to get the nth derivative.</p>