akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch index cumulative sum operation

username: nios93
cooked: <p>Is there any way to create the torch.index_add behavior but for a cumulative summation?</p>
<p>For example instead of the final vector output  in the following aggregation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c09881be5fcfdce2624c530fce66cf7d4fd5d022.png" alt="image" data-base62-sha1="rtMpXkUpjdtnVf31dvrGS7Weixc" width="461" height="280"></p>
<p>I would like a cumulative index sum output vector as:</p>
<blockquote>
<p>output: [5, 5+1, 7 ,5+1+2 , 3, 3+2, 1, 1+3]</p>
</blockquote>
<p>Thank you!</p>
Is it possible to convert Transformer Tokenizers into Onnx format?

username: Yermukhamet_Medetov
cooked: <p>Lets say I have some Transformer model LayoutLM (but it could be any model from hugging face) then I would like to convert it to ONNX format , but before applying some Sentence we have to tokenize it via Bert or other tokenizer, I already know how to wrap model into Onnx but I have troubles with converting tokenizer to onnx , any suggestions ?</p>
Why the Transformer model size is the same for different nhead?

username: odats
cooked: <p>I have just started learning Transformer architecture. I suppose the number of heads should increase the number of QKV matrices. As a result, the number of learnable parameters should increase.</p>
<p>Why does the following code produce the same output?</p>
<pre><code class="lang-auto">import torch
from torch import nn

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(count_parameters(nn.Transformer(nhead=8))) # 44140544
print(count_parameters(nn.Transformer(nhead=1))) # 44140544
</code></pre>

username: vdw
cooked: <p>The model size is actually the size of the QKV matrices, the latter sizes are scaled by the number of heads. In therms of source code, it looks something like that.</p>
<pre><code class="lang-auto">qkv_size = max(model_size // num_heads, 1)
</code></pre>
<p>That the number of parameters should not increase with the number of heads is by design. The also means that you will see an error of the <code>model_size</code> cannot be properly be divided by <code>num_heads</code>.</p>

username: odats
cooked: <p>Do I understand correctly: the nhead is like a parameter to balance between single big QKV matrices (nhead=1) or multiple smaller QKV matrices?</p>
<p>Then how should I create a new Transformer with the same size QKV matrices as given but with 8 more heads? (it should be 8 times larger).</p>
<pre><code class="lang-auto">t1 = nn.Transformer(nhead=1)
t2 = nn.Transformer(qkv_size same as in t1 but 8 times more heads)
count_parameters(t1) * 8 == count_parameters(t2)
</code></pre>

username: vdw
cooked: <p>You can check the original <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noopener nofollow ugc">Attention is all You Need paper</a>:</p>
<blockquote>
<p>In this work we employ h = 8 parallel attention layers, or heads. For each of these we use<br>
dk = dv = dmodel/h = 64. Due to the reduced dimension of each head, the total computational cost<br>
is similar to that of single-head attention with full dimensionality.</p>
</blockquote>
<p>It doesn’t have to be done this way, but that’s how <code>nn.Transformer</code> is implemented anyway.</p>
<p>For learning purposes, I’ve implemented my own Transformer architecture from scratch, and there I could easily keep <code>model_size</code> without dividing by the number of heads.</p>
<p><strong>EDIT:</strong> Yes, with <code>nn.Transformer</code> you would need to multiple your intended <code>model_size</code> by the number of heads.</p>
Why dataloader can not use global vatiable?

username: zo_Aoo
cooked: <p>Suppose the following is python script</p>
<pre><code class="lang-auto">import torch
import os
import scipy.io as sio
from torch.utils.data import TensorDataset
root = '.'
data_img = sio.loadmat(os.path.join(root, 'mirflickr25k-iall-vgg-rand.mat'))['XAll']
data_txt = sio.loadmat(os.path.join(root, 'mirflickr25k-yall-rand.mat'))['YAll']
labels = sio.loadmat(os.path.join(root, 'mirflickr25k-lall-rand.mat'))['LAll']
def MIRFlickr25K_fea(partition):
    test_size = 2000
    train_size = 10000
    if 'test' in partition.lower():
        data_img, data_txt, labels = data_img[-test_size::], data_txt[-test_size::], labels[-test_size::]
    elif 'train' in partition.lower():
        data_img, data_txt, labels = data_img[0: train_size], data_txt[0: train_size], labels[0: train_size]
    else:
        data_img, data_txt, labels = data_img[0: -test_size], data_txt[0: -test_size], labels[0: -test_size]

    return torch.from_numpy(data_img), torch.from_numpy(data_txt), torch.from_numpy(labels)

d1,d2,d3 = MIRFlickr25K_fea('test')
mydata= TensorDataset(d1,d2,d3)
c = torch.utils.data.DataLoader(
        mydata
    )
print(c)

</code></pre>
<p>Why DataLoader can not find the global variable even though VSCODE debuger can find these values.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55945c35cf89e21309956ad51cad611f36ecb984.png" data-download-href="https://discuss.pytorch.org/uploads/default/55945c35cf89e21309956ad51cad611f36ecb984" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55945c35cf89e21309956ad51cad611f36ecb984_2_690x350.png" alt="image" data-base62-sha1="cd4rBvIf24ck9SZVOpeEGGiFfxO" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55945c35cf89e21309956ad51cad611f36ecb984_2_690x350.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55945c35cf89e21309956ad51cad611f36ecb984_2_1035x525.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55945c35cf89e21309956ad51cad611f36ecb984_2_1380x700.png 2x" data-dominant-color="322726"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1686×856 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Create smaller model from pruning mask

username: dens2101
cooked: <p>Hello,</p>
<p>We know that pytorch pruning does not actually speed up inference.</p>
<p>I plan to do the following:</p>
<ul>
<li>train a faster rcnn resnet18 on my dataset</li>
<li>apply group pruning (from the pruning module) on the backbone, in order to select kernel filters to be pruned</li>
<li>iterate on layers, and create a new backbone containing only non-pruned filters, and copy weights</li>
<li>retrain on the smaller model</li>
</ul>
<p>In this way, I hope that I can speed up inference.<br>
Is my approach reasonnable ?</p>
<p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Training diverges if torch.autocast is not used

username: Assefa_Seyoum
cooked: <p>I was training a face recognition baseline model without changing any hyper parameters. After training 100 epochs, the equal error rate (EER)  on a validation set was <strong>29.61567</strong>.</p>
<p>Using the same hyper parameters and using torch.autocast() context manager, the exact same model has EER of <strong>13.34276</strong> in just 15 epochs.</p>
<pre><code class="lang-auto">         with autocast():
               loss = self.model(data.cuda(), label.cuda())

          torch.cuda.amp.GradScaler()(loss).backward();
          torch.cuda.amp.GradScaler().step(self.optimizer);
          torch.cuda.amp.GradScaler().update();
</code></pre>
<p>I have repeated the experiment several times, and the results are consistent.</p>
<p>What exactly does torch.autocast() do?</p>

username: ptrblck
cooked: <p><code>toch.cuda.amp.autocast()</code> is a mixed-precision training util. and allows for op-specific <code>dtype</code> casts to speed up the training while maintaining accuracy as described in the <a href="https://pytorch.org/docs/stable/amp.html?highlight=autocast#torch.autocast">docs</a>. It should neither improve nor decrease the model accuracy if properly used.<br>
Based on your code snippet you are recreating the <code>GradScaler</code> in each forward pass, which is also wrong, so take a look at <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">these examples</a> to see how <code>amp</code> should be used.</p>

username: Assefa_Seyoum
cooked: <p>You are right about recreating the GradScaler. It is actually initialized only once. I wanted make it explicit in the snippet which looks awfully wrong.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    ...
    self.scaler = GradScaler()
    ...
    def forward(self, x, **kwargs): 
        ...
        with autocast():
           loss = self.model(data.cuda(), label.cuda())
        self.scaler.scale(loss).backward()
        self.scaler.step(self.optimizer)
        self.scaler.update()
        ...
</code></pre>
<p>However, the accuracy is still very low if I don’t use mixed precision training.</p>
Pad several dimensions at the same time

username: Aeryan
cooked: <p>I want to pad with zeros a tensor along different dimensions at the same time.</p>
<p>For example, given a tensor of shape <strong>[3,3,3,7]</strong>, I want to obtain a tensor of shape <strong>[5,5,5,7]</strong>, where the new entries contain a constant value (e.g., zero). For example, the value at the index [2,<strong>4</strong>,2,6] would contain 0.0.</p>
<p>Additionally, I would like for the method to be as efficient as possible and also preserve gradients (e.g., the element at the position [1,0,2,4] would have its gradient preserved since it was contained in the old tensor, before padding occurred).</p>

username: ptrblck
cooked: <p><code>F.pad</code> should work:</p>
<pre><code class="lang-python">x = torch.randn(3, 3, 3 ,7, requires_grad=True)
out = F.pad(x, (0, 0, 1, 1, 1, 1, 1, 1), mode="constant", value=0.0)

print(out.shape)
# torch.Size([5, 5, 5, 7])

print(out[2,4,2,6])
# tensor(0., grad_fn=&lt;SelectBackward0&gt;)

loss = out.mean().backward()
print(x.grad[1,0,2,4])
# tensor(0.0011)
</code></pre>
Use Sigmoid Top prediction model but i didn't the probabality always output 0.0

username: Youness_EL_BRAG
cooked: <p>hey to all of you<br>
i would to like to know why. when i am using torch.sigmoid(Y_hat) to convert to probability, always get 0.0 without extra values</p>

username: ptrblck
cooked: <p><code>torch.sigmoid</code> will output zeros for large negative values as seen here:</p>
<pre><code class="lang-python">torch.sigmoid(torch.ones(10) * -1000)
# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>I don’t know which “extra values” you are expecting so could you describe your use case and issue a bit more, please?</p>
Loss function depends on the output gradients

username: D12138
cooked: <p>I would like calculate the loss in the following form:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4918383a2bf8c59b7f36d243dc0f34c6f00408a0.jpeg" alt="stackoflow" data-base62-sha1="aqCKenJdg53e69eYDdNCl0GQCME" width="359" height="74"><br>
where u_bc and \hat{u}_bc are the predicted and exact values of x_1, u’‘_r and \hat{u}’'_r are the predicted and exact second derivatives of the output from x_2. x_1 and x_2 are different samples.<br>
I am trying to implement in the following way:</p>
<pre><code># forward pass to calculate the first loss component
u_bc_pred = self.forward(self.X_u)
loss_bcs = self.loss_fnc(u_bc_pred, self.Y_u)

# the second loss component involves the second derivative of output
u_r_pred = self.forward(self.X_r)
u_x = torch.autograd.grad(u_r_pred, self.X_r, torch.ones_like(u_r_pred), retain_graph=True, create_graph=True)[0] / self.sigma_x
u_xx = torch.autograd.grad(u_x, self.X_r, torch.ones_like(u_x), retain_graph=True, create_graph=False)[0] / self.sigma_x
loss_res = self.loss_fnc(u_xx, self.Y_r)

# Total loss
loss = loss_res + loss_bcs
self.loss_log.append(loss.data)

# backpropagation
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>It does not seem that my code is decreasing u_xx loss. Is there anything wrong with the way I write the second loss term involving solution gradients?<br>
Can someone please help take a look?<br>
Thanks a lot!</p>
How to get a tensor of diagonal vectors

username: Ananyapam_De
cooked: <p>I have a tensor which has some diagonal matrices inside it.</p>
<pre><code class="lang-auto">tensor([[[-0.2920,  0.0000],
         [ 0.0000, -0.8663]],

        [[-0.2920,  0.0000],
         [ 0.0000, -0.7568]],

        [[-0.3373,  0.0000],
         [ 0.0000, -0.5953]],

        [[-0.0068,  0.0000],
         [ 0.0000, -0.8065]],

        [[-0.1260,  0.0000],
         [ 0.0000, -0.7147]]], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>I want to do an operation on this to change this tensor to the following such that I do not lose the gradients because I need to call backward later.</p>
<pre><code class="lang-auto">tensor([[-0.2920, -0.8663],

        [-0.2920, -0.7568],

        [-0.3373, -0.5953],

        [-0.0068, -0.8065],

        [-0.1260, -0.7147]])
</code></pre>
<p>I saw <code>diagonal_embed</code> but it didnt help much. How do I do this?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/ananyapam_de">@Ananyapam_De</a>,</p>
<p>You should be able to do something like,</p>
<pre><code class="lang-auto">diagonal_tensors = torch.randn(10, 2).diag_embed() #creates batch of diagonal matrices
diagonal_only = torch.diagonal(diagonal_tensors, 0, -2, -1) #get 0 = diagonal and -2,-1 is over the last two dimensions. 
#returns 
"""
tensor([[ 1.4204,  0.3541],
        [-0.0236, -0.3348],
        [ 0.0364,  0.7933],
        [ 0.4506,  1.5018],
        [-1.2026, -0.0438],
        [ 1.8065,  1.8191],
        [-1.2577,  0.7627],
        [ 0.6864, -0.3432],
        [-0.0303,  0.0384],
        [ 0.0991,  0.4471]])
"""
</code></pre>
<p>if you want it the same shape as your example (with the spacing between the rows), you’ll have to <code>unsqueeze</code> the results, i.e.,</p>
<pre><code class="lang-auto"> torch.diagonal(diagonal_tensors, 0, -2, -1).unsqueeze(-2)
"""
#returns 
tensor([[[ 1.4204,  0.3541]],

        [[-0.0236, -0.3348]],

        [[ 0.0364,  0.7933]],

        [[ 0.4506,  1.5018]],

        [[-1.2026, -0.0438]],

        [[ 1.8065,  1.8191]],

        [[-1.2577,  0.7627]],

        [[ 0.6864, -0.3432]],

        [[-0.0303,  0.0384]],

        [[ 0.0991,  0.4471]]])
"""
</code></pre>
<p>But you can just check the shapes via <code>x.shape</code> if you need to.</p>
Normalization does not work as expected!

username: papoo13
cooked: <p>I am trying to normalize CUB_200 images by ImageNet stats. This is my code:</p>
<pre><code class="lang-auto">def _read_images_from_list(imagefile_list):

    imgs = []
    mean=[0.485, 0.456, 0.406]
    std= [0.229, 0.224, 0.225]

 
    
    Transformations = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean, std)])
    
    for imagefile in imagefile_list:
        
        print("Reading img: ", imagefile)
        img = cv2.imread(imagefile).astype(np.float32) # cv2 returns images with channels in BGR order. This is reverse to the order that torchvision pretrained channels accept - (500, 357, 3) np.ndarray
         
        img = cv2.resize(img, (224, 224))
        # Convert BGR to RGB
        img_r, img_g, img_b = np.split(img, 3, axis=2) # the output should in fact be b, g, r 
        img = np.concatenate((img_b, img_g, img_r), axis=2) # the images are in BGR # here we can use Image.merge("RGB", (r, g, b)) (from PIL import Image)
        img = Transformations(img)
        imgs += [img]
    return imgs
</code></pre>
<p>This way I have images with very large max and very small min. What am I doing wrong?</p>
To(device) changed value of input

username: wilsonng
cooked: <pre><code class="lang-auto">for images, _in train_dataloader:
     print(images.min(), images.max())
     images= images.to(device)
     print(images.min(), images.max())
        ...
</code></pre>
<p>What could be the potential reason that the range of the images changes from [-1, 1] to [-inf, inf] after images= images.to(device)?</p>
<p>To be specific, after to(device), the value of images could change to any arbitrary values including Nan.</p>
<p>In the images shown below, after to(device), there could be horizontal or vertical lines appear on the new image and those pixel values are arbitrary.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52c8db1a52668067297a02f2309941450e381c9c.png" data-download-href="https://discuss.pytorch.org/uploads/default/52c8db1a52668067297a02f2309941450e381c9c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52c8db1a52668067297a02f2309941450e381c9c_2_690x369.png" alt="image" data-base62-sha1="bOlu1CVJ3ab0Ty7BM35wBAlxREg" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52c8db1a52668067297a02f2309941450e381c9c_2_690x369.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52c8db1a52668067297a02f2309941450e381c9c_2_1035x553.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52c8db1a52668067297a02f2309941450e381c9c.png 2x" data-dominant-color="AAAAAA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1092×584 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>This would usually indicate a software issue (e.g. a missing synchronization), a driver issue, or a hardware issue.<br>
I would thus recommend to update PyTorch as well as your CUDA driver first to check if this helps. If not, try to use your GPU in another system and check if you are seeing the same issues (assuming it’s easy to reproduce). If not, it could be any other part of your local setup.</p>

username: ptrblck
cooked: <p>Also, double post from <a href="https://discuss.pytorch.org/t/to-device-changes-the-value-of-input/167491/2">here</a>, so disregard my post as you’ve already narrowed down the root cause without updating this double post.<br>
Please avoid creating multiple posts in the future.</p>
CUDA error: CUBLAS_STATUS_EXECUTION_FAILED on cuda 11.8

username: Merlo
cooked: <p>Hello,<br>
I’m trying to run pytorch using cuda 11.8 on my device using an RTX 3060 card:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   54C    P8    12W /  N/A |      5MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2515      G   /usr/lib/Xorg                       4MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>however, any time I run my code I get a</p>
<pre><code class="lang-auto">File "/home/merlo/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>Nothing I could find online helped, what could be the issue? I’m running pytorch:</p>
<pre><code class="lang-auto">pytorch                   1.13.0          py3.10_cuda11.7_cudnn8.5.0_0    pytorch
</code></pre>
<p>installed directly from the official channel. Can I somehow update to use cuda 11.8?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Merlo" data-post="1" data-topic="167493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/merlo/48/55383_2.png" class="avatar"> Merlo:</div>
<blockquote>
<p>I’m trying to run pytorch using cuda 11.8</p>
</blockquote>
</aside>
<p>You are using the CUDA 11.7 runtime since you’ve installed the corresponding PyTorch binary. Your local CUDA toolkit will only be used if you are building PyTorch from source or a custom CUDAExtenstion.</p>
<p>In any case, could you post a minimal, executable code snippet to reproduce the issue, please?</p>
Code runs good once then is always broken. Help?

username: Drayhem
cooked: <p>Trying to get started with Kaggle and I’ve ran into what appears to be a somewhat unique problem. The code can be found here if I did everything right:<br>
<a href="https://drive.google.com/file/d/1WU9FB3mDAdQxIq61rhyh-5XjzsinGIsS/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1WU9FB3mDAdQxIq61rhyh-5XjzsinGIsS/view?usp=sharing</a><br>
You’ll have to check it out to see what I’m talking about since I’m new and can only embedded one thing &gt;_&gt;</p>
<h1>
<a name="problem-1" class="anchor" href="#problem-1"></a>Problem:</h1>
<p>BCEWithLogitsLoss becomes unstable while training a four hidden layer FC network, giving nan’s after anywhere between the first and 10th call, breaking the training. I have been googling a bunch and trying to figure out where the destabilization is coming from, but to no avail. <strong>Please help!</strong></p>
<p><span class="hashtag">#Further</span> details:<br>
For the “understanding stage” of solving this problem, I’ve been plotting the value of the loss function, and the test and training data set accuracy. On a typical run, the losses gives no proper plot. This comes from the function turn to nan’s fairly quickly. In the training set accuracy,  the model initially learns for an epoch or two and then oscillates as the loss function becomes useless.</p>
<p>It’s worth noting that I have tried some of the tricks out there on stack exchange for nan-ing loss functions. Interestingly, when I add a small value (1e-10) to the predictions or labels, this is enough to allow training to complete one time. When you track the value of the loss function and train/test acc, the successful train gives this:<br>
<img src="https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12625083%2Fa157a237057f49c958f2798bae44bc08%2Ftitanic_problems%201.png?generation=1669611099322523&amp;alt=media" alt="" role="presentation" width="690" height="224"> While this may feel like a success, the terrible truth is that this only sometimes works once. After you’ve made a successful run, rerunning the <em>exact same code</em> gives failed results where nan’s creep in somewhere along the training.</p>
<p>One final note. In the documentation for BCEWithLogitsLoss, it specifies that the target values should be between zero and one. To that end, I tried mapping the 0’s and 1’s in the “survived” column to points within the interval (e.g. 0-&gt;1/3, 1 -&gt;2/3), but this always gives even worse results! Unfortunately, the accuracy becomes fully zero when doing then, which certainly isn’t what I’m after.</p>
<p>If anyone can offer insight or help, that would be greatly appreciated. This is code and a network that successfully trains on the wine UCI dataset and I cannot figure out what’s different here.</p>

username: Drayhem
cooked: <p>I’m embarrassed. This was a data processing error. There were some nan’s in a column that I didn’t expect</p>
Pytorch 2.0 Compile to native code

username: Peter_Featherstone
cooked: <p>It seems pytorch has had the idea many times to convert models to IR representations. It looks like <code>torch.fx</code> kind of does this and now <code>torch.compile</code> too.</p>
<p>It would be really cool if pytorch converted models to LLVM IR, then used LLVM to compile to native code which could either be used as a static library, shared library or python module.</p>
<p>So workflow would be train → export to LLVM IR → LLVM compile →  {model.h/libmodel.a, model.h/libmodel.so} with runtime dynamic linking to libcuda.so using dlopen</p>
<p>Then pytorch would essentially become its own language and be a proper compiler. This is inline with software 2.0 where machine learning just becomes another programming language. It emits libraries or binary which can be consumed by other software. Sounds great.</p>
Newbie, non linear regression not converging

username: archocron
cooked: <p>First im new to pytorh and DL, I want to create a simple non linear regression model, but apparently is not converging, i tried to change some hyperparams without sucess. This is the code, i guess im making wrong something obvius.</p>
<pre><code>    import torch
    
    x1 = torch.arange(1,600,1,  dtype=torch.float32).view(-1,1)
    y1 = x1*x1
    
    model = torch.nn.Sequential(
        torch.nn.Linear(1,500),
        torch.nn.ReLU(),
        torch.nn.Linear(500,1)
    )
    
    criterion = torch.nn.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
    
    for i in range (10000):
        y_pred = model(x1)
        loss = criterion(y_pred, y1)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
        if i%100 == 0:
            print(loss)
</code></pre>
<p>print(model(torch.tensor([6], dtype=torch.float32)))</p>

username: ptrblck
cooked: <p>Your code works fine if I use <code>Adam</code> and increase the number of epochs.<br>
The loss starts with a huge value since you are not normalizing the inputs (and outputs), which could help training the model.<br>
E.g. if I use this native normalization:</p>
<pre data-code-wrap="python"><code class="lang-python">x1 = torch.arange(1,600,1,  dtype=torch.float32).view(-1,1)
y1 = x1*x1

mean = x1.mean()
std = x1.std()
x1 = x1 - mean
x1 = x1 / std

mean = y1.mean()
std = y1.std()
y1 = y1 - mean
y1 = y1 / std

model = torch.nn.Sequential(
    torch.nn.Linear(1,500),
    torch.nn.ReLU(),
    torch.nn.Linear(500,1)
)

criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for i in range (10000):
    y_pred = model(x1)
    loss = criterion(y_pred, y1)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i%100 == 0:
        print(loss)
        
        
import matplotlib.pyplot as plt

plt.plot(y1 * std + mean)
plt.plot(y_pred.detach() * std + mean)
</code></pre>
<p>I can overfit the model pretty quickly:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35bdd18fdd2f0f75e2cec5a4b24f2a7c628aaca4.png" alt="image" data-base62-sha1="7FpZp8Rcj4s3rOSroeNGdAFo0gQ" width="393" height="248"></p>

username: archocron
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> many thanks.</p>
Pytorch profiler not exporting any stack information?

username: Vedant_Roy
cooked: <p>I’m trying to export some stack information to generate flamegraphs by following this tutorial:</p>
<p>However, I can’t seem to get it work. Specifically, the generated file is always empty.</p>
<p>Here’s a snippet to reproduce:</p>
<pre><code class="lang-auto">#! /usr/bin/env python3
import torch
import torchvision.models as models
from torch.profiler import profile, record_function, ProfilerActivity

model = models.resnet18().cuda()
inputs = torch.randn(5, 3, 224, 224).cuda()

with profile(activities=[ProfilerActivity.CUDA], with_stack=True) as prof:
    model(inputs)

prof.export_chrome_trace("trace.json")
prof.export_stacks("stacks.txt", metric="self_cuda_time_total")
</code></pre>
<p>stacks.txt is always empty. Does anyone have suggestions on what I should do?</p>
Problem in training a GAN network

username: Mahdi_Amrollahi
cooked: <p>I have used a DCGAN network from this <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">tutorial</a> to create my simple images. The images are <code>(35,35,3)</code> which there is a colored circle with random radius in them.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2d79d1ad8e873264784e868a5a2b5d5a97ec046.png" alt="image" data-base62-sha1="wmJReafAyu5wXDxKLOw8NS3bURU" width="84" height="80"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e05e33cceb7cd0a2230f3c1f967ae2270bf8571e.png" alt="image" data-base62-sha1="w0QMq4fyfWXskOsdIY0B2hKWG6i" width="84" height="76"></p>
<p>During the training, the discriminator works well and the loss decreases in each step. However, there is a problem with Generator which the output is not desirable and it just makes some noises and in each epoch the loss increases dramatically from the beginning.<br>
I have tried tuning hyper-params , changing the Generator model and increasing the samples but all the same.<br>
May I ask how is your idea and where the problem is?</p>
Global variables

username: robinho
cooked: <p>With the following code<br>
def f1():<br>
x.requires_grad = True</p>
<p>def f2(a,b,…):<br>
x=1<br>
f1()</p>
<p>calling f1() doesn’t seem to recognize x, defined in f2, as a global variable to f1?</p>

username: ConvolutionalAtom
cooked: <p>Hi,<br>
In your case, when f1 is defined your are trying to access x from global scope when it is not defined in global scope. To access variable from the scope of f2, I would do the following:</p>
<pre><code class="lang-auto">
def f2(a,b,…):
    x=1
    def f1():
        x.requires_grad = True
    f1()
</code></pre>
<p>Here, when defining f1, it access x from the scope of f2. But, of course you can also pass x as parameter to f1 and do the following,</p>
<pre><code class="lang-auto">def f1(x):
    x.requires_grad = True  # x should be torch.Tensor

def f2(a,b,…):
    x=1   #  this would not work since x integer and torch.Tensor
    f1(x)
</code></pre>
<p>Regarding scoping, I would suggest the following reading: <a href="https://www.datacamp.com/tutorial/scope-of-variables-python" rel="noopener nofollow ugc">https://www.datacamp.com/tutorial/scope-of-variables-python</a></p>

username: robinho
cooked: <p>thanks so much. Then with my original lines I can only add global like the following if I want to put f1 outside of f2?<br>
def f1():<br>
global x<br>
print(x)</p>
<p>def f2(a,b,…):<br>
global x<br>
x=1<br>
f1()</p>

username: ConvolutionalAtom
cooked: <p>Yes, global keyword puts variable into global scope. Therefore, it would work. But, (as always) be careful with the use of global variables.</p>
<p>Note: I would also suggest to use ``` symbol to show code (rather than plain text), it makes it easier to read for others</p>
Why do neural networks with more layers perform better than a single layer?

username: laro
cooked: <p>Why do neural networks with more layers perform better than a single layer MLP with a number of neurons that leads to the same number of parameters?</p>
<p>I read this post:</p>
<p><a href="https://www.quora.com/Why-do-neural-networks-with-more-layers-perform-better-than-a-single-layer-MLP-with-a-number-of-neurons-that-leads-to-the-same-number-of-parameters" rel="noopener nofollow ugc">https://www.quora.com/Why-do-neural-networks-with-more-layers-perform-better-than-a-single-layer-MLP-with-a-number-of-neurons-that-leads-to-the-same-number-of-parameters</a></p>
<p>and still I’m not sure it always true.</p>
<p>For example:</p>
<p>Assume with have model with 2 linear layers (for simplicity with no bias) and RELU as the final layer, the model look:</p>
<pre><code>   RELU (w1x + w2x)
</code></pre>
<p>and the number of parameters to optimize is (w1 + w2)</p>
<p>We can see that:</p>
<pre><code>  RELU (w1x + w2x) =  RELU ((x(w1 + w2)) = RELU (w3x)
</code></pre>
<p>so <code>w3 = w1+w2</code></p>
<p>i.e second model with 1 linear layers has less parameters to optimize.</p>
<ol>
<li>In my example is it still better to use one layer or 2 layers ?</li>
<li>Am I right that the second model (with <code>w3</code>) has less parameters?</li>
<li>Is it easier to optimize the second model (<code>w3</code>) ?</li>
</ol>

username: ptrblck
cooked: <p>I’m not sure how to interpret your “model” definition:</p>
<pre><code class="lang-python">RELU (w1x + w2x)
</code></pre>
<p>as it seems you are trying to sum the outputs of both layers, which is unusual.<br>
Assuming you want to use two consecutive linear layers with a final <code>ReLU</code>, you model would be defined as:</p>
<pre><code class="lang-python">out = relu(w2 @ (w1 @ x))
</code></pre>
<p>In this case you are right and the two linear layers can be seen as a single one, since no activation function was used.<br>
However, collapsing these layers is not possible if you are using activation functions, as would be the common approach:</p>
<pre><code class="lang-python">out = relu(w2 @ relu(w1 @ x))
</code></pre>
Where is Sparse Indexing Defined?

username: Wimmerer
cooked: <p>Hi folks,</p>
<p>I’m looking for the definitions of the get_index functions for COO (coalesced and uncoalesced) as well as compressed sparse tensors. I’ve found the index set function (I think): <a href="https://github.com/pytorch/pytorch/blob/825f4e602b766545a4ee6dfd971056e24c7dbbe8/aten/src/ATen/SparseTensorImpl.cpp#L80" class="inline-onebox" rel="noopener nofollow ugc">pytorch/SparseTensorImpl.cpp at 825f4e602b766545a4ee6dfd971056e24c7dbbe8 · pytorch/pytorch · GitHub</a>, but can’t find its complement.</p>
<p>Thanks!</p>
Understanding BatchNorm usage in DCGAN Tutorial

username: Max_Muster
cooked: <p>Hello,</p>
<p>in the DCGAN Tutorial <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">here</a> I do not understand why there is no BatchNorm layer after the first Conv Layer in the Discrimenator:</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.main(input)
</code></pre>
<p>I do not have a good understanding of the usage of BatchNorm layers. But from the looks it seems like it is being used after every Conv Layer, so why not also after the first one?</p>
The loss function is either very large or very small.

username: Xige
cooked: <p>I’m trying to classify two very similar fonts using a binary classification network. I used BCEwithlogitLoss, but the training loss is either very large (e.g., 0.69) or very small (e.g., 0.002). Can you tell me what could be the reason for this?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afddf2c11fb2c33f2cfd9b8aa807fcbc54546af5.png" alt="1670052617436" data-base62-sha1="p5N4UKUzzevmVSxRdIvu8quRAih" width="574" height="228"></p>
How to create simple images?

username: Mahdi_Amrollahi
cooked: <p>I was trying to build a very simple model to create simple circle images with 5 different colors and random radius, but I did not get a good result. I have created 500 images for each color for the training set.<br>
I am just wondering that the logic of the model is correct.</p>
<p>The input shape is : <code> one_hot([1,0,0,0,0])</code> and the output shape is <code>(35,35,3)</code>.<br>
Here is my simple model:</p>
<pre><code class="lang-python">cls_model(
  (model): Sequential(
    (0): Linear(in_features=5, out_features=1024, bias=True)
    (1): ReLU()
    (2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
    (3): Linear(in_features=1024, out_features=961, bias=True)
    (4): ReLU()
  )
  (tconv1): ConvTranspose2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
  (tconv2): ConvTranspose2d(1, 3, kernel_size=(3, 3), stride=(1, 1))
)
</code></pre>
<pre><code class="lang-python">criterion = nn.MSELoss()
optim = torch.optim.Adam(net.parameters(), lr=1e-3)
</code></pre>
Cowardly refusing to serialize non-leaf tensor which requires_grad

username: spyroot
cooked: <p>Hi Folks,</p>
<p>I cross-checked a couple of times. I never serialize tensor with grad, but I still get this error.<br>
Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing .</p>
<p>How can I find what tensor actually causing this ?  Becase if multiprocess so that couldn’t happen on a backward pass.   I never actually put anything on the queue with grad.</p>
<p>It is either something in /popen_spawn_posix.py", line 47, in _launch or torch somehow reconstructed somewhere tensor in another process, and I believe they are connected.  (in a graph)</p>
<p>How I can trace this tensor ?</p>
Mismatch in bias addition

username: Prakhar_Pradhan1
cooked: <p>Hey Folks,<br>
Recently I came across strange stuff in bias addition. Bias can be added in 2 ways :</p>
<ol>
<li>
<p>Acc = F.conv2d(input, weights , bias) : bias is added inside the convolution layer in PyTorch</p>
</li>
<li>
<p>Acc = F.conv2d(input, weights , torch.zeros(bias.shape))<br>
Acc = Acc + bias<br>
here bias is added externally</p>
</li>
</ol>
<p>Surprisingly I noticed that the Acc after bias addition in the two cases above are different and bias addition in case 2 is more accurate <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Can anyone help me in explaining if PyTorch uses any optimization algorithm while doing bias addition inside a convolutional later?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Could you explain how you’ve measured the accuracy in both cases and what the ground truth was, please?</p>

username: jondapper
cooked: <p>Can you provide a minimal example which reproduces the problem?</p>

username: Prakhar_Pradhan1
cooked: <p>Sure,<br>
A1 = F.conv2d(input, weight, torch.zeros_like(bias)) = tensor([128276)]<br>
bias = tensor([16777215])<br>
A1 + bias = tensor([16905492])</p>
<p>A2 = F.conv2d(input, weight, bias) = tensor([16905484])</p>
<p>so A1 is not equals to A2 and A1 is close to the actual addition value</p>

username: ptrblck
cooked: <p>Your used numbers are close to the range where integers start to be rounded to multiples of 2 (<code>[2**24, 2**25]</code>) as seen <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">here</a>.<br>
Depending on the order of operations these rounding errors are expected. If you need more precision in this range, you could use <code>float64</code>.</p>

username: Prakhar_Pradhan
cooked: <p>Thank You <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
Copy a model's gradients to another model

username: rkzhou
cooked: <p>Hi all,</p>
<p>I am trying to implement the communication of gradients between two models with same architecture, like copying model A’s gradients to model B.</p>
<p>In model A, I use:<br>
gradients = {}<br>
inputs = inputs.to(device)<br>
labels = labels.to(device)<br>
outputs = modelA(inputs)<br>
optimizerA.zero_grad()<br>
loss = criterion(outputs, labels)<br>
loss.backward()<br>
for name, param in modelA.named_parameters():<br>
gradients[name] = param.grad.clone()</p>
<p>In model B, I use:<br>
optimizerB.zero_grad()<br>
for name, param in modelB.named_parameters():<br>
param.grad = gradients[name]<br>
optimizerB.step()</p>
<p>I am not sure if there is any error in this part, but the testing accuracy in cifar10 dataset remains about 10% ~ 15% after 50 epochs. Really need help from someone.</p>

username: ptrblck
cooked: <p>The core looks correct (at least I don’t see any obvious issues).</p>
<aside class="quote no-group" data-username="rkzhou" data-post="1" data-topic="167416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rkzhou/48/55361_2.png" class="avatar"> rkzhou:</div>
<blockquote>
<p>I am not sure if there is any error in this part, but the testing accuracy in cifar10 dataset remains about 10% ~ 15% after 50 epochs. Really need help from someone.</p>
</blockquote>
</aside>
<p>Could you explain why this approach should work at all?<br>
Assuming both models are using different parameters, I would expect to see a failure in <code>modelB</code>’s training if <code>modelA</code>’s gradients are used.</p>

username: rkzhou
cooked: <p>Sorry, I forgot to mention that I would synchronize the weights of these two parameters after each epoch training, by passing modelB’s state_dict to modelA, and modelA will load the state_dict. So, if modelA can converge to a minimum point of the loss function, modelB could converge to the same point. If my mind is correct, I should that should work. I am sure that modelA can reach at least 70% testing accuracy if I only train it solely.</p>
<p>By the way, I define modelA and modelB, optimizerA and optimizerB as global variables in two different  python files. Would that be the possible problem?</p>
Hyperparameters tuning with cross validation

username: John4
cooked: <p>Hello, I applied gridSearch (with cross validation) method to find the best parameters; then I fit again the model with the best parametes:</p>
<pre><code>svc = svm.SVC(class_weight='balanced')
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.3, random_state=109)
clf = GridSearchCV(svc, par, cv=cv)
search = clf.fit(Xtrain, ytrain)
b_par = search.best_params_
svc_best = svm.SVC(C=best_C, kernel=best_kernel, gamma=best_gamma, probability=True, 
class_weight='balanced')
svc_best.fit(X_train, y_train)
y_pred = svc_best.predict(X_test)
</code></pre>
<p>My question is: is there a way to re-fit the model with the addition of the cross validation?</p>

username: ptrblck
cooked: <p>I don’t understand the question, since you are already using the training set (which was previously used to run the cross-validation).<br>
This question in unrelated to PyTorch, so you might get a faster and better answer in a scikit-learn discussion board.</p>
Model Memory Requirement Calculation for Inference?

username: anujd9
cooked: <p>Hi, I need some help/guidance on how to calculate the amount of memory required when performing a model inference.<br>
So, as per my understanding, the memory will be used by the input image (uint8/float), model weights (FP32) and the activations generated during the Forward Pass. ( Please feel free to correct me if I am wrong here. )<br>
But how do I calculate this using PyTorch for a specific model, say ResNet18?<br>
Thanks.</p>
Multiprocessing.shared_memory, autograd and debug flag CUDA

username: spyroot
cooked: <p>Hi Folks,</p>
<p>Can anyone please share the resource where I can check all torch auto grad and CUDA debugging env flags? I’m debugging multiprocessing cases where the model set shared_memory, and it is a bit complex second-order auto grad in each consumer.</p>
<p>Thank you !</p>
How to determine the value of the parameters in the loss functions

username: msr3384
cooked: <p>J_1 = 0.5 * torch.norm(output.mT - output.mT@weight, p = ‘fro’)**2<br>
J_2 =  torch.sum(torch.sum(abs(weight)))<br>
J_3 = 0.25 * torch.trace((torch.square(output@output.mT - torch.eye(len(output)))))</p>
<p>loss =  gamma *J_2 + lamda * J_3+ J_1</p>
<p>The loss function involves two parameters. Since the values of J_1, J_2 and J_3 all have different scale in terms of the magnitude, so the right values of the parameters are very important for loss function to converges. Now without knowing the values of J_1, J_2, and J_3 how to determines the values of the parameters? Is there any general techniques to determine these parameters?</p>
How to deal with a 200go dataset with 1M samples?

username: tuturta
cooked: <p>Hello,</p>
<p>I’ve created a huge dataset of 200go to train my GNN model. This dataset contains 1M of samples.<br>
I would like to use Dataset, Dataloader class with it but the dataset is too large to fit into my memory.<br>
I also tried to put all my samples in separate files to follow <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html" rel="noopener nofollow ugc">this solution</a> but writing 1M files in my SSD made it crash.</p>
<p>Then, I just split my dataset into 20 files of 10 go that can fit into my memory but know I was wondering how can I load this file using the DataLoader class? Is my solution a good solution or should I do something else?</p>

username: JuanFMontesinos
cooked: <p>Numpy’s memory map allows to read on-demand those segments that compose ur batch.</p>

username: brunompacheco
cooked: <p>Maybe you could store your data into a HDF5 file, which allows you to read the data chunk-by-chunk, without loading everything to memory. HDF5 also supports compression, so the chunks do not occupy your whole SSD. <a href="https://docs.h5py.org/en/stable/" rel="noopener nofollow ugc">h5py</a> has a nice introduction to it, if you are interested.</p>
<p>In this case, your custom dataset could be something like:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, hdf5_fpath):
        super().__init__()

        self._fpath = hdf5_fpath

    def __len__(self):
        with h5py.File(self._fpath, 'r') as h:
            return h.shape[0]

    def __getitem__(self, index):
        with h5py.File(self._fpath, 'r') as h:
            X = h['X'][index]
            y = h['y'][index]

        return X, y
</code></pre>
<p>which can be used to create a DataLoader as usual.</p>