akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why call self.attention.forward

mst72: <p>I am learning llama, where self.attention.forward is explicitly called. But normally, we don’t write code that explicitly calls forward, but why would llama authors explicitly call it here? Is it just because of habit?</p>
<p>Explicitly called here:  <code> h = x + self.attention.forward(self.attention_norm(x), start_pos, freqs_cis, mask)</code></p>
<pre><code class="lang-auto">class TransformerBlock(nn.Module):
    def __init__(self, layer_id: int, args: ModelArgs):
        super().__init__()
        self.n_heads = args.n_heads
        self.dim = args.dim
        self.head_dim = args.dim // args.n_heads
        self.attention = Attention(args)
        self.feed_forward = FeedForward(
            dim=args.dim,
            hidden_dim=4 * args.dim,
            multiple_of=args.multiple_of,
            ffn_dim_multiplier=args.ffn_dim_multiplier,
        )
        self.layer_id = layer_id
        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention.forward(
            self.attention_norm(x), start_pos, freqs_cis, mask
        )
        out = h + self.feed_forward.forward(self.ffn_norm(h))
        return out
</code></pre>

ptrblck: <p>It’s unclear what their intentions are (and you might want to ask in their GitHub repository) as it will skip potentially registered forward hooks.</p>
Does padded rows (fake inputs) affect backpropagation?

jupyter: <p>Each row data doesn’t have the same size. It looks like this:</p>
<p>Ideally, the shape of input data will be: <code>(batch_size, N, dim)</code></p>
<p>But each row in the batch is not equal dimension. E.g: it can be <code>(k, dim)</code> k &lt; N</p>
<p>To feed to my model I have to add some fake rows called padded rows.</p>
<p>These inputs go through some functions, layers in my model.</p>
<p>In the end, the loss function takes the mean of each row as input → <code>(batch_size, dim)</code> . But I don’t want to consider the padded row in the reduction.</p>
<p> → So I computed the mean as:</p>
<ul>
<li>assigned all the padded rows to zero.</li>
<li><code>torch.sum(..)/(number_of_non_padded_rows)</code></li>
</ul>
<p>My question is: As the padded rows is included in the parameter of function torch.sum(…), does the model try to modify the weights base on these fake (padded) rows?<br>
My second assumption is because I assigned the padded row by a constraint value which is zero, then it somehow get rid of gradients of these padded rows during training?</p>
Unexpected Reboot

eaah: <p>I am currently facing an unexpected reboot of my system when running a pytorch model.</p>
<p>The model just calculates an embeding for 200 images every 1 second. I load the images using cv2 and process them either with a ViT or a VGG16 model.</p>
<p>Both model load to GPU correctly and the code runs for various iterations.</p>
<p>The sytem hardwar is built with the following components:</p>
<p>-1080ti<br>
-2080ti<br>
-Ryzen 7<br>
-850W PSU</p>
<p>I have tested two different models thinking that size of the model or computation complexity of the model might be causing the reboot. The tests ran with a ViT and a VGG16. Unfortunately, the shutdown occurs with both models and it happenes at random time when running the sequence of images.</p>
<p>Ubuntu 22.04<br>
NVIDIA drivers installed:<br>
CUDA12.2 Driver: 535<br>
NVCC --version: cuda 12.2</p>
<p>Pytorch version: 2.2</p>
<p>I am monitoring the gpus with nvidia-smi and none of the cards memory rises more than 4GB used. I am controlling the temperatures and both cards operates at a maximum of 47°C during the tests.</p>
<p>Any suggestions how can I debbug or fix this unexpected reboot?</p>
<p>Thanks in advanced for all responses.</p>

marksaroufim: <p>This either sounds like a power issue something like a loose GPU to PSU cable or not a big enough PSU (Although I think yours should be enough) or a CPU temperature issue. It sounds like you built your own PC in which case are you sure your CPU fan is working otherwise your CPU will get really hot and shut down</p>
<p>I would suspect you can repro the shutdown using software like <a href="https://geeks3d.com/furmark/" class="inline-onebox">FurMark Homepage</a> and some CPU stress test you can find online.</p>

eaah: <p>Thanks I perform the tests by removing one of the GPUs and it worked fine. It seemed to be PSU issue.</p>
Can't find entry point caffe2@@3GB in c10.dll

Dom83: <p>Hello.<br>
Using Automatic1111. Everything was working fine, but I tried updating xformers, which seems to have updated pytorch too and caused this trouble. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Can’t find entry point ?int_ptr_metadata_index@detail@caffe2@<span class="mention">@3GB</span> in c10.dll.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de2749326a2156c63b0f5c13e28a35823ad9a895.jpeg" alt="pytorch_error" data-base62-sha1="vHgaqm7nMYYl9dkQQaMTm7je4zH" width="496" height="184"></p>
<p>Can somebody help, please?</p>
What happens to the gradients if the output is multiplied by zero?

Siladittya_Manna: <p>Suppose, <em>h</em> is the output embedding vector of a network. Now, out of the D dimensions of <em>h</em>, I multiply the last ‘dr’ dimensions with 0 before passing it to the loss function, while keeping the rest of the (D - dr) dimensions unchanged. That is, <code>h = [h1, h2, h3, ... hk, 0, 0, 0, ..., 0, 0]</code></p>
<p>This is done for all the output embeddings in a batch.</p>
<p>Now, the question is, when the gradients are back-propagated, do the gradients along those ‘dr’ dimensions become zero too?</p>

KFrank: <p>Hi Siladittya!</p>
<aside class="quote no-group quote-modified" data-username="Siladittya_Manna" data-post="1" data-topic="198125" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siladittya_manna/48/26984_2.png" class="avatar"> Siladittya_Manna:</div>
<blockquote>
<p><code>h = [h1, h2, h3, ... hk, 0, 0, 0, ..., 0, 0]</code><br>
…<br>
do the gradients along those ‘dr’ dimensions become zero too?</p>
</blockquote>
</aside>
<p>Yes (because the gradient of zero is zero).</p>
<p>You can check this with a simple script:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.2.1'
&gt;&gt;&gt; t = torch.arange (1.0, 11.0, requires_grad = True)
&gt;&gt;&gt; m = torch.ones (10)
&gt;&gt;&gt; m[7:] = 0.0
&gt;&gt;&gt; m
tensor([1., 1., 1., 1., 1., 1., 1., 0., 0., 0.])
&gt;&gt;&gt; (t**2 * m).sum().backward()
&gt;&gt;&gt; t.grad
tensor([ 2.,  4.,  6.,  8., 10., 12., 14.,  0.,  0.,  0.])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Image generation experiment with mnist images - not working quite as expected

dvijay: <p>I’m trying a simple experiment using mnist images. Not sure if this is a viable idea or not. It’s not working quite as expected so I thought I’d get some feedback.<br>
What I’m doing is basically take mnist images, ignore the labels but instead have a simple model that takes a random image (just generated with torch.rand) and uses the mnist images as output. I trained the model with the entire training dataset (60k). Now, when I run the model with a random image as input, it always generates a fixed output image, which happens to be the number 9.<br>
Is this a reasonable approach? Since every single input image in the training is random, I’m not sure what the model would actually learn. But it is generating a valid output at the end, just not one of the 10 different numbers as I’d want. Instead it only generates the number 9.<br>
Any thoughts or explanation of what I’m seeing is appreciated.</p>
EfficientNet Error - RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1000 and 1408x512)

Nitya_Rakhe: <p>I am trying to train an Efficientnet model in PyTorch. Below is the code I have written for the model. In the following code I get the error "RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1000 and 1408x512)</p>
<pre><code class="lang-auto">backbone_model = EfficientNet.from_pretrained('efficientnet-b2', include_top=False)
x = torch.randn(1,3,150,150)
features = backbone_model(x)
num_channels = features.shape[1]
print(num_channels, "num_channels")

num_classes = len(CLASSES)


class CustomModel(nn.Module):
    def __init__(self):
        super(CustomModel, self).__init__()
        self.backbone_model = models.efficientnet_b2(pretrained=True)
        self.backbone_model._avg_pooling = nn.AdaptiveAvgPool2d(1)
        self.dropout1 = nn.Dropout(p=0.2)
        self.fc1 = nn.Linear(1408, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc_out = nn.Linear(256, num_classes)
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, x):
        x = self.backbone_model(x)
        print(x.shape, "shape1")
        x = torch.flatten(x, 1)
        print(x.shape, "shape2")
        print("fc1 input size", self.fc1.in_features)
        x = self.dropout(x)
        x = self.fc1(x)
        x = torch.relu(x)
        x = self.dropout(x)
        x = self.fc2(x)
        x = torch.relu(x)
        x = self.fc_out(x) 
</code></pre>
<p>print(num_channels, “num_channels”) → 1408 num_channels</p>
<p>print(x.shape, “shape1”) → torch.Size([4, 1000]) shape1</p>
<p>print(x.shape, “shape2”) → torch.Size([4, 1000]) shape2</p>
<p>print(“fc1 input size”, self.fc1.in_features) → fc1 input size 1408</p>
<p>How can I solve this error?</p>

th3gentlecod3r: <p>From the looks of it, I would try to change the <code>self.fc1 = self.fc1 = nn.Linear(1408, 512)</code> to <code>self.fc1 = nn.Linear(1000, 512)</code>. This is just an intuition.</p>
<p>UPDATE:</p>
<p>Upon further investigation, I found out that the classifier block of the EfficientNet is still set to 1000 as its output shape and the two fully connected layers that you used are just getting built on top of it. That is to say, the classifier output is still there.</p>
Penalizing cosine similarity between kernels

tal123: <p>I have a fully convolutional network, (like YOLOv3 or SSD).<br>
The last layer uses K 1x1 kernels to produce a tensor of K predictions all over the feature map (i.e., a BxKxHxW tensor).</p>
<p>I want those kernels to be linearly independent, i.e., that there will be no predictor that can be computed as a linear combination of the others.</p>
<p>For that reason, I want the optimizer to be rewarded for learning kernels that will be orthogonal to each other, by adding their pairwise cosine similarity to the loss.</p>
<p>The loss function only receives the network output. How do I add the cosine similarity of the kernels to the loss? Should I implement a special mode where the network will include them in the output of its <code>forward</code> method, so the loss function will have access to it?</p>
<p>Thanks</p>

ptrblck: <p>You don’t need to manipulate the <code>forward</code> method and can add any auxiliary loss to the already computed loss as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv2d(1, 64, 1)
x = torch.randn(1, 1, 24, 24)
target = torch.randint(0, 64, (1, 24, 24))

criterion = nn.CrossEntropyLoss()

output = conv(x)
loss = criterion(output, target)
aux_loss = 1 - F.cosine_similarity(conv.weight[0], conv.weight[1])
loss = loss + aux_loss
loss.backward()
</code></pre>

tal123: <p>Thank you so much for the comprehensive answer. I am going to use the auxiliary loss approach.</p>
<p>Just note that in my case, <code>aux_loss</code> will be the cosine similarity itself, or actually its absolute value, rather than <code>1 - CS</code>. I want the loss to grow as the kernels are getting more similar to each other.</p>
<p>When two vectors have the same direction, their cosine similarity will be 1, while when they are exactly opposite (but still very related, actually a negative scalar multiplication of each other), it will be -1.</p>
<pre><code>&gt;&gt;&gt; c = nn.Conv2d(3, 64, 1) 
&gt;&gt;&gt; F.cosine_similarity(c.weight[0], c.weight[0], dim = 0)
tensor([[1.0000]], grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; F.cosine_similarity(c.weight[0], -c.weight[0], dim = 0)
tensor([[-1.0000]], grad_fn=&lt;SumBackward1&gt;)
</code></pre>
<p>Therefore, the absolute cosine similarity is the loss I’m looking for. In my code for 3 kernels, it will look like this:</p>
<pre><code>conv = nn.Conv2d(3, 64, 1)  # 3 kernels of 1x1 shape, feature dimensionality is 64
aux_loss = 0
# The shape of conv.weight[i] is (64,1,1). Therefore, specifying dim=0
aux_loss += torch.abs(F.cosine_similarity(conv.weight[0], conv.weight[1], dim = 0))
aux_loss += torch.abs(F.cosine_similarity(conv.weight[0], conv.weight[2], dim = 0))
aux_loss += torch.abs(F.cosine_similarity(conv.weight[1], conv.weight[2], dim = 0))
</code></pre>
<p>I.e., every pair of 2 kernels is rewarded for being in different directions (striving to being orthogonal).</p>
How to initialize a quantized.Embedding?

HuangLED: <p>The API doc says quantized.Embedding implements the same interface as nn.Embedding thus it should be easily replaceable.</p>
<p>I tried to use it but found that in my model, it previously use “init.uniform_” to initialize Embedding. And this init looks like not compatible with quantized version embedding. What is the recommended way to initialize quantized.embedding?</p>
<p>error message:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
self.reset_parameters()<br>
File “/data00/home/user/byted/test/poc/test_dw.py”, line 148, in reset_parameters<br>
init.uniform_(self.node_embed.weight.data, -init_range, init_range)<br>
AttributeError: ‘function’ object has no attribute ‘data’</p>
</blockquote>
<p>Thank you!</p>

HuangLED: <p>Figured out by reading code from github, there is slight difference between two implementations, now I made the initialization work.</p>
<p>Previously my model works perfectly fine with nn.Embedding.  Now when it runs with quantized.Embedding, I got this optimizor error.  Any suggestions?</p>
<p>Traceback (most recent call last):<br>
File “/data00/home/user/byted/poc/test_dw.py”, line 246, in <br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)<br>
File “/home/user/anaconda3/envs/egs39/lib/python3.9/site-packages/torch/optim/sgd.py”, line 27, in <strong>init</strong><br>
super().<strong>init</strong>(params, defaults)<br>
File “/home/user/anaconda3/envs/egs39/lib/python3.9/site-packages/torch/optim/optimizer.py”, line 273, in <strong>init</strong><br>
raise ValueError(“optimizer got an empty parameter list”)<br>
ValueError: optimizer got an empty parameter list</p>
Yolox object detection improvement

shakhub: <p>I am using yolox-s architecture for 11 class object detection model which is trained from scratch. My training data is well balanced and the input image size is 160x96. The training data is about 50,000 images. Average object size is 20x10px. I am able to get mAP of 0.88</p>
<p>Two things I have observed are:</p>
<ol>
<li>Two of the 11 classes under perform in mAP metrics.</li>
<li>loss_cls, loss_reg and loss_iou reduces but loss_obj starts increasing after 40 epochs.</li>
</ol>
<p>How to improve the performance of the two classes. Why is loss_obj increasing after 40 epochs.</p>
RuntimeError: does not have a grad_fn

Limalka_Sadith: <p>I’m trying to use the model output as indexes to create a 3d point cloud. Because in order to calculate the loss I want to convolve it.<br>
<strong>code:</strong></p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    optimizer.zero_grad()  # Zero the gradients
    output = model(masked_coords)
    points_3d = gen_3d(200,200,35, mask, output)
    psf_output = conv_3D(points_3d,IPSF)
    loss = loss_fn(psf_output, target)
    loss. Backward()  # Backpropagation
    optimizer.step()  # Update weights
</code></pre>
<p><strong>This is my gen_3d function:</strong></p>
<pre><code class="lang-auto">def gen_3d(x,y,z, mask, depths):
    pc = torch.zeros(x,y,z)
    _pc = pc.view(-1,z)
    m = mask.flatten()
    d = (depths*z).int().flatten()
    _pc[m, d] += 1
    return torch.nn.functional.pad(pc, (0, 0, 10, 10, 10, 10))
</code></pre>
<p><strong>Error:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37753bb8dc97af6a2c6144040227483718ecef3a.png" data-download-href="https://discuss.pytorch.org/uploads/default/37753bb8dc97af6a2c6144040227483718ecef3a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37753bb8dc97af6a2c6144040227483718ecef3a.png" alt="image" data-base62-sha1="7UBqML0Qv6AXWNqaRP2rc5VBneO" width="630" height="499" data-dominant-color="5E3223"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">780×619 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>seems like the process is not differentiable. Is there a way to solve this?</p>

ptrblck: <p>Converting <code>depth</code> to an integer detaches the tensor from the computation graph so the error is expected. I don’t know how the point cloud will be used and if there is a valid solution so let’s wait for other inputs.</p>

Limalka_Sadith: <p>Thanks for the reply. Is there a way to bypass this and connect the gradients? (ex. coping the gradients into respective indices)</p>
Issue with torch.batch_norm(): Cannot convert a MPS Tensor to float64

alpha_bootis: <p>Hi! i’m new to the forums, i hope to formulate the issue correctly:<br>
I am working with the pytorch maskrcnn model which contains a bunch BatchNorm2d layers.<br>
When trying to run the model with my Macbook M1 i run into this error:<br>
<code>TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.</code><br>
The problem seems to lie in the torch.batch_norm() function, as i’m pretty sure all tensors i’m using are torch.float32.<br>
These are the easy steps to replicate the issue:</p>
<pre><code class="lang-auto">import torch

device = "mps"

test_input = torch.zeros([0, 256, 14, 14], dtype=torch.float32).to(device, dtype=torch.float32)
weight = torch.zeros([256], dtype=torch.float32).to(device, dtype=torch.float32)
bias = torch.zeros([256], dtype=torch.float32).to(device, dtype=torch.float32)
running_mean = torch.zeros([256], dtype=torch.float32).to(device, dtype=torch.float32)
running_var = torch.zeros([256], dtype=torch.float32).to(device, dtype=torch.float32)
momentum=0.1
eps=0.1
training=False

torch.batch_norm(test_input, weight, bias, running_mean, running_var, training, momentum, eps, True)
</code></pre>
<p>Thanks in advance and let me know if i can improve the post in any way.</p>
Resnet with SVM output layer

xolotl18: <p>Hi everyone, for a project I find myself having to recreate the experimets of this paper( <a href="https://arxiv.org/abs/1306.0239" rel="nofollow noopener">https://arxiv.org/abs/1306.0239</a>), where they integrate the svm in the structure of the network in order to fine tune even the first layers with the svm objective.<br>
My problem is that I have little to no idea on how implement this feature. In particular I am using the implementation of resnet of this repository( <a href="https://github.com/akamaster/pytorch_resnet_cifar10" rel="nofollow noopener">https://github.com/akamaster/pytorch_resnet_cifar10</a>) in the other parts of the project, so I need to use it also here for consistency<br>
another thing that I would like to ask you is how can I use the pretrained network as a feature extractor for the svm in order to compare the results with the other mothod.<br>
thank you in advance</p>

ptrblck: <p>I’m not familiar with the paper, but have a look at <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MultiLabelMarginLoss"><code>MultiLabelMarginLoss</code></a>, which implements the multi-class hinge loss and might thus be seen as the “SVM loss”.</p>

xolotl18: <p>thank you for answering, I am a little bit confused about the whole thing…<br>
do you think that the result of implementing hinge loss is going to be the equivalent of replacing the fc layer with a support vector classifier?<br>
And another thing, how can I use the net that I alread trained on the dataset as a feature extractor and then use those features to train an svm classifier?</p>

ptrblck: <p>I’m not sure for the first question, but have seen use cases which use the proposed loss instead of an SVM.</p>
<p>If you want to use a “real” SVM, you could store all extracted features as numpy arrays (with their target), and train e.g. an SVM from <code>sklearn</code> on it. This would of course not retrain the feature extractor anymore, but might be a valid use case, if you just want to train the SVM.</p>

xolotl18: <p>I would lilke to do both things.<br>
First I would like to use an svm classifier instead of the fc layer, in order to train the network with it, and I was wondering if it was enough to use a different loss in order to do it, or if I had to do something else.<br>
I also would like to try and use the sklearn implementation of the svm using the in_features of the fc layer as its inputs. In this case my question regards how to retrieve these features from the trained net</p>

Seba_AlNasser: <p>Hello,<br>
Did you find out how to integrate SVM in the neural network structure? I’m currently trying to implement similar model.</p>
Issue with Fine-Tuning SageFormer Model: CUDA Error

Scbas_scias: <p>I’m encountering a problem while fine-tuning the <em>SageFormer</em> model from Hugging Face in a Colab notebook. After a certain number of epochs, I consistently encounter the following error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>My training data consists of images sized at (640, 640, 3), and I’m using an image processor to align them with the model. Despite trying various approaches, I’m still unable to resolve this issue.</p>
<pre><code class="lang-auto">checkpoint = "nvidia/mit-b0"
image_processor = AutoImageProcessor.from_pretrained(checkpoint, reduce_labels=False)
</code></pre>
<p>Could anyone offer insights into why this error might be occurring and how I could potentially troubleshoot it?</p>

ptrblck: <p>Rerun your script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace to isolate the failing method. Often these kind of issues are raised in an embedding layer receiving inputs containing values of of the expected range (<code>[0, num_embeddings-1]</code>).</p>
What is the correct way to qat a conv layer with weight norm

kunle12: <p>Hi all,</p>
<p>It seems that if I prepare_qat on a weight_norm(conv) layer, the prehook for weight norm is disabled/removed, only normal quantized weight remains (due to module swapping?). What is the right way to carry out quantization in this case? do I need to create a custom quantization module for this.</p>
<p>Thanks,</p>
<p>Xun.</p>

jerryzh168: <p>yeah I think this is due to module swapping, right I think you need to create a custom QAT module and quantized module for this module, and connect this with the quantization flow.</p>
<p>If you are using fx graph mode quantization, you can use the BackendConfig api (<a href="https://pytorch.org/docs/master/generated/torch.ao.quantization.backend_config.BackendConfig.html#torch.ao.quantization.backend_config.BackendConfig" class="inline-onebox" rel="noopener nofollow ugc">BackendConfig — PyTorch master documentation</a>), we will publish a tutorial later for this cc <a class="mention" href="/u/andrewor">@andrewor</a>.</p>

cxy22: <p>hi!<br>
Have you solved it? I don’t know how to solve it yet.</p>