akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
NotImplementedError: Could not run 'aten::as_strided' with arguments from the 'SparseCUDA' backend.

username: csuwjf524
cooked: <p>NotImplementedError: Could not run ‘aten::as_strided’ with arguments from the ‘SparseCUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::as_strided’ is only available for these backends: [CPU, CUDA, Meta, QuantizedCPU, QuantizedCUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].</p>

username: kumpera
cooked: <p>Hi,</p>
<p>This is a limitation of Pytorch’s sparse tensor backend. I’d ask this question on the right forum as this is not related to distributed features.</p>
<p>I’d suggest you provide test code that reproduces this issue as part of asking your question as there’s not enough context from just the error message to know what you’re trying to accomplish and what’s the best way to address it.</p>
<p>Given my understanding of how sparse tensors work, the concept of strides don’t apply to them so it makes sense you cannot use as_strided on them.</p>

username: Jonas_De_Schouwer
cooked: <p>Any update on this? Did you ask the question in another forum?</p>
Problems with loading the model with libtorch

username: Kris1
cooked: <p>I trained a CNN model with libtorch. And I successfully save the model using “torch::save(model,“model.pt”)”. But when I load the model in other project and use it for inference" model-&gt;forward(input)", the code throws out error “Method forward is not defined.”. I tried several ways to save  model like OutputArchive, but none of them works.<br>
By the way, I know the .pt file can be unzipped. After unzipping it , in “./code/<strong>torch</strong>.py”, there is no defination for forward.<br>
Any one can give me some help? Thanks!</p>

username: Kris1
cooked: <p>Thanks! I tried <code>torch::save</code>, but when I load the model with <code>torch::load</code>, there is an error “The operator &lt;&lt; should be overloaded.” And I changed the defination of the model from “class” to “struct” and added “TORCH_MODULE”. Everything works well.<br>
As for <code>torch::jit::trace</code>, my compiler asserts "There is no class or namespace <code>trace</code> in<code> torch::jit</code> ". However, i found there is “trace” method in namespace <code>torch::jit::tracer</code>.</p>

username: Anqin
cooked: <p>I got the same problem. how did you make it work?</p>
More efficient face alignment using affine/similarity transform in pytorch

username: rafale77
cooked: <p>I have been searching for a solution to do this more efficiently entirely with torch tensors but have not found one so I am posting here to see if some expertise could help.<br>
I am running inference on a facial detection model which then needs an alignment to then be an input for recognition.<br>
The flow goes from: cv2 image → torch → detection model -&gt;torch landmarks → numpy(cpu) -&gt;estimate transform (cv2 or skimage) (cpu) → execute transform on cv2 image (cpu) → torch for next model</p>
<p>All torch tensors are actions are done on GPU. Now I would like to avoid doing this back and forth between numpy and torch.</p>
<p>In order to do this I would need to get torch equivalent functions to the two cv2 functions below:</p>
<pre><code>        tfm, _ = cv2.estimateAffinePartial2D(src_pts.cpu().numpy(), ref_pts)
        face_img = cv2.warpAffine(image, tfm, out_size)
</code></pre>
<p>Are there any such functions? I have found ways to convert the transform matrix to F.affine_grid which would still require to generate the transformation matrix in numpy:</p><aside class="quote quote-modified" data-post="1" data-topic="19522">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f475e1/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/affine-transformation-matrix-paramters-conversion/19522">Affine transformation matrix paramters conversion</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi all, 
I want to rotate an image about a specific point. First I create the Transformation matrices for moving the center point to the origin, rotating and then moving back to the first point, then apply the transform using affine_grid and grid_sample functions. But the resulting image is not what it should be. Once I tested these parameters by applying them on the image using scipy and it works. 
import torch.nn.functional as F
mat_move = torch.eye(3)
mat_move[0,2] = -x_center*2/my_image.shap…
  </blockquote>
</aside>
<p>
but would like to avoid this altogether by generating the similarity transformation directly from torch tensors and apply it to a tensor.</p>

username: Geneous22
cooked: <p>Follow my script <a href="https://gist.github.com/duonglong289/79392e7062cbe8517294266ef7670703" class="inline-onebox" rel="noopener nofollow ugc">face_alignment_torch.py · GitHub</a></p>
USE_CUDNN set to zero after upgrade to libtorch 2.0

username: Lin_Jia
cooked: <p>Hey I just upgraded to libtorch 2.0, but it says that it compiles with USE_CUDNN = 0, and will compile without cudnn support. I did not change anything on my codebase including CMAKEFile. I am wondering anyone else experiencing this?</p>

username: ptrblck
cooked: <p>Could you describe your workflow a bit more?<br>
Are you trying to build <code>libtorch</code> or an application based on <code>libtorch</code>?<br>
Would <code>export USE_CUDNN=1</code> work?</p>

username: Lin_Jia
cooked: <p>I directly downloaded the Libtorch from Pytorch front page: <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>    .<br>
Then libtorch is integrated into my C++ codebase via CMake<br>
Cuda can be found, but the configuration stage of building tells that build using USE_CUDNN=0, and<br>
exporting USE_CUDNN=1 does not work</p>

username: Xavier31
cooked: <p>Same error message after upgrading to libtorch2.0:<br>
<code>USE_CUDNN is set to 0. Compiling without cuDNN support</code><br>
although it does not seem to change anything in practice since cuda is available, and libtorch does run on the GPU.<br>
In my case, I still have cuda 11.5 and cudnn8.4 so it may be the cause if the error message</p>

username: benraziel
cooked: <p>I encountered the same issue after upgrading to pytorch 2.0</p>
<p>setting <code>CAFFE2_USE_CUDNN=1</code> solved it for me.</p>
<p>The relevant cmake code where pytorch searchs for cudnn is located here: <a href="https://github.com/pytorch/pytorch/blob/87f9160b6727ba554520bbd9f8905c7cf3c8654a/cmake/public/cuda.cmake#LL239C21-L239C21" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/87f9160b6727ba554520bbd9f8905c7cf3c8654a/cmake/public/cuda.cmake#LL239C21-L239C21</a></p>

username: Lin_Jia
cooked: <p><a class="mention" href="/u/benraziel">@benraziel</a></p>
<p>Welcome to the community. Thanks for the tip. It works!!!</p>

username: Joey_Guo
cooked: <p>you save my day.</p>
<pre><code class="lang-auto"># Find PyTorch
set(CAFFE2_USE_CUDNN ON)
set(USE_CUSPARSELT OFF)
find_package(Torch REQUIRED)
</code></pre>
Autocast/ Mixed Precision nondeterministic behavior during converting into half (float16)

username: AKT_TARAFDER
cooked: <pre><code class="lang-auto">
import torch
import torch.nn as nn
import itertools
import torch.nn.functional as F
import datetime
import argparse
import numpy as np
from torch.cuda.amp import autocast, GradScaler


x= torch.randn(5).cuda()
x=x.to(torch.float16)

print('x type: ', x.dtype)

use_fp16= True

with autocast(enabled = True, dtype=torch.float16):
    y= torch.exp(x)
    print('y type: ', y.dtype)
    z= torch.sum(x)
    print('z type; ', z.dtype)

    w=x+1
    print('w type: ', w.dtype)
</code></pre>
<p>gives the following output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe2e9b48f29c26ecc39d06d972cfd57f0a4a4d16.png" alt="image" data-base62-sha1="AgB8a0l0COdM4kM7KwBDUWvtVki" width="452" height="110"></p>
<p>(1) The question is why exp/sum producing a 32 bit floating point number?<br>
(2) When the kernels are called during Mixed-precision training, some operations might start calling 32 bit kernels, rather than 16 bit ones. How to know what is expected behavior of what operation?<br>
(3) Does anybody know that even for float16, are they doing float32 operations first and writing them, then converting it to 16 bits using another kernel (for matrix multiplication, GEMM, SpMM)?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/amp.html#op-eligibility">This section of the docs</a> might be helpful as it describes which operators are eligible.</p>

username: AKT_TARAFDER
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> There are 2 categories in the documentation. One of the listed ones for GPU is: can auto-cast to float32. But this does not detail when it goes to float32, and when it does not. I am a little confused about the selection criteria. Or does it mean these do not go into float16 at all?<br>
I noticed exp does not go into float16, when the input is float16 as shown in the code above. Exp is in “can autocast to float32”</p>
<p><a href="https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float32" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float32</a></p>
Batch size of test dataloader influences model performance even in eval() mode

username: Bodkens
cooked: <p>I have model with 3 LSTM layers and one full connected layer, I use MPS on MacBook Air with M2 processor.</p>
<p>I have set batch size of train dataloader at 64, and I have different results of model performance evaluation when I set batch size of test dataloader at 64 and 256, although I have set model in eval() mode</p>
<p>Training model code:</p>
<pre><code class="lang-auto">    for epoch in range(epochs):

        model.train()
        print(f'Epoch {epoch}')
        __train_loop(model, train_dataloader, loss_function, optimizer, scheduler, verbose, device=device)

        if test_per_epoch:

            model.eval()
            train_loss, train_accuracy, train_f_score = test_model(model, train_dataloader, loss_function, device=device)
</code></pre>
<pre><code class="lang-auto">def test_model(model, test_dataloader: DataLoader, loss_function, device='cpu'):
    loss = 0
    y_pred_all = []
    y_all = []
    with progressbar.ProgressBar(max_value=(len(test_dataloader))) as bar:
        with torch.no_grad():

            for batch_id, (X, y) in enumerate(test_dataloader):
                X, y = X.to(device), y.to(device)

                y_pred = model(X)
                loss += loss_function(y_pred, y.to(device)).item()
                y_pred = torch.argmax(y_pred, dim=1)

                y_pred_all.append(y_pred.cpu().numpy())
                y_all.append(y.cpu().numpy())
                bar.update(batch_id)

    loss = 0  

    y_pred_all = np.hstack(y_pred_all).flatten()
    y_all = np.hstack(y_all).flatten()

    cr = classification_report(y_all, y_pred_all, output_dict=True)
    f_score = cr['macro avg']['f1-score']
    accuracy = cr['accuracy']

    return loss, accuracy, f_score
</code></pre>
<p>Model:</p>
<pre><code class="lang-auto"> (lstm): LSTM(3, 25, num_layers=3, batch_first=True, dropout=0.7)
 (dense): Linear(in_features=25, out_features=2, bias=True)
</code></pre>
<p>For loss function I use <code>torch.nn.CrossEntropyLoss()</code> . For optimizer I use <code>torch.optim.Adam()</code></p>
<p>learning rate is 0.001 with <code>ExponentialLR(gamma=0.95)</code> scheduler</p>
<p>Model performance after 6th epoch when batch size of test dataloader is 64</p>
<p>Train accuracy: 0.9087542087542088<br>
Train F-Score: 0.908342717859852</p>
<p>Test accuracy: 0.8906356801093643<br>
Test F-Score: 0.8901527949844201</p>
<p>Model performance after 6th epoch when batch size of test dataloader is 256</p>
<p>Train accuracy: 0.8942760942760942<br>
Train F-Score: 0.8939527432555182</p>
<p>Test accuracy: 0.5974025974025974<br>
Test F-Score: 0.5557444197838286</p>
<p>There is no such effect on Nvidia CUDA graphic cards, so I created this topic in MPS category. Why it works like that? Thanks a lot in advance for answer!</p>

username: Aniruth_Sundararaja1
cooked: <p>batch normalization layers (if any) might  exhibit different behaviors based on batch size due to their internal running statistics.</p>
<p>do u have any batch normalization layer</p>

username: Bodkens
cooked: <p>No, my model has only LSTM layer and Linear layer</p>
Add_bias_kv in MultiheadAttention

username: Vanda_Alicia
cooked: <p>The shape of tensor <code>k</code> is <code>(seq_len, batch_size, embed_dim)</code>.</p>
<p>In <code>activation.py</code>:<br>
<code>bias_k = Parameter(torch.empty((1, 1, embed_dim)))</code></p>
<p>In <code>functional.py</code>:</p>
<pre><code class="lang-auto"># add bias along batch dimension (currently second)
k = torch.cat([k, bias_k.repeat(1, bsz, 1)])
</code></pre>
<p>However, the <code>cat</code> operation concat these two tensors in <code>seq_len</code> dimension.<br>
Is this annotation <code># add bias along batch dimension (currently second)</code> wrong?<br>
I think it should be add bias along sequence length dimension</p>