akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Transformer for time series classification with classification costs

abhandari73: <p>Hi,<br>
I am using transformer to classify time series data into 5 categories and I have balanced data.<br>
I would like to input my own misclassification matrix so that the misclassification costs are unequal.<br>
Example: True Category 1 - Predicted Category 5 is worse than True Category 1 and Predicted Category 2.</p>
<p>Categories are ordinal in nature. So classifying 5 as 1 is worse than classifying 5 as 4.</p>
<p>How can I do that.<br>
Thanks</p>

alaria: <p>I have no experience with this in particular but the loss function you use should provide a parameters defining the weights for all classes.</p>
Help compile pytorch 2.0.1 with CUDA 11.4

RedHead: <p>Attempting to compile pytorch 2.01 with CUDA 11.4 on Xavier NX,<br>
using :</p>
<pre><code class="lang-auto">#!/bin/sh


    export DEBUG=1
    export BUILD_TEST=0
    export USE_CUDA=ON
    export ENABLE_CUDA=ON
    export CUDACXX=/usr/local/cuda/bin
    export CMAKE_CUDA_ARCHITECTURES=native
    export CMAKE_CUDA_FLAGS=-std=c++17
    export CUDNN_INCLUDE_DIR=/usr/include/aarch64-linux-gnu/
    export CUDNN_LIB_DIR=/usr/lib
    export USE_DISTRIBUTED=0
    export USE_MKLDNN=0
    export USE_FBGEMM=0
    export USE_NNPACK=0
    export USE_QNNPACK=0
    export USE_XNNPACK=0
    export BUILD_CAFFE2_OPS=0
    export PATH=/home/nvidia/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin
python setup.py build --cmake-only
cmake --build build
</code></pre>
<p>but in resulting  CMakeCache.txt</p>
<pre><code class="lang-auto">//Use CUDA
USE_CUDA:BOOL=OFF
</code></pre>
<p>and therefore pytorch compiles without CUDA<br>
same result with</p>
<pre><code class="lang-auto">export USE_CUDA=1
</code></pre>
<p>Any pointer appreciated, am I missing something obvious here?</p>
<p>Thanks</p>
Will there ever be provided builds for Torch+CUDA for AWS aarch64?

aostang: <p>The new Graviton servers are so much cheaper to run but building from source has proven to be incredibly painful. I’ve managed to get things mostly working but I still cannot get torchvision to build with image support no matter what I’ve tried.</p>

ptrblck: <p>Yes, we are working on binaries but in the meantime you can use <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">PyTorch NGC containers</a> supporting ARM for some time already.</p>

aostang: <p>Yeah, I had discovered that a couple days ago. I was fighting with getting the containers to recognize CUDA even with the Nvidia container stuff installed. I probably just need to clean slate a server and try that from scratch at this point since I’ve done installing and uninstalling for weeks on this server trying to brute force getting torch and torchvision working with CUDA. I finally got torch working but torchvision has refused to work with image support and I’ve about thrown in the towel on it because the error messages are absolutely no help even with looking at the code. Everything according to the build logs says it worked, it just doesn’t.</p>
Will the following script build wheels with CUDA support?

aostang: <aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/builder/tree/main/aarch64_linux">
  <header class="source">

      <a href="https://github.com/pytorch/builder/tree/main/aarch64_linux" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/builder/tree/main/aarch64_linux" target="_blank" rel="noopener nofollow ugc"></a></h3>

  <p><a href="https://github.com/pytorch/builder/tree/main/aarch64_linux" target="_blank" rel="noopener nofollow ugc">//github.com/pytorch/builder/tree/main/aarch64_linux</a></p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If they will, is there anything special I need to do? I’m trying to get around the pain of building from source where I finally have torch working but I cannot for the life of me get a build of torchvision that has image support to work.</p>

ptrblck: <p>No, these scripts are used to build the current ARM CPU-only binaries.</p>

aostang: <p>Ok thanks for the quick reply so that I did not waste my time!</p>
AssertionError with autograd vmap

Saimon: <p>Hi,<br>
I am trying to understand how to calculate the higher-order gradients using vmap and grad because it seems helpful in controlling the dimensions. However, vmap is showing error in my code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29afb89f52a7301f8a92eda90d432d3ff33700cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/29afb89f52a7301f8a92eda90d432d3ff33700cb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29afb89f52a7301f8a92eda90d432d3ff33700cb_2_517x164.png" alt="image" data-base62-sha1="5WM3xOcAi5sXPPow78TrIkO44vV" width="517" height="164" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29afb89f52a7301f8a92eda90d432d3ff33700cb_2_517x164.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29afb89f52a7301f8a92eda90d432d3ff33700cb_2_775x246.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29afb89f52a7301f8a92eda90d432d3ff33700cb_2_1034x328.png 2x" data-dominant-color="EEF1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1134×361 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/015f6e1c1a0325d421f988a833af4796599ffe6b.png" data-download-href="https://discuss.pytorch.org/uploads/default/015f6e1c1a0325d421f988a833af4796599ffe6b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/015f6e1c1a0325d421f988a833af4796599ffe6b_2_463x499.png" alt="image" data-base62-sha1="c8W0g9JhWLgSn6zzNomKMWr7RN" width="463" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/015f6e1c1a0325d421f988a833af4796599ffe6b_2_463x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/015f6e1c1a0325d421f988a833af4796599ffe6b_2_694x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/1/015f6e1c1a0325d421f988a833af4796599ffe6b.png 2x" data-dominant-color="F3F5EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">877×947 66.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance!</p>
CPU => GPU => CPU automatic memory management

rcecchini: <p>Hello,<br>
I have a very ugly idea that aims to allow me to inference a pretty large pretrained model on my 8GB vram. I would like to convert CPU tensors to GPU immediately before the execution of a tensors-related function and bring the result back to cpu. Is a pretty bottlenecks-creator solution that, anyway, seems theoretically faster than use only CPU. I tried to create a wrapper of the Python class Tensor, but is not a good idea. Now I’m looking for at some point of C++ code of torch where I can intercept all tensor related function where to do this operation without wrapping manually every possible function. For example, the ATen/core/dispatch zone could have an intermediate function that is called to execute these math functions? In a few words, I’m looking for a solution that doesn’t imply to edit too much the torch code, obviously also because I’m not able to manage a similar huge project through its development. Finally, this is only an experiment, but I don’t think to have too many alternative without the possibility to buy a more powerful video card. I hope there is some expert of the torch core source code that can give me an advise about how navigate in the source code and create an unique simple wrapper.</p>
<p>Thank you</p>
<p>P.S.<br>
From a “serious” point of view, I think that this could be a reflection for a real development of another class, a “Memory Stream Management” where the user can activate a similar function, but also choose the behaviour about where use CPU or GPU in basis of the available memory or recurrence of the operation etc. I know that appears dirty create a “tensor reference object” where track the same tensor among different devices, but I think that it could allow a strongest optimization about the use of accelleration devices. Another example could be a middleware function that can be setted to handle the cases where an operation involves tensors allocated on different devices, so the developer can choose where to move what tensor to continue the operation without breaking the code execution. Finally, also the possibility to move automatically a tensor to another device in base of currently available memory could be a good reason to the creation of a TensorRef class, maybe compatible with the interface of Tensor class…</p>
How nn.Unfold works?

hema_rathore: <pre><code class="lang-auto">&gt;&gt;&gt;unfold = nn.Unfold(kernel_size=(2, 3))
&gt;&gt;&gt; input = torch.randn(2, 5, 3, 4)
&gt;&gt;&gt; output = unfold(input)
</code></pre>
<p>Can someone explain the exact operation of using F.unfold please.  Like I read doc but its not clear I need to understand the whole maths behind.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>Unfold</code> uses a sliding window approach as seen in this code snippet:</p>
<pre><code class="lang-python">unfold = nn.Unfold(kernel_size=(2, 3))
input = torch.randn(2, 5, 3, 4)
output = unfold(input)

output_manual = []
kernel_size = [2, 3]
# sliding window approach
for i in torch.arange(input.size(2)-kernel_size[0]+1):
    for j in torch.arange(input.size(3)-kernel_size[1]+1):
        # index current patch
        tmp = input[:, :, i:i+kernel_size[0], j:j+kernel_size[1]]
        # flatten and keep batch dim
        tmp = tmp.contiguous().view(tmp.size(0), -1) # has a shape of [2, 30] afterwards
        output_manual.append(tmp)
    
# stack outputs in dim2
output_manual = torch.stack(output_manual, dim=2)

# compare
print((output_manual == output).all())
# &gt; tensor(True)
</code></pre>
<p>The manual approach doesn’t consider padding, dilation etc. and just uses the nested for loops to create the current patch/window via indexing.</p>

hema_rathore: <p>Okay <a class="mention" href="/u/ptrblck">@ptrblck</a> I think its lil clear now, actually sir in my case I am applying this unfold operation on a tensor of A as given below:</p>
<pre><code class="lang-auto">A.shape=torch.Size([16, 309,128])
A = A.unsqueeze(1) # that's I guess for making it 4 dim for unfold operation
A_out= F.unfold(A, (7, 128), stride=(1,128),dilation=(3,1))
A_out.shape=torch.Size([16, 896,291])
</code></pre>
<p>I am not getting this 291 <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> If dilation factor is not there then as per your nested loop theory it would be [16,896,303] right ? Thanks for reply</p>

JuanDeAguero: <p>This works great! How would you implement “fold”?</p>
How do create a combined loss?

alaria: <p>I’m trying to perform a classification task on time series data. I want to first predict the continuation of the time series and then classify all timeslots. I’m using a default PyTorch transformer to continue the time series and I think the transformer has too much liberty with its results if only the classification in the end is considered for the loss.</p>
<p>Example Input: (1,2,3,4,5)<br>
Example Continuation: (6,7,8)<br>
Example Classification: (True, False, True)      # simple example: just classify all even numbers as true</p>
<p>Can I somehow create two different losses and combine them? I’m using BCEWithLogitsLoss for the classification, which one should I use for the transformer? How do I combine them?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/alaria">@alaria</a>,</p>
<p>If define two loss functions in your script, you can just compute the losses individually and add them together. Autograd will backprop through both loss functions without a problem.</p>

alaria: <blockquote>
<p>If define two loss functions in your script, you can just compute the losses individually and add them together. Autograd will backprop through both loss functions without a problem.</p>
</blockquote>
<p>So literally just total_loss = loss1 + loss2? Nice, thank you.</p>
<p>Any recommendation for the type of loss function that is desirable for the transformer/time series part?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="alaria" data-post="3" data-topic="198577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/57b2e6/48.png" class="avatar"> alaria:</div>
<blockquote>
<p>So literally just total_loss = loss1 + loss2? Nice, thank you.</p>
</blockquote>
</aside>
<p>Exactly,</p>
<pre><code class="lang-auto">loss1  = loss_fn1(*args)
loss2 = loss_fn2(*args)

loss = loss1 + loss2
loss.backward() #etc...
</code></pre>
<aside class="quote no-group" data-username="alaria" data-post="3" data-topic="198577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/57b2e6/48.png" class="avatar"> alaria:</div>
<blockquote>
<p>Any recommendation for the type of loss function that is desirable for the transformer/time series part?</p>
</blockquote>
</aside>
<p>No idea unfortunately, but there might a tutorial on transformer for time-series data in the documentation somehow. Good luck!</p>

alaria: <p>I see, thank you. Would I usually have to reduce the learning rate when adding more loss? I already somewhat tuned my hyper params and I’m wondering if I just change them again.</p>
Debugging nan gradients: what am I doing wrong?

thanrl: <p>I see nan gradients in my model parameters.</p>
<p>I set <code>torch.autograd.set_detect_anomaly(True)</code> and it points to that <code>Function 'DivBackward0' returned nan values in its 1th output</code> on this line</p>
<pre><code class="lang-auto">div = x / scale
</code></pre>
<p>So I try to print the nan gradient by doing</p>
<pre><code class="lang-auto">x.requires_grad_()
print(f"x: {x.isnan().any().item(), x.isinf().any().item()}", flush=True)
x.register_hook(lambda grad: print(f"x: {grad.isnan().any().item(), grad.isinf().any().item()}", flush=True))
        
scale.requires_grad_()
print(f"scale: {scale.isnan().any().item(), scale.isinf().any().item()}", flush=True)
scale.register_hook(lambda grad: print(f"scale: {grad.isnan().any().item(), grad.isinf().any().item()}", flush=True))
        
div = x / scale

div.requires_grad_()
print(f"div: {div.isnan().any().item(), div.isinf().any().item()}", flush=True)
div.register_hook(lambda grad: print(f"div: {grad.isnan().any().item(), grad.isinf().any().item()}", flush=True))
</code></pre>
<p>But all I got was <code>False</code>. What am I doing wrong?</p>

soulitzer: <p>Could you share the full script where this is happening?</p>

thanrl: <p>I have not found the exact cause of the nans, but the question about debugging I have figured it out.</p>
<p>To see the nans printed, I should have registered the hooks <code>scale.register_hook(print)</code> without setting <code>torch.autograd.set_detect_anomaly(True)</code>. Otherwise, the anomaly detection would stop the program before the backward hook is called.</p>
How bad is it to use torch.ops.aten?

gngdb: <p>They’re not documented much aside from <a href="https://pytorch.org/docs/stable/torch.compiler_ir.html" rel="noopener nofollow ugc">this page</a>. Is it recommended to use them in practice? Specifically, if I want to replicate the functionality called in the backward pass I can find the <code>backward</code> op associated with a given forward op and call it separately to get the same result and it should be exactly the same. For example, this replicates the functionality of <code>F.embedding</code>’s backward pass (approximately):</p>
<pre><code class="lang-auto">class EmbeddingTest(torch.autograd.Function):
    # replicate the behavior of the embedding backward
    @staticmethod
    def forward(ctx, input, weight):
        ctx.save_for_backward(input, weight)
        return F.embedding(input, weight)

    @staticmethod
    def backward(ctx, grad_output):
        input, weight = ctx.saved_tensors
        grad_input = grad_weight = None
        if ctx.needs_input_grad[0]:
            raise NotImplementedError('non-differentiable in general')
        if ctx.needs_input_grad[1]:
            grad_weight = torch.ops.aten.embedding_dense_backward(grad_output, input, weight.size(0), -1, False)
        return grad_input, grad_weight
</code></pre>
<p>What are the potential pitfalls to calling <code>torch.ops.aten...</code>?</p>

soulitzer: <blockquote>
<p>What are the potential pitfalls to calling <code>torch.ops.aten...</code> ?</p>
</blockquote>
<p>There is less stability there, things can be subject to change</p>
Batch-wise Gradient Computation using autograd

Saimon: <p>Hello,<br>
My loss function includes the partial differential equations (PDE). For each batch, the input size (xy) is of torch.Size([batch, 2, points]) and output (u) is of size torch.Size([batch, 1, points]); which means the prediction is about the response at each point of the batch element. Model is taking the input as torch.Size([batch, channels, points]) and providing output torch.Size([batch, response, points]).<br>
The PDE includes d2u/dx2 and d2u/dy2. Essentially I need the derivative at each point of each element in the batch. Batch dimension is confusing in the calculation. Is there anyone with suggestions? Also, I am trying to avoid loop.</p>

soulitzer: <p>You can compose torch.func.vmap and torch.func.grad to compute per-sample gradients <a href="https://pytorch.org/docs/stable/func.whirlwind_tour.html#vmap-auto-vectorization" class="inline-onebox" rel="noopener nofollow ugc">torch.func Whirlwind Tour — PyTorch 2.2 documentation</a></p>
How to calculate gradient w.r.t the specific input element?

Porn_Jack: <p>for instance, I’ve a input tensor matrix is (4,2).<br>
I want to obtain gradient w.r.t the input element (1,1), and all of other element’s gradient is 0.<br>
because I want to know how each element changing the output independent.</p>

soulitzer: <p>You can do backprop normally, then index onto the grad, e.g. inp.grad[1, 1]</p>
How to pass batch of Graphs in GATv2Conv?

Sar2580P: <p>I have a batched input to GATv2Conv with node matrix of shape [batch_sz , num_nodes , node_feature_dim] , but the GATv2Conv accepts input of dim 2  ,searching through the internet , I found some solution … (not the one I want)</p>
<pre><code class="lang-auto">data_list = [Data(x= torch.squeeze(torch.index_select(x, dim= 0, index = torch.tensor([idx]))) , 
                              edge_index= self.edge_indices ,
                              edge_attr= torch.squeeze(torch.index_select(adj_mats[i], dim= 0, index = torch.tensor([idx])))) 
                         for idx in range(self.batch_sz)] 
            batch  = Batch.from_data_list(data_list)
</code></pre>
<p>But using above solution , the distinction between graphs got lost , becuase :<br>
<code>batch.x.shape</code> gave <code>[batch_sz * num_nodes , node_feature_dim]</code>…</p>
<p>It simply put all nodes of all graphs in one single graphs… Now there is shared calculations between different graphs , which is strictly undesirable… As when applying some graph pooling layer, I don’t know which nodes belonged to which graph…</p>
<p>Pls suggest some fix for this issue …</p>
<p>Thanks in advance</p>
Help compile pytorch from source

RedHead: <p>Attempting to compile pytorch 1.13 from source witrh CUDA 11.4 on Xavier NX</p>
<p>[40/593] Slimming NCCL<br>
FAILED: nccl/lib/libnccl_slim_static.a /mnt/sda1/pytorch/build/nccl/lib/libnccl_slim_static.a<br>
cd /mnt/sda1/pytorch/build/nccl &amp;&amp; /usr/bin/cmake -E make_directory /mnt/sda1/pytorch/build/nccl/objects &amp;&amp; cd objects &amp;&amp; /usr/bin/ar x /mnt/sda1/pytorch/build/nccl/lib/libnccl_static.a &amp;&amp; for obj in all_gather_* all_reduce_* broadcast_* reduce_*.o; do /usr/bin/objcopy --remove-relocations .nvFatBinSegment --remove-section __nv_relfatbin $obj; done &amp;&amp; /usr/bin/ar cr /mnt/sda1/pytorch/build/nccl/lib/libnccl_slim_static.a *.o &amp;&amp; /usr/bin/ar xN 1 /mnt/sda1/pytorch/build/nccl/lib/libnccl_static.a net.o &amp;&amp; /usr/bin/ar q /mnt/sda1/pytorch/build/nccl/lib/libnccl_slim_static.a net.o &amp;&amp; cd - &amp;&amp; /usr/bin/cmake -E remove_directory /mnt/sda1/pytorch/build/nccl/objects</p>
<p>/usr/bin/objcopy: stx6dsTj: symbol `__nv_relfatbin’ required but not present<br>
/usr/bin/objcopy: stx6dsTj: no symbols</p>
<p>last error repeated hundreds of times</p>
<p>Any help appreciated</p>
<p>Thanks</p>

RedHead: <p>Removed objdump part in line 228 of build.ninja</p>
<p>Obviously that particular error doesn’t show up anymore but this is a bad patch so would still like to understand …</p>
Question About Pytorch Tensor Behaviour On GPU On Apple M1 Pro Chip

Nihar_Jani: <p>If I a modify A Pytorch tensor on GPU(M1 Pro) , and store in the same variable as before, then there will be two copies of tensor, the original one and the modified one or just the modified one on THE GPU . For Example Y is a Pytorch tensor on Gpu, I modified it and store again in Y only.</p>
<p>The What will be the behaviour of Pytorch Tensor?</p>
<p>Reason Behind Asking The Question:<br>
Limited Resources (GPU And RAM).</p>
Cuda OOM When using torch.cumsum

bigtuna: <p>I am implementing a model that uses <code>x = torch.cumsum(x, dim = 2)</code> in one of the modules. As I understand it, this operation should be fast and, if done in place, have low memory requirements. However, I get the oom error below. When I try <code>torch.cumsum(x, dim = 2, out = x)</code>, there is an error since x requires grad and this operation is apparently not differentiable. Is there any way to compute torch.cumsum() inplace/without blowing up memory requirments?</p>
<pre><code class="lang-auto">File "/home/sww/wikipedia_training/mixer_model.py", line 119, in forward
    x = torch.cumsum(x, dim = 2)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.50 GiB (GPU 1; 39.44 GiB total capacity; 23.79 GiB already allocated; 6.73 GiB free; 31.29 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
Set weights of Conv layer and make them non trainable

Rituraj_19_Dutta_100: <p>I’m writing a module that includes some Conv2D layers and I want to manually set their weights and make them non-trainable. My module is something like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


def SetWeights():
    ## manual function to set weights
    return
## Returns a 4D tensor

class Module(nn.Module):
    def __init__(self):
        super().__init__()

        self.Conv1 = nn.Conv2d(128,128,kernel_size=(2,2),padding='same')
        self.Conv2 = nn.Conv2d(128,128,kernel_size=(3,3),padding='same')
        self.Conv3 = nn.Conv2d(128,128,kernel_size=(5,5),padding='same')
        self.Conv4 = nn.Conv2d(128,128,kernel_size=(7,7),padding='same')


    def forward():

        ## usual forward pass
        return
</code></pre>
<p>In Keras it’s something like:</p>
<pre><code class="lang-auto">     Conv1.set_weights([weights1])
     Conv2.set_weights([weighs2])
     Conv3.set_weights([weights3])
     Conv4.set_weights([weights4])
     Conv1.trainable = False
     Conv2.trainable = False 
     Conv3.trainable = False
     Conv4.trainable = False
</code></pre>
<p>How can I do that in Pytorch?</p>
<p>P.S. The weights are numpy tensors</p>

ptrblck: <p>You can either assign the new <code>weights</code> via:</p>
<pre><code class="lang-python">with torch.no_grad():
    self.Conv1.weight = nn.Parameter(...)
    # or
    self.Conv1.weight.copy_(tensor)
</code></pre>
<p>and set their <code>.requires_grad</code> attribute to <code>False</code> to freeze them or alternatively you could also directly use the functional API:</p>
<pre><code class="lang-python">x = F.conv2d(input, self.weight)
</code></pre>

Rituraj_19_Dutta_100: <p>Okk so if I use torch.no_grad() I need to use it inside the  <code>__init__()</code> and what about the <code>forward()</code> do we need to make any changes in that function?</p>

ptrblck: <p>If you freeze the parameters, you won’t have to change the forward and could just apply the conv layers.<br>
You could try to also use <code>no_grad()</code> in the <code>forward</code>, but since the parameters should not be trained at all, the right approach would be to just freeze them.</p>

Rituraj_19_Dutta_100: <p>Ok thanx I got my answer</p>

Abhishek_Gupta: <p>In case I want to set each and every layer non-trainable, is there any shorthand to make it or i have wite the above line for each and every layer</p>

ptrblck: <p>To freeze the entire model, you would iterate the parameters and set their <code>.requires_gradient</code> attribute to <code>False</code>.</p>
Libtorch 2.1.1 build error in linux

jayden1: <p>Hi all,<br>
I tried to include torch in my script, by:</p>
<p><span class="hashtag-raw">#include</span> &lt;torch/script.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;torch/torch.h&gt;</p>
<p>but I got the following errors when I tried to build a C++ project using libtorch:</p>
<p>In file included from /usr/local/libtorch/include/torch/script.h:3:<br>
In file included from /usr/local/libtorch/include/torch/csrc/api/include/torch/types.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/ATen.h:9:<br>
In file included from /usr/local/libtorch/include/ATen/DeviceGuard.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/core/IListRef.h:632:<br>
In file included from /usr/local/libtorch/include/ATen/core/IListRef_inl.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/core/List.h:490:<br>
In file included from /usr/local/libtorch/include/ATen/core/List_inl.h:4:<br>
In file included from /usr/local/libtorch/include/ATen/core/ivalue.h:4:<br>
/usr/local/libtorch/include/ATen/core/TensorBody.h:891:42: error: default arguments can only be specified for parameters in a function declaration<br>
at::Tensor logit(c10::optional eps=c10::nullopt) const;<br>
^   ~~~~~~~~~~~~<br>
/home/{path to another file. h}:12:24: note: expanded from macro ‘logit’<br>
<span class="hashtag-raw">#define</span> logit(x) (log((x) / (1 - (x))))<br>
^<br>
In file included from /home/{path to file3.cc}:61:<br>
In file included from /home/{path to file4.h}:3:<br>
In file included from /usr/local/libtorch/include/torch/script.h:3:<br>
In file included from /usr/local/libtorch/include/torch/csrc/api/include/torch/types.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/ATen.h:9:<br>
In file included from /usr/local/libtorch/include/ATen/DeviceGuard.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/core/IListRef.h:632:<br>
In file included from /usr/local/libtorch/include/ATen/core/IListRef_inl.h:3:<br>
In file included from /usr/local/libtorch/include/ATen/core/List.h:490:<br>
In file included from /usr/local/libtorch/include/ATen/core/List_inl.h:4:<br>
In file included from /usr/local/libtorch/include/ATen/core/ivalue.h:4:<br>
/usr/local/libtorch/include/ATen/core/TensorBody.h:891:20: error: C++ requires a type specifier for all declarations<br>
at::Tensor logit(c10::optional eps=c10::nullopt) const;<br>
^<br>
/usr/local/libtorch/include/ATen/core/TensorBody.h:891:14: error: expected ‘)’<br>
at::Tensor logit(c10::optional eps=c10::nullopt) const;</p>
<p>Versions:<br>
libtorch version: 2.1.1<br>
cuda: 12.1<br>
C++：17<br>
OS: Ubuntu 22.04.3 LTS(X86_64)</p>

jayden1: <p>This problem has been solved, a macro ‘logit’ defined in my project conflicted with the Python script.</p>
Graph Depth First Search

Ruben_Band: <p>Dear all,</p>
<p>I’m working on a depth first search algorithm using PyTorch Geometric Data objects. The idea is that I have a Minimal Spanning Tree as a (2, E) Tensor and a graph object that contains normal vectors per graph node. We sometimes flip the normal vectors in certain conditions (This is part of an algorithm to construct normal vectors for a pointcloud). I don’t like that the algorithm contains a loop and recursion call, because this is slow in python… Is there a way I can speed this up? Code below and thank you for reading!</p>
<pre><code class="lang-auto">@classmethod
    def flipNormals(cls, graph: tg_data_Data, mst_edge_index: torch_Tensor) -&gt; None:
        r"""
        Flips normals in the graph based on the MST.

        Args:
            graph (Torch Geometric Data): Graph that contains normals that need to be flipped.
            mst (Torch Tensor (E,)): Indices that point to the edges that are included in the Minimal Spanning Tree.
        """

        _n = graph.n
        def dfs_from_node(src, visited):
            visited[src] = True

            # Get neighbors of the current node
            neighbors = mst_edge_index[1, mst_edge_index[0] == src]

            # Recursive DFS on unvisited neighbors
            for dest in neighbors:
                if not visited[dest]:
                    if (_n[src] * _n[dest]).sum(dim=0) &lt; 0:
                        _n[dest] *= -1
                    dfs_from_node(dest, visited)

        N = graph.num_nodes
        visited = torch_zeros(N, dtype=bool)
        start_node = torch_argmax(graph.pos[:, 2])
        if _n[start_node, 2] &lt; 0:
            _n[start_node] *= -1
        dfs_from_node(start_node, visited)
</code></pre>
Selecting values in a multi dimension tensor using indices

anukr95: <p>I have a matrix A of shape (batch, dim1, dim2, K). K is the number of classes. I have another matrix B which contains the indices of the most probable values of A. This B is not derived by using argmax on A.</p>
<p>How can I collect matrix A’s values from matrix B’s indices?  Let this matrix be C. I hope C’s shape will be the same as that of B.</p>
I3D and X3D_XS inference speed comparison

LewsTherin: <p>I’ve been testing the I3D and X3D_XS models from PytorchVideo to classify short video sequences.</p>
<p><a href="https://pytorchvideo.readthedocs.io/en/latest/model_zoo.html" rel="noopener nofollow ugc">This table</a> and a manual inspection of the models show that X3D_XS has about 1/10 of the parameters of I3D (3M against 30M).<br>
Based on this, I was expecting X3D_XS to have a much higher inference speed than I3D, also considering that X3D_XS accepts sequences with a minimum of 4 frames, whereas I3D only works if the sequence length is &gt;= 8 frames. However, I found the opposite, i.e. I3D seems to run inference faster.</p>
<p>Here is an example of what I’ve been doing:</p>
<pre><code class="lang-auto">import numpy as np
import torch


def time_model(model, num_frames):

    times = []
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)

    # dummy input data
    input_data = \
        (torch.tensor(np.zeros([4, 3, num_frames, 224, 224],dtype=np.float32)).half().to(device))

    for _ in range(100):
        start.record()
        _ = model(input_data)
        end.record()
        torch.cuda.synchronize()
        times.append(start.elapsed_time(end))

    # check the mean inference time, remove first entries (no model warm-up)
    return np.mean(times[10:])


def main():

    i3d_model = (
        torch.hub.load("facebookresearch/pytorchvideo", "i3d_r50", pretrained=True).half().to(device))
    x3d_xs_model = (
        torch.hub.load("facebookresearch/pytorchvideo", "x3d_xs", pretrained=True).half().to(device))

    # timing I3D with 4 sequences of 8 frames
    avg_time_i3d = time_model(i3d_model, 8)
    # timing X3D_XS with 4 sequences of 5 frames
    avg_time_x3dxs = time_model(x3d_xs_model, 5)

    print(f"I3D: {avg_time_i3d}")
    print(f"X3D_XS: {avg_time_x3dxs}")


if __name__ == "__main__":

    if torch.cuda.is_available():
        device = "cuda:0"
    else:
        device = "cpu"
    device = torch.device(device)

    main()
</code></pre>
<p>In my tests (using a rtx_8000 GPU), I see that x3d is even slightly slower than I3D:</p>
<pre><code class="lang-auto">I3D: 21.59845754835341
X3D_XS: 23.11600530412462
</code></pre>
<p>Am I doing something wrong, or are these results reasonable? Was my initial assumption on the expected inference speed for the models wrong?</p>
How to find K-nearest neighbor of a tensor

Saifullahi_Aminu: <p>Hi. I have a tensor of shape (Batch, 40, 128, 3). How can I find the k nearest neighbor of a given constant data point that is 3-dimension, so that I get a tensor of shape (Batch, 40, k, 3)? Thanks</p>

Marco_Piccoli: <p>It’s a bit late but since I went throw the same problem here’s the answer <img src="https://discuss.pytorch.org/images/emoji/apple/tipping_hand_man/3.png?v=9" title=":tipping_hand_man:t3:" class="emoji" alt=":tipping_hand_man:t3:"></p>
<pre><code class="lang-auto">def kNN(cloud, center, k):
    center = center.expand(cloud.shape)
    
    # Computing euclidean distance
    dist = cloud.add( - center).pow(2).sum(dim=3).pow(.5)
    
    # Getting the k nearest points
    knn_indices = dist.topk(k, largest=False, sorted=False)[1]
    
    return cloud.gather(2, knn_indices.unsqueeze(-1).repeat(1,1,1,3))
</code></pre>
<p>where cloud is a 4 dimension tensor and center is a tensor with the three coordinates x-y-z.</p>
<p>The function is based on a reply by <a class="mention" href="/u/ptrblck">@ptrblck</a> in  <a href="https://discuss.pytorch.org/t/k-nearest-neighbor-in-pytorch/59695/2" class="inline-onebox">K nearest neighbor in pytorch</a></p>

JosueCom: <p>Late as well haha but for anyone who might still be looking, I implemented NN, KNN and KMeans on a project I am working on only using PyTorch. You can find the implementation here with an example: <a href="https://gist.github.com/JosueCom/7e89afc7f30761022d7747a501260fe3" class="inline-onebox" rel="noopener nofollow ugc">Nearest Neighbor, K Nearest Neighbor and K Means (NN, KNN, KMeans) only using PyTorch · GitHub</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch as th
&gt;&gt;&gt; from clustering import KNN
&gt;&gt;&gt; data = th.Tensor([[1, 1], [0.88, 0.90], [-1, -1], [-1, -0.88]])
&gt;&gt;&gt; labels = th.LongTensor([3, 3, 5, 5])
&gt;&gt;&gt; test = th.Tensor([[-0.5, -0.5], [0.88, 0.88]])
&gt;&gt;&gt; knn = KNN(data, labels)
&gt;&gt;&gt; knn(test)
tensor([5, 3])
</code></pre>

JosueCom: <p>This is an updated link for the project (<a href="https://gist.github.com/wzjoriv/7e89afc7f30761022d7747a501260fe3" rel="noopener nofollow ugc">Nearest Neighbor, K Nearest Neighbor and K Means (NN, KNN, KMeans) implemented only using PyTorch · GitHub </a>). It is under a new username.</p>
Where and what are the design rules for optimal CNNs

NZ1: <p>Being new to Pytorch, I’ve run a few CNNs on my multichannel time series data, but I’m keen to understand how the design rules for an optimal classification CNN are related to the input signal characteristics. After reading books (by eg Goodfellow, Brunton, Geron, Chollet etc) and appreciating the wide variety of CNNs from the various *Net competitions, I’m amazed there’s so little on the internet and in books about the design rules, eg. what decides the optimal depth and width of a CNN. Might there be any useful texts that could help me in this area. NZ</p>
Add a custom loss in a public github repository (distillation)

Giovanni_Cena: <p>Hi,</p>
<p>I’m trying to implement a custom loss in a public repository regarding knowledge distillation. The link to the repository is the following:</p>
<p>" <a href="https://github.com/DefangChen/SimKD" class="inline-onebox" rel="noopener nofollow ugc">GitHub - DefangChen/SimKD: [CVPR-2022] Official implementation for "Knowledge Distillation with the Reused Teacher Classifier".</a> "</p>
<p>The main issue is regarding how to add the new loss because simply adding it to the actual loss (/helper/loops.py) doesn’t produce any change in the output (I’ve already made an issue in the git repo)</p>
<p>In simpler terms, without looking into the repository, do you have any advice on how should be done this in Pytorch? The reasoning is the following:</p>
<blockquote>
<p>Criterion = Nn.MSEloss()</p>
<p>Loss_custom = criterion( torch.tensor(map1, requires_grad=True), torch.tensor(map2, requires_grad=True)).cuda()</p>
</blockquote>
<p>map1 and map2 are numpy.ndarray()</p>
<p>and then,</p>
<blockquote>
<p>Total_loss = weight_1 * Main_Loss + weight_2 * Loss_custom</p>
</blockquote>
<p>If I set the weight_2 to zero or a very high number the final result is the same.<br>
The implementation should use something in particular? (e.g., nn.Parameter, a specific grad function or this like that)</p>
<p>Thanks in advance</p>

ptrblck: <aside class="quote no-group" data-username="Giovanni_Cena" data-post="1" data-topic="198504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giovanni_cena/48/16672_2.png" class="avatar"> Giovanni_Cena:</div>
<blockquote>
<p>doesn’t produce any change in the output (I’ve already made an issue in the git repo)</p>
</blockquote>
</aside>
<p>This is expected, since <code>map1</code> and <code>map2</code> are numpy arrays and thus not tracked by Autograd. Creating new <code>tensor</code>s via <code>torch.tensor(map1, requires_grad=True)</code> creates new leaf-variables without any gradient history.</p>
<p>You would need to compute <code>map1</code> and <code>map2</code> via differentiable operations in PyTorch. If you need to use a 3rd party library, such as <code>numpy</code>, you would need to implement a custom <code>autograd.Function</code> including the <code>backward</code> pass as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Causal CNN explanation

Utkarsh_tiwari: <p>Hi i want to understand how does Causal work for CNN , i know padding the sequence would introduce causality to the network but i am unclear how does it exactly work as padding would mean to pad the whole sequence and when stride happens it would still be the same set of pad.</p>
<p>assume the input to be ( 1,1,8,16,16) where 18 is the number of frames of video and 16 is H and W.</p>
<p>could anybody explain what should be done for causality in conv2d and conv3d both</p>
<p>Thanks</p>
Grad is None when `requires_grad=True`, but only for some epochs

Itamar_Katz: <p>I am logging the norm of the gradient of my loss with respect to one of the inputs <code>Z</code> (the latent random vector in a GAN-like architecture). To this end I create the tensor <code>Z</code> using <code>requires_grad = True</code>, and after the <code>backward</code> pass I accumulate the norm, for each batch in each epoch. This works fine for about 70 epochs, but then for some batches I start getting a <code>None</code> gradient in <code>Z.grad</code>, and as the training continues, the number of such batches grows, for example when the problem starts it happens in 1 batch out of 50, and after 200 epochs, it is around 25 batches out of 50.</p>
<p>I read these related topics:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/after-loss-backward-requires-grad-true-no-gradient-can-be-found-on-some-variables/43627" class="inline-onebox">After loss.backward(requires_grad=True), no gradient can be found on some variables</a></li>
<li><a href="https://discuss.pytorch.org/t/grad-is-none-even-when-requires-grad-true/29826" class="inline-onebox">Grad is None even when requires_grad=True</a></li>
<li><a href="https://discuss.pytorch.org/t/none-type-return-while-trying-to-find-gradient-of-models-parameters/160786" class="inline-onebox">None type return while trying to find gradient of model's parameters</a></li>
</ul>
<p>but my problem seems different.<br>
I am using Python 3.9.12, Pytorch 2.0.1 (for cuda 11.8), and Ubuntu.<br>
Any ideas?</p>

Itamar_Katz: <p>Ok, I solved my own problem, writing it here in case it is relevant to someone:<br>
The discriminator block has a condition on the loss, and it performs the <code>backward</code> step only if the loss is higher than some threshold. As the training continues and the discriminator is getting better, the number of cases where this loss is small enough, also grows. When the <code>backward</code> step is not happening, the gradient is not computed.<br>
That is easily solved by accumulating the gradient before the step that optimizes the discriminator.</p>
Having trouble plotting labels in draw_bounding_boxes

martinfloo: <p>Hi!</p>
<p>I’m having some trouble plotting the labels with my bounding boxes. When i plot the label with corresponding to the image, I get the error as seen in the picture below. I have tried specifying a font, but with no better result. Ive plotted the tensors that I use in the <code>draw_bounding_boxes(img, boxes)</code></p>
<p>First time posting here, so please direct me if this is the wrong place to post. Thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba" title="Skjermbilde 2024-03-08 kl. 18.23.19"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba_2_523x500.png" alt="Skjermbilde 2024-03-08 kl. 18.23.19" data-base62-sha1="seMaePohSd9VBecp3DYN086q0b0" width="523" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba_2_523x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba_2_784x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e87cf4811af6a8dc7aabd01b30e7eabf5d08ba_2_1046x1000.png 2x" data-dominant-color="5E595B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Skjermbilde 2024-03-08 kl. 18.23.19</span><span class="informations">1592×1520 300 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Find winning unit between 2 torch tensors of different shapes

grid_world: <p>I am trying to implement a Self-Organizing Map where for a given input sample, the best matching unit/winning unit is chosen based on (say) L2-norm distance between the SOM and the input. To implement this, I have:</p>
<pre><code># Input batch: batch-size = 512, input-dim = 84-
z = torch.randn(512, 84)

# SOM shape: (height, width, input-dim)-
som = torch.randn(40, 40, 84)

# Compute L2 distance for a single sample out of 512 samples-
dist_l2 = np.linalg.norm((som.numpy() - z[0].numpy()), ord = 2, axis = 2)

# dist_l2.shape
# (40, 40)

# Get (row, column) index of the minimum of a 2d np array-
row, col = np.unravel_index(dist_l2.argmin(), dist_l2.shape)

print(f"BMU for z[0]; row = {row}, col  = {col}")
# BMU for z[0]; row = 3, col  = 9
</code></pre>
<p>So for the first input sample of ‘z’, the winning unit in SOM has the index: (3, 9). I can put this in a for loop iterating over all 512 such input samples, but that is very inefficient.</p>
<p>Is there an efficient vectorized PyTorch manner to compute this for the entire batch?</p>

ptrblck: <p>This should work:</p>
<pre data-code-wrap="python"><code class="lang-python"># Input batch: batch-size = 512, input-dim = 84-
z = torch.randn(512, 84)

# SOM shape: (height, width, input-dim)-
som = torch.randn(40, 40, 84)

# Compute L2 distance for a single sample out of 512 samples-
dist_l2 = np.linalg.norm((som.numpy() - z[0].numpy()), ord = 2, axis = 2)
out = np.linalg.norm(som.numpy()[:, :, None, :] - z[None, None, :, :].numpy(), ord=2, axis=3)

print((out[:, :, 0] == dist_l2).all())
# True

# Using PyTorch
out = torch.linalg.norm(som[:, :, None, :] - z[None, None, :, :], ord=2, dim=3)
print((out[:, :, 0] - torch.from_numpy(dist_l2)).abs().max())
# tensor(1.9073e-06)
</code></pre>
Linear behave differently with 2D and 3D input

XinJiade: <p>Hi guys,<br>
I run the following code on torch 1.13.1, but got the inequal results. Why Linear behave differently with 2D and 3D input?</p>
<pre><code class="lang-auto">a = torch.randn(3, 4, 5)
l1 = torch.nn.Linear(5, 7, bias=True)
torch.equal(l1(a)[0, 0, :], l1(a[0, 0, :]))    # False
torch.equal(l1(a)[0:2, 0:4, :], l1(a[0:2, 0:4, :]))   # True
torch.equal(l1(a)[0:2, 0:3, :], l1(a[0:2, 0:3, :]))   # False
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/xinjiade">@XinJiade</a>,</p>
<p>The difference is most likely due to floating point precision, which for <code>torch.float32</code> precision is around <code>1e-7</code>. Although the operation are mathematically identical, their floating point operation will differ.</p>
<pre><code class="lang-auto">a = torch.randn(3, 4, 5)
l1 = torch.nn.Linear(5, 7, bias=True)
torch.allclose(l1(a)[0, 0, :], l1(a[0, 0, :])) #True
torch.allclose(l1(a)[0:2, 0:4, :], l1(a[0:2, 0:4, :])) #True
torch.allclose(l1(a)[0:2, 0:3, :], l1(a[0:2, 0:3, :])) #True
</code></pre>
<p>You can print out the difference between the terms and see that they have elements which differ by <code>1e-8</code>, when checking the difference between 2 tensors use <code>torch.allclose</code> rather than <code>torch.equal</code></p>

XinJiade: <p>Thank you for your response.<br>
In my situation, these differences will be acummulated as the layers go deeper. I wonder why the operation orders of 2D and 3D linear are different? Are there any documents about the operation orders of 2d and 3d linear?</p>

J_Johnson: <p>The bulk of machine learning is a minimization problem. Thus small discrepancies in rounding can act as a form of regularization(albeit, minor) - so it can help prevent overfitting. For this reason, it’s better to use float16 or bfloat16.</p>

ptrblck: <aside class="quote no-group" data-username="XinJiade" data-post="3" data-topic="197997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xinjiade/48/33175_2.png" class="avatar"> XinJiade:</div>
<blockquote>
<p>n my situation, these differences will be acummulated as the layers go deeper.</p>
</blockquote>
</aside>
<p>This is expected and neither result is “more correct” in such a case and should show a similar error to a wider dtype. If these expected numerical errors, caused by the limited floating point precision, are causing issues for your use case, you might want to use e.g. <code>float64</code>.</p>