akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to concat two sequential()?

username: zzuczy
cooked: <p>Suppose I define two sequential():</p>
<pre><code class="lang-auto">a = nn.Sequential(
...
)

b = nn.Sequential(
...
)
</code></pre>
<p>Now I want to define another Sequential() that can concat the two Sequential() a and b above. What am I supposed to do?</p>

username: tom
cooked: <p>If you don’t want to stack sequentials, you could do</p>
<pre><code class="lang-auto">s_cat = torch.nn.Sequential(*(list(s1)+list(s2)))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: zzuczy
cooked: <p>Thanks a lot!<br>
BTW, what should I do if want to stack sequentials?</p>

username: tom
cooked: <p><code>s_stacked = torch.nn.Sequential(s1, s2)</code></p>

username: zzuczy
cooked: <p>Oh, sorry, I meant the wrong thing. Let me ask you another way. Suppose the output dim of <code>s1</code> is [a, b, c] and the output dim of<code>s2</code> is [a, b, c] too. Then what is the output dim of  <code>s_cat</code> by</p>
<pre><code class="lang-auto">s_cat = torch.nn.Sequential(*(list(s1)+list(s2)))
</code></pre>

username: tom
cooked: <p>Oh, you want the tensors concatenated?<br>
I think just running the two and then concatenating the result is the best option.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zzuczy
cooked: <p>Thank you very much!</p>

username: Mugen
cooked: <p>Hi Thomas,</p>
<p>What is the difference between stacking vs concatenating nn.Sequentials?</p>

username: tom
cooked: <p>Maybe this is not the best terminology here.<br>
You can run Sequential and concatenate/stack the resulting tensors.<br>
You can also make a combined sequential of two of them. If you come from viewing them as functions, you might call that composing the two, or if you come from viewing them as lists, you might say concatenate.</p>
<p>I would probably recommend to think about what you want to do and then see how this might be achieved.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Mugen
cooked: <p>Thank you very much for your detailed explanation.</p>

username: epistoteles
cooked: <p>I know this thread is old, but just for reference (and because it tends to pop up again), I want to mention that you can also do</p>
<p><code>a.extend(b)</code></p>
<p>which modifies a (maybe this is okay for you), but is much simpler syntax.</p>
<p>Alternatively, you could do:</p>
<p><code>c = torch.nn.Sequential().extend(a).extend(b)</code></p>
<p>which does not modify a.</p>
How to install pytorch CPU in requirements.txt?

username: kgblitz
cooked: <p>We can install pytorch’s CPU version via <code>pip install torch==2.1.1 --index-url https://download.pytorch.org/whl/cpu</code>.</p>
<p>How can I add this to requirements.txt so I can install CPU version of it using <code>pip install -r requirements.txt</code>?</p>

username: tom
cooked: <p>This is more Python than PyTorch, but you can either use --index-url (but this is global, so a bit tricky for requirements.txt) or give specific packages (whl archives) with <code>@</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>Here are the details:<br>
<a href="https://pip.pypa.io/en/stable/reference/requirements-file-format/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pip.pypa.io/en/stable/reference/requirements-file-format/</a></p>

username: kgblitz
cooked: <p>Thanks a lot, <code>torch @ </code> format worked well!</p>
PyTorch unable to fully utilize cpu when performing SVD

username: Steve-Stewart
cooked: <p>When performing SVD of a large matrix, I find that PyTorch is unable to utilize all cores of the CPU of my machine (Apple M1 Pro, 6+2 CPU cores. I am using PyTorch 2.1.1).</p>
<pre><code class="lang-python">import torch
torch.set_default_dtype(torch.float64)

a = torch.rand((1024, 1024), dtype=torch.complex128)
u, s, v = torch.linalg.svd(a, full_matrices=False)
</code></pre>
<p>PyTorch can only use about 230% CPU, and <code>%timeit</code> reports 1.3s execution time of the SVD. However, if I run similar code in NumPy, the CPU usage can go up to about 550% (but curiously the execution time is about the same as PyTorch).</p>
<pre><code class="lang-python">import numpy as np

a = np.random.rand(1024, 1024) + 1.0j * np.random.rand(1024, 1024)
u, s, v = np.linalg.svd(a, full_matrices=False)
</code></pre>
<p>Julia can also use all cores. <code>@btime</code> with <code>BenchmarkTools</code> report 1.04s of execution time.</p>
<pre data-code-wrap="julia"><code class="lang-plaintext">using LinearAlgebra

a = rand(ComplexF64, (1024, 1024))
u, s, v = svd(a)
</code></pre>
<p>How to make PyTorch utilize all cores and possibly speed up the SVD process?</p>
Unable to call c++ op which has a `int64_t` parameter

username: nahso
cooked: <p>I created a c++ function and registered it as follow:</p>
<pre><code class="lang-auto">void to_csr_index(torch::Tensor &amp;row_ptr, const torch::Tensor &amp;edge_index_i, int64_t num_rows) {
    auto *prow_ptr = row_ptr.data_ptr&lt;int32_t&gt;();
    for (uint32_t i = 0; i &lt; edge_index_i.size(0); ++i) {
        prow_ptr[edge_index_i[i].item&lt;int32_t&gt;() + 1]++;
    }
    for (int i = 0; i &lt; num_rows; i++) {
        prow_ptr[i + 1] += prow_ptr[i];
    }
}

TORCH_LIBRARY(my_ops, m) {
    m.def("to_csr_index", to_csr_index);
}
</code></pre>
<p>Then I called it with:</p>
<pre><code class="lang-auto">num_nodes = data[AtomicDataDict.NODE_FEATURES_KEY].shape[0]
row_ptr = torch.zeros(num_nodes, dtype=edge_index_i.dtype)
torch.ops.my_ops.to_csr_index(row_ptr, edge_index_i, num_nodes)
</code></pre>
<p>Here is the error message:</p>
<pre><code class="lang-auto">  File "/home/dym/code/torch-ff/Torch-FF/src/models/components/encoder/_edge.py", line 71, in to_csr
    torch.ops.my_ops.to_csr_index(row_ptr, edge_index_i, num_nodes)
  File "/home/dym/.conda/envs/torch2.1.0/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: expected scalar type Int but found Long

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
</code></pre>
<p>How to fix it?</p>

username: tom
cooked: <p>The problem is not that the int64_t parameter, but that you pass a int64 tensor <code>row_ptr</code> and/or <code>edge_index_i</code> and then access it with the incompatible <code>.data_ptr&lt;int32_t&gt;()</code> and <code>.item&lt;int32_t&gt;()</code>.<br>
Either cast the tensors to torch.int32 aka torch.int or use int64_t in the template parameter.</p>
<p>Best regards</p>
<p>Thomas</p>

username: nahso
cooked: <p>Thank you! it works!</p>
Model parameters and MACs

username: torch2.0
cooked: <p>Dear All,</p>
<pre><code>        I had trained and saved a model with its parameters by default in float32. Upon loading the saved model and doing model.half(), I was able to reduce the model precision to float16 (half). But the number of parameters and MACs obtained using the profile(model, inputs = (,)) from thop framework remains the same for both float32 and float16. Kindly, explain this. Is my understanding of quantization wrong?
</code></pre>

username: ptrblck
cooked: <p>Transforming the <code>dtype</code> will not change the needed operations and thus the reporting sounds reasonable. Internal algorithms could change of course.</p>

username: torch2.0
cooked: <p>Thanks for the response, sir. Can you please provide a possible direction towards cutting down MACs and PARAMs?</p>

username: ptrblck
cooked: <p>You could run experiments by reducing the actual layer size and check how it would affect the accuracy of your model.</p>
Pairwise KL divergence of a batch of Multivariate Gaussian Distributions

username: Elita_Lobo
cooked: <p>Hi,<br>
Can someone please suggest an efficient way to calculate pairwise KL divergence of a batch of multivariate gaussian distributions?</p>
<p>Thanks,<br>
Elita</p>

username: guzy0324
cooked: <p>Hi! Have you figured out how to calculate it?</p>

username: guzy0324
cooked: <p><a href="https://discuss.pytorch.org/t/how-to-efficiently-compute-a-pairwise-kl-divergence-matrix-of-a-batch-of-gaussian-distributions/192519/2">This</a> should be the solution.</p>
Training loss constant with variable input size and few samples per class

username: ml_noob
cooked: <p>I have a CRNN based model that I am training with audio data. My model works well with AudioSet, but now I am training it with a private dataset that has only 87 samples (about 10 samples per class) and variable input length. The audio files’ length varies between 30 seconds and 10 minutes. Since I got a collate_fn error during stacking of the input in the batch, I am training with batches of size 1. However I see that the training loss drops in the first epoch and then stays constant or slightly goes up and down in the next epochs. I trained it for 100 epochs, but the trend remains the same. The validation loss is also staying constant to 0.6. I am new in deep learning, so I am not quite clear why is this happening and how can I address this issue.</p>
How do I place torch.fx.Node on different gpu

username: jrt-20
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/106a19bb2e4194957b17e5ab081a5f90105f2f2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/106a19bb2e4194957b17e5ab081a5f90105f2f2e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/106a19bb2e4194957b17e5ab081a5f90105f2f2e.png" alt="image" data-base62-sha1="2lcXrJegOighZ2RznXw8i0lEllQ" width="690" height="236" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1740×597 29.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I use torch.fx to trace nn.Module, then I can the node of graph, how do i place this node on different gpu</p>
Modify gradients of an FSDP model

username: ekurtic
cooked: <p>Hi everyone,<br>
I have an FSDP model which has zeros in some of the <code>torch.nn.Linear.weight</code> parameters. During the training I would like to keep those parameters fixed to zeros, and to zero-out their gradients as well. The specific use-case is: I am loading a pruned model and I want to fine-tune it with FSDP while keeping the pruning mask fixed.</p>
<p>To achieve this I need to do two things:</p>
<ol>
<li>multiply parameters with the mask before the forward pass (so that all pruned weights remain pruned),</li>
<li>multiply gradients of pruned parameters after the backward pass (so that gradients of pruned weights are zeros)</li>
</ol>
<p>In the standard DDP training I would achieve this by:</p>
<ol>
<li>registering forward pre-hook on <code>torch.nn.Linear</code> modules and multiplying weights with the mask before each forward pass,</li>
<li>registering a hook on the parameter <code>torch.nn.Linear.weight</code> and multiplying its gradient with the mask.</li>
</ol>
<p>For example:</p>
<pre><code class="lang-python">def keep_param_pruned(mask, module, input):
    with torch.no_grad():
        module.weight.data.mul_(mask.to(module.weight.device))

def keep_grad_pruned(mask, grad):
    return grad.mul_(mask.to(grad.device))

for n, m in model.named_modules():
    if isinstance(m, torch.nn.Linear):
        mask = m.weight &gt; threshold
        m.register_forward_pre_hook(partial(keep_param_pruned, mask))
        m.weight.register_hook(partial(keep_grad_pruned, mask))
</code></pre>
<p>However, I am struggling to modify this idea to work with FSDP. For forward pass, it seems like my hook is being called but unfortunately the hook for gradients is not and I can’t zero them out after the backward pass. Any suggestions/ideas on what I am doing wrong or if there is a simpler way to achieve this without playing with hooks?</p>

username: 111357
cooked: <p>Hi <a class="mention" href="/u/h-huang">@H-Huang</a>, I have the same question, do you have any comments about it?</p>

username: zyh3826
cooked: <p>Hi <a class="mention" href="/u/ekurtic">@ekurtic</a> did you find the solution? I also want manipulate the model parameters during training, but can’t found the right way</p>
Error in forward() Expected a value of type 'Tuple[Tensor, Tensor]'

username: iGian
cooked: <p>Hello,<br>
trying to test libtorch with Qt following <a href="https://discuss.pytorch.org/t/how-to-convert-an-opencv-image-into-libtorch-tensor/90818/2">this useful answer from PyTorch forum</a>, but when it comes to forward I’m catching the following exception:</p>
<blockquote>
<p><code>forward() Expected a value of type 'Tuple[Tensor, Tensor]' for argument 'inputs' but instead found type 'Tensor'.</code><br>
<code>Position: 1</code><br>
<code>Declaration: forward(__torch__.detectron2.export.caffe2_modeling.Caffe2RetinaNet self, (Tensor, Tensor) inputs) -&gt; ((Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor))</code></p>
</blockquote>
<ul>
<li>The loaded model is a detectron2 exported via <code>caffe_tracer.export_torchscript()</code>.</li>
<li><a href="https://detectron2.readthedocs.io/en/latest/modules/export.html#detectron2.export.Caffe2Tracer.__init__" rel="noopener nofollow ugc">Caffe2Tracer input</a> was just an image of <code>torch.Size([3, 128, 80])</code></li>
<li>The input tensor used to forward the network has shape<code> [3, 128, 80]</code>, matching the tensor using to export the model.</li>
<li>The input is then unqueezed to shape <code>[1, 3, 128, 80]</code></li>
</ul>
<p><strong>My question:</strong><br>
How can I make a proper input tensor (<code>Tuple[Tensor, Tensor]</code>) from my image?</p>
<p>Thanks.</p>

username: FenglongSong
cooked: <p>I’m also struggling with this problem. Have you found a solution? I will appreciate any suggestions.</p>

username: lastlaugher
cooked: <p>You need to use <code>toTuple()</code> with the result of <code>forward</code> instead of <code>toTensor()</code>. And then use <code>toTensor()</code> with each tuple’s element. Below is sample code.</p>
<pre><code class="lang-auto">auto tuple = model.forward(input_to_net).toTuple();
auto output = tuple-&gt;elements()[0].toTensor();
</code></pre>
Detectron2 Installation on windows issue

username: manju_kannan
cooked: <p>Hello,<br>
I am trying to install Detector2 on my Windows 11 machine. I am facing below error can you guide me to solve this issue.</p>
<p>These are the libraries installed in virtual environment :<br>
anyio==3.6.1<br>
attrs==21.4.0<br>
click==8.1.3<br>
colorama==0.4.5<br>
cycler==0.11.0<br>
Cython==0.29.30<br>
fastapi==0.78.0<br>
fonttools==4.33.3<br>
h11==0.13.0<br>
idna==3.3<br>
importlib-resources==5.8.0<br>
jsonschema==4.6.0<br>
kiwisolver==1.4.3<br>
matplotlib==3.5.2<br>
numpy==1.23.0<br>
packaging==21.3<br>
Pillow==9.1.1<br>
pycocotools-windows==2.0.0.2<br>
pydantic==1.9.1<br>
pyparsing==3.0.9<br>
pyproject-toml==0.0.10<br>
pyrsistent==0.18.1<br>
python-dateutil==2.8.2<br>
six==1.16.0<br>
sniffio==1.2.0<br>
starlette==0.19.1<br>
toml==0.10.2<br>
torch==1.8.1+cpu<br>
torchaudio==0.8.1<br>
torchvision==0.9.1+cpu<br>
typing_extensions==4.2.0<br>
uvicorn==0.18.1<br>
zipp==3.8.0</p>
<p>I have already cloned detectron2 git repo.</p>
<p>I am using below command to install detectron2.<br>
python -m pip install -e detectron2</p>
<p><strong>Error</strong>:</p>
<p>error: subprocess-exited-with-error</p>
<p>× Building wheel for pycocotools (pyproject.toml) did not run successfully.<br>
│ exit code: 1<br>
╰─&gt; [14 lines of output]<br>
running bdist_wheel<br>
running build<br>
running build_py<br>
creating build<br>
creating build\lib.win-amd64-cpython-38<br>
creating build\lib.win-amd64-cpython-38\pycocotools<br>
copying pycocotools\coco.py → build\lib.win-amd64-cpython-38\pycocotools<br>
copying pycocotools\cocoeval.py → build\lib.win-amd64-cpython-38\pycocotools<br>
copying pycocotools\mask.py → build\lib.win-amd64-cpython-38\pycocotools<br>
copying pycocotools_<em>init</em>_.py → build\lib.win-amd64-cpython-38\pycocotools<br>
running build_ext<br>
skipping ‘pycocotools_mask.c’ Cython extension (up-to-date)<br>
building ‘pycocotools._mask’ extension<br>
error: Microsoft Visual C++ 14.0 or greater is required. Get it with “Microsoft C++ Build Tools”: <a href="https://visualstudio.microsoft.com/visual-cpp-build-tools/" class="inline-onebox" rel="noopener nofollow ugc">Microsoft C++ Build Tools - Visual Studio</a><br>
[end of output]</p>
<p>note: This error originates from a subprocess, and is likely not a problem with pip.<br>
ERROR: Failed building wheel for pycocotools<br>
Failed to build pycocotools<br>
ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>Your toolchain might be too old:</p>
<pre><code class="lang-auto">error: Microsoft Visual C++ 14.0 or greater is required.
</code></pre>

username: hasanradi93
cooked: <p><a class="mention" href="/u/manju_kannan">@manju_kannan</a>  are you solved bro?<br>
if you solved can you tell me how?<br>
Thanks</p>
Torch.jit.trace vs torch.fx.symbolic_trace

username: Ars_ML
cooked: <p>What is the difference between these tracers? Does it mean that torch.jit.trace is “non-symbolic” in any sense?</p>

username: tom
cooked: <p>No. One (torch.jit.trace) produces a (legacy) TorchScript representation while <code>torch.fx.symbolic_trace</code> gives you a shiny new FX one. (But if you ask, maybe torch.compile is the function you actually want…)</p>
<p>Best regards</p>
<p>Thomas</p>
Is it possible to replace gradients with stale ones to mitigate communications

username: huahuanZ
cooked: <p>Here’s my thoughts:</p>
<ol>
<li>Inter-node communication is usually the bottleneck for multi-nodes DDP;</li>
<li>For small bandwidth inter-node connection, the gradient all reducing operation could take even longer than forward pass (in extreme case);</li>
<li>pytorch currently has some optimization on overlapping some gradient all reducing ops and backward computation.</li>
</ol>
<p>And I wonder, if we use grad_t-1 (which was calculated after last step) instead of grad_t to update parameters, during forward pass of step t, the all reducing for grad_t-1 could run in parallel, so that we’re able to overlap the communications with both backward and (next step) forward passes, and mitigate the communication issue.</p>
<p>What I tried so far: add a custom hook via register_comm_hook(), where I impl a “queue” of gradient buckets, something like this</p>
<pre><code class="lang-auto">def custom_hook(state, gradbucket):
    # a torch.futures.Future object with .value() = previous gradients after all reduced.
    last_fut = state.pull(gradbucket.index())
    state.push(gradbucket)
    return last_fut
</code></pre>
<p>But it didn’t work, no any time benefits observed in my experiment. And since everything about grad reducer are wrapped as blackbox and in lack of documentation, it’s difficult to do debugging.</p>
<p>Any help or suggestion will be appreciated.</p>
How to make Cross Entropy Loss work with Cutmix & Mixup?

username: sandrodand
cooked: <p>Hi everyone,<br>
I’m trying to follow the steps of the official tutorial on how to implement both cutmix and mixup during training to perform augmentation but when I start training i get this runtime error from the criterion call.</p>
<p><code>0D or 1D target tensor expected, multi-target not supported</code></p>
<p>The following is my training code</p>
<pre><code class="lang-auto">    cutmix = v2.CutMix(num_classes=NUM_CLASSES)
    mixup = v2.MixUp(num_classes=NUM_CLASSES)
    cutmix_or_mixup = v2.RandomChoice([cutmix, mixup])
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=args.learning_rate)    

    for epoch in range(args.epochs):
        train_losses = [] 
        train_acc = 0.0
        total=0
        print(f"[Epoch {epoch+1} / {args.epochs}]")
        
        model.train()
        pbar = tqdm(data_loader)
        for i, (x, y) in enumerate(pbar):
            image = x.to(args.device)
            label = y.to(args.device) 
            image, label = cutmix_or_mixup(image, label)
            optimizer.zero_grad()

            output = model(image)
            label = label.squeeze()
            
            loss = criterion(output, label)
            loss.backward()
            optimizer.step()

            train_losses.append(loss.item())
            total += label.size(0)

            train_acc += acc(output, label)

        epoch_train_loss = np.mean(train_losses)
        epoch_train_acc = train_acc/total

        print(f'Epoch {epoch+1}') 
        print(f'train_loss : {epoch_train_loss}')
        print('train_accuracy : {:.3f}'.format(epoch_train_acc*100))
</code></pre>
<p>What am I missing? The tutorial says that I can pass the transformed labels as-is to a loss function like cross entropy.<br>
Tutorial: <a href="https://pytorch.org/vision/master/auto_examples/transforms/plot_cutmix_mixup.html" rel="noopener nofollow ugc">link</a></p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>If you are passing one-hot encoded labels, make sure they are passed as a floating point tensor. This feature was introduced a few releases ago and allows you to pass “soft” labels to <code>nn.CrossEntropyLoss</code>.</p>

username: sandrodand
cooked: <p>I added <code>label = label.to(torch.float)</code> before the criterion call but I keep getting the same error.</p>

username: ptrblck
cooked: <p>The tutorial works fine for me, so could you post a minimal and executable code snippet reproducing the issue, please?</p>
After wrapped by FSDP, most of my model's weights are empty using named_parameters()

username: zyh3826
cooked: <p>I have a LLM and trained it using transformers with FSDP, I found that after being wrapped by <code>FSDP</code>, most of my model’s weights are empty using <code>named_parameters()</code>, see figure below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e40c81d0710342519dfd66643c95aa0ba619ae33.png" data-download-href="https://discuss.pytorch.org/uploads/default/e40c81d0710342519dfd66643c95aa0ba619ae33" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e40c81d0710342519dfd66643c95aa0ba619ae33.png" alt="image" data-base62-sha1="wxpEToxVkgefmiYfxHlmGqomrrJ" width="690" height="152" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1650×364 11.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Could you tell me what is the correct way to get all model <code>{name: param}</code> dict after being wrapped by <code>FSDP</code>, thanks a lot</p>
What determines the stride of the output of einsum?

username: marcsgil
cooked: <p>Hi!</p>
<p>I’m constructing an image dataset using <code>torch.einsum</code>, and I found a behavior that I could not understand.</p>
<p>The following dummy code reproduces what puzzles me:</p>
<pre><code class="lang-python">import torch

rho = torch.randn(10,4,4)
basis = torch.randn(4,2,64,64)
result = torch.einsum('imn,mjkl,njkl-&gt;ijkl',rho,basis,basis)
</code></pre>
<p>The variable <code>result</code>, which has shape <code>(10,2,64,64)</code>, would be my dataset. There are 10 images, with 2 color channels, with resolution 64x64.</p>
<p>When I started training neural networks with this dataset, I found that the process was around 3x slower compared to when I used other datasets with the same shape and datatype. By investigating further, I found out that <code>result.stride() = (1, 40960, 640, 10)</code>, which explains the slowness: during the batching process, each batch is not contiguously stored on memory. By correcting this dataset using <code>result.contiguous()</code> I get back to normal speeds.</p>
<p>I would like, therefore, to understand what determines the stride of the result of <code>einsum</code>. Is there a better way to obtain the stride that I want?</p>
<p>Thank you very much!</p>

username: tom
cooked: <p>The stride result of a computation is an implementation detail. What happens (in the most common case) is that PyTorch rearranges dimensions to express the contraction of 2 operands as a batch matrix multiplication and then permutes the axes back to the desired output.</p>
<p>One easy thing to try is to call <code>.contiguous()</code> right after the <code>einsum</code>.<br>
If you are on the CPU, another thing you could try is to use numpy.einsum, which can be much more sophisticated than PyTorch’s implementation.</p>
<p>Best regards</p>
<p>Thomas</p>
torch.load() occur _pickle.UnpicklingError: A load persistent id instruction was encountered

username: biggiantpigeon
cooked: <p>I was doing fine with torch 1.0.0, I can save and load the models I train by my self.<br>
then I transfer my program to torch 1.9.1(I do not save under 1.0.0 but load under 1.9.1, I do both in 1.9.1), and I save model like this:</p>
<pre><code class="lang-auto">       for model_name, model in self.models.items():
            save_path = os.path.join(save_folder, "{}.pth".format(model_name))
            to_save = model.state_dict()
            torch.save(to_save, save_path)
        save_path = os.path.join(save_folder, "{}.pth".format("adam"))
        torch.save(self.model_optimizer.state_dict(), save_path)
</code></pre>
<p>when I load my model by</p>
<pre><code class="lang-auto">loaded_dict_enc = torch.load(encoder_path, map_location='cpu')
</code></pre>
<p>this error occur:</p>
<pre><code class="lang-auto">  File "/home/***/examples/myproject/example.py", line 27, in &lt;module&gt;
    loaded_dict_enc = torch.load(encoder_path, map_location='cpu')
  File "/home/***/anaconda3/envs/py3.6-tch1.0/lib/python3.6/site-packages/torch/serialization.py", line 367, in load
    return _load(f, map_location, pickle_module)
  File "/home/***/anaconda3/envs/py3.6-tch1.0/lib/python3.6/site-packages/torch/serialization.py", line 528, in _load
    magic_number = pickle_module.load(f)
_pickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.
</code></pre>
<p>I think this is due to version update because I don’t change a bit of code.<br>
Any clue about this?</p>

username: biggiantpigeon
cooked: <p>Sorry! it was my own fault. When transfer I didn’t change the python SSH intepreter to the envs contain torch-1.9.1.</p>

username: Niloofar_Azari
cooked: <p>May you explain more?</p>

username: Benjamin_Loison
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/61442" rel="noopener nofollow ugc">GitHub PyTorch post</a>.</p>
Attribute Error when using GridSearchCV and Pytorch + SklearnPipelines

username: rookiearun
cooked: <p>I defined a class TorchRegressionModule using pytorch and then later I created a pipe line and tried to pass it through GridSearchCV as estimator but I got the following error there.</p>
<blockquote>
<pre><code class="lang-auto">f16_elevators_pipeline_steps = [
    ("selector", sklearn.feature_selection.SelectKBest(
        score_func=sklearn.feature_selection.mutual_info_regression,
        k=desired_features
    )),
    ("scaler", sklearn.preprocessing.MinMaxScaler()),
    ("estimator", skorch.NeuralNetRegressor(
        TorchRegressionModule,
        optimizer=torch.optim.Adam,
        lr=0.01,
        max_epochs=50,
        callbacks=[skorch.callbacks.EarlyStopping(
            patience=10)],
        batch_size=10,
    ))
]
</code></pre>
</blockquote>
<p>Then if I try to pass it under GridSearchCV,</p>
<pre><code class="lang-auto">&gt; grid = GridSearchCV(f16_elevators_pipeline, param_grid=param_grid, n_jobs=-1, cv=3)
&gt; grid_result = grid.fit(f16_elevators_X_train.to_numpy().astype('float32'), f16_elevators_y_train.values.astype('float32').reshape(-1,1))
</code></pre>
<pre><code class="lang-auto">&gt; AttributeError: Can't get attribute 'TorchRegressionModule' on &lt;module '__main__' (built-in)&gt;
&gt; """
&gt; 
&gt; 
&gt; The above exception was the direct cause of the following exception:
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44cab74ad206e469cbae4e3a589fa38344bc8314.png" data-download-href="https://discuss.pytorch.org/uploads/default/44cab74ad206e469cbae4e3a589fa38344bc8314" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44cab74ad206e469cbae4e3a589fa38344bc8314.png" alt="image" data-base62-sha1="9OyMFucxM4Ohqm8EmG4kUEuLRbK" width="690" height="333" data-dominant-color="5A2E2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">894×432 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone let me know what should I do to fix this issue, as I am unsure of what to do here, so that I can pass my pipeline with Neural network as one of the arguments in pipeleine?</p>
HeteroData and HGTLoader

username: SKTheLearner
cooked: <p>Hi<br>
I have a large HeteroData like:<br>
HeteroData(<br>
mol={ x=[876, 384] },<br>
gene={ x=[18211, 18211] },<br>
(mol, perts, gene)={<br>
edge_index=[2, 11181554],<br>
edge_label=[11181554],<br>
},<br>
(gene, rev_perts, mol)={<br>
edge_index=[2, 11181554],<br>
edge_label=[11181554],<br>
}<br>
)</p>
<p>can someone guide me to  create mini-batches for my training…</p>
<p>I tried the following…<br>
a. split the data into train,val,set using RandomLinkSplit<br>
b. created HGTLoader using train like  so<br>
train_loader = HGTLoader(<br>
train,<br>
# Sample 512 nodes per type and per iteration for 4 iterations<br>
num_samples={key: [16] * 2 for key in train_data.node_types},<br>
# Use a batch size of 128 for sampling training nodes of type paper<br>
batch_size=16,<br>
input_nodes=(‘mol’,None),<br>
)</p>
<pre><code> c. Then in train loop i used for t in train_loader:
     but model throws error...

     Input In [12], in EdgeDecoder.forward(self, z_dict, edge_label_index)
    19 def forward(self, z_dict, edge_label_index):
    20     row, col = edge_label_index
    ---&gt; 21     z = torch.cat([z_dict['mol'][row], z_dict['gene'][col]], dim=-1)
   23     z = self.lin1(z).relu()
   24     z = self.lin2(z)

   IndexError: index 859 is out of bounds for dimension 0 with size 32
   quite obvious it is not finding the edge_label_index in the batch...
    (how can I make sure I pass the correct label_indices for the current batch?)
</code></pre>
<p>QUESTIONS:<br>
a. am I using HGT Loader correctly or is HGTLoader is the right one to use ?<br>
b. am I iterating thru the batches correctly ?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> et al</p>
How to create a empty tensor with numel == 0 and data_ptr == nullptr?

username: cokespace2
cooked: <p>I create a empty tensor by <code>at::Tensor tensor = at::empty({}, options.device(kCUDA).dtype(dtype));</code></p>
<p>but the <code>tensor</code> has attributes like:</p>
<pre><code class="lang-c++">tensor.numel() == 1 // true
tensor.data_ptr() == nullptr // false
</code></pre>
<p>and I tried with <code>Tensor tensor = at::Tensor().to(dtype).to(kCUDA);</code>, which fails of a Runtime error: tensor does not have a device.</p>
<p>how could I just create a tensor with <code>numel == 0</code> and <code>data_ptr == nullptr</code>?</p>

username: ptrblck
cooked: <p>What does <code>data_ptr</code> return in your case?<br>
In Python I see <code>0</code>:</p>
<pre><code class="lang-python">torch.empty(0, device="cuda").data_ptr()
# 0
</code></pre>

username: tom
cooked: <aside class="quote no-group" data-username="cokespace2" data-post="1" data-topic="192361">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cokespace2/48/51244_2.png" class="avatar"> cokespace2:</div>
<blockquote>
<p><code>at::Tensor tensor = at::empty({}, options.device(kCUDA).dtype(dtype));</code></p>
</blockquote>
</aside>
<p>This is a scalar (and thus 1-element) tensor.<br>
As Piotr mentions, you would want to provide a size-0 dimension to get a 0 element tensor:<br>
<code>at::Tensor tensor = at::empty({0}, options.device(kCUDA).dtype(dtype));</code></p>
<p>Best regards</p>
<p>Thomas</p>

username: cokespace2
cooked: <p>In my case the <code>data_ptr</code> is just a normal pointer which point to the one-element data, while I expected a <code>nullptr</code>.</p>

username: cokespace2
cooked: <p>Okk, seems the only difference is the shape of <code>{0}</code> instead of <code>{}</code>, I’ll give a try, thx!</p>
ResNet-50 takes 10.13GB to run with batch size of 96

username: HaydenW
cooked: <p>I have been working on using a ResNet-50 and have images of shape (3, 256, 256) and I’m trying to run it in batch size of 96, but I keep getting an error stating that it ran out of CUDA memory. I have a 3090 so I find that hard to believe since it has 24GB. So I then tested a very simple structure with this code, which showed that it takes 10.13GB to run with a shape (96, 3, 256, 256). Is this correct, I thought it would take a lot less allocated memory to run such a network? I checked and CUDA is running on my device and is available.</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models

device = torch.device('cuda:0'if torch.cuda.is_available() else 'cpu')

def print_allocated_memory():
   ...:     print("{:.2f} GB".format(torch.cuda.memory_allocated() / 1024 ** 3))

resnet50_input = torch.ones(96, 3, 256, 256).float().to(device)

resnet50 = models.resnet50(pretrained = False).to(device)

encoded = resnet50(resnet50_input)

print_allocated_memory()

10.13 GB
</code></pre>

username: ptrblck
cooked: <p>A memory usage of ~10GB would be expected for a ResNet50 with the specified input shape.<br>
Note that the input itself, all parameters, and especially the intermediate forward activations will use device memory.<br>
While the former two might be small the intermediates (which are needed for the gradient computation) would most likely use the majority of the memory.<br>
A quick way of checking it would be to register forward hooks and check all output activations.<br>
Note that this approach is not exact, as you would need to check which operations are performed inplace etc., but would at least give you an approx. estimation:</p>
<pre><code class="lang-python">nb_acts = 0
def count_output_act(m, input, output):
    global nb_acts
    nb_acts += output.nelement()

[...]
for module in resnet50.modules():
    if isinstance(module, nn.Conv2d) or isinstance(module, nn.Linear) or isinstance(module, nn.BatchNorm2d):
        module.register_forward_hook(count_output_act)
</code></pre>
<p>Using this approach, I get an estimation of:</p>
<pre><code class="lang-python">nb_params = sum([p.nelement() for p in resnet50.parameters()])
nb_inputs = resnet50_input.nelement()
print('input elem: {}, param elem: {}, forward act: {}, mem usage: {}GB'.format(
    nb_inputs, nb_params, nb_acts, (nb_inputs+nb_params+nb_acts)*4/1024**3))

&gt; 10.127978801727295
input elem: 18874368, param elem: 25557032, forward act: 2787211008, mem usage: 10.548689991235733GB
</code></pre>
<p>which comes close to the reported value.<br>
If you don’t want to train the model and would like to save memory by not storing the intermediates, wrap the forward pass into <code>with torch.no_grad():</code>, which will use ~0.167GB.</p>

username: HaydenW
cooked: <p>Great! Thank you so much, glad to know I set up everything correctly</p>

username: Hurizma1003
cooked: <p>Could you share full code for doing this on transformer based model<br>
That would be great</p>
RuntimeError: Trying to backward through the graph a second time. Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward

username: Sai_Kishore
cooked: <p>Here is the forward code, Im calculating a custom loss function in the Model class and returning the loss value, what are the in-place operations am i doing that is causing these errors, where can i fix them.</p>
<pre><code class="lang-auto">class MatrixModel(nn.Module):
  def __init__(self, num_matrices=10,layers = 10,img_shape=(32,32),lamda=1.0) -&gt; None:
    super().__init__()
    self.l = lamda
    modules = []
    self.W = torch.randn((num_matrices,img_shape[1],img_shape[1]))
    for i in range(layers):
      modules.append(nn.Conv2d(num_matrices,num_matrices,kernel_size=3,stride=1,padding=1))
    self.model = nn.Sequential(*modules)
    modules = self.model.named_children()
    for name, module in modules:
      # module.register_full_backward_hook(backward_hook_fn)
      module.register_forward_hook(lambda m, input, output: torch.nan_to_num(output,nan=0.0))


  def calculate_loss(self,X,X_w,W,lamda=1.0):
    X_w = X - X_w
    X_w = torch.einsum("ijk,ijk-&gt;ijk",X_w,X_w)

    X_w = torch.nan_to_num(X_w)
    X_w = torch.einsum("ijk-&gt;",X_w)
    X_w = 0.5*X_w.clone()

    W = torch.einsum("ijk,ijk-&gt;ijk",W,W)

    W = torch.einsum("ijk-&gt;jk",W)
    W = torch.sqrt(W+1e-6)
    W = torch.einsum("jk-&gt;",W)
    W = lamda*W.clone()
    return W + X_w

  def forward(self,x):
    W = self.W.clone()
    W = torch.unsqueeze(W,0)
    W = torch.nan_to_num(W,nan=0.0)
    W = W.to(device)

    W = self.model(W)

    x = torch.mean(x,dim=1)
    x_expanded = x.unsqueeze(1)

    X_w = torch.matmul(x_expanded, W)
    X_w = torch.nan_to_num(X_w)
    X_w = torch.sum(X_w,dim=1)

    X_w = torch.nan_to_num(X_w)
    W = torch.squeeze(W)
    loss = self.calculate_loss(x,X_w,W,self.l)
    self.W = W.clone()
    return loss

</code></pre>

username: ptrblck
cooked: <p>It seems you are assigning the model output to <code>self.W</code> and are then reusing it. You might need to <code>.detach()</code> the tensor before assigning it instead of cloning.</p>
Debugging CUDA out of memory (16GB GPU, 14+ GB reserved)

username: motown_dad
cooked: <p>I’m pretty new and want to learn how to debug GPU memory allocation.</p>
<p>My setup:</p>
<ul>
<li>paperspace, machine with A4000 16G GPU</li>
<li>single notebook running</li>
<li>playing with <a href="https://github.com/facebookresearch/dinov2" rel="noopener nofollow ugc">DINOv2</a>, just using the embedding part with pre-trained weights</li>
<li>inspecting the model, it has ~427M params, so even with float32 that should be around 1.7GB</li>
<li>loading 280x280 images which I want to get embedding, 100 images x 280x280x3, with float32 should be under 100MB</li>
</ul>
<p>I’m still getting <code>RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 15.73 GiB total capacity; 13.80 GiB already allocated; 23.12 MiB free; 14.15 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>Here is code to reproduce:</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms
from PIL import Image

m = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14_reg')
m.eval()
m.cuda()

pil2tensor = transforms.ToTensor()
files = [] # 100 file paths to png files
x = torch.stack([pil2tensor(Image.open(f).resize((14*20,14*20))) for f in files])
x.shape == (100, 3, 280, 280)

bs = 10 # tried many different batch sizes
x_embs = []
for i in range(0, len(files), bs):
    batch = x[i:i+bs]
    batch = batch.cuda()
    x_emb = m(batch)
    x_embs.append(x_emb.cpu())
</code></pre>
<p>Usually it fails on the <code>x_emb = m(batch)</code> line, which is when running the model inference. I tried different batches. I tried calling <code>torch.cuda.empty_cache()</code> everywhere, but nothing helps.</p>
<p>It works fine on CPU.</p>
<p>Any advice on how to figure out why is so much memory “reserved”?</p>

username: ptrblck
cooked: <p>Besides the parameters and inputs intermediate forward activations could allocate a lot of memory as explained in e.g. <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">this post</a>.</p>
Debug a custom cuda kernel

username: santisy
cooked: <p>Hi all,</p>
<p>I am wondering if there is any instruction on how to debug custom cuda kernels which are loaded using <code>torch.utils.cpp_extension.load</code>. What I want to do is: 1. start a gdb session; 2. set breakpoints in the <code>.cu</code> files; 3. start the python script running (where the specified nvcc flags passed to <code>torch.utils.cpp_extension.load</code> contain the <code>-G</code> flag).</p>
<p>I failed to let the program stop at the desired place.</p>
<p>Looking forward to your suggestion.</p>
<p>Best</p>

username: ptrblck
cooked: <p>Use <code>cuda-gdb</code> to debug CUDA kernels, which should allow you to set breakpoints in your kernel.</p>
Why does CUDA memory usage suddenly increase when using torch.sub()

username: Wyongbo
cooked: <p>Recently, in my deep learning model, I attempted to use the torch.sub() function or ‘-’ to subtract two tensors, but the memory consumption has doubled. Before torch. sub() or ‘-’, I set *'with torch. no_ Grad() '*can make the graphics memory consumption return to normal, so I guess it was an abnormal consumption of graphics memory in gradient calculation, but I don’t know how to solve it</p>
<blockquote>
<p>Blockquoteclass edge_generator(nn.Module):<br>
def <strong>init</strong>(self, feature_dim=256, num_classes=9):<br>
super(edge_generator, self).<strong>init</strong>()<br>
self.feature_dim = feature_dim<br>
self.sim_net = nn.Sequential(<br>
nn.Linear(self.feature_dim, 2<em>self.feature_dim),<br>
nn.LayerNorm(2</em>self.feature_dim),<br>
nn.ReLU(),<br>
nn.Linear(2*self.feature_dim, 1)<br>
)<br>
self.edge_loss = BCEFocalLoss( )<br>
self.num_classes = num_classes</p>
</blockquote>
<pre><code>def forward(self, node_feat, label):
    num_data = node_feat.size(0)
    feat_size = node_feat.size(1)
    
    # distance based
    x_i = node_feat.unsqueeze(1)
    x_j = x_i.transpose(0, 1)
    x_i = x_i.expand(num_data, num_data, feat_size)
    x_j = x_j.expand(num_data, num_data, feat_size)
    # x_ij = torch.abs(x_i - x_j)
    x_ij = torch.sub(x_i, x_j)
    x_ij = torch.abs(x_ij)
    # compute similarity/dissimilarity (num_data x num_data)
    edge_feat = torch.sigmoid(self.sim_net(x_ij)).squeeze( )

    force_edge_feat = torch.eye(num_data).cuda( )
    edge_feat = edge_feat + force_edge_feat
    edge_feat = edge_feat + 1e-6
    edge_feat = edge_feat / torch.sum(edge_feat, dim = 1).unsqueeze(1)

    one_hot_labels = self.one_hot(label)
    edge_labels = torch.matmul(one_hot_labels, one_hot_labels.transpose(0, 1))

    edge_loss = self.edge_loss(edge_feat, edge_labels.long( ))

    return edge_feat, edge_loss
def one_hot(self, x):
    return torch.eye(self.num_classes)[x.long(), :].cuda()
</code></pre>
<p>After running the torch. sub() function and obtaining the x_ij result, the memory usage will increase</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet to reproduce the issue?</p>

username: Wyongbo
cooked: <p>Sorry, I forgot to share the code during the questioning. I have already made some additions during the questioning. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
Tensor.is_cuda not working?

username: starkj
cooked: <p>I cannot get a value of True from Tensor.is_cuda, even though moving the tensor to the GPU seems to go with no problem.  Here is a simple test program:</p>
<pre><code class="lang-auto">import torch

avail = torch.cuda.is_available()
print("Cuda available: ", avail)

a = torch.randn(2, 2)
print(a)
print("Moving a to the GPU...")
a.cuda()
verif = a.is_cuda
print("verification = ", verif)
</code></pre>
<p>Output looks like this:</p>
<pre><code class="lang-auto">(cda) $ !py
python ttt.py
Cuda available:  True
tensor([[0.8693, 2.1210],
        [1.4204, 0.1404]])
Moving a to the GPU...
verification =  False
</code></pre>
<p>I am running Ubuntu 22.04 with a RTX-3080.  Python is 3.11 and torch version is 2.1.0+cu118.  nvidia-smi shows CUDA version 11.2 and driver version 535.113.01.  I just did a new rebuild of everything a couple weeks ago.  What might be going wrong here?</p>
<p>This is not just an academic exercise. When I try running some torch-based NN training code, it complains (in F.linear()) that not all tensors are on the same device, even though I explicitly move them there with Tensor.cuda().</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><code>.cuda()</code> and <code>.to()</code> calls on tensors are not executed inplace and you would need to assign the result.<br>
This should fix it:</p>
<pre><code class="lang-python">a = a.cuda()
</code></pre>

username: starkj
cooked: <p>Got it - thanks for the quick response!</p>