akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Compiling a model that uses sdp_kernel to enable the backends does not work

raulc: <p>the model uses “with sdp_kernel(enable_math=True, enable_flash=True, enable_mem_efficient=True):”</p>
<p>when calling compile on this model, it generated the following error:</p>
<blockquote>
<p>torch._dynamo.exc.Unsupported: torch.* op returned non-Tensor _GeneratorContextManager call_function &lt;function sdp_kernel at 0x7e3c02b79fc0&gt;</p>
</blockquote>
<p>how can i use sdp_kernel with compile</p>
<p>thanks</p>

ptrblck: <p><code>torch.compile</code> works for me but raises the deprecation warning:</p>
<pre data-code-wrap="python"><code class="lang-python">query = torch.rand(64, 12, 77, 64, dtype=torch.float16, device="cuda")
key = torch.rand(64, 12, 77, 64, dtype=torch.float16, device="cuda")
value = torch.rand(64, 12, 77, 64, dtype=torch.float16, device="cuda")
attn_mask = torch.rand(64, 1, 77, 77, dtype=torch.float16, device="cuda")


def fun(query, key, value, attn_mask):
    with torch.backends.cuda.sdp_kernel(enable_math=True, enable_flash=True, enable_mem_efficient=True):
        out = torch.nn.functional.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask)
    return out

out = fun(query, key, value, attn_mask)
# FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.

fun_compiled = torch.compile(fun)
out = fun_compiled(query, key, value, attn_mask)
# FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.
</code></pre>

bdhirsh: <p>Can you move the context manager so it is outside of the compiled region? E.g</p>
<p>With sdp_kerbel(…):<br>
out = compiled_model(inp)</p>
<p>Although cc <a class="mention" href="/u/drisspg">@drisspg</a> (we should make sure the new, non-deprecated context manager works inside of a compiled region)</p>

raulc: <p>sorry, forgot to mention that it was with fullgraph.</p>
<blockquote>
<p>fun_compiled = torch.compile(fun, fullgraph=True)</p>
</blockquote>
<p>it yields:</p>
<blockquote>
<p>File “/home/…/env/lib/python3.12/site-packages/torch/_dynamo/exc.py”, line 297, in unimplemented<br>
raise Unsupported(msg, case_name=case_name)<br>
torch._dynamo.exc.Unsupported: torch.* op returned non-Tensor _GeneratorContextManager call_function &lt;function sdp_kernel at 0x7fa07c0c8400&gt;</p>
<p>from user code:<br>
File “/home/…/test.py”, line 10, in fun<br>
with torch.backends.cuda.sdp_kernel(enable_math=True, enable_flash=True, enable_mem_efficient=True):</p>
</blockquote>

raulc: <p>i guess i can do that as a workaround. will try.</p>
Pytorch start training process stucked when blocking GPU 0

forestbat: <p>The GPU0 in my server has been occupied by others’ processes, so I blocked GPU0 and use <code>mp.spawn</code> to train my model, but it failed to create train process.</p>
<p>This is reproducible example:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
import os
import torch.distributed as dist

def train(rank, gpu_ids):
    device_id = gpu_ids[rank]
    dist.init_process_group(
        backend="nccl", init_method="env://", world_size=len(gpu_ids), rank=device_id)
    torch.cuda.set_device(device_id)
    print(f"Rank {rank} is using device {device_id}")
    model = nn.Linear(10, 1).to(device_id)
    optimizer = optim.SGD(model.parameters(), lr=0.01)
    input_data = torch.randn(5, 10).to(device_id)
    target = torch.randn(5, 1).to(device_id)
    for epoch in range(5):
        optimizer.zero_grad()
        output = model(input_data)
        loss = nn.MSELoss()(output, target)
        loss.backward()
        optimizer.step()
        print(f"Rank {rank}, Epoch {epoch}, Loss: {loss.item()}")

def test_main():
    os.environ['CUDA_VISIBLE_DEVICES'] = '1,2'
    os.environ["MASTER_ADDR"] = '127.0.0.1'
    os.environ["MASTER_PORT"] = '12345'
    gpu_ids = [1]
    world_size = len(gpu_ids)
    mp.spawn(train, args=(gpu_ids,), nprocs=world_size, join=True)
</code></pre>
<p>In my test, the <code>train()</code> method stucked in here:</p>
<pre><code class="lang-auto">dist.init_process_group(backend="nccl", init_method="env://", world_size=len(gpu_ids), rank=device_id)
</code></pre>
<p>So what happened and how to solve it?</p>

mreso: <p>Hi <a class="mention" href="/u/forestbat">@forestbat</a>,</p>
<p>this is because you’re giving device_id as rank to init_process_group which is 1 in your case. And as world size is 1 in your case (len(gpu_ids)) it probably waits for rank 0 to show up.</p>
<p>If you switch to rank=rank instead of device_id it works but there is another issue. As soon as you define CUDA_VISIBLE_DEVICES you can just address the GPUs with 0,1,2,… etc. The mapping from cuda:0 to gpu1 happens automatic. In your code your model will actually end up on gpu2 as cuda:1 maps to gpu2.</p>
LoRA + DP-SGD tutorial

iden-kalemaj: <p>Dear Opacus users,</p>
<p>We have updated our tutorial on DP fine-tuning of a language model to demonstrate the usage of LoRA (low-rank adaptation) with DP-SGD: <a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_text_classifier.ipynb" rel="noopener nofollow ugc">https://github.com/pytorch/opacus/blob/main/tutorials/building_text_classifier.ipynb</a></p>
<p>LoRA is a parameter-efficient fine-tuning method that allows for training significantly fewer parameters while maintaining on-par accuracy. You can combine it with Opacus training with a few lines of code and no conceptual changes to the privacy analysis.</p>
<p>Feel free to take a look!</p>
How to apply layernorm to image data with shape N C H W?

LuoXin-s: <p>I want to apply a normalization along channel dimension with data shape NCHW. However, <code>torch.nn.LayerNorm</code> only support normalize over last D dimensions (I wanna skip spatial dimension).</p>
<p>This requirement occurs in networks like <strong>NAFNet</strong> (a popular network for image restoration), they have to implement a special layernorm by themselves, but that would slower than possible <strong>fast cuda implementation</strong> provided by torch teams.</p>

neworldemancer: <p>If you want each spatial location - vector of C elements - to be standardized - you have to reshape your tensor to NHWC first, apply the LN, then go back to NCHW.<br>
Not sure what exactly is the performance penalty of reshaping in the current torch implementation.</p>
<p>The other rpossibility is to apply LN to each channel independently. Then ou set dimensions to [H,W]. Better to also disable <code>elementwise_affine</code></p>
Checking termination condition is exceptionally slow

ViktorAJStein: <p>In my code I perform an iterative procedure (for n in range(iterations): …) and in every iteration I create the long tensor unscaled_h_star_grad on the GPU. If want to stop if its norm is small, so I wrote</p>
<blockquote>
<p>Dfnulambda_grad = 1 / N * torch.norm(unscaled_h_star_grad)**2<br>
cond3 = Dfnulambda_grad &lt; 1e-6<br>
if cond3:<br>
print(f’time derivative of the objective = {Dfnulambda_grad.item()} &lt; 1e-6. Stopping iteration’)<br>
break</p>
</blockquote>
<p>However, this takes a lot of time, it contributes roughly a third of the total run time of my algorithm.<br>
Apart from only checking this condition only say, every 25th iteration, is there a way to handle this better? I suspect there is some issue with GPU-CPU transfer or synchronization here.</p>
<p>Kind regards</p>

ptrblck: <aside class="quote no-group" data-username="ViktorAJStein" data-post="1" data-topic="214324">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/47e85d/48.png" class="avatar"> ViktorAJStein:</div>
<blockquote>
<p>However, this takes a lot of time, it contributes roughly a third of the total run time of my algorithm.</p>
</blockquote>
</aside>
<p>PyTorch executes CUDA kernels asynchronously.<br>
Your code uses a data dependent condition is is thus synchronizing your code and accumulates the GPU execution time from already scheduled and launched kernels.</p>

ViktorAJStein: <p>Ok, understood, thank you for your fast response. I guess that checking if some loss is small enough in every k-th iteration is a quite common thing to do. What is the runtime optimal way to execute it?</p>
Relationship of libtorch and pytorch (Do I need Libtorch if I already pip installed pytorch)

Weixuan: <p>Hi, I am trying to understand the relationship between the libtorch and precomipled pytorch, thus figuring out whether I need Libtorch if I already have pytorch installed using pip.<br>
Does pytorch already contain libtorch? I am asking this because “print(torch.utils.cmake_prefix_path)” gives me “home/weixuan/.local/lib/python3.6/site-packages/torch/share/cmake”, and I can also set this as the CMAKE_PREFIX_PATH and compile my C++ code. Also see the instruction here: <a href="https://pytorch.org/cppdocs/installing.html" class="inline-onebox" rel="noopener nofollow ugc">Installing C++ Distributions of PyTorch — PyTorch master documentation</a><br>
So what is the difference between the libtorch and the shared libraries inside the pytorch packages? The purpose of libtorch is for those who do not have pytorch installed on their system and therefore independant of the python?<br>
In addition, it seems that the shared libraries in the pytorch 1.9.0 package are not compiled with cxx11-abi, as pointed out here: <a href="https://discuss.pytorch.org/t/sos-libtorch-conflict-with-ros/48977" class="inline-onebox">SOS! libtorch conflict with ROS!</a> This leads to conflict with ROS package. Is there a way of revolving this without having to compile pytorch from source?  Thanks!</p>

Hanshuo_WANG: <p>Actually, there is no need to download c++ libtorch if you have already downloaded pytorch using pip. Assume that you are using conda, libtorch.so will be automatically installed in “/root/miniconda3/envs/your_env/lib/python3.10/site-packages/torch/lib”, alongside with many other libraries that c++ can use. Using this libtorch directly in your c++ code and cmake will cause no version confliction when importing it to python by pybind11, because they come from the same place. And pybind11 has no version issue, so any download method will be fine (git clone from source or pip) if you are using it.</p>
Cutlass custom kernel integration in Pytorch

Izan_C_G: <p>Hello, I would like to know if, in addition to importing a custom kernel as detailed in <a href="https://github.com/NVIDIA/cutlass/blob/main/examples/python/02_pytorch_extension_grouped_gemm.ipynb" rel="noopener nofollow ugc">this example</a>, it is possible to import it into PyTorch and then reuse it not for a specific operation but to perform inference on a neural network model that uses these operations. This would be similar to what is described <a href="https://pytorch.org/vision/main/models.html" rel="noopener nofollow ugc">here</a>, where there are pre-trained models in PyTorch, such as ResNet50, which includes up to 50 convolutions among other operations.</p>

ptrblck: <aside class="quote no-group" data-username="Izan_C_G" data-post="1" data-topic="214304">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/izan_c_g/48/68535_2.png" class="avatar"> Izan_C_G:</div>
<blockquote>
<p>to import it into PyTorch and then reuse it not for a specific operation but to perform inference</p>
</blockquote>
</aside>
<p>I don’t understand this part of the question. Could you describe how the cutlass kernel should be used for “inference” but not a specific operation (matmul)?</p>

Izan_C_G: <p>Sorry <a class="mention" href="/u/ptrblck">@ptrblck</a> I didn’t explain myself well. Is it possible to use that custom convolution kernel from CUTLASS with a neural network model like ResNet50, which contains multiple convolution layers?</p>
<p>In the examples I’ve seen, such as <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>, you can define new ops or import them using C++ kernels. However, it seems that they can only be used as single operations, for instance:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

X = torch.randn(batch_size, input_features)
h = torch.randn(batch_size, state_size)
C = torch.randn(batch_size, state_size)

rnn = LLTM(input_features, state_size) 
new_h, new_C = rnn(X, (h, C))
</code></pre>
<p>However, suppose I want all the convolution layers inside a pre-trained PyTorch model like ResNet50 to use my imported CUTLASS convolution kernel. In that case, I’m not sure how to do that or whether it’s even possible.</p>
<p>Therefore, I have two questions: first,  how to import a custom cutlass kernel with PyTorch and second, how to use or link it to allow pre-trained models and their operators (convolutions in my case) instead of the default convolution use it by PyTorch.</p>
Optimizing difference of two outer products help

Nyakov: <p>Hi.<br>
I have following computation:<br>
<code>res = torch.outer(po, x) - torch.outer(o, px)</code></p>
<p>Is there a way to make it with only one outer product?</p>

soulitzer: <p>Sure you can concat po and o and px with x, but its probably only more efficient if your tensors are really small, e.g. when kernel launch overhead is the main bottleneck.</p>
Quantized GLU not implemented?

iliasslasri: <p>Hi there,<br>
I’m currently working on quantizing to int8 denoiser speech enhacement model by facebook research, here : <a href="https://github.com/iliasslasri/denoiser/blob/main/quantization/linear-q.ipynb" class="inline-onebox" rel="noopener nofollow ugc">denoiser/quantization/linear-q.ipynb at main · iliasslasri/denoiser · GitHub</a><br>
and I could not find an implementation of nn.GLU() activation in the quantized module in pytorch. Does anyone have an idea of how that works or if there’s any implementation out there or if I should just implement it ?</p>
<p>Thank you for the help guys.</p>

iliasslasri: <p>I wrote a quick implementation. Please fell free to comment!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class QuantizedGLU(nn.Module):
    def __init__(self, last_layer=False):
        super().__init__()

    def forward(self, x) -&gt; torch.Tensor:
        a, b = x.chunk(2, dim=1) # dim to adapt
        output = a * torch.sigmoid(b)
        return output.to(torch.int8)

</code></pre>
Temporal Fusion Transformer Error

himanshu_birla: <p>I am using TFT for forecasting . It is throwing me the below error when i put return_y=True while making prediction (baseline_predictions = Baseline().predict(val_dataloader,return_y=True)). It works fine if i set return_y = False.</p>
<p>Error : RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 320 for tensor number 1 in the list.</p>
<p>Can you please suggest where am i going wrong?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error, please?</p>

Juan_pablo_Mazo_quin: <p>Hello,</p>
<p>I’m encountering a similar issue while working with the Temporal Fusion Transformer (TFT). My dataset consists of ~5,000 unique combinations (e.g., SKU, store, etc.) spanning 24 timesteps (2 years of monthly data). My goal is to forecast the next month for each combination.</p>
<p>When I try to calculate metrics like the Mean Absolute Error (MAE) using the <code>Baseline().predict()</code> function with <code>return_y=True</code>, I encounter the following error:</p>
<pre><code class="lang-auto">RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1280 but got size 641 for tensor number 4 in the list.
</code></pre>
<p>If I set <code>return_y=False</code>, the code runs without issue. I’ve tried the following setup, but the issue persists:</p>
<pre><code class="lang-auto"># Create validation set (predict=True) to predict the last max_prediction_length points in time for each series
validation = TimeSeriesDataSet.from_dataset(training, df, predict=True, stop_randomization=True)

# Create dataloaders for the model
batch_size = 128  # Set this between 32 to 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=0)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size * 10, num_workers=0, drop_last=True)

# Define baseline predictions
baseline_predictions = Baseline().predict(val_dataloader, return_y=True)

# Calculate MAE
 MAE()(baseline_predictions.output, baseline_predictions.y)
</code></pre>
<p>I configured the model with the following parameters:</p>
<ul>
<li><code>max_prediction_length = 2</code></li>
<li><code>max_encoder_length = 12</code></li>
</ul>
<p>It seems to be a dimension mismatch issue, but I’m unsure how to resolve it. Could anyone provide guidance on how to fix this?</p>
<p>Thank you!</p>
Taking advantage of sparsity with FlexAttention

cajoek: <p>Hi!<br>
I am wondering if and how I should replace scaled_dot_product_attention with FlexAttention. I am mostly interesting in benefits from sparsity, this is my usecase:<br>
I do cross-attention between two sequences of length 256 with a boolean mask that is identical for all heads i.e. of shape: <code>[batch, 1, 256, 256]</code>. I get the mask from a preprocessing step and it is 95% sparse and will appear uniformly distributed. How do I best create a <code>BlockMask</code> from my current mask and how do I get as much benefit as possible from sparsity?</p>
<p>Thanks!</p>
Any faster and memory-efficient alternative of torch.autograd.functional.jacobian(model.decoder, latent_l)?

Jimut123: <p>I have a decoder <code>model.decoder</code>, which is comprised of a series of Convolutional Batchnorm and ReLU layers. I have a latent vector <code>latent_l</code>, which is a 8 dimensional latent vector, say, has the dimension (1, 8, 1, 1), where 1 is the batch size. I am doing <code>torch.autograd.functional.jacobian(model.decoder, latent_l)</code>, which is taking a huge amount of time, is there any fast approximation for this jacobian?</p>
<p>There is <code>jacrev</code>, but I am not sure if that works for this example where we pass a decoder as a whole and compute the jacobian of the decoder with respect to the latent vector.</p>
<p>When I use <code>torch.autograd.functional.jacobian(model.decoder, latent_l, vectorize=True)</code>, the memory consumption of the GPU increases drastically, leading to the crashing of the program. Is there any efficient way of doing this using Pytorch?</p>

soulitzer: <aside class="quote no-group" data-username="Jimut123" data-post="1" data-topic="214045">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimut123/48/29864_2.png" class="avatar"> Jimut123:</div>
<blockquote>
<p>which is taking a huge amount of time</p>
</blockquote>
</aside>
<p>If the dimensionality of your output is greater than 8 (you should use strategy=“forward-mode”) and in that case, your jacobian column-by-column with forward mode with 8 forward passes, with each forward pass taking slightly longer than a normal forward pass.</p>
<p>Given that, the Jacobian taking 10x ~ 15x the amount of time it takes to compute a single forward seems expected.</p>

soulitzer: <p>If you see it taking much longer than 10x forward, one explanation is that your output dimensionality &gt;&gt; input dimensionality and you are using “reverse-mode”.</p>
<ul>
<li>the default strategy for is “reverse-mode” btw, so you’d need to explicitly pass in “foward-mode”</li>
</ul>

Jimut123: <p>looks like <code>torch.autograd.functional.jacobian(model.decoder, latent_l, strategy="forward-mode", vectorize=True)</code> seems to do the job for me! Thanks!</p>
Can MPS profiles be inspected by non closed-source software? Where are they stored?

Aknw_Fen: <p>PyTorch doesn’t yet offer <a href="https://github.com/pytorch/pytorch/issues/99037" rel="noopener nofollow ugc">ProfilerActivity.MPS</a> but is there any way to inspect torch.mps.profiling without XCode? Where are these files located?</p>
<p>It <a href="https://pytorch.org/docs/stable/_modules/torch/mps/profiler.html#profile" rel="noopener nofollow ugc">only mentions XCode</a>:</p>
<blockquote>
<p>The generated OS Signposts could be recorded and viewed in XCode Instruments Logging tool.</p>
</blockquote>
Questions Regarding the Direction of Half Padding for Odd-Sized Inputs in PyTorch Conv2D and ConvTransposed2D

mx34kryce: <p>Hello, I have a question regarding the behavior of padding in convolution layers in PyTorch’s CNN implementation. According to the official PyTorch documentation, for details on how convolutions operate, it refers us to the following paper: (<a href="https://arxiv.org/abs/1603.07285" class="inline-onebox" rel="noopener nofollow ugc">[1603.07285] A guide to convolution arithmetic for deep learning</a>). While reading that paper, I noticed something about the way padding is handled in ConvTransposed2D compared to the regular Conv2d function.</p>
<p>In general, consider a scenario where <code>stride = 2</code> and the kernel size is an odd number (let’s say 3), and you have an odd-by-odd 2D tensor as input. According to the paper, half padding is applied toward the bottom-right direction. I believe the code also implements this in the same way. If that’s the case, then for the Conv2D function as well, when <code>stride = 2</code>, the kernel size is odd, and the input is an odd-by-odd 2D tensor, you would need to apply half padding in a similar manner as with ConvTransposed2D. In that situation, in which direction would the half padding be applied for the Conv2D function?</p>
<p>I’ve tried looking into the relevant parts of the code, but PyTorch is so well-encapsulated that I can’t seem to find the portion that handles this exact behavior🤣.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbbaac4cedae04ffa60904b9c583ad66ddfa697c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c_2_690x302.png" alt="image" data-base62-sha1="qMJeokE13HhY8ka1P9NvXeuZJq4" width="690" height="302" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c_2_690x302.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c_2_1035x453.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c.png 2x" data-dominant-color="D6E5E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1123×493 69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I think the figure above clearly illustrates the problematic situation.</p>

mx34kryce: <p>The question I created seems completely incorrect. I don’t know how to delete it…<br>
I’ll revise it and ask again.</p>
How to Fine-Tune Pre-Trained Stable Diffusion Models Using Custom Images

Aditya_Jhaveri: <p>Problem statement:<br>
I am utilizing Stable Diffusion XL Base 1.0 for image generation, but it does not accept my custom input image. I would like to generate a new image based on my input image and the specified prompt.</p>
<p>Description:<br>
I need a solution to generate anthropomorphic pet portraits from user-uploaded pet photos. Specifically, I want the AI to use the pet’s face from the uploaded image and create the rest of the body based on a given prompt, such as a king, doctor, or lawyer. The generated portraits should retain the pet’s unique features, making it clear they are the same pets.</p>
<p>The problem I’m encountering is that when I input a custom pet image into the pre-trained Stable Diffusion model with an anthropomorphic prompt, the model generates an image based on its dataset instead of using my custom image. I want the AI to generate new images using the provided pet photos, incorporating the given prompt, rather than creating random images from its own dataset.</p>
<p>How can I fine-tune a pre-trained Stable Diffusion model, or any relevant model, with our custom images so that it uses these images to generate new portraits according to the given input and prompt?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1340ad5d0cea40b16cb810d73a3e143c484f920.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a1340ad5d0cea40b16cb810d73a3e143c484f920" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1340ad5d0cea40b16cb810d73a3e143c484f920_2_690x354.jpeg" alt="image" data-base62-sha1="n04n7sMoMjEIDjdRJCUTvo09HuU" width="690" height="354" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1340ad5d0cea40b16cb810d73a3e143c484f920_2_690x354.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1340ad5d0cea40b16cb810d73a3e143c484f920_2_1035x531.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1340ad5d0cea40b16cb810d73a3e143c484f920_2_1380x708.jpeg 2x" data-dominant-color="9C9E9C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1792×921 285 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ritabratamaiti: <p>Looks like you are looking for controlnet</p>
Problems with fork() in DataLoader with num_workers > 1

tanhevg: <p>Throughout my work I have run into multiple issues with hung Dataloaders with num_workers &gt; 1. Turns out this is a known issue with fork(), see, for example, <a href="https://pythonspeed.com/articles/python-multiprocessing/" rel="noopener nofollow ugc">this article</a>. I see lots of material on the Internet suggesting <code>fork</code> is a bad practice (e.g. <a href="https://docs.pola.rs/user-guide/misc/multiprocessing/" rel="noopener nofollow ugc">polars</a>). <code>spawn</code> is rumoured to become the default method for starting child processes in Python 3.14. Wouldn’t it make sense to make <code>spawn</code> the default method for child processes for DataLoaders in PyTorch (on Linux, I understand other OSes already use spawn)? Or at least mention the potential for deadlocks and the workaround with spawn in the docs…</p>

ptrblck: <p>I don’t know what kind of errors you are seeing, but the <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing"><code>multiprocessing</code> docs</a> already ask users to use <code>spawn</code> in case you want to use CUDA in your workflow (e.g. your data loading):</p>
<blockquote>
<p>The CUDA runtime does not support the <code>fork</code> start method; either the <code>spawn</code> or <code>forkserver</code> start method are required to use CUDA in subprocesses.</p>
</blockquote>

tanhevg: <p>Thanks. The errors I am getting are deadlocks, followed by a killed DataLoader worker - see the <a href="https://discuss.pytorch.org/t/ddp-num-workers-0-cuda-error/191096/3">linked post</a>.</p>
<p>Yes, you are right that the documentation warns against using <code>fork</code>. Although I generally operate with CPU tensors in subprocesses, and move them to CUDA in the main process. What is confusing is that in this scenario <code>fork</code> actually works, at least for a very trivial example. With a more complex use case it hangs. At the same time <code>fork</code> is the default in Linux. So, it is very easy to run into a hard to debug issue by just setting <code>num_workers</code> to a positive value in the <code>DataLoader</code> constructor, without realising what you are doing wrong.</p>
<p>For example, <a href="https://docs.pola.rs/user-guide/misc/multiprocessing/" rel="noopener nofollow ugc">polars</a> is raising a warning in a similar scenario. Wonder if pytorch should raise a warning, too, or maybe even an error.</p>
I meet 3 API-calls not listed in the NVIDIA-DOC

shysuen001: <p>When I used the ltrace command to trace the PyTorch training process of ResNet50, I encountered 3 API-calls that were not listed in the NVIDIA documentation. So what these API-calls are for?</p>
<p>NVIDIA-DOC link:<a href="https://docs.nvidia.com/cuda/archive/12.1.1/cublas/index.html#cublaslt-api-reference" class="inline-onebox" rel="noopener nofollow ugc">CUDA cuBLAS</a></p>
<p>ltrace command: ltrace -f -x ‘cu*’ -x ‘__cu*’ -n 4 -o outputfile.txt  -tt python resnet50.py</p>
<p>3 API-calls:<br>
cublasLtCtxInit<br>
cublasLtSSSMatmul<br>
cublasLtSSSMatmulAlgoGetHeuristic</p>
How to output weight

bille_du: <p>I need to know all the weight values，How can I output the weight of the training process？</p>
<blockquote>
<p>criterion = nn.CrossEntropyLoss().cuda()</p>
</blockquote>
<blockquote>
<p>optimizer = torch.optim.SGD(model.parameters(), args.lr,<br>
momentum=args.momentum,<br>
weight_decay=args.weight_decay)</p>
</blockquote>
<blockquote>
<p>if args.evaluate:<br>
<code>validate(val_loader, model, criterion)</code></p>
</blockquote>
<blockquote>
<p>for epoch in range(args.start_epoch, args.epochs):<br>
adjust_learning_rate(optimizer, epoch)</p>
</blockquote>
<blockquote>
<pre><code>    # train for one epoch
train(train_loader, model, criterion, optimizer, epoch)
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # evaluate on validation set
prec1 = validate(val_loader, model, criterion)
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # remember best prec@1 and save checkpoint
is_best = prec1 &gt; best_prec1
best_prec1 = max(prec1, best_prec1)
save_checkpoint({
    'epoch': epoch + 1,
    'state_dict': model.state_dict(),
    'best_prec1': best_prec1,
}, is_best, filename='/home/zl/models/checkpoint_{}.pth.tar'.format(epoch))
</code></pre>
</blockquote>

colesbury: <pre><code class="lang-auto">for param in model.parameters():
  print(param.data)
</code></pre>

bille_du: <p>Thank you very much!<br>
If I want to output the output value(x) after the first layer convolution，not parameters ,what should I do？</p>

xwgeng: <p>You can solve it with the method <a href="http://pytorch.org/docs/nn.html?highlight=hook#torch.nn.Module.register_forward_hook" rel="nofollow noopener">register_forward_hook</a></p>

bille_du: <p>Very grateful, but the function of the official website is very simple to introduce, there is no example, I tried it or did not use it。Can you give me an example?</p>

xwgeng: <p>look at <a href="http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks" rel="nofollow noopener">Forward and Backward Function Hooks</a></p>

anirvirnn_mishra: <p>you can use:<br>
model.layername.weight<br>
or<br>
model.layername.weight.data</p>
<p>in second one you can also change the weights <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

guen_gn: <aside class="quote no-group" data-username="anirvirnn_mishra" data-post="9" data-topic="2796">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anirvirnn_mishra/48/26299_2.png" class="avatar"> anirvirnn_mishra:</div>
<blockquote>
<p>model.layername.weight</p>
</blockquote>
</aside>
<p>Sorry, I wanted to revive and chip in:<br>
How to print dimensions of particular layer, that is length of weight array in each axis?<br>
torch.nn.parameter.Parameter</p>

ptrblck: <p>You can print the shape of the <code>weight</code> parameter via <code>print(model.layername.weight.shape)</code>.</p>

Chicago_Girl: <p>why this command gives me the initial value every time?</p>
Average tensor unbind times vary drastically depending on implementation of unrelated code flows

JPK314: <p>I am running into a strange issue in my code base. The code is complicated but essentially I was noticing that calling unbind(0) on tensors took longer in one commit vs another, and it was noticeably slowing down performance. The old commit has an average per-call runtime of 8.727e-5, while the new commit has an average per-call runtime of  2.36e-4.</p>
<p>I was playing around with unbinding and I noticed that something seems to be getting cached… subsequent calls to unbind(), even on different tensors, seems to be consistently faster than the first call, no matter which tensor is put first.</p>
<p>The new code could potentially be messing with whatever cache exists and making the unbind call slower, but I am uncertain how to build around this. Is it possible to preserve the cache somehow? Are there any devs who might have insight onto what’s going on here?</p>

ptrblck: <p>I don’t know how you are profiling your code and if you are using a GPU, but note that CUDA operations are executed asynchronously. If you are using host timers you would need to synchronize the code before starting and stopping the host timers.</p>

Yougrianes: <p>try using pytorch profiler ?</p>

JPK314: <p>Yep, seems like the issue is due to unbind forcing a sync, and my code in the newer commit was faster making the forced sync require more waiting time. Thanks. I have tried avoiding this where possible (although it is worthwhile to note that calling unbind(0) is still faster than creating a view for each index of the tensor’s first dimension, at least for my use case)</p>
Does autocast create copies of tensors on the fly?

Aknw_Fen: <p><a href="https://pytorch.org/docs/stable/amp.html" rel="noopener nofollow ugc">From autocast docs</a>, it appears that:</p>
<ul>
<li>Ops will autocast within an autocast block - if the Ops support it.</li>
</ul>
<p>But tensors don’t change type, see example below, so I assume a copy must be created.</p>
<p>So, for example here:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

l  = torch.nn.Linear(10,12)
t = torch.randn((10,1))

with torch.autocast(device_type="cpu"):
  print(l(t.transpose(1,0)).type())
  print(next(l.parameters()).type())
</code></pre>
<p>The result of appplying the linear layer <code>l</code> to the tensor <code>t</code> is in half or bhalf precision, but the parameters are all in float32 (default) unless one explicitly sets them to other dtype.</p>
<ul>
<li>
<p>So my simple question is: does it mean the tensors (input, weight and bias) are “casted” for the operations within the layer, but do not change the original tensor and hence they are a new copy created on the fly?</p>
</li>
<li>
<p>I’m assuming also that tensors using <code>int64</code> by default won’t be autocasted? (only Ops are mentioned in docs.)</p>
</li>
<li>
<p>Would creating tensors directly in the desired type run <em>much</em> faster?</p>
</li>
</ul>

soulitzer: <blockquote>
<ul>
<li>So my simple question is: does it mean the tensors (input, weight and bias) are “casted” for the operations within the layer, but do not change the original tensor and hence they are a new copy created on the fly?</li>
</ul>
</blockquote>
<p>Yes. Autocast does have a cache that is enabled by default though to avoid creating extra copies, e.g. for example parameters.</p>
<blockquote>
<p>I’m assuming also that tensors using <code>int64</code> by default won’t be autocasted? (only Ops are mentioned in docs.)</p>
</blockquote>
<p>That is correct</p>
<blockquote>
<ul>
<li>Would creating tensors directly in the desired type run <em>much</em> faster?</li>
</ul>
</blockquote>
<p>It depends on what proportion of runtime is taken up by those copies</p>

Aknw_Fen: <p>Thanks; I realise about that after reading your comment in a different post.</p>
How does Forward-mode AD work behind the scenes in Pytorch?

Milk: <p>I am following this <a href="https://pytorch.org/tutorials/intermediate/forward_ad_usage.html#usage-with-modules" rel="noopener nofollow ugc">tutorial on Forward-mode AD</a>, particularly the “Usage with Modules” part. And I have looked into the <code>torch.autograd.forward_ad</code> file as well.</p>
<p>I was curious on how exactly the forward computations are happening. As in, how and where are the intermediate jvp’s getting computed? Is it something related to the implementations of <code>@staticmethod def jvp(ctx, gI):</code>? If so, where can I find those definitions for the <code>nn.module</code>’s of an arbitrary model?</p>

soulitzer: <p>Derivatives for forward mode (and reverse mode) are actually implemented at the operator level rather than the module level. You can find them both here <a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml" class="inline-onebox" rel="noopener nofollow ugc">pytorch/tools/autograd/derivatives.yaml at main · pytorch/pytorch · GitHub</a></p>
<p>When writing a custom autograd Function, <code>@staticmethod def jvp(ctx, gI)</code> is what you can use to define the jvp rule.</p>
Generate a sub-2D tensor by reference

weitaoliu: <p>Dear all,</p>
<p>I would like to inquire whether it is possible to generate a sub-matrix (2D, torch::Tensor) from a large matrix (torch::Tensor) using a list of row and column indices, such that the sub-matrix serves as a reference to the original matrix rather than a copy. Specifically, I am looking for a solution where any changes made to the sub-matrix directly reflect in the original matrix.</p>
<p>Best, Weitao.</p>

ptrblck: <p>Yes, this is possible if a view can be created as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.zeros(8, 8)
b = a[2:4, 2:4]
b.fill_(1.)

print(a)
# tensor([[0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 1., 1., 0., 0., 0., 0.],
#         [0., 0., 1., 1., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>However, not all passed indices can create views:</p>
<pre data-code-wrap="python"><code class="lang-python">c = a[[0, 4, 7], [0, 2, 3]]
c.fill_(2.)

print(a)
</code></pre>
Autograd for duplicated index put operation

skpighhh: <p>I’m confused about index put operation when given indices are duplicated. Here is a code snippet.</p>
<pre><code class="lang-auto">import torch

torch.manual_seed(0)
mapping = torch.randn((3,3))

a = torch.ones((3), requires_grad=True)
index = torch.tensor([0,1,1,1])

# Version 1
b = a @ mapping
b[index] += 1
print("index put with duplicated indices", b)
loss = b.sum()
loss.backward()
print("a.grad", a.grad)

# Version 2
a.grad = None
b = a @ mapping
b[torch.unique(index)] += 1
print("index put with unique indices", b)
loss = b.sum()
loss.backward()
print("a.grad", a.grad)
</code></pre>
<p>The outputs are:</p>
<pre><code class="lang-auto">index put with duplicated indices tensor([ 3.5128,  0.4601, -4.2966], grad_fn=&lt;IndexPutBackward0&gt;)
a.grad tensor([-1.5181, -4.0837,  2.1982])
index put with unique indices tensor([ 3.5128,  0.4601, -4.2966], grad_fn=&lt;IndexPutBackward0&gt;)
a.grad tensor([-0.9312, -1.9147,  0.5221])
</code></pre>
<p>So the outputs of these two index put operations are identical, while the gradients are different. I’m confused about this phenomenon.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.index_put_.html#torch.Tensor.index_put_">docs</a>:</p>
<blockquote>
<p>If <code>accumulate</code> is <code>True</code>, the elements in <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.values.html#torch.Tensor.values"><code>values</code></a> are added to <code>self</code>. If accumulate is <code>False</code>, the behavior is undefined if indices contain duplicate elements.</p>
</blockquote>
<p>Your current code depends on undefined behavior, so use:</p>
<pre data-code-wrap="python"><code class="lang-python">a.grad = None
b = a @ mapping
b.index_put_((index,), torch.tensor(1.), accumulate=True)
print("index put with unique indices", b)
loss = b.sum()
loss.backward()
print("a.grad", a.grad)
</code></pre>
<p>instead to accumulate the duplicated indices.</p>
Export to ONNX: a better story

Peter_Featherstone: <p>What are the better ways to convert models to ONNX?<br>
I usually do:</p>
<ul>
<li>trace model with torch.jit.trace()</li>
<li>use traced model with torch.onnx.export()<br>
This almost never works out the box with dynamic shapes.<br>
I have to decorate certain functions with <span class="mention">@torch.jit.script</span>.<br>
I have to remove certain libraries like einops and other stuff.<br>
I have to refactor a bunch of layers. For example, torch.stft,  nn.TransformerEncoderLayer are not onnx-export able. The list of unsupported stuff is very long.<br>
How do we practically export stuff? Are there a set of guides or libraries we can use to help us?</li>
</ul>

ionymikler: <p>Right now I am having issues because I need to export a model to ONNX and it has a bunch of einops operation. What approach has worked for you to rewrite this lines?</p>
Installing from source error about third party eigen library

furkandurmus: <p>Hi. I need to install pytorch via source. However, there is a problem about the eigen third party library, I checked the similar issues and suggested <code>git submodule update --init</code> still doesnt work.</p>
<p>Here is the error message. Any suggestion would be appreciated.</p>
<pre><code class="lang-auto">error: RPC failed; HTTP 524 curl 22 The requested URL returned error: 524
fatal: the remote end hung up unexpectedly
fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path '/home/furkan/pytorch/third_party/eigen' failed
Failed to clone 'third_party/eigen' a second time, aborting
</code></pre>

ptrblck: <p><code>524</code> seems to be a timeout error. Are you able to reach the URL in your browser or manually <code>git clone</code> it? I cannot reproduce the issue locally so would guess that either the repository hosting had a few issues or your setup is timing out.</p>

jingshengL: <pre><code class="lang-auto">https is blocked, use git instead.

vi .git/config
vi .gitmodules

changge
    url = https://gitlab.com/libeigen/eigen.git
to
    url = git@gitlab.com:libeigen/eigen.git
</code></pre>
Looking to Contribute: Understanding PyTorch Architecture/Framework

Emmett_Bicker: <p>Hi everyone,</p>
<p>I’m Emmett and I would love to contribute to PyTorch! Before I start I would love to have a strong understanding of the architecture PyTorch is built around. I’ve read the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html" rel="noopener nofollow ugc">PyTorch contribution guide</a> and its linked resources, but I can’t find a singular resource to fully understand the architecture and how the codebase works from a contributor’s perspective.</p>
<p>I would greatly appreciate any resources, advice, or other kinds of help :).</p>

soulitzer: <p>Some resources that might be useful (if you haven’t found them already) are:</p>
<p><a href="https://pytorch-dev-podcast.simplecast.com/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-dev-podcast.simplecast.com/</a><br>
<a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/</a><br>
<a href="http://blog.ezyang.com/2019/05/pytorch-internals/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://blog.ezyang.com/2019/05/pytorch-internals/</a></p>
Adding softmax layer with crossentropy

mathwseg: <p>If i used this method for extract the features with loss = CrossEntropy . Should i add softmax layer but this way ?</p>
<pre><code class="lang-auto">
def get_probabilities(outputs):
    return F.softmax(outputs, dim=1)  
</code></pre>
<pre><code class="lang-auto">
class First (nn.Module):
    def __init__(self, modelA, modelB, 5):
        super(First, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        self.modelA.heads = nn.Identity()
        self.modelB.head. = nn.Identity()
        self.classifier = nn.Linear(1024 + 768, 5)

    def forward(self, x):
        x1 = self.modelA(x.clone())
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        x = self.classifier(F.relu(x))
        return x
</code></pre>
<p>what are the conditions that i need to apply the softmax layer and where ?</p>

soulitzer: <p>The CrossEntropy loss takes the unnormalized logits <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 2.5 documentation</a> and will apply the softmax for you - you should not apply softmax before yourself</p>
Unable to update a latent vector using custom loss function

Jimut123: <p>I am trying to implement this function but have had no luck. There is a VAE model that I am using, and along with it, there are encoder and decoder. I’m freezing the weights of the VAE decoder, and trying to change a latent vector which is updated using the function <em><strong>optimize_latent_vector(model, inp__, num_epochs=50, learning_rate=0.01)</strong></em>. Now, there is some error regarding this piece of code: <em><strong>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</strong></em></p>
<pre data-code-wrap="py"><code class="lang-py">
class VAE_GD_Loss(nn.Module):
    def __init__(self):
        super(VAE_GD_Loss, self).__init__()

    def forward(self, bad_seg, recons_mask, vector):
        # l2 normed squared and the soft dice loss are calculated
        loss = torch.sum(vector**2)+Soft_Dice_Loss(recons_mask, bad_seg)
        return loss

def optimize_latent_vector(model, inp__, num_epochs=50, learning_rate=0.01):
    inp__ = inp__.to(device).requires_grad_(True)
    # Encode and reparameterize to get initial latent vector
    with torch.no_grad():
        mu, log_var = model.encoder(inp__)
        z_latent_vect = model.reparameterize(mu, log_var)
    optimizer_lat = torch.optim.Adam([z_latent_vect], lr=learning_rate)
    dec_only = model.decoder
    
    for epoch in range(num_epochs):
        optimizer_lat.zero_grad()
        dec_only.eval()
        # Decode from latent vector
        recons_mask = dec_only(z_latent_vect)
        # Calculate loss
        VGLoss = VAE_GD_Loss()
        loss = VGLoss(inp__, recons_mask, z_latent_vect)
        # loss = Variable(loss, requires_grad=True)
        # Backpropagation
        loss.backward()
        optimizer_lat.step()
        print(f"Epoch {epoch}: Loss = {loss.item()}")
    
    return z_latent_vect
</code></pre>
<p>If we uncomment the line <em><strong>loss = Variable(loss, requires_grad=True)</strong></em>, then the code runs, but it doesn’t minimize the loss whatsoever. I want to update the latent vector in such a way so that it follows the constraint set in the loss function. Any leads would help!</p>

soulitzer: <p>Typically how you debug these situations is to bisect your model and figure out where the model is first disconnected.</p>
<p>From reading your code, it looks like z_latent_vect is the output of an operation in no-grad mode, which means it will have requires_grad=False. Does making that requires_grad=True fix things.</p>
<p>(also you shouldn’t use “Variable”, it’s deprecated - the equivalent code here is .requires_grad_(True)).</p>