akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Wonders of how to use flex attention

Haoran_Cheng: <p>Hi there, we may encounter an issue of using flex attention version <a href="https://download.pytorch.org/whl/nightly/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu118</a>.<br>
We created sliding window with<br>
generate_sliding_window(window_size=self.window_size)<br>
and attention output computation with:<br>
y = torch.nn.attention.flex_attention.flex_attention(q, k, v, block_mask=block_mask).<br>
However, when we measure overall gpu memory use and compare with manual implementation of sliding-window mask, flex attention doesn’t show improvement in running speed:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2a97aae156c6bfab57a083e5d2d02091ec4395.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6b2a97aae156c6bfab57a083e5d2d02091ec4395" title="Screen Shot 2024-10-30 at 11.22.45 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2a97aae156c6bfab57a083e5d2d02091ec4395_2_690x266.jpeg" alt="Screen Shot 2024-10-30 at 11.22.45 PM" data-base62-sha1="fi2lhqCdbGOfGCblfyZdTuelanz" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2a97aae156c6bfab57a083e5d2d02091ec4395_2_690x266.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2a97aae156c6bfab57a083e5d2d02091ec4395_2_1035x399.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2a97aae156c6bfab57a083e5d2d02091ec4395_2_1380x532.jpeg 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2024-10-30 at 11.22.45 PM</span><span class="informations">1494×578 48.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
wondering how we make better use of flex attention cuz it looks really useful in utilizing sparsity.</p>

ss42: <p>Did you ever resolve this?</p>
How to do instance segmentation and keypoint detection at the same time?

Rafa: <p>Hi all,</p>
<p>I’d like to train a model capable of doing both instance segmentation and keypoint detection at the same time. Is there anything available off the shelf right now?</p>
<p>If not, how could I implement it myself? Torchvision has <a href="https://pytorch.org/vision/main/models/mask_rcnn.html" rel="noopener nofollow ugc">Mask-RCNN</a> for instance segmentation and <a href="https://pytorch.org/vision/main/models/keypoint_rcnn.html" rel="noopener nofollow ugc">Keypoint-RCNN</a> for keypoint detection. How could I, for example, extract the heads of Keypoint-RCNN and add it to a parallel branch of Mask-RCNN? And then train for a new case</p>

PremRaj_Kala: <pre><code class="lang-auto">
#Example code

from torchvision.models.detection import maskrcnn_resnet50_fpn
from torchvision.ops import MultiScaleRoIAlign
from torchvision.models.detection.keypoint_rcnn import KeypointRCNNHeads, KeypointRCNNPredictor
import torchvision


def get_model(num_classes=2,num_keypoints=2):
    model  = maskrcnn_resnet50_fpn(pretrained=True)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes)  # 1 class + background
    
    
    model.roi_heads.keypoint_roi_pool = MultiScaleRoIAlign(featmap_names=["0", "1", "2", "3"], output_size=14, sampling_ratio=2)
    keypoint_layers = tuple(512 for _ in range(8))

    out_channels = model.backbone.out_channels
    model.roi_heads.keypoint_head = KeypointRCNNHeads(out_channels, keypoint_layers)

    keypoint_dim_reduced = 512  # == keypoint_layers[-1]
    model.roi_heads.keypoint_predictor = KeypointRCNNPredictor(keypoint_dim_reduced, num_keypoints)

    return model
</code></pre>
How to get data on the same device as fsdp model during training?

tcm: <p>Hi,</p>
<p>I’m currently learning how to use FSDP to shard model across gpus, and I have a toy example as follows where I shard a simple Multilayer Perceptron using <code>FullyShardedDataParallel</code>:</p>
<pre><code class="lang-auto">if __name__ == "__main__":

    torch.cuda.set_device(torch.cuda.current_device())
    rank = int(os.environ["RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    dist.init_process_group(backend = "nccl")

    model = MLP()
    fsdp_model = FullyShardedDataParallel(
        model,
        device_id = rank,
        auto_wrap_policy = size_based_auto_wrap_policy,
        cpu_offload = CPUOffload(offload_params = True)
    )
    optim = torch.optim.Adam(fsdp_model.parameters(), lr = 0.0001)
    custom_dataset = gen_data(100)
    dataloader = DataLoader(custom_dataset, batch_size = 10, shuffle = True)
    device = next(fsdp_model.parameters()).device
    for i in range(NUM_EPOCHS):
        fsdp_model.train()
        for batch_idx, (ts_batch, label_batch) in enumerate(dataloader):
            ts_batch = ts_batch.to(device)
            label_batch = label_batch.to(device)
            optim.zero_grad()
            logits = fsdp_model(ts_batch)
            loss = F.cross_entropy(logits, label_batch)
            loss.backward()
            optim.step()
</code></pre>
<p>My cluster environment is single-node with 2 A100s, so I use:</p>
<pre><code class="lang-auto">torchrun --nproc_per_node=2 --nnodes=1 train.py
</code></pre>
<p>My first question is that when I pass <code>device_id</code> arg in <code>FullyShardedDataParallel()</code> as <code>torch.cuda.current_device()</code>, the error <code>Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 7000</code> occurs so then I set it to the rank which solves the error. The <a href="https://pytorch.org/docs/stable/fsdp.html" rel="noopener nofollow ugc">FSDP doc</a> states that this arg specifies the cuda device to initialize model, shard parameters, etc., and they suggest that we may pass <code>torch.cuda.current_device</code>, but it results in the error above. When I print out both the rank and <code>torch.cuda.current_device</code>:</p>
<pre><code class="lang-auto">In rank 0: cuda.current_device: 0
In rank 1: cuda.current_device: 0
</code></pre>
<p>Thus, I’m confused why <code>torch.cuda.curren_device()</code> returns 0 for both ranks even if they are in the same node. And for the FSDP definition, is passing <code>torch.cuda.current_device</code> to <code>device_id</code> arg correct?</p>
<p>For the second question, before the training loop, I attempted to get the device via:</p>
<pre><code class="lang-auto">device = next(fsdp_model.parameters()).device
</code></pre>
<p>and in the body of training, I move the data tensors to this device. But it turns out that this device is just <code>cpu</code>, and only after the forward pass, the device of <code>logits</code> is either <code>cuda:0</code> or <code>cuda:1</code>. But to correctly transfer the data tensors to the same device as <code>fsdp_model</code>, shouldn’t we need the cuda device first?</p>
<p>Thanks!</p>
AttributeError: 'Image' object has no attribute 'shape'

Mert_Arda_Asar: <p>I tried to apply RandomErase transform for every images using the following code</p>
<pre><code class="lang-auto">
my_transforms = transforms.Compose([

  transforms.Resize((192,192)),

  transforms.RandomErasing(p=1),

  transforms.ToTensor(),

  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))                                    

])

dataset = datasets.ImageFolder(root=DATA_PATH, transform=my_transforms)
</code></pre>
<p>I tried to display first image from the dataset using following line</p>
<p><code>plt.imshow(dataset[0][0].permute(1,2,0))</code></p>
<p>And I get this error</p>
<p><code>AttributeError: 'Image' object has no attribute 'shape'</code></p>

mxahan: <p>You need to send the</p>
<pre><code class="lang-auto">dataset = datasets.ImageFolder(root=DATA_PATH, transform=transform)
data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, drop_last=True, **kwargs)
</code></pre>
<p>You can iterate over the dataset to find the instances</p>
<pre><code class="lang-auto">image_sample = next.itere(data_loader)
</code></pre>

Mert_Arda_Asar: <p>Still getting the same error. If I remove transforms.RandomErasing(p=1) from my transforms it works. But I need to put random erasing each image.</p>

Andrei_Cristea: <p>RandomErasing is a transform on a Tensor, not an Image, per <a href="https://pytorch.org/vision/stable/transforms.html#transforms-on-torch-tensor-only" rel="noopener nofollow ugc">the docs</a>.</p>
<p>You should try again with the transforms in this order:</p>
<pre><code class="lang-auto">transforms.ToTensor(),
transforms.RandomErasing(p=1),
</code></pre>
<p>Hope this helps!</p>

Mert_Arda_Asar: <p>Yes, that works. Thanks a lot.</p>

Ziaul_Hoque: <p>thanks man, it worked, but why it worked and why got the error didn’t understood</p>

Andrei_Cristea: <p>A transform is a function, and different functions expect different kinds of inputs.</p>
<p>An image can be represented as either an object of type Image (in your CPU memory) or Tensor (which can be sent to GPU memory for fast operations). That particular transform expects a Tensor, not an Image. So, if you’re starting off with an Image, you must first convert it to a tensor via transforms.ToTensor before you can pass it to RandomErasing.</p>

Ziaul_Hoque: <p>thanks man,<br>
I appreciate your help, i was stacked on that problem now all is good, thanks to you.</p>
Pytorch is slower when with multiple threads ( on CPU)

Rojin: <p>I noticed that pytorch is slower when I set the number of threads to more than 1 (on cpu) with the following line of code:</p>
<pre><code class="lang-auto">torch.set_num_threads(30)
</code></pre>
<p>I was wondering if anyone has had the same issue?</p>

Skalwar: <p><a class="mention" href="/u/rojin">@Rojin</a> how many cores do your cpu have??<br>
To know maximum number of threads that your pc can run simultaneously use this formula:-<br>
Number of threads = number of cores * 2</p>

mailcorahul: <p>The reason could be you use more number of threads(in your case, 30) than the actual number of threads that can be processed parallely by your machine.<br>
Do</p>
<pre><code class="lang-auto">nproc
</code></pre>
<p>which gives the Core(s) per socket * Thread(s) per core<br>
Pass that value to torch.set_num_threads().</p>
<p>You can also use torch.get_num_threads() to get the number of threads which will be used for parallelizing.<br>
P.S: Not sure how to interpret the output given by torch.get_num_threads().</p>

Rojin: <p><a class="mention" href="/u/mailcorahul">@mailcorahul</a> <a class="mention" href="/u/skalwar">@Skalwar</a></p>
<p>Thanks for your responses. I have 56 cores. which mean 112 threads.  I set the number of threads to 112, and it defiantly slowed down the speed compared to using only 1 thread</p>

Ramy_Nagah_Eskander: <p>I am facing the same dilemma, Have you got to know the reason behind this?</p>

khalil: <p>Firstly, you must confirm there are many threads which run on the different CPUs.</p>
<p>Then you should be attention to  memory latency, if your machine is based on numa.</p>

gitgithan: <p>In <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#avoid-cpu-oversubscription" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 2.6 documentation</a>, it talks about <code>torch.set_num_threads(floor(N/M))</code></p>
<p>If i understand the article right,<br>
N = os.cpu_count()  # Number of vCPUs available on the machine<br>
M = args.num_processes  # Number of processes to use, passed as argument</p>
<p>Is it better to have small M less processes but more threads per process, or big M more processes but less threads per process?</p>
<p>How does number of process or number of threads per process affect data loading and subsequent augmentation/computation speed?</p>
<p>If computer has:<br>
Thread(s) per core:                 2<br>
Core(s) per socket:                 12</p>
<p>Should M value be 12? (Determined by Core(s) per socket)</p>
<p>Is M = 11 wasting 2 threads per core?<br>
I assume the 2 threads per core from the 12th core cannot be re-assigned to the other 11 cores, thus wastage.<br>
Or is this assumption debunked by this “thread-migration” concept mentioned in <a href="https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/os-thread-migration.html" rel="noopener nofollow ugc">https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/os-thread-migration.html</a> ?</p>
<p>Is M = 13 impossible? (If the code doesn’t break, what is happening under the hood?)</p>
TorchServe Cannot Find Files in Subfolders Inside .mar File

natalia_meira: <p>I have a model converted to TorchScript and generated a <code>.mar</code> file to upload with TorchServe in a container. My model requires several files that are organized in subfolders. These subfolders are included inside my <code>.mar</code> file. However, when I run TorchServe, it cannot find the files located in the subfolders.</p>
<p>How can I resolve this issue?</p>
USing Quantization tutorial,but the result different

BambooKui: <p>here is web:<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.6.0+cu124 documentation</a><br>
In the final speed - testing section, the tutorial mentions:<br>
“Running this locally on a MacBook Pro yielded 61 ms for the regular model, and just 20 ms for the quantized model, illustrating the typical 2 - 4x speedup we see for quantized models compared to floating - point ones.”<br>
However, when I ran it, I got 15 ms for the regular model and 30 ms for the quantized model. The quantized model runs slower.<br>
Why? I’m asking for help.</p>

anantguptadbl: <p><a class="mention" href="/u/bambookui">@BambooKui</a> theoretically this is possible in the following scenarios</p>
<ul>
<li>You have almost exhausted your machine memory and when you ran the prediction on the quantized model, the memory bottleneck kicked in</li>
</ul>
<p>In order to test this hypothesis</p>
<ul>
<li>Save both the models in separate files</li>
<li>Terminate the python session</li>
<li>Open another session, load just the quantized model and check the benchmark timings</li>
</ul>

BambooKui: <aside class="quote no-group" data-username="anantguptadbl" data-post="2" data-topic="216178">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anantguptadbl/48/17784_2.png" class="avatar"> anantguptadbl:</div>
<blockquote>
<p>theoretically this is possible in the following scenarios</p>
<ul>
<li>You have almost exhausted your machine memory and when you ran the prediction on the quantized model, the memory bottleneck kicked in</li>
</ul>
<p>In order to test this hypothesis</p>
<ul>
<li>Save both the models in separate files</li>
<li>Terminate the python session</li>
<li>Open another session, load just the quantized model and check the benchmark timings</li>
</ul>
</blockquote>
</aside>
<p>Thank you for your reply. I ran the inference part in a separate process. The quantized model was twice as fast, which was faster than the non - quantized model, but the improvement wasn’t as significant as described in the tutorials.</p>
Data types on quantized models

watchmexplode: <p>I’m currently running some customized conv2d layers of a ResNet18 quantized model. My goal is just to express the convolution in terms explicit matrix multiplication.</p>
<p>For the first convolution layer, for example, I notice that the input tensor is of type torch.uint8, and the filters are of type torch.int8. The output of the function ops.quantized.conv2d(input_tensor, conv params) are of type torch.uint8.</p>
<p>Given that a convolution is essentially the product of inputs (type  torch.uint8) by the filters (type torch.int8), I don’t understand how this computation is performed. I would expect the output to be of a signed type with at least 16 bits. Why, so, the outputs are of type torch.uint8?</p>
Hooks, dynamo, and unsupported builtins

sternj: <p>Hello!</p>
<p>I have some functions that I’d like to attach as pre/post hooks to instances of Module, but when I do so, I get “UserWarning: Graph break due to unsupported builtin”. I’d like to be able to use these hooks for forward_pre, forward, full_backward_pre, and full_backward. None of the code in the callbacks should be analyzed by the compiler, but the calls can’t be optimized out.</p>
<p>How would I go about this?</p>

sternj: <p>What ended up seeming to work is decorating the hooks with <code>@torch._dynamo.disable</code>. Is there any reason not to use this approach?</p>
<p>For reference, I want to be able to execute arbitrary C code in these hooks.</p>

bdhirsh: <p>Marking the hooks with disable is probably reasonable (it accomplishes your goal of running the custom c-bound code in eager mode while compiling other parts of your model)</p>
<p>If you’re willing to do a bit more work to make your model compile friendly, you can wrap your custom c function into a PyTorch custom op as per this guide: <a href="https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Custom Operators — PyTorch Tutorials 2.6.0+cu124 documentation</a></p>
<p>This will avoid graph breaks and generally improve performance (and can open the doors to eg exporting your model and running in a no-python environment if that’s something you want)</p>
Torch.compile: Generated Triton kernel seems wrong

Shihab_Shahriar: <p>My original code:</p>
<pre><code class="lang-auto">import torch


def square(x):
    x = torch.square(x)
    if x.sum() &gt; 0:
        x = x-1
    return torch.square(x)


opt_square = torch.compile(square)
opt_square(torch.randn(10000, 10000).cuda())
</code></pre>
<p>generated triton kernel:</p>
<pre><code class="lang-auto">@triton.jit
def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
    xnumel = 100000000
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:]
    xmask = xindex &lt; xnumel
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (x0), xmask)
    tmp1 = 1.0
    tmp2 = tmp0 - tmp1
    tmp3 = tmp2 * tmp2
    tl.store(out_ptr0 + (x0), tmp3, xmask)
''', device_str='cuda')
</code></pre>
<p>That is, for given input:</p>
<pre><code class="lang-auto">torch: (x**2-1)**2
triton: (x-1)**2
</code></pre>

ptrblck: <p>I cannot reproduce an issue on a 3090 with a current nightly binary:</p>
<pre data-code-wrap="python"><code class="lang-python">def square(x):
    x = torch.square(x)
    if x.sum() &gt; 0:
        x = x-1
    return torch.square(x)


opt_square = torch.compile(square)
x = torch.randn(10000, 10000).cuda()
out = opt_square(x)
ref = square(x)

print((out - ref).abs().max())
# tensor(0., device='cuda:0')
</code></pre>

Shihab_Shahriar: <p>I dont see any issue either using your code on my 4060. But I generated and checked triton again. It’s same as above. Could be something silly.</p>

Shihab_Shahriar: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edd1ae3c77e2745e7c34ee89bf425c852509e7c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/edd1ae3c77e2745e7c34ee89bf425c852509e7c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edd1ae3c77e2745e7c34ee89bf425c852509e7c6_2_690x202.png" alt="image" data-base62-sha1="xVQoKgioy1nje2qyYBH7mg7ThMG" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edd1ae3c77e2745e7c34ee89bf425c852509e7c6_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edd1ae3c77e2745e7c34ee89bf425c852509e7c6_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edd1ae3c77e2745e7c34ee89bf425c852509e7c6_2_1380x404.png 2x" data-dominant-color="3C1830"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1753×515 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Shihab_Shahriar: <p>Found it: It’s generating multiple files and kernels and I was only looking at one.</p>
Using libtorch C++ on Windows, MKL Installed

RickF: <p>Hi,</p>
<p>On one Windows machine, I run fine.</p>
<p>On another, I get a linker error because I have MKL installed.</p>
<p>My project uses CMake. It seems like find_package(Torch REQUIRED)<br>
is picking up the MKL I have installed…</p>
<p>Is there a workaround for this?</p>
<p>Thanks</p>
Optimizer.step() calls torch.cuda.synchronize()

eliird: <p>I want to ask a question which might seem a bit stupid. So during training each iteration would need updated model after calling <code>optimizer.step()</code> function. But dont we need to call <code>torch.cuda.synchronize</code> at end of each iteration to ensure that the weights have been updated or is it called internally somewhere especially in case of distributed training. or is it called implicitly? i looked at the opitmizer codes but did not find synchronization call anywhere.</p>
<pre data-code-wrap="python"><code class="lang-python">
for i in range(iterations):
    model.train()  # Set model to training mode

    # Generate random data: 32 samples, each with 10 features
    inputs = torch.randn(32, 10)
    targets = torch.randn(32, 1)  # Random target values

    # Zero the gradients from the previous step
    optimizer.zero_grad()

    # Forward pass
    outputs = model(inputs)

    # Compute loss
    loss = criterion(outputs, targets)

    # Backward pass (calculate gradients)
    loss.backward()

    # Update model weights
    optimizer.step()

    # synchronize
    torch.cuda.synchronize()
</code></pre>

ptrblck: <p>No, you don’t need to synchronize your code. PyTorch enqueued the compute kernels to the default CUDAStream which makes sure kernels are executed in order.</p>

eliird: <p>Thank you for reply, I understand we do not have to call it explicitly but somewhere it needs to happen right if so where? or am I mistaken.</p>

ptrblck: <p>No, you don’t need to synchronize the code at all unless you want to read an output value. In this case PyTorch will synchronize the code for you and want until the GPU is done with all compute kernels.<br>
You can think about the general workflow (without implicit or explicit synchronizations) as the CPU scheduling CUDA kernels using data pointers only. No need to synchronize anything, since all kernels will be launched on the same CUDAStream and are thus guaranteed to not cause any race conditions. The <code>optimizer.step()</code> method will then enqueue the CUDA kernels needed for the actual weight update (again into the same stream). These kernels cannot be executed before the other already enqueued kernels are finished.</p>
<p>This story differs if you are using custom CUDA streams, as you would be responsible for the synchronizations, since multiple kernels can overlap on the GPU.</p>
Torch.compile triggers anomaly detection

Nava_Schulmann: <p>I’m setting <code>trainer.detect_anomaly</code> to <code>False</code>, but when switching from eager training to <code>torch.compile</code> with the default backend, I receive the following warning:</p>
<p>UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.</p>
<p>Has anyone else encountered this?</p>

bdhirsh: <p>I haven’t been seeing this locally. Does this repro for you on a recent nightly?</p>
The difference between `torch.ops.aten.addmm.default` and `torch.ops.aten.linear.default`

111357: <p>Given a <code>torch.nn.Linear</code> module, I noticed that the <code>aot_autograd</code> transform the torch IR captured by dynamo into  <code>torch.ops.aten.addmm.default</code>, while the <code>pre_dispatch_eager</code> backend transform it into <code>torch.ops.aten.linear.default</code>.</p>
<p>I suspect this might have something to do with the <code>pre_dispatch</code>, but I am not full understand the concept of <code>pre_dispatch</code> and how it work. Could someone provide some guidance on where to find documentation or code examples that explain <code>pre_dispatch</code> in more detail?</p>
<p>Thank you very much for any help!</p>
<p>May related owner <a class="mention" href="/u/ezyang">@ezyang</a> <a class="mention" href="/u/chillee">@Chillee</a> <a class="mention" href="/u/bdhirsh">@bdhirsh</a> (please ignore it if not <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> )</p>
<pre data-code-wrap="python"><code class="lang-python">from typing import List
from torch._dynamo.backends.common import aot_autograd
import torch
torch._dynamo.config.suppress_errors = True


class Bar(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(5, 10)
        self.fc2 = torch.nn.Linear(5, 10)

    def forward(self, inputs):
        a = self.fc1(inputs)
        b = self.fc2(inputs)
        return a * b


# Use pre_dispatch_eager
user_model = Bar()
compiled_func = torch.compile(user_model, backend="pre_dispatch_eager")
print(compiled_func)

inp1 = torch.randn(1, 5)
inp2 = torch.randn(1, 5) * -1
out = compiled_func(inp1)
out2 = compiled_func(inp2)
print(out)
print(out2)


# Use aot_autograd
def custom_backend(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]):
    print("custom backend called with FX graph:")
    gm.print_readable()
    return gm.forward


aot_backend = aot_autograd(fw_compiler=custom_backend)

compiled_func = torch.compile(Bar(), backend=aot_backend)
print(compiled_func)

inp1 = torch.randn(1, 5)
inp2 = torch.randn(1, 5) * -1
out = compiled_func(inp1)
out2 = compiled_func(inp2)
print(out)
print(out2)


</code></pre>

GodHforever: <p>i am also curious about this.<br>
i also noticed that if linear’s bias is none, the transformed ir is <code>aten.mm</code> , which is different with eager mode.</p>

bdhirsh: <p>Hopefully this answers the question: <a href="https://discuss.pytorch.org/t/decomposition-to-aten-ir/216033/2" class="inline-onebox">Decomposition to Aten IR - #2 by bdhirsh</a>, but let me know if you want any more detail</p>
Replace torch.rfft() code (PyTorch 1.6)

ptrch_c_m: <p>I want to use a repository that was built with PyTorch 1.6, with newer versions of PyTorch (preferably the latest). This repository uses</p>
<p><code>torch.rfft()</code></p>
<p>which was deprecated to <code>torch.fft.rfft</code> in later PyTorch versions.</p>
<p>The snippet of code that I want to replace is</p>
<pre><code class="lang-auto">v = torch.cat([x[:, ::2], x[:, 1::2].flip([1])], dim=1)
Vc = torch.fft.rfft(v, 1, onesided=False)
</code></pre>
<p>where <code>x</code> is a two dimensional tensor. In the comments around that piece of code, it is written that</p>
<blockquote>
<p>Discrete Cosine Transform, Type II (a.k.a. the DCT)</p>
</blockquote>
<p>I switched <code>torch.rfft()</code> with <code>torch.fft.rfft</code> but then I get the error</p>
<p><code>TypeError: fft_rfft() got an unexpected keyword argument 'onesided'</code></p>
<p>Does anyone know the exact equivalent of this code to the current PyTorch version?</p>
<p>Additionally, I have also found this line of code</p>
<p><code>v = torch.irfft(V, 1, onesided=False)</code></p>
<p>which also needs an update.</p>
<p>Thank you in advance.</p>

ptrch_c_m: <p>I found the equivalent code for that line. Since <code>oneseded=False</code>, the corresponding function to use is <code>torch.fft.fft</code>. Additionally, since PyTorch supports complex values, to produce the same output with that of the deprecated version, <code>torch.view_as_real</code> must be used. Below is the equivalent code of <code>Vc = torch.fft.rfft(v, 1, onesided=False)</code></p>
<pre><code class="lang-auto">Vc = torch.fft.fft(v)
Vc = torch.view_as_real(Vc)
</code></pre>

anantguptadbl: <p><a class="mention" href="/u/ptrch_c_m">@ptrch_c_m</a>  i found one reference post <a href="https://discuss.pytorch.org/t/fourier-transform-in-latest-pytorch/145848" class="inline-onebox">Fourier transform in latest pytorch</a></p>
<p>Very difficult to find a runtime env where we can install <code>torch==1.6.0</code> to verify the results unfortunately. It would be great if you can try the above article’s suggestions from your end</p>
How to create a DataLoader with nested tensors

Parijat_Das: <p>I am trying to use nested tensors to build my transformer block. However when I am trying to feed my data, which is converted into nested tensor from regular tensors, to the Dataloader and trying to iterate over it, I am getting the error: Internal error: NestedTensorImpl doesn’t support sizes. To solve this I have to revert back to the padded tensor, So how can I really take advantage of the nested tensors?</p>
FSDP clarifying questions

Samir_Char: <p>Hello everyone, I’ve been reading the FSDP paper and the tutorials and have a couple of questions I could use some help with.</p>
<p>My understanding is FSDP does the forward pass of each FSDP Unit, one at a time, then it does the backprop again one at a time, and finally it updates weights. Both in the forward and backward of a given FSDP Unit all the sharded parameters for that unit are fully materialized for a moment and then discarded when no longer needed.</p>
<ol>
<li>
<p>If all the weights of an FSDP unit are fully materialized, why even bother using a sharding factor F &gt; 1? Why can’t we just always replicate all the weights of the unit in all GPUs since we know it will fit?</p>
</li>
<li>
<p>Initially before the process begins are all shards stored in CPU and each FSDP unit is sent to the GPU one at a time and then back to CPU?</p>
</li>
<li>
<p>After backprop for all layers, how are the parameters of the shards updated? Does PyTorch iterate again across all shards and update one at a time in a loop?</p>
</li>
<li>
<p>Finally, the diagram that is shown in all the tutorials is not making sense for me. It looks like that the first FSDP Unit of layers 1:N is processed with forward and then backward, and then (all the way to the right of the image) the algorithm moves to the next FSDP Unit where it does forward and backward again. I don’t think this makes sense, since we need to do full forward of all layers first before doing backprop over all layers. Perhaps I’m misunderstanding something?</p>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e4f544ff3e53365869083460811a12c04e40f1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/7e4f544ff3e53365869083460811a12c04e40f1b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4f544ff3e53365869083460811a12c04e40f1b_2_690x311.png" alt="image" data-base62-sha1="i1o7FaL6Nn6lSIiJgsAV4LlM96j" width="690" height="311" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4f544ff3e53365869083460811a12c04e40f1b_2_690x311.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4f544ff3e53365869083460811a12c04e40f1b_2_1035x466.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4f544ff3e53365869083460811a12c04e40f1b_2_1380x622.png 2x" data-dominant-color="B4C8CC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">4372×1975 560 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks !!</p>
Explicit casting inside autocast context

avi_singhal: <p>Can we use explicit casts inside an autocast enabled context? i.e. input=input.to(torch.float32), input=input.to(torch.bfloat16). Based on the documentation here: <a href="https://pytorch.org/docs/stable/amp.html#:~:text=When%20entering%20an%20autocast%2Denabled%20region%2C%20Tensors%20may%20be%20any%20type.%20You%20should%20not%20call%20half()%20or%20bfloat16()%20on%20your%20model(s)%20or%20inputs%20when%20using%20autocasting" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.6 documentation</a>. , it seems like we cant use .half() and .bfloat16() on the inputs/models inside the enabled context. No reasoning for that is provided, I think the reasoning is that .half() and .bfloat16() creates a new tensor which is not tracked by autograd? and thats why it might be a problem during the backward as the wrong dtype might be used.</p>
<p>Based on my understanding, autocast works on a per op basis, so if I want a specific op which is a part of the autocast policy to execute in a different dtype, then it must be done in an autocast disabled context. However, if an op is not a part of the policy, then its fine to perform explicit casting using .to() . Could anyone confirm if this understanding is correct? There is very little documentation related to the effects of explicit casting in autocast context.</p>

ptrblck: <p>Disable <code>autocast</code> via a nested context as described in the docs, if you want to explicitly change the <code>dtype</code>:</p>
<blockquote>
<p><code>autocast(enabled=False)</code> subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular <code>dtype</code>. Disabling autocast gives you explicit control over the execution type.</p>
</blockquote>

avi_singhal: <p>So we should disable the autocast context even if the op is not a part of the autocast policy? I understand the context needs to be disabled in the scenario where an op is part of the policy and needs a different dtype.</p>
<p>I was looking at some base where autocast is used, i do not see this happening i.e. looking at Nvidia Nemo: <a href="https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/megatron/transformer.py#L427" class="inline-onebox" rel="noopener nofollow ugc">NeMo/nemo/collections/nlp/modules/common/megatron/transformer.py at main · NVIDIA/NeMo · GitHub</a>, ParallelMLP uses RPL and CPL layers. these layers internally cast all of the inputs to BF16 if autocast is enabled, however, this casting is done inside the enabled context as shown here:<br>
<a href="https://github.com/NVIDIA/apex/blob/master/apex/transformer/tensor_parallel/layers.py#L448" class="inline-onebox" rel="noopener nofollow ugc">apex/apex/transformer/tensor_parallel/layers.py at master · NVIDIA/apex · GitHub</a></p>
<p>So, is this a bug in Nemo? Or we can explicitly cast inside the enabled autocast context, but we need to be careful about it.</p>
Importing torchtext

Dristro: <p>Trying to import torch along with torchtext to download the WikiText-2 dataset.</p>
<p>torch version: 2.5.1<br>
torchtext version: 0.18.0<br>
platform: macOS (mps)</p>
<p>Not sure if its a compatibility error b/w torch versions or something.</p>
<p><strong>Getting the following error:</strong><br>
OSError: dlopen(/anaconda3/envs/llm/lib/python3.9/site-packages/torchtext/lib/libtorchtext.so, 0x0006): Symbol not found: __ZN3c105ErrorC1ENSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES7_PKv<br>
Referenced from:  /anaconda3/envs/llm/lib/python3.9/site-packages/torchtext/lib/libtorchtext.so<br>
Expected in:      /anaconda3/envs/llm/lib/python3.9/site-packages/torch/lib/libc10.dylib</p>

vdw: <p><code>torchtext</code> is no longer maintained, and only compatible up to PyTorch version 2.3.0.</p>
Dynamically loading symbols from libtorch.so

Theophile_Champion: <p>Hello,</p>
<p>I have a python module <code>relab</code> and a C++ extension <code>librelab.so</code>. The python module depends on <code>torch</code> and the C++ extension depends on <code>libtorch.so</code>. I am trying to create a package on PyPI, I believe the only way to achieve that is by dynamically loading symbols from <code>libtorch</code>.</p>
<p>To do so, I am trying to use <code>dlopen</code> and <code>dlsym</code>, but my problem start when trying to load a simple function such as <code>torch::zeros</code>. Indeed, there is no symbol <code>zeros</code> in <code>libtorch.so</code>, instead these symbols seems to be defined in <code>libtorch_cpu.so</code> and <code>libtorch_cuda.so</code>.</p>
<p>Does that means that I need to load different symbols depending on whether I want to create a tensor on the CPU or on the GPU?</p>
<p>Is there a way to load the symbol corresponding to the standard <code>torch::zeros</code>?</p>
Is there a way to force some functions to be run with FP32 precision?

Dazitu616: <p>Currently I want to train my model using FP16. However, I have a model which utilizes some CUDA ops borrowed from others’ repo. Those ops only accept FP32 input. Since I’m not familiar with CUDA, I don’t want to modify them. A workaround I’ve came up with is to always apply FP32 in the forward function of these ops, but apply FP16 on the other part of the model. So I’m wondering is there any way in PyTorch to do so? Something like a decorator would be extremely helpful:</p>
<pre><code class="lang-auto">class CUDAOPWarper(nn.Module):

    ...

    @torch.amp.forcefp32()  # a decorator that indicates this function should be run in full-precision
    def forward(self, input):
        ...

</code></pre>
<p>Thanks!</p>

ptrblck: <p>Yes, you can disable <code>autocast</code> for custom method as described in this <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#autocast-and-custom-autograd-functions">example</a>.</p>

Dazitu616: <p>Ahhh yes. That’s exactly what I want! Thanks so much for the reply.</p>

FiorenzoParascandolo: <p>Interesting, but does it is sufficient to disable the autocast in the forward pass if I use common operators in the custom block (such as multiplication for nn.Parameter)? Or something is required at backward level?</p>

ptrblck: <p>The link shows two examples:</p>
<ol>
<li>The first one showing how to disable <code>autocast</code> and explicitly cast the tensors to the desired <code>dtype</code> expected in your custom function. No <code>backward</code> implementation is shown and needed.</li>
<li>The second example shows the addition of <code>@custom_fwd</code> and <code>@custom_bwd</code> to an <code>autograd.Function</code> which can be added if you are the author of this custom function.</li>
</ol>
<p>Which approach is used depends on your workflow and control over custom <code>autograd.Function</code>s:</p>
<blockquote>
<p>If you’re the function’s author (or can alter its definition) a better solution is to use the <a href="https://pytorch.org/docs/stable/amp.html#torch.amp.custom_fwd"><code>torch.amp.custom_fwd()</code></a> and <a href="https://pytorch.org/docs/stable/amp.html#torch.amp.custom_bwd"><code>torch.amp.custom_bwd()</code></a> decorators as shown in the relevant case below.</p>
</blockquote>
How to use torch.jit.trace to trace CausalLMOutputWithPast?

GoodJob: <p>Call torch.jit.trace to trace the transformers.models.model_qwen2 model by the following code:</p>
<pre><code class="lang-auto">    huggingface_model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
    torch_model = modeling_qwen2.Qwen2ForCausalLM.from_pretrained(
        huggingface_model_name,
        config=llm_config,
        ignore_mismatched_sizes=False,
    )
    ...
    pt_model = torch.jit.trace(torch_model.to("cpu"), model_inputs.input_ids)
</code></pre>
<p>The error happened:<br>
<code>RuntimeError: Tracer cannot infer type of CausalLMOutputWithPast(loss=None, logits=tensor...)</code></p>
<p>The CausalLMOutputWithPast isn’t a class inherited from nn.Module, please refer to:<br>
<a href="https://huggingface.co/docs/transformers/main_classes/output" rel="noopener nofollow ugc">Model outputs</a></p>
<p>How can I handle this case?</p>
<p>Thanks for the great work of pytorch team!</p>

MyCenturaHealth: <p>Extract logits from CausalLMOutputWithPast, then trace the model with torch.jit.trace on a function returning only logits<a href="https://xn--bk1bp4r.info/" rel="noopener nofollow ugc">.</a></p>
Quession about including torch/extension

p3n9u1n: <p>Hi, I’m trying to create a custom c++ extension but get stuck with <span class="hashtag-raw">#include</span> &lt; torch/extension.h &gt; with the error:</p>
<pre><code class="lang-auto">interrpolation.cpp:1:29: fatal error: torch/extension.h: No such file or directory
 #include &lt;torch/extension.h&gt;
                             ^
compilation terminated.
</code></pre>
<p>I’m just wondering that torch can been detected just like below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6c1d6014497944935fb34ae023787bba5fff013.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b6c1d6014497944935fb34ae023787bba5fff013" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c1d6014497944935fb34ae023787bba5fff013_2_690x99.jpeg" alt="image" data-base62-sha1="q4KbR8N8jlZykM3qg2ucFfAhlvB" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c1d6014497944935fb34ae023787bba5fff013_2_690x99.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6c1d6014497944935fb34ae023787bba5fff013.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6c1d6014497944935fb34ae023787bba5fff013.jpeg 2x" data-dominant-color="3B3E44"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">884×128 34.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And I also input the correct includePath in c_cpp_properties.json like below:</p>
<pre><code class="lang-auto">{
    "configurations": [
        {
            "name": "Win32",
            "includePath": [
                "${workspaceFolder}/**",
                "C:/Users/USER/anaconda3/envs/cppcuda/Lib/site-packages/torch/include",
                "C:/Users/USER/anaconda3/envs/cppcuda/Lib/site-packages/torch/include/torch/csrc/api/include",
                "C:/Users/USER/anaconda3/envs/cppcuda/Lib/site-packages/torch/include"
            ],
            "defines": [
                "_DEBUG",
                "UNICODE",
                "_UNICODE"
            ],
            "windowsSdkVersion": "10.0.22621.0",
            "cStandard": "c17",
            "cppStandard": "c++14",
            "intelliSenseMode": "windows-msvc-x64"
        }
    ],
    "version": 4
}
</code></pre>
<p>But somehow torch/extension can’t be included.<br>
Any guidance is much appreciated, thank you.</p>
On pull requests... progress

kuraga: <p>Good day!</p>
<p>Wishing to follow the traditions of development and the code of conduct, what are my actions on PR staling?</p>
<p>One of it <a href="https://github.com/pytorch/pytorch/pull/136218#event-16056743113" rel="noopener nofollow ugc">was closed</a> by <code>@github-actions</code> bot without any comment.</p>
<p>There is <a href="https://github.com/pytorch/pytorch/pull/138628" rel="noopener nofollow ugc">another one</a>…</p>
<p>Thanks.</p>

ptrblck: <p>Thanks for creating these PRs!<br>
I think the PRs lack a bit of explanation and what exactly is being improved.<br>
E.g. in <span class="hashtag-raw">#136218</span> you change the assignment to a comparison:</p>
<blockquote>
<p>At <code>groups == 1</code> …</p>
</blockquote>
<p>but why was the previous code unclear, misleading, or wrong? (I’m not saying your change is invalid, but also don’t see why the current docs are wrong.)</p>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for the reply!<br>
Mostly both PRs add or fix highlighting of attrs/arguments in the (generated) documentation.<br>
If I find some content changes, I’ll write there.</p>

kuraga: <p>The first PR has been <a href="https://github.com/pytorch/pytorch/pull/146247" rel="noopener nofollow ugc">reborn</a>.</p>
<p>The second PR has been merged.</p>
IG for VQA using VLMs from transformers

nkossmann: <p>Hi there,</p>
<p>I would like to use IG for Image and Text to display the results of VLM models such as <a href="https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf" rel="noopener nofollow ugc">LLava-OneVision</a> or <a href="https://huggingface.co/microsoft/Phi-3.5-vision-instruct" rel="noopener nofollow ugc">Phi-3.5-Vision-instruct</a> on a VQA task with multiple images.<br>
I have already created a <a href="https://colab.research.google.com/drive/1ztArzir46kUhHp_STRYt0CrtnPva0KaN?usp=sharing" rel="noopener nofollow ugc">Google Colab notebook</a> for a simple model from Transformers. However, I cannot pass the pixel_values and input_embeds together when using the llava model. Do you have any ideas how to overcome this?<br>
Also, the results for the <a href="https://huggingface.co/dandelin/vilt-b32-finetuned-vqa" rel="noopener nofollow ugc">dandelin/vilt-b32-finetuned-vqa model</a> don’t look quite right somehow.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa13899c19040456dc3e00574bb3ef76856ad38.png" data-download-href="https://discuss.pytorch.org/uploads/default/2fa13899c19040456dc3e00574bb3ef76856ad38" title="vqa_ig"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2fa13899c19040456dc3e00574bb3ef76856ad38_2_690x435.png" alt="vqa_ig" data-base62-sha1="6NlR3nd0ZHSOvqQFTs24dxAzXVS" width="690" height="435" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2fa13899c19040456dc3e00574bb3ef76856ad38_2_690x435.png, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa13899c19040456dc3e00574bb3ef76856ad38.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fa13899c19040456dc3e00574bb3ef76856ad38.png 2x" data-dominant-color="A2A29C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">vqa_ig</span><span class="informations">806×509 261 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nkossmann: <p>I have created another <a href="https://colab.research.google.com/drive/1ltcXy6yLldew94gb58wFNNn3wen9uh8u?usp=sharing" rel="noopener nofollow ugc">notebook</a> for llava-one-vision. Do you have an idea how to calculate the input_embedding in parallel with the pixel values for ig?<br>
Also, could someone with a bigger GPU (only had T4 with 15 GB) validate the results with more than two steps?</p>
Passing tensors between c++ and python

vsnuhsn: <p>Im trying to write a C++ method that takes in torch::Tensor as a parameter and makes some modifications to the tensor and returns it back. i build this with cmake and expose it to python, but im getting errors when i try to run this, how do i pass tensors between c++ and python?</p>

ptrblck: <p>Writing a custom C++ extension as described <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a> will work.</p>
How can I combine my custom conv2D in CUDA C and autograd of pytorch

Aakira: <p>Recently, I learned how to use CUDA C to write a few of my custom layers (simple tasks like calling a kernel for matrix multiplication, for example). I wonder how I can leverage PyTorch’s autograd to turn my CUDA C layers into an autograd layer like the classes inherited from nn.Module in PyTorch.</p>
<p>I have read many discussions on forums, as well as the official PyTorch documentation <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">here</a>. However, I find those examples a bit complex (involving many concepts mentioned as prerequisites, such as pybind, “just in time,” etc.). Is there anywhere that provides an example of writing a Conv2D in CUDA C and the steps to make it work as an nn.Module (i.e., with autograd) like in PyTorch? If not, where should I start learning to do this?</p>
<p>Thank you very much for your time.</p>