akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Clarity on version suffices

Ram5: <p>Hi Team,</p>
<ol>
<li>Does pytorch use 2.X.0a0 type of version suffixes?</li>
<li>Is nightly build suffices X.Y.0.devdate &gt; X.Y.0a0 ?</li>
<li>Does custom pytorch build versions use .0a0 type of suffices</li>
</ol>

ptrblck: <p>The versions are created according to <a href="https://peps.python.org/pep-0440/">PEP 440</a>.</p>
Thread pool exectuor + PyTorch

AlexP4499: <p>Has anyone used Thread Pool Executor with Pytorch?</p>
<p>Details:</p>
<ul>
<li>I am using a single GPU (Nvidia/MPS)</li>
<li>I am trying to predict around 80 parameters (using 80 similar models). I have found that my current model and training workflow are not fully utilising the GPU.</li>
<li>Using RayTune / and simply just running the script multiple times, I have noted that the total runtime would be less, if I could run multiple training loops at once, probably only around 2, maybe 3.</li>
<li>In my production environment, I cannot package RayTune, so using it is not an option.</li>
<li>I would ordinarily use Process Pool Executor, but this isn’t a possibility due to the GUI framework that I am using.</li>
</ul>
<p>Is it possible to use Thread Pool Executor to create multiple training loops at the same time? (All the tasks are independent, and none rely on another)</p>

AlexP4499: <p>Can confirm - both approaches work.</p>
<p>However, Thread Pool Executor results in limited performance improvements (as expected given the GIL).</p>
<p>Process Pool executor is the way to go.</p>
<p>The results are particularly pronounced on Apple silicon when using MPS, the efficiency gains are nearly 4x running 4 fairly sizeable models in parallel.</p>
Why N-D Parallelism?

Ravi_Teja2: <p>I see in latest pytorch distributed paradigms are moving into combining Data Parallel + Tensor Parallel+ Pipeline Parallel. Making the training a “N-D Parallel”. But No major updates on improving API’s (RPC framework) for “Parameter Server Strategy”??<br>
In PS Strategy:</p>
<ol>
<li>If a model doesn’t fit in memory I can shard the parameters and store on several parameter servers and make workers call only required parameters.</li>
<li>If a worker fails it doesn’t gracefully stop all my other workers(Which torchrun does for synchronous training). Ofcourse this leads to some degradation in performance simple put.</li>
</ol>
<p>But isn’t this a great solution? and improve the communication bottlenecks in this API?</p>
Inductor backend report error when add 'device='cuda'' attribute

Yong2024: <p>Hello,</p>
<p>I am a new comer in torch. I meet a problem when run torch.compile with ‘device=‘cuda’’ attribute.</p>
<p>Below is the example code in torch.compile tutorial doc.  It can be run successfully.</p>
<pre><code class="lang-auto">import torch

@torch.compile
def foo(x, y):
    a = torch.sin(x)
    b = torch.cos(y)
    return a + b

opt_foo = torch.compile(foo)
print(opt_foo(torch.randn(10), torch.randn(10)))
</code></pre>
<p>But when I run it in GPU by adding cuda tag, some error raised.</p>
<p>The code as below (the only difference is ‘device=‘cuda’’ in torch.randn),</p>
<pre><code class="lang-auto">import torch

@torch.compile
def foo(x, y):
    a = torch.sin(x)
    b = torch.cos(y)
    return a + b

opt_foo = torch.compile(foo)
print(opt_foo(torch.randn(10, device='cuda'), torch.randn(10, device='cuda')))
</code></pre>
<p>The error info as below,</p>
<pre><code class="lang-auto">  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
TypeError: Signature keys must be string
</code></pre>
<p>torch.cuda.is_available() return true in my env. So the cuda software is ready.</p>
<p>Seems like “device=‘cuda’” is not string?  To be honest, I have no idea about this error.</p>
<p>I also tried to find out whether there have been any same issue by google, but have not any useful information.</p>
<p>So I ask for help here. Thanks</p>
<p>BR</p>

Yong2024: <p>Python version in my env is 3.12.3. As triton is not bundled with Python at this version, so the triton in my env is compiled from source code (the latest triton 3.0) by myself.</p>
<p>The torch version is 2.4.1+cu124.</p>

Yong2024: <p>I think I have found the reason now.</p>
<p>I replaced the triton package by the official release (triton-3.0.0-1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl). And now it can work.</p>
<p>But I still have no idea why it can not work with the triton by compiled from source code. Maybe some mistake in my compiling instruction, or the triton version mismatch with torch (my triton source code is based on main stream, not a stable branch).</p>
Prior art on implementing a "print" op on hardware

jkshtj: <p>Hi folks,<br>
Are there existing hardware backends that support a “print” op? If yes, could someone with direct/indirect experience implementing them speak to how such ops work?</p>
<p>Specifically, I’m interested in understanding <strong>how a tensor partitioned across the memory hierarchy printed without significantly changing the compute graph or introducing expensive “collective” operations</strong>?</p>
<p>I understand that a lot of this would depend on the hardware features available. For instance, in some hardware backends you cannot print from device and the tensor to be printed must always be sent to the host first. <strong>I’m more interested in the scenarios where the device does support printing.</strong></p>
<p>P.S. - Perhaps even CPUs with a cache hierarchy run into similar challenges while printing a value. Any relevant insights here would be appreciated.</p>
PINN-pde_loss-error-in-residual

radik_rammohan: <p>I am trying to calculate the Pde residual for 1dheat transfer equation.</p>
<p>The function is as follows:-</p>
<pre data-code-wrap="python"><code class="lang-python">ef pde_loss(model,x,t,T_S,T_L):
    # u_pred.requires_grad = True
    x.requires_grad = True
    t.requires_grad = True
    
    u_pred = model(x,t).requires_grad_()

    

    u_t = torch.autograd.grad(u_pred, t, 
                                torch.ones_like(u_pred).to(device),
                                create_graph=True,
                                allow_unused=True,
                                )[0] # Calculate the first time derivative
    if u_t is None:
        raise RuntimeError("u_t is None")

    u_x = torch.autograd.grad(u_pred, 
                                x, 
                                torch.ones_like(u_pred).to(device), 
                                create_graph=True,
                                allow_unused =True)[0] # Calculate the first space derivative
            
    u_xx = torch.autograd.grad(u_x, 
                                x, 
                                torch.ones_like(u_x).to(device), 
                                create_graph=True,
                                allow_unused=True,
                                )[0]
    
    T_S_tensor = torch.tensor(T_S, dtype=torch.float32, device=device)
    T_L_tensor = torch.tensor(T_L, dtype=torch.float32, device=device)

    
    
    residual = u_t - alpha_t * u_xx
   
    
    loss = nn.MSELoss()(residual,torch.zeros_like(residual))
    
    

    return loss
```python

When I train the model for data loss only using MSE error, the model trains well, but the PDE residual shows error. Ideally if the data loss is less then Pde loss should also be less as this will lead to satisfying physics equation based on which data was generated.</code></pre>
Combined binary and multiclass model

Vince_Faller: <p>Hello, first post, yay.</p>
<p>I’m doing some defect detection with my primary metric being pass/fail recall/precision.  But they also want the cross pareto.  The problem I’m running into is a balance problem for the model.  I either have it balanced for multiclass or pass fail.  I just want to see if my thought process makes sense or if there’s a better way.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
counts = {"class1": 10, "class2": 10, "class3": 9, "Pass":10}
n = len(counts)
weights = [1]*(n-1) + [n-1] # assume pass is last
weights = torch.tensor(weights)
weights = weights/((n-1)*2)
weights
mc_criterion = torch.nn.CrossEntropyLoss()
binary_criterion = torch.nn.CrossEntropyLoss(weight=weights)
# then in the training loop, the loss is some weighted sum of the two
.2*mc_criterion(outputs, labels) + .8*binary_criterion(outputs, labels)
</code></pre>
<p>I’ve also started looking into hierarchical modeling where I run pass/fail first, and if it fails, go to a secondary model for defect classification.</p>
Casting complex linear layers to cfloat or chalf using model.to(torch.float) or model.to(torch.half)

Laurent1: <p>Hi,</p>
<p>In my model, I have linear transformations, some of type <code>torch.float</code>, some of type <code>torch.cfloat</code>. I set the model precision using either <code>model.to(torch.float)</code> or <code>model.to(torch.half)</code>. By doing so, pytorch over-writes the types of all complex linear transformation to <code>torch.float</code> or <code>torch.half</code>. I was expecting that, for a complex linear transformation,  pytorch would cast the type to <code>torch.cfloat</code> or <code>torch.chalf</code>.</p>
<p>Is there a way to achieve this ?</p>
<p>Thanks for the help.</p>

Laurent1: <p>Thank you for this detailed answer!</p>
DistributedSampler - shuffle and size

whatdhack: <p>Couple of very simple questions.   When shuffle=True, does DistributedSampler shuffle over the full dataset or the just the shard  of the replica/rank ?   What happens when dataset size is not divisible by num_replicas ?</p>

ptrblck: <p>The shuffled indices will be created baden on the length of the entire dataset as seen <a href="https://github.com/pytorch/pytorch/blob/38114ec860b33ae4bdce5af6a54cdb3a72539d12/torch/utils/data/distributed.py#L111">here</a>.</p>
<p>If <code>drop_last=True</code> is used the indices will be dropped to make the dataset indices divisible by the number of ranks. Otherwise the indices will be padded with repeated samples as seen <a href="https://github.com/pytorch/pytorch/blob/38114ec860b33ae4bdce5af6a54cdb3a72539d12/torch/utils/data/distributed.py#L115">here</a>.</p>

whatdhack: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for the precise answer.</p>
Torch don't see full vram on Ubuntu

Kupert: <p>Previously on Windows when working with GPU - torch saw 6030 mb of memory (6144 mb available on GPU). After switching to linux I noticed that only 5930 mb is available. How to solve this problem? Who has faced this problem?<br>
I used to work on Windows 11<br>
Now on Linux Ubuntu</p>

Soumya_Kundu: <p>Is this using <code>nvidia-smi</code> ?</p>
Saving a model compiled with Torch-TensorRT with dynamic inputs

Rishi_C: <p>I can successfully compile a model with dynamic inputs using Torch-TensorRT, as specified in <a href="https://pytorch.org/TensorRT/user_guide/dynamic_shapes.html" rel="noopener nofollow ugc">the docs</a>:</p>
<pre><code class="lang-auto">import torch_tensorrt as trt
inputs = [trt.Input(min_shape=(1, 1, 28, 28), 
                   opt_shape=(50, 1, 28, 28), 
                   max_shape=(64, 1, 28, 28), 
                   dtype=torch.float32)]

exp_program = trt.dynamo.trace(model_from_state, inputs)
trt_gm = trt.dynamo.compile(exp_program, inputs=inputs)
</code></pre>
<p>However, if I try to save the model using <code>torch_tensorrt.save</code> (as specified <a href="https://pytorch.org/TensorRT/user_guide/saving_models.html" rel="noopener nofollow ugc">here</a>), the <code>inputs</code> parameter requires torch tensors:<br>
<code>trt.save(trt_gm, "trt_model.ep", inputs=inputs)</code><br>
This code throws:</p>
<pre><code class="lang-auto">ValueError: Not all inputs provided are torch.tensors. Please provide torch.tensors as inputs
</code></pre>
<p>What is the correct way to save a model compiled AOT with Torch-TensorRT which has dynamic inputs? In theory, providing the dynamic inputs using torch_tensorrt.Input is supposed to handle the dynamic “Dim” object creation internally and attach it to the compiled graph, but I don’t see a way to save the resultant model.</p>
Where can I find pytorch tests suites results?

ccoulombe: <p>Hello,</p>
<p>I am running numerous tests and some are failing but I’d like to compare with the “official” test suite ran.</p>
<p>Where can I find information on which tests is success/failed, is it expected to fail, etc.?<br>
For instance, some cudnn tests are failing and I’m trying to understand if it is the cudnn version (9.2) or a stale tests.</p>
<p>For example, if anyone have insights on this test</p>
<pre><code class="lang-auto">__________________________________________________________________________________________ TestNN.test_cudnn_weight_tying ___________________________________________________________________________________________
AssertionError: Tensor-likes are not close!

Mismatched elements: 486 / 800 (60.8%)
Greatest absolute difference: 0.00012448430061340332 at index (2, 1, 8) (up to 1e-05 allowed)
Greatest relative difference: 0.0016339612193405628 at index (3, 0, 12) (up to 1.3e-06 allowed)

The failure occurred for item [0]

To execute this test, run the following from the base repo dir:
     python test/test_nn.py -k TestNN.test_cudnn_weight_tying

</code></pre>
<p>Thanks,</p>
Is there a way to precompute the size of the computation graph a model will need?

Levente_Sipos: <p>Hi,</p>
<p>I am training a UNet3D model with MRI data, and the images and the model itself are quite large. To fit into the 24GB VRAM available to me I need to decrease the input image size significantly. I was wondering how much VRAM I would need to use the original image size, but I obviously can’t just try to run it as I will very quickly get an OOM error.<br>
So as a solution I was wondering if there was a way to compute how much VRAM the model’s computation graph and the loaded data will need (taking into account the batch size)?</p>

ptrblck: <p><a href="https://dev-discuss.pytorch.org/t/how-to-measure-memory-usage-from-your-model-without-running-it/2024">This util.</a> might be helpful using <code>TorchDispatchMode</code> to measure the memory usage without running the actual model.</p>
The model still updates the weight of the freezing layers in training

hasan11085: <p>Hi, I’m facing a strange problem. After unfreezing a little portion of the part in model, the model starts to update the whole model though the variance is small.  When i took the output from the previous layer before model.inception the output got changed. Theoretically or mathematically it should not go in this way. Can you suggest which problem caused this?</p>
<p>My code is here:<br>
model =oldmodel()<br>
model.load_state_dict(torch.load(MODEL_PATH))<br>
for param in model.parameters():<br>
param.requires_grad = False<br>
for param in model.inception.parameters():<br>
param.requires_grad = True</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>
Compute batched L2-distance between 2 vectors - torch.cdist()

grid_world: <p>I have 2 vectors ‘a’ and ‘b’ of shapes: a = (batch_size, embed_dim) and b = (num_neurons, embed_dim), and I want to compute L2-distance between them such that the distance has the shape: (batch_size, num_neurons).</p>
<pre><code class="lang-auto">nr_units = 200
z_dim = 128

a = nn.Embedding(num_embeddings = nr_units, embedding_dim = z_dim)

# Initialize weights of 'a'-
a.weight.data.uniform_(-np.sqrt(1 / z_dim), np.sqrt(1 / z_dim))

b = torch.rand(1024, z_dim)

# Compute L2-distance using torch.cdist()-
dist_cdist = torch.cdist(
    x1 = a, x2 = b.weight,
    p = 2, compute_mode = 'donot_use_mm_for_euclid_dist'
)
</code></pre>
<p>While searching for similar questions, I came across <a href="https://github.com/pytorch/pytorch/issues/42479" rel="noopener nofollow ugc">this</a> thread which has this comment:</p>
<blockquote>
<p>This is an expected behavior, given that mm distance is calculated as a^2-2<em>a</em> b+b^2. If a and b are close, then catastrophic cancellation happens and result is imprecise (can even be negative, but we clamp it to be non-negative).<br>
Using matrix multiplication dramatically improves performance, but if accuracy is not enough for your application, then you should not use it.</p>
</blockquote>
<p>In my use-case, ‘a’ and ‘b’ should be very close to each other and according to this comment, catastrophic cancellation might ensue?! So, I am using:</p>
<blockquote>
<p>compute_mode = ‘donot_use_mm_for_euclid_dist’</p>
</blockquote>
<p>in torch.cdist() computation. Is this the correct approach to handle the case when a and b are close?</p>
<p>I found <a href="https://discuss.pytorch.org/t/efficient-distance-matrix-computation/9065">another thread</a> which discusses pair-wise distance computation but seems to be old.</p>
<p>Therefore, which is the new and correct pair-wise batched way/method to compute L2-distance between ‘a’ and ‘b’ even when they are quite close to each other?</p>
Event handler for CUDA OOM?

drscotthawley: <p>Is there a way to have an event handler or callback that triggers whenever (and wherever) a CUDA OOM error occurs?</p>
<p>For example I’d like to trigger a printout of the current node name on which the error occurs, because I know my run “fits” in VRAM, but sometimes my cluster will give me “bad nodes” and I’d like to know which nodes to exclude when I re-run the job.</p>
<p>Thanks.</p>
<p>PS-</p>
<p>I asked ChatGPT this question, and it suggested the following, haha:</p>
<pre><code class="lang-python">import socket
import torch

def oom_callback():
    print(f"OOM error occurred on node {socket.gethostname()}")

torch.cuda.register_cudart_error_callback(oom_callback)
</code></pre>
<p>And when I told it there was no method <code>torch.cuda.register_cudart_error_callback</code>, ChatGPT would respond by saying that it was introduced in a later version of PyTorch than… whichever version I had  (e.g. it would say it was introduced in 1.10.0, and I replied by saying I had 1.13.0, it would say something like 'oh sorry i meant 1.14.0`, lol.)</p>

eqy: <p>Could you achieve this by catching the exception on the node and propagating it somehow?</p>
<pre><code class="lang-auto">import torch

try:
  a = torch.empty(2**40, device='cuda')
except torch.cuda.OutOfMemoryError as _:
  print("propagate this OOM exception")
</code></pre>

drscotthawley: <p>Well, yes but then wouldn’t you have to place copies such code throughout your entire training system?<br>
because it my experience, once that exception happens, execution grinds to a halt.</p>

marksaroufim: <p>This is how fairseq does it btw <a href="https://github.com/facebookresearch/fairseq/blob/50a671f78d0c8de0392f924180db72ac9b41b801/fairseq/trainer.py#L283" class="inline-onebox" rel="noopener nofollow ugc">fairseq/trainer.py at 50a671f78d0c8de0392f924180db72ac9b41b801 · facebookresearch/fairseq · GitHub</a></p>

drscotthawley: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> thanks very much!  I will give such a thing a try…</p>

ngimel: <p>You can register oom observer that can make arbitrary callback on oom <a href="https://github.com/pytorch/pytorch/blob/df5bbc09d191fff3bdb592c184176e84669a7157/test/test_cuda.py#L3848" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/test_cuda.py at df5bbc09d191fff3bdb592c184176e84669a7157 · pytorch/pytorch · GitHub</a></p>
Iterative relation with a network at previous epochs

aaronpim: <p>Hi, I’m new to pytorch and neutral networks and am having an issue devising a memory efficient. I want to implement the following pseudo-code:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(self.net_params_pinn, lr=adam_lr)
for n in range(max_epoch):
            loss, boundary_loss, saved_loss = self.Method()
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            if n % 100 == 0:
                self.z = self.z + rho*self.u_net    
</code></pre>
<p>I am training a neural net that outputs a function <code>self.u_net</code> (that I am training using a PINNs scheme, that uses the function <code>self.z</code>) that I wish to use compute a function <code>self.z</code> using the above iterative relation.</p>
<p>The issue is that I am not well versed enough to understand how best to implement this final step. How can I go about doing this? Is there a way to make this memory or computationally efficient?</p>
IterableDataset support in Opacus

Pooja_Agarwal1: <p>Hi,</p>
<p>I saw that <a href="https://github.com/pytorch/opacus/blob/4823344e47a79367fd42e54fe2b283d521d7e9a9/opacus/data_loader.py#L220" rel="noopener nofollow ugc">DPDataLoader currently does not support IterableDataset</a>. How are streaming datasets and/or large datasets that cannot be fully loaded in memory due to memory limitation handled in Opacus?</p>
<p>Thanks,<br>
Pooja</p>

HuanyuZhang: <p>One potential get-around is to turn off the Poisson subsampling in which case you do not need DPDataLoader. I am not aware of an efficient implementation of Poisson subsampling+IterableDataset for the whole DP community.</p>
Inference for detection on huge images

FiReTiTi: <p>Hi,<br>
I trained a Mask R-CNN model on patches/crops 512x512 to detect an object.<br>
Now I need to apply this model on huge images, 50K x 50K.<br>
Is there a function that would take care of it? Because objects partially detected on the border will create problems.<br>
Thanks in advance.</p>

AviZ: <p>It is called sliding / sliced / tiled inference.<br>
The package <a href="https://github.com/obss/sahi" rel="noopener nofollow ugc">SAHI: Slicing Aided Hyper Inference</a> should do what you’re after.</p>

FiReTiTi: <p>Thank you I’ll take a look</p>
Re-directing Questions to Github Issues

HuanyuZhang: <p>Dear Opacus users,</p>
<p>We kindly request that you redirect your questions to our Github issue page (<a href="https://github.com/pytorch/opacus/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/opacus · GitHub</a>) if you would like attention from the Opacus team. Due to limited bandwidth and a desire to consolidate efforts, we will not be able to provide any guarantee on response time for Pytorch Forum.</p>
<p>Best regards,<br>
The Opacus team</p>

richard:
Filtering the labels of mobilenetV3

felipequentino: <p>I have a project that i need to detect Persons, and i’m using the fasterrcnn_mobilenet_v3_large_fpn model to do that. But this model has several classes:</p>
<p>[<br>
‘<strong>background</strong>’, ‘person’, ‘bicycle’, ‘car’, ‘motorcycle’, ‘airplane’, ‘bus’,<br>
‘train’, ‘truck’, ‘boat’, ‘traffic light’, ‘fire hydrant’, ‘stop sign’,<br>
‘parking meter’, ‘bench’, ‘bird’, ‘cat’, ‘dog’, ‘horse’, ‘sheep’, ‘cow’,<br>
‘elephant’, ‘bear’, ‘zebra’, ‘giraffe’, ‘backpack’, ‘umbrella’, ‘handbag’,<br>
‘tie’, ‘suitcase’, ‘frisbee’, ‘skis’, ‘snowboard’, ‘sports ball’, ‘kite’,<br>
‘baseball bat’, ‘baseball glove’, ‘skateboard’, ‘surfboard’, ‘tennis racket’,<br>
‘bottle’, ‘wine glass’, ‘cup’, ‘fork’, ‘knife’, ‘spoon’, ‘bowl’, ‘banana’,<br>
‘apple’, ‘sandwich’, ‘orange’, ‘broccoli’, ‘carrot’, ‘hot dog’, ‘pizza’,<br>
‘donut’, ‘cake’, ‘chair’, ‘couch’, ‘potted plant’, ‘bed’, ‘dining table’,<br>
‘toilet’, ‘tv’, ‘laptop’, ‘mouse’, ‘remote’, ‘keyboard’, ‘cell phone’,<br>
‘microwave’, ‘oven’, ‘toaster’, ‘sink’, ‘refrigerator’, ‘book’, ‘clock’,<br>
‘vase’, ‘scissors’, ‘teddy bear’, ‘hair drier’, ‘toothbrush’<br>
]</p>
<p>And i’m showing the detections only if the bounding box represents a Person, but I think that isn’t a smart thing to do, because in the end, all classes are being computed, only aren’t being showed. It is a good idea apply a Transfer Learning to only detect Persons? I saw several tutorials of people doing transfer learning on mobilenet, but their purpose is different: they want to add new classes to the model and remove others, I just want to continue detecting Persons and remove the other labels.</p>
CUDA 12.4 for PyTorch build 1.9

rahul_ol: <p>I am trying to run the following repository for text detection: <a href="https://github.com/clovaai/CRAFT-pytorch" rel="noopener nofollow ugc">CRAFT-pytorch text detection</a></p>
<p>The requirements include PyTorch=1.9 which I am trying to install in a conda environment this is resulting in errors since 1.9 install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.</p>
<p>My current GPU is A100-SXM4-40GB with CUDA VERSION 12.4<br>
NVIDIA A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the my desired pytorch 1.9.</p>
<p>Is there anyway I can install 1.9 using my current GPU CUDA VERSION on a conda environment?</p>

ptrblck: <p>I would recommend updating PyTorch to a recent release which ships with CUDA 12.4 runtime dependencies. Did you try updating and if so did you see any issues in your code?</p>
Any way to perform forward pass by utilizing the DAG created?

yf1: <p>Hi all,</p>
<p>I have the following program:</p>
<ol>
<li>create a vector x.</li>
<li>Perform a finite-element physical simulation on x to obtain P. (P = f(x)). I make a DAG of interactions the environment makes on x.</li>
<li>Compute error, E. (E = ||P_actual - P_simulated||^2 )</li>
<li>Compute ∇E wrt x using E.backward()</li>
<li>Update x using optimizer.step()</li>
<li>Repeat steps 2 to 5.</li>
</ol>
<p>The problem is, step 2 (the finite element simulation) takes way too much time to run everytime. Is there a method available that allows me to directly compute E by using the DAG already present? Since the DAG is used to compute the gradient, it should also allow me to perform a “forward pass” that completely bypasses having to run the finite-element simulation for every iteration, right?</p>
<p>So in essence, I simply wish to just run the finite-element simulation once to get a DAG, and use that DAG to perform my computations for every subsequent iteration.</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Parallel k-fold cross validation on one GPU

Doug_J: <p>My model training uses k-fold cross-validation, and I’m exploring whether it’s possible to parallelize the k-fold process on a single GPU. Specifically, can each fold run in a separate stream on the same GPU, dispatching folds until the cores are fully utilized? For instance, if a GPU can handle 3 streams at once, and I have 6 folds, parallelism could theoretically reduce the cross-validation time to a third.</p>
<p>Does this approach make sense, or does PyTorch already handle parallelization for cross-validation?</p>
<p>Thanks!</p>

Soumya_Kundu: <p>I don’t believe there is native support for this.</p>
<p>Assuming you have enough VRAM for the above, pre-making the folds and submitting “Separate Jobs” should be a simplistic way to achieve this?</p>
Fluctuating inference time using LibTorch

Kazuma_Kiryuu: <p>Hi, I’ve been working on implementing my own program that supports inference using either TensorRT or LibTorch. Currently the whole pipeline works well with TensorRT as the inference time looks smooth all the time but when I run the inference using a .pt model with LibTorch, the inference time obviously fluctuates.</p>
<p>The phenomenon mentioned above HAPPENS ONLY WHEN I take input images from a USB CAMERA. If I instead take inputs from a video clip, the inference time also looks smooth when using a .pt model with LibTorch like that with TensorRT.</p>
<p>Not sure what I’m doing incorrectly and currently no idea what to do to improve this.<br>
I’m showing some of the code snippets I implemented for LibTorch inference.</p>
<pre><code class="lang-auto">cv::Mat predict(cv::Mat&amp; image)
{
    if (m_libtorch_model_loaded == true)
    {
        torch::NoGradGuard no_grad;
        preprocess_libtorch(image);
        m_output_tensor = m_libtorch_model.forward({m_input_tensor}).toTensor();
        torch::cuda::synchronize();
    } 

    postprocess();
    return m_colormap_image;    
}
</code></pre>
<pre><code class="lang-auto">preprocess_libtorch(cv::Mat&amp; image)
{
    cv::resize(image, m_input_image, cv::Size(m_input_w, m_input_h));
    m_input_tensor = torch::from_blob(m_input_image.data, {1, m_input_image.rows, m_input_image.cols, 3}, torch::kByte);

    m_input_tensor = m_input_tensor.permute({0, 3, 1, 2});  // [N, C, H, W]
    m_input_tensor = (m_input_tensor / m_scalar_255 - m_mean_tensor) / m_std_tensor;

    if (torch::cuda::is_available())
    {
        m_input_tensor = m_input_tensor.to(torch::kCUDA);
    }

    return;
}
</code></pre>
The forward function of a multi-layer Elman RNN from tutorial has two errors

zhangliquan: <p>one is that while layer=2 x[t] should be substituted by h_t[layer-1],<br>
and the other is that output.append(h_t[-1]) has overwriten the previous elements in the list, so here it should be rewritten as temp = h_t[-1].clone()  and output.append(temp).</p>
Torchrun command specifies the GPU

TTYee: <p>When I use the “torchrun” command to run .sh file in Single-node multi-worker, it seems like it will start training on the fisrt n GPU by default by using “–nproc-per-node=n”. For some reason, my GPU1 has been in use. I’m wondering how to use torchrun command to get files training on the specified GPU. (e.g. GPU2, 3,4,5)</p>

H-Huang: <p>To my knowledge, <code>torchrun</code> doesn’t distinguish between which device to use and treats the python script you are running as a blackbox. So in your user code you are probably doing something like <code>.cuda()</code> which defaults to the first device, but instead you should do something like <code>.to(local_rank)</code> to set it to a specific device (1 rank for each device)</p>
<p>Alternatively you can set the default device via code or environment variable <code>CUDA_VISIBLE_DEVICES </code> <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_device.html" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.set_device — PyTorch 2.0 documentation</a></p>

saner-cakir: <p>You can use CUDA_VISIBLE_DEVICES=1<br>
for example</p>
<pre><code class="lang-auto">PYTHONPATH=$(git rev-parse --show-toplevel) CUDA_VISIBLE_DEVICES=1 torchrun models/scripts/chat.py $CHECKPOINT_DIR
</code></pre>
<p>for running llama on gpu 1</p>
Calling torch.compile inside the training loop

limsanky: <p>Hello!</p>
<p>I had a question regarding how one should use torch.compile inside the training loop.<br>
Currently, I am calling torch.compile(model) for every iteration of the training loop.</p>
<p>For example, it is currently something like this (PS. I am actually training a diffusion model):</p>
<pre><code class="lang-auto">for (img, label) in data:
    model = torch.compile(model)  #  &lt;- torch.compile is going to be called multiple times
    output = model(img)
    loss = mse_loss(output, label)
    ...
    (rest of the code)
</code></pre>
<p>Is this the correct way of using it, though? Would it be okay to call torch.compile multiple times like this?</p>
<p>The docs do not contain any information about this, so I am posting the question here.</p>
<p>Thank you so much!<br>
Best regards</p>

ptrblck: <p>No, I think calling it once outside the training loop is the right approach.<br>
CC <a class="mention" href="/u/marksaroufim">@marksaroufim</a> in case there is a use case for multiple calls</p>

marksaroufim: <p>You can actually compile the entire training loop, otherwise I typically recommend compiling the model the first time outside of the training loop</p>
In-place operations in gradient computation in nested PyTorch loops

Vinicius_Moreira: <p>I’m working on a PyTorch model that involves a double loop for calculations. The problem is that I’m getting an error related to in-place operations when I try to perform backpropagation.</p>
<p>Here’s a simplified example of my code:</p>
<pre><code class="lang-auto">import torch

length = 1000
dx = 100
num_rows = 5
num_cols = int(length / dx) + 1

C = torch.rand(num_rows, requires_grad=True)
D = torch.rand(num_rows, requires_grad=True)
qobs = torch.rand(num_rows)

Caux = 1 + C + D
C0 = (-1 + C + D) / Caux
C1 = (1 + C - D) / Caux
C2 = (1 - C + D) / Caux

qprop = torch.zeros((num_rows, num_cols))
qprop[:, 0] = qobs

# causes the problem
for j in range(1, num_cols): 
    for n in range(1, num_rows):
        term1 = C2[n] * qprop[n-1, j-1]
        term2 = C1[n] * qprop[n, j-1]
        term3 = C0[n] * qprop[n-1, j]
        qprop[n, j] = term1 + term2 + term3  

loss = qprop.sum()
loss.backward()  # This raises an error
</code></pre>

ptrblck: <p>You could try to work around this issue by cloning the <code>termX</code> tensors:</p>
<pre data-code-wrap="python"><code class="lang-python">for j in range(1, num_cols): 
    for n in range(1, num_rows):
        term1 = C2[n] * qprop[n-1, j-1].clone()
        term2 = C1[n] * qprop[n, j-1].clone()
        term3 = C0[n] * qprop[n-1, j].clone()
        qprop[n, j] = term1 + term2 + term3  
</code></pre>
Share parameter between two modules in a nn.Sequential

fouronnes: <p>I’m trying to share a single parameter between two modules used in a <code>nn.Sequential</code>. Each module uses the parameter differently. For example, imagine a function, parametrized by w:</p>
<p><code>F(x) = x+w</code></p>
<p>Let’s say I’m trying to find w so that <code>x + w - w/2</code> is minimal, I can express this as a <code>Sequential(F(w), F(-w/2))</code>.</p>
<p>This is what I tried:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

class F(nn.Module):
    def __init__(self, w):
        super().__init__()
        self.w = w

    def forward(self, x):
        return x + self.w

w = nn.Parameter(torch.tensor([5.0]), requires_grad=True)

model = nn.Sequential(F(w), F(-w/2))
optimizer = optim.Adam([w], lr=1e-1)

for epoch in range(10):
    loss = torch.pow(model(torch.tensor([2.0])) - 10.0, 2)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    print(epoch, loss.item())
</code></pre>
<p>and I’m getting:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
</blockquote>
<p>Is there a trick to make this kind of parameter sharing work? Of course this is a minimal example, my actual function is much more complex.</p>

ptrblck: <p>The issue is caused by performing a differentiable operation on <code>w</code> in: <code>F(-w/2)</code>. The internal <code>self.w</code> attribute won’t be a trainable <code>nn.Parameter</code> anymore but is the output of the division.</p>
<p>You would work around this issue e.g. by passing a <code>scale</code> argument to the <code>__init__</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">class F(nn.Module):
    def __init__(self, w, scale=1.0):
        super().__init__()
        self.w = w
        self.scale = scale

    def forward(self, x):
        return x + self.w * self.scale

w = nn.Parameter(torch.tensor([5.0]), requires_grad=True)

model = nn.Sequential(F(w), F(w, scale=-0.5))
optimizer = optim.Adam([w], lr=1e-1)

for epoch in range(10):
    loss = torch.pow(model(torch.tensor([2.0])) - 10.0, 2)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    print(epoch, loss.item())
</code></pre>
`CUDNN_BACKEND_OPERATION: cudnnFinalize Failed cudnn_status: CUDNN_STATUS_BAD_PARAM`

Jerome_Ku: <p>Trying to use <code>torch.nn.grad.conv1d_{input,weight}</code> as part of a custom backwards pass.  The operation runs when the dtype is <code>bfloat16</code> or <code>float32</code> but fails with the aforementioned error when using <code>float16</code>.</p>
<p>Any ideas why?</p>
<p>Here is a minimal repro:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.nn.grad import conv1d_input, conv1d_weight
# Shapes
bs = 2
seqlen = 32
d = 64
g = 2
hl = 4
dg = d // g

# NOTE: Changing this to float16 results in `RuntimeError: CUDNN_BACKEND_OPERATION: cudnnFinalize Failed cudnn_status: CUDNN_STATUS_BAD_PARAM`
dtype = torch.bfloat16

# Inputs
x = torch.randn(bs, seqlen, g, dg, device="cuda", dtype=dtype)
x2 = x.reshape(bs, seqlen, -1).permute(0, 2, 1)  # bs, d, seqlen
h = torch.randn(g, 1, hl, device="cuda", dtype=dtype)

h_grouped = h.repeat_interleave(dg, dim=0)  # (d, 1, hl)
assert h_grouped.shape == torch.Size([d, 1, hl])

padding = hl - 1
groups = d

# depthwise causal conv
y = torch.nn.functional.conv1d(x2, h_grouped, groups=d, padding=padding)[..., :-padding]
assert y.shape == torch.Size([bs, d, seqlen])

dy = torch.randn_like(y)

# These ops will fail if dtype is set to `float16`
dx = conv1d_input(x2.shape, h_grouped, dy, padding=padding, groups=groups)
dh_grouped = conv1d_weight(x2, h_grouped.shape, dy, padding=padding, groups=groups)
</code></pre>

ptrblck: <p>Could you post the device you are using as well as the PyTorch version?</p>

Jerome_Ku: <p>Torch version: <code>2.5.0.dev20240814+cu121</code><br>
Device: <code>NVIDIA H100 80GB HBM3</code></p>

ptrblck: <p>Thanks! I’m able to reproduce the issue and forwarded it.</p>
Dataloader to load batches alternatively from multiple datasets

Vaibhav_Sharma: <p>I am training my network with multiple datasets for multitask learning, using gradient accumulation from batches of these datasets and then doing the optimization step.<br>
I found out that using multiple data loaders for each dataset is not optimal as each data loader may prefetch the batch and consume GPU memory.<br>
Is there way to have a single data loader for multiple datasets, which alternatively sample from these datasets.</p>

ptrblck: <p>You could write your custom <code>sampler</code>, which is responsible to create the indices passed to <code>Dataset.__getitem__</code> selecting the sample, and iterate the different datasets according to your logic.<br>
The common use case is also to load and process the samples on the CPU and to move the processed batch to the GPU inside the <code>DataLoader</code> loop. If you stick to this approach multiple <code>DataLoader</code>s won’t consume any GPU memory.</p>
Evaluation mode behavior for training-specific parameters for pytorch model

Hanumanth04: <p>Hi All,</p>
<p>I have the below clarification question related to pytorch <code>module.eval()</code>:</p>
<p>The document below briefly mentions the <code>evaluation mode</code> behavior for the <code>Dropout</code> and <code>Batchnorm</code> operators. It then advises readers to look into the details of the respective modules for the <code>evaluation mode</code> behavior for all other operators.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.eval" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.eval</a></p>
<p>I am looking at the <code>EmbeddingBag</code> module behavior in <code>evaluation mode,</code> and some of the properties of this module are training-specific. For instance, the <code>sparse</code> parameter seems training-specific. I was wondering what the expected behavior is for this property and all other training-specific properties for this module and, in general, after <code>the module.eval()</code>. Do we expect training-specific properties to get flipped to <code>eval()</code> mode? For instance, do we expect the value of <code>sparse</code> to be set to <code>false</code> even if users set it to <code>True</code> during training mode?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#embeddingbag" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#embeddingbag</a></p>
<p>Some context for asking this question:</p>
<p>We are trying to use the <code>torch-mlir</code> repo, and for some of the pytorch models in the repo, the MLIR legalization (torch-to-linalg) pass fail if operators/modules contain training only properties after <code>model.eval()</code>. I noticed this behavior while running the MLR <code>torch-to-linalg</code> pass for the <code>dlrm</code> model as the <code>EmbeddingBag</code> module in the model includes some training-specific parameters even after <code>module.eval()</code>. We can fix the respective lowering from our end to handle this situation. However, I wanted to understand what the expected behavior is after <code>model.eval()</code>. Do we expect all the training-specific parameters to get flipped to <code>evaluation mode</code>? Our use case is to generate code only for <code>evaluation mode</code> for now, so we don’t care about training-specific parameters in the model.</p>
<p>Best,<br>
Hanumanth</p>
How to compute batched vector jacobian product?

ChenyangWu: <pre data-code-wrap="python"><code class="lang-python">
import torch
import torch.nn as nn
import time
from congrad.torch import cg_batch
from torch.func import grad, vmap, vjp, functional_call

# Check if GPU is available and set the device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Define a larger neural network with 5 layers and 512 hidden units
class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(24, 512)  # Input size is the sum of observation and action dimensions
        self.fc2 = nn.Linear(512, 512)
        self.fc3 = nn.Linear(512, 512)
        self.fc4 = nn.Linear(512, 512)
        self.fc5 = nn.Linear(512, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.relu(self.fc3(x))
        x = torch.relu(self.fc4(x))
        return self.fc5(x)

# Initialize the neural network and move it to the GPU
net = NeuralNet().to(device)
params = dict(net.named_parameters())

# Mock data, moved to GPU
torch.manual_seed(0)  # For reproducibility
batch_size = 1024
x = torch.rand(batch_size, 24, device=device)  # 1024 observations, 17 features each

# Define the necessary functions
def fcall(params, x):
    return functional_call(net, params, x.unsqueeze(0)).squeeze()

def compute_gradient_and_its_copy(params, x):
    gradient = grad(fcall, argnums=1)(params, x)
    return gradient, gradient

def compute_vjp_fn_and_gradient(params, x):
    _, vjp_fn, gradient = vjp(lambda x: compute_gradient_and_its_copy(params, x), x, has_aux=True)
    return vjp_fn, gradient

with torch.no_grad():
    vjp_fn, gradient = vmap(compute_vjp_fn_and_gradient, in_dims=(None, 0))(params, x)
    solution, solve_info = cg_batch(lambda x: vjp_fn(x)[0], gradient)
</code></pre>
<p>I am trying to compute a batch of vector jacobian products and use them to solve the linear equation Hess x=grad with the conjugate gradient method. This approach avoids the computation of the Hessian matrix and should, hopefully, be faster. However, <code>vmap</code> does not support return a batch of functions. I am wondering if there is any workaround for this.</p>
<p>Thanks in advance for your precious time!</p>
Getting NaNs only on GPU training

Anshul_Gupta: <p>I am getting NaNs for loss after some time only on GPU training, the same code(with the same config) was working till last month. The cuda driver and PyTorch were updated during this period.</p>
<p>The loss increases exponentially with each step only on GPU.</p>
<pre><code class="lang-auto">Train Epoch: 1 [0/7146 (0%)] Loss: 0.176097 Train Acc: 0.187500
tensor(0.1642, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(8.2784, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(21.9132, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(51.5612, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(415.4600, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(17203.3320, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor(nan, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
</code></pre>
<p>When I used autograd anomaly detection this was the output:<br>
<code>RuntimeError: Function 'CdistBackward' returned nan values in its 0th output.</code></p>
<p>I ran the training on CPU for 15 hours and it ran perfectly.</p>
<pre><code class="lang-auto">Train Epoch: 1 [0/7146 (0%)] Loss: 0.172225 Train Acc: 0.062500
tensor(0.1619, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1835, grad_fn=&lt;DivBackward0&gt;)
tensor(0.2013, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1784, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1805, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1624, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1773, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1839, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1797, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1745, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1901, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1658, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1688, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1681, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1857, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1821, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1936, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1817, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1565, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1748, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1710, grad_fn=&lt;DivBackward0&gt;)
tensor(0.1846, grad_fn=&lt;DivBackward0&gt;)
Train Epoch: 1 [352/7146 (5%)] Loss: 0.184633 Train Acc: 0.062500
.....
</code></pre>

ptrblck: <p>Did you change anything else besides the device and do you have a reproducible code snippet?</p>

Anshul_Gupta: <p>No just changed the device to CPU.<br>
I’ll try and create a snippet to reproduce the error.</p>

Anshul_Gupta: <p>I downgraded pytorch version to 1.1.0 and it started working again on GPU.</p>

ptrblck: <p>Good to hear it’s working now, but I’m a bit concerned that you might have discovered a regression in the newer version(s).<br>
Could you try to update to <code>1.3.0</code> and run the code again, please?</p>

Anshul_Gupta: <p>Sure. I’ll do this by tomorrow.</p>

Anshul_Gupta: <p>It works absolutely fine on <code>1.3.0</code>.</p>

ptrblck: <p>Thanks for checking!</p>

nimaaminz: <p>I have this problem on version 2.4.0+cu124</p>
Proper way to make siamese clip not loosing information from text and image encoders

breadfan: <p>I have my own tuned CLIP using convnext_tiny and rubert-tiny. I want to make siamese net for further comparison of pair of objects. Each object has it’s name, description and image.</p>
<p>I’m using outputs of the BERT model:</p>
<pre><code class="lang-auto">x = bert(input_ids=input_ids, attention_mask=attention_mask)
x = x.last_hidden_state[:, 0, :]
x = final_ln(x)
</code></pre>
<p>where bert takes outputs from tokenizer and <code>final_ln</code> is the final layer of size (312x768), which was also fitted while fitting CLIP model with.<br>
For image encoder I take <code>create_model</code> from <code>timm</code> with <code>num_classes=0</code> for reducing linear head. Then I’m concatenating it in two big vectors:</p>
<pre><code class="lang-auto">first_emb = torch.cat([image_emb1, name_emb1, desc_emb1], dim=1)
second_emb = torch.cat([image_emb2, name_emb2, desc_emb2], dim=1)
</code></pre>
<p>And then forwarding trough MLP layer, which consists of dense layers, dropout and batchnorm with relu as activation:</p>
<pre><code class="lang-auto">out1 = head(first_emb)
out2 = head(second_emb)
</code></pre>
<p>My questions are:</p>
<ol>
<li>aren’t this operations with bert like <code>.last_hidden_state</code> are equivalent of just pooling therefore loosing some information?</li>
<li>Is there better way to use bert for further using this models with subsequent MLP?</li>
<li>Is it proper way to make siamese clip network?</li>
</ol>
Which at_dispatch can work for bfloat16?

coincheung: <p>Hi,</p>
<p>I wrote cuda kernel myself, and I can use <code>AT_DISPATCH_FLOATING_TYPES_AND_HALF</code> to let it support double/float32/float16, but it does not support bfloat16. How can I let a kernel to work with all of the sereis of double/float32/float16/bfloat16 ?</p>

ptrblck: <p><code>AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, ...)</code> is used internally.</p>

coincheung: <p>Hi,</p>
<p>I tried this, and my op can support double/float/fp16/bf16 now. However, I got to find that double/float/fp16 has same training results as pytorch autograd, but bf16 mode brings a big gap between pytorch autograd and my cuda kernel. Besides, bf16 run much slower than the other 3. Do you know why bf16 is different from the other three?</p>