akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Combine vmap, func.grad with tensordict

Roy-Kid: <p>Hi everyone,</p>
<p>I am trying to move my code to tensordict(td) and vmap along batch according to <a href="https://discuss.pytorch.org/t/how-to-apply-vmap-on-a-heterogeneous-tensor/214109/5" class="inline-onebox">How to apply vmap on a heterogeneous tensor - #5 by soulitzer</a>. Since vmap not support batched tensordict, now I just use batchsize = 1.</p>
<p>Nevertheless, everything worked super well until I needed to rewrite derivative module. I have a td with distance between atoms and predict energy by model, and the I need to calculate force by derivating energy w.r.t distance.</p>
<p>Before I use <code>torch.grad</code>, very straightforward to use. But within vmap, it raises: <code>element 0 of tensors does not require grad and does not have a grad_fn</code>. I find <code>Batchedtensor</code> of energy has <code>grad_fn</code>, but “vmaped” tensor, the real tensor required_grad is false.</p>
<p>Apparently(?), we can not combine vmap with torch.grad, but <code>torch.func.grad</code>, according to <a href="https://discuss.pytorch.org/t/use-vmap-and-grad-to-calculate-gradients-for-one-layer-independently-for-each-input-in-batch/187556/2" class="inline-onebox">Use vmap and grad to calculate gradients for one layer independently for each input in batch - #2 by AlphaBetaGamma96</a>, and <a href="https://discuss.pytorch.org/t/simple-use-case-compete-per-sample-gradient-with-autograd/207317/4" class="inline-onebox">Simple use case: Compete per sample gradient with autograd - #4 by AlphaBetaGamma96</a></p>
<p>Since <code>torch.func.vmap</code> can only specify the position of arguments, how can I derivate <code>energy</code> w.r.t <code>distance</code>. Here is my pesudo code:</p>
<pre><code class="lang-auto">inputs = td({'distance': tensor})
inputs = model(inputs)  # predict energy
dedx = torch.grad(inputs['energy'], inputs['distance'], torch.zeros(len(energy), 3))
# how to replace torch.grad with torch.vmap?
</code></pre>
<p>So thx for your help!</p>
<p>Also post in discussion of tensordict <a href="https://github.com/pytorch/tensordict/discussions/1167" class="inline-onebox" rel="noopener nofollow ugc">Combine vmap, func.grad with tensordict · pytorch/tensordict · Discussion #1167 · GitHub</a></p>
Yolo model does not propose any targets during training

Sam_Tux: <p>I wrote a minimal example of a training step for yolo11n (or yolo8n) model. It does not propose any targets (no binding boxes, no classes, no labels). Hence, I cannot do backpropagation. I’ve tried decreasing model.conf, but it had no effect. Same thing happens on my custom dataset.</p>
<p>What should I tweak to get some predicted targets, so that the backprop (hence, training) can actually start?</p>
<pre><code class="lang-auto">from os import environ
import torch
from ultralytics import YOLO
from torch.optim import Adam


def dummy_train_step():
    images = torch.rand(2, 3, 640, 640).cuda()
    gt_boxes = [torch.tensor([[0, 0.5, 0.5, 0.2, 0.2]]).cuda(), torch.tensor([[0, 0.7, 0.7, 0.3, 0.3]]).cuda()]
    model = YOLO("yolov8n.pt")
    model.args['data'] = None
    model.conf = 0.0000001  # or 0.1, or 0.00000000000001
    optimizer = Adam(model.parameters(), lr=0.001)
    for step in range(100):
        print(step)
        predictions = model(images)
        print(f"Predictions: {predictions}")
        pred_boxes = predictions[0].boxes.xywh
        pred_scores = predictions[0].boxes.conf
        pred_labels = predictions[0].boxes.cls
        preds_flat = torch.cat([pred_boxes, pred_scores.unsqueeze(-1), pred_labels.unsqueeze(-1)], dim=-1)
        if preds_flat.size(0) == 0:
            print("No detections in the current batch.")
            continue
        targets_flat = torch.cat(gt_boxes, dim=0)
        print('Shape of targets_flat: ', targets_flat.shape)
        loss = model.loss(preds_flat, targets_flat)
        print('Loss: ', loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


if __name__ == "__main__":
    environ['OMP_NUM_THREADS'] = '1'
    dummy_train_step()

</code></pre>
<p>orig_shape: (640, 640)<br>
path: ‘image1.jpg’<br>
probs: None<br>
save_dir: ‘runs\detect\predict’<br>
speed: {‘preprocess’: 0.0, ‘inference’: 5.001425743103027, ‘postprocess’: 1.2627840042114258}]<br>
No detections in the current batch.</p>
Pairwise distance calc for large tensors

abrahamezzeddine: <p>Hello,</p>
<p>I am wondering if anyone has an idea on how to memory efficiently calculate pairwise distance between points? My batch has lots of data, approx 150000 points per batch. I realise that with torch.cdist, it creates an enormous tensor due to (N^2). I have tried looking around for a solution but it seems that I am not able to find it.</p>
<p>I would be very happy if anyone could share a code snippet for this? I found keops, but unfortunately it’s not supported on Windows.</p>
<p>I know I could also use FAISS knn, but that limits the query windows as it is k-limited, as I would like the points to attend to all points inside the batch.</p>
<p>I am still new and learning this concept, so please bear with me. =) thanks!!</p>

KFrank: <p>Hi Abraham!</p>
<p>Could you explain your use case a little more?</p>
<aside class="quote no-group" data-username="abrahamezzeddine" data-post="1" data-topic="215065" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abrahamezzeddine/48/74944_2.png" class="avatar"> abrahamezzeddine:</div>
<blockquote>
<p>My batch has lots of data, approx 150000 points per batch. I realise that with torch.cdist, it creates an enormous tensor due to (N^2).</p>
</blockquote>
</aside>
<p>With the simplest interpretation of what you seem to be doing, you will need a lot<br>
of memory.  If you need to store all pairwise distances of 150000 point in memory<br>
at the same time, it will take about 90 GB (assuming four-byte <code>floats</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>I would like the points to attend to all points inside the batch.</p>
</blockquote>
</aside>
<p>Normally by “batch” we mean a batch of independent samples that don’t interact in<br>
a substantive way with one another.  They are batched together so that they can be<br>
passed though the network at the same time to make more efficient use of the gpu<br>
(or cpu).</p>
<p>If I understand you correctly, you want to compute the distance between one sample’s<br>
point and another sample’s point, but now these samples wouldn’t be being treated<br>
independently.  Do you really need to compute <em>all</em> pairwise distances?  Do you really<br>
need to store them all in memory at the same time?  If so, please provide a little more<br>
detail about your use case,and we’ll see if we can offer any practical (if complicated)<br>
suggestions for how to reduce your peak memory usage.</p>
<p>Best.</p>
<p>K. Frank</p>
Pytorch binaries with cuda version 12.2/12.0

gal_kesten: <p>I want to know if PyTorch version 2.5.1+cu124 (with the CUDA 12.4 binary) will work on a computer where the CUDA Version is 12.2 or 12.0 as indicated by <code>nvidia-smi</code>.</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used as PyTorch binaries ship with their own CUDA runtime. The driver requirements are explained <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#id1">here</a>.</p>
Why doesn't torch have a memmap-like function?

Rain_river: <p>Hi all, I’m working with large datasets and noticed that while <code>numpy</code> offers a <code>memmap</code> function to map large arrays from disk to memory without fully loading them, torch doesn’t seem to have a similar feature. Is there a particular reason for this?</p>

JuanFMontesinos: <p>Whereas I’m not a core developer, I’d say pytorch is not focused on providing data engineering tools. What’s the point of maintaining such a tool when it’s already implemented for parquet, numpy, hdf5 and many other frameworks?</p>
Running inference of a model as part of loading training data for another model

jhnthndr: <p>Hi,</p>
<p>I have been training a primary model in pytorch lightning. Now I would like to make modifications to my training data by running inference of another already trained secondary model on the training data and using the output for the modifications. I have tried to include the call to the secondary model both in the collate function and in the Dataset class but in both cases I get an error</p>
<pre><code class="lang-auto">"RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with
multiprocessing, you must use the 'spawn' start method
</code></pre>
<p>This led me to try adding <code>mp.set_start_method('spawn', force=True)</code> at different places in my start script but this causes another problem related to multiprocessing and pickling:</p>
<pre><code class="lang-auto">AttributeError: Can't pickle local object 'convert_frame.&lt;locals&gt;._convert_frame'
</code></pre>
<p>Searching I have seen that lambda functions may be the cause of this but I have not been able to identify such a source of the problem. Also that stack trace is from deep within pytorch lightning and multiprocessing package code.</p>
<p>I would think that wanting to run inference of one model during data loading for training another model would not be that uncommon but I have found close to nothing on the subject. Am I on an impossible mission or does anyone here have suggestions to how I could get this to work?</p>
<p>Alternatively I could preprocess my dataset with the output of the secondary model but I find that inflexible and unelegant so I would prefer my initial approach</p>
Is model.forward(x) the same as model.__call__(x)?

ToruOwO: <p>When I worked with Tensorflow, I used to define a model’s forward pass and other customizations under its <code>def __call__(self, x)</code> function. If I want to implement the same thing in PyTorch, should I do it in <code>def forward(self,x)</code> instead?</p>

kira: <p>It is just a naming convention. Why do you want to define <strong>call</strong>(x) again inside the forward function? It isn’t needed, just use the forward function to get the outputs for the input data. Check this tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>

ptrblck: <p><code>__call__</code> is already defined in <code>nn.Module</code>, will register all hooks and call your <code>forward</code>. That’s also the reason to call the module directly (<code>output = model(data)</code>) instead of <code>model.forward(data)</code>.</p>

cosmosaa: <p>The problem with using <code>__call__</code> is that it is impossible for the IDE to know that <code>__call__</code> calls <code>forward</code>. Therefore, the IDE cannot help you with passing the correct arguments to <code>forward</code> for example.</p>
<p>This has created major unknown bugs in my code before. For example, I accidentally passed the wrong tensor to <code>forward</code>. If the IDE knew the correct arguments to <code>forward</code> then this issue would have been caught.</p>

Rishi_Malhotra: <p>wondering if you found a way around this issue?</p>
<p>currently, i am using .forward everywhere in my code until i’m ready to train. then i’ll bulk find and replace .forward for <strong>call</strong></p>

cosmosaa: <p><a class="mention" href="/u/rishi_malhotra">@Rishi_Malhotra</a> that’s an interesting solution.</p>
<p>I haven’t done this before, but I guess you could override the <code>__call__</code> method to call the parent <code>__call__</code> method and specify the arguments in the overridden one. Example:</p>
<pre><code class="lang-auto">class Foo(nn.Module):
    def forward(self, bar: torch.Tensor):
        print(bar.shape)

    def __call__(self, bar: torch.Tensor):
        super().__call__(bar=bar)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f918b731e9f32e245dbd28a71cf55f278e8d7ba3.png" alt="Screenshot 2024-08-18 at 8.35.32 AM" data-base62-sha1="zxBQgO4AOqLaqv9HkLvlqS8V52r" width="237" height="148"></p>

milmillin: <p>For annotating <code>__call__</code> to be the same as <code>forward</code>. You can see my workaround on a Github issue here<br>
<a href="https://github.com/pytorch/pytorch/issues/74746#issuecomment-2574443618" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/74746#issuecomment-2574443618</a></p>
Predicting pregnancy codes with transformer

random42: <p>Im trying to predict pregnancy codes with a basic transformer model architecture. These pregnancy codes are like following prg001, prg002 to prg030. Prg001 would be antenatal screening and prg030 would be maternal outcome of delivery.</p>
<p>The source is codes in one year (2022 for example) and the target is the codes in the following year (2023 for example).</p>
<p>My dataset has about 42k rows. About half (22k rows) have empty target (no code because the pregnancy ended in the previous year).</p>
<p>The model is basically predicting zero code no matter what sequence im testing. For example, its not predicting that if a person has prg008 in the prior year, they would have prg030 (the code that represents delivery) in the target because they would give birth in the following year.</p>
<h1><a name="p-461560-hyperparameters-1" class="anchor" href="#p-461560-hyperparameters-1"></a>Hyperparameters</h1>
<pre><code>max_seq_length = 20
max_gen_length = 20
learning_rate = 0.001
weight_decay = 0.01
dropout = 0.1
d_model = 128
nhead = 4
num_layers = 3
dim_feedforward = 512
batch_size = 256
train_num_samples = 800000
valid_num_samples = 200000
num_epochs = 10
</code></pre>
<p>Im using crossentropyloss. Any suggestion on how i can improve this?</p>

deepsthecoder: <aside class="quote no-group" data-username="random42" data-post="1" data-topic="214296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e8c25b/48.png" class="avatar"> random42:</div>
<blockquote>
<p>The model is basically predicting zero code no matter what sequence im testing</p>
</blockquote>
</aside>
<p>Dear random42,<br>
If I were you I would test the following.</p>
<ul>
<li>
<p>check if my implementation is correct and not any bugs in the code. Right from Dataloader and until the loss.backward(), check if the backpropagation is happening.</p>
</li>
<li>
<p>If everything is correct then i will use weighted Cross entropy or Focal loss such that minority classes have high weights while loss calculation.</p>
</li>
</ul>

random42: <p>Hi!<br>
I used the same architecture to test a simple text reversal task and it worked correctly so i think architecture is implemented correctly.<br>
Does pytorch have FocalLoss? Or do you have an example of how to use weighted cross entropy?</p>
GNN for electrical distribution systems, HELP

Diego_Paul_Guachichu: <p>Hello, I’m Diego G and I’m interested in the application of graph neural networks in power flow in electrical distribution systems, or in state estimation. I have to admit that I’m new to this topic of graph neural networks, but I’m very interested.<br>
I need your help to create a correctly structured algorithm or if you can help me with material so I can read it and implement it in my work. I’ve implemented a code but it seems to me that it’s not structured correctly.</p>
<p>I appreciate your help and your time in advance.</p>
Multiple GPU code isn't initializing DDP model

lbadi: <p>When I run below code with world size=1, everything runs fine but when I run with world size&gt;1, model = DDP(model, device_ids=[rank]) isn’t executed.</p>
<p>Here is the code I’m running:</p>
<pre><code class="lang-auto">import os
import numpy as np
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, Dataset, DistributedSampler
from tqdm import tqdm
from torchvision.datasets import MNIST
from torchvision import transforms

bsize = 5000

def seed_everything(seed):
    import random
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '12345'
    dist.init_process_group("nccl", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def train(rank, world_size):    
    setup(rank, world_size)
    
    torch.cuda.set_device(rank)
    device = torch.device(f"cuda:{rank}")
    
    model = torch.nn.Linear(10, 10).to(device)
    
    model = DDP(model, device_ids=[rank])
    
    
    dataset = MNIST(root='./data', download=False, transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ]), train=False)    
    sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank, drop_last=True)
    dataloader = DataLoader(dataset, batch_size=bsize, sampler=sampler)
    

    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
    
    for epoch in range(5):
        sampler.set_epoch(epoch)
        epoch_loss = 0.0
        for inputs, targets in tqdm(dataloader, desc=f"Rank {rank}, Epoch {epoch}"):
            inputs, targets = inputs.to(device), targets.to(device)
            optimizer.zero_grad()
            inputs = inputs.view(inputs.size(0), -1)
            outputs = model(inputs)
            loss = criterion(outputs, targets)
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
        
        print(f"Rank {rank}, Epoch {epoch}, Loss: {epoch_loss:.4f}")
    
    cleanup()

if __name__ == "__main__":
    
    seed_everything(1)
    world_size = torch.cuda.device_count()
    seed = 42  # Set a fixed seed for reproducibility    
    mp.spawn(train, args=(world_size, seed), nprocs=world_size, join=True)

</code></pre>

agu: <p>Can you try with <code>os.environ['MASTER_PORT'] = '29500'</code> instead? This seems to fix the issue for me.</p>
Torch.distributed.all_reduce causes memory trashing

SzymonOzog: <p>I’ve noticed a lot of cuda memory frees when profiling my code. Doing memory profiling it seems like the output of all reduce doesn’t get reused by the caching allocator but gets accumulated and then freed.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5838d755d5ca5296ddc8137c78a5014be93befef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5838d755d5ca5296ddc8137c78a5014be93befef" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5838d755d5ca5296ddc8137c78a5014be93befef_2_690x284.jpeg" alt="image" data-base62-sha1="cArNxogFOCVoqpt9Ni3sNSd56fd" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5838d755d5ca5296ddc8137c78a5014be93befef_2_690x284.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5838d755d5ca5296ddc8137c78a5014be93befef_2_1035x426.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/5838d755d5ca5296ddc8137c78a5014be93befef.jpeg 2x" data-dominant-color="443C38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1362×562 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tested it out and disabling all reduce after my linear layer fixes the issue.</p>
<p>Tested on torch 2.4.0 and 2.5.1</p>

SzymonOzog: <p>Okay using</p>
<p><code>TORCH_NCCL_AVOID_RECORD_STREAMS=1</code></p>
<p>fixed it, shouldn’t it be the default? Refering to <a href="https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573" class="inline-onebox">CUDA allocation lifetime for inputs to distributed.all_reduce</a></p>

agu: <p>Yes, this env var is the correct fix. I think we want it to be the default, but it is a major change and requires some effort from someone to drive the flip.</p>
Custom kernels for Intel GPUs

Matthias_Moller: <p>Is it possible to write (simple) custom kernels for Intel GPUs similar to the approach for NVIDIA GPUs using CUDA extensions?</p>
<p>I am used to programming in C++ and started looking into sycl and some of the XPU functionality in PyTorch. However, I miss a minimal example showing how basic functions like Tensor.pow() are implemented for XPU backends.</p>

EikanWang: <p>Hi <a class="mention" href="/u/matthias_moller">@Matthias_Moller</a> , we are working on this to provide the <code>C++</code> extension for Intel GPU and we will keep you posted as soon as the PR being landed.</p>

Matthias_Moller: <p>Hi <a class="mention" href="/u/eikanwang">@EikanWang</a>, I am very happy to hear that this is WIP. Is there maybe some part of the LibTorch source code (C++ API) that I could already look at to see how the existing kernels are implemented? As said (maybe in my other post), I am not using the Python API but directly the C++ API, which might make it easier to integrate a few extra kernels.</p>

EikanWang: <p>By now, we have not distributed the libtorch for XPU. I think the <a href="https://github.com/intel/intel-extension-for-pytorch/tree/release/xpu/2.5.10" rel="noopener nofollow ugc">Intel Extension for PyTorch</a> may be another example to demonstrate how to extend aten operation through SYCL. However, it is sort of heavy to understand the logic. Is the <code>libtorch</code> must-to-have for your case?</p>

EikanWang: <p>I noticed your other post and we will update there.</p>

Matthias_Moller: <p>Yes, libtorch is crucial as my entire code is written in c++. When installing PyTorch as described in <a href="https://pytorch.org/docs/stable/notes/get_start_xpu.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started on Intel GPU — PyTorch 2.5 documentation</a> I can find the libtorch libraries somewhere in the python installation. Except for my few custom kernels my code runs fine.</p>

EikanWang: <p><a href="https://github.com/pytorch/pytorch/pull/132945" rel="noopener nofollow ugc">xpu: support sycl with torch.utils.cpp_extension APIs by dvrogozh · Pull Request #132945 · pytorch/pytorch</a> FYI</p>
Installing Torch globally on Linux/Ubuntu

RubixCube: <p>Hello all,</p>
<p>I have a small question regarding installing Torch on Linux, in particular on Ubuntu/Ubuntu-based distributions. It’s mostly a Python/pip management question, still strongly related to Torch, thus I think it’s a good point to ask here (and I hope I selected an appropriate category).</p>
<p>On Windows, usually my globally installed Python packages are all organized by <code>pip3 install</code> (maybe extended by a few preinstalled ones). However on Linux, some of them are installed by the package manager, either explicitly by the user or implicitly as a dependency of installed programs. Therefore, I always try to install the required Python packages using the package manager (i.e. apt on Ubuntu), and of course avoid global pip installations. This works fine with almost all packages (numpy, pandas, etc.). Now, Torch is a special case here. First, it is not available by a apt PPA (i.e. installation requires pip) and it has a quite a lot of dependencies that could be provided partially by the system (e.g. numpy), while others could still be required but are not installed so far.</p>
<p>Clearly, the go-to solution on Linux is using virtual envorionments and install Torch inside one, which works fine. However using multiple virtual environments or using multiple users on the same machine, this approach installs Torch multiple times (at least to the best of my knowledge). Thus, I am looking for a solution to install Torch globally using pip, but to use the provided System packages (e.g. numpy) as much as possible to avoid redundancies and incompatabilities. To avoid inferences with system packages, a valid installation location is <code>usr/local/lib/pythonX.XX</code> (at least on Ubuntu and derived distros), i.e. I use:</p>
<pre><code class="lang-auto">sudo pip3 install --target '/usr/local/lib/python3.12/dist-packages' &lt;list of packages to install&gt;
</code></pre>
<p>Installing Torch this way works, but there are some issues: It installs numpy-2.X, whereas Ubuntu’s <code>python3-numpy</code> Debian package is still numpy-1.X. To the best of my knowledge, there are also Torch builds available that work with numpy-1.X.</p>
<p>Therefore, I would like to use some <code>pip3 install</code> command with some option <em>try to use the existing debian packages and only install dependencies that are not resolved by those</em> to avoid conflicts like parallel installing numpy-1 (by apt to <code>/usr/lib/python3</code>) and numpy-2 (by pip to <code>/usr/local/lib/python3.12</code>). In case of incompatabile selections (e.g. preinstalled numpy is 1.X whereas some package <em>requires</em> 2.X and is not compatible with 1.X), either install the dependency or exit with an error (I’d prefer the error). This would be a great way of maintining system-wide Python packages (in particular Torch) parallel to the system-wide ones installed by the system’s package manager.</p>
<p>Is this possible somehow?</p>
<p>Appreciating any comments and thanks in advance!</p>
Does PyTorch provide packages with _GLIBCXX_USE_CXX11_ABI=1?

risemeup1: <p>Are the packages provided on the PyTorch website defaulted to <code>_GLIBCXX_USE_CXX11_ABI=0</code> ? If I need a PyTorch package with <code>_GLIBCXX_USE_CXX11_ABI=1</code> , do I have to build it from source?</p>

ptrblck: <p>Building from source would work, but you can also try out the nightly binaries with CUDA 12.6.3 support as these should already be built with CXX11 ABI.</p>

risemeup1: <p>thanks very much,thanks very much</p>
Is pytorch Rprop implementation incorrect?

qq-me: <p>Hi, I believe RProp implementation is slightly incorrect (according to <a href="https://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.2016/pdfs/Rprop.pdf" rel="noopener nofollow ugc">https://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.2016/pdfs/Rprop.pdf</a>).</p>
<p>here is an example on a quadratic function <code>f(x) = x^2</code></p>
<pre data-code-wrap="py"><code class="lang-py">param = torch.tensor(2., requires_grad=True)
opt = torch.optim.Rprop([param], 1)
last_param = param.clone()
loss = param**2
print(f'param = {round((last_param).item(), 3)}, loss = {round(loss.item(), 3)}')
for i in range(10):
    opt.zero_grad()
    loss.backward()
    opt.step()
    loss = param**2
    update = round((param-last_param).item(), 3)
    last_param = param.clone()
    print(f'param = {round((last_param).item(), 3)}, loss = {round(loss.item(), 3)}, grad = {round(param.grad.item(), 3)}, {update = }')
</code></pre>
<p>those are the first six output lines:</p>
<pre data-code-wrap="py"><code class="lang-py">param = 2.0, loss = 4.0
param = 1.0, loss = 1.0, grad = 4.0, update = -1.0
param = -0.2, loss = 0.04, grad = 2.0, update = -1.2
param = -0.2, loss = 0.04, grad = -0.4, update = 0.0
param = 0.4, loss = 0.16, grad = -0.4, update = 0.6
param = 0.4, loss = 0.16, grad = 0.8, update = 0.0
</code></pre>
<p>notice the last line. When the derivative changes, Rprop is supposed to perform a ‘backtracking’ weight-step.<br>
<code>If the partial derivative changes sign, i.e. the previous step was too large and the minimum was missed, the previous weight-update is reverted. Due to that 'backtracking' weight-step, the derivative is supposed to change its sign once again in the following step.</code></p>
<p>However pytorch doesn’t perform a backtracking step and instead doesn’t perform a step at all with that weight.</p>
<p>Update -1.2 moved parameter from 1 to -0.2, which changes the sign of the gradient to minus. Rprop is supposed to undo that update and move back to 1, while multiplying the update by n-. The adaption is disabled because that changes sign of the gradient back to plus. So the next update will be -0.6, which will move param to 0.4. However pytorch doesn’t perform the backtracking step, and keeps parameter at -0.2.</p>
<p>Here is what I believe it should look like</p>
<pre data-code-wrap="py"><code class="lang-py">param = 2.0, loss = 4.0
param = 1.0, loss = 1.0, grad = 4.0, update = -1.0
param = -0.2, loss = 0.04, grad = 2.0, update = -1.2
param = 1.0, loss = 1.0, grad = -0.4, update = 1.2
param = 0.4, loss = 0.16, grad = 2.0, update = -0.6
param = -0.32, loss = 0.102, grad = 0.8, update = -0.72
</code></pre>

soulitzer: <p>Good observation, its actually using a different variant of rprop that doesn’t do the backtracking.</p>
<p><a href="http://www.neuro.nigmatec.ru/materials/themeid_18/igel00improving.pdf" class="onebox" target="_blank" rel="noopener nofollow ugc">http://www.neuro.nigmatec.ru/materials/themeid_18/igel00improving.pdf</a></p>
<p>Maybe its worth mentioning this on the docs.</p>
Cant pip install torch on MacOS Sonoma 14.6.1

error_torcz: <p>Got this error trying to install through pip and pip3:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
</code></pre>
<p>These are my Python versions:<br>
Python 3.9.6<br>
Python 3.10.9</p>
<p>I use VSC.</p>

soulitzer: <p>Could you share the command that you used to install torch?</p>
Register a python object with TORCH_LIBRARY_IMPL

ChanHeo: <p>Hello.</p>
<p>Before I ask, I apologize for my lack of knowledge as this is my first time asking questions about open source projects.</p>
<p>I’ve seen in the manual how to use the <code>TORCH_LIBRARY_IMPL</code> macro to register a kernel pointer that maps to a specific backend key.<br>
However, this is only available for the C++ implementation.</p>
<p>Is it possible to register a <strong>python object</strong> inside the <code>TORCH_LIBRARY_IMPL</code> C++ macro?</p>
<p>Inevitably, I need to call the python object implemented at the python level like a kernel, which I’ve found can only be registered by using <code>library.impl</code> or equivalent decorator at the python level.</p>
<p>So, while analyzing the source code, I found <code>at::impl::PythonOpRegistrationTrampoline</code>, which seemed to can possible to register a Python module in TORCH_LIBRARY_IMPL in C++.</p>
<p>Would it be okay to use this? and if it works, would it be okay to ask review the code for upstream?</p>

soulitzer: <p>Are you looking for something like <a href="https://pytorch.org/docs/stable/library.html" class="inline-onebox" rel="noopener nofollow ugc">torch.library — PyTorch 2.5 documentation</a></p>

ChanHeo: <p><code>torch.library</code> you mentioned allows you to register a python object as a kernel for a specific backend at the python level.<br>
But, I’m looking for a way to register Python objects at C++ level. so, I’m looking for a way to register a Python kernel using a C++ macro like <code>TORCH_LIBRARY_IMPL</code>.</p>

soulitzer: <p>It certainly seems possible as you point out, but out of curiosity, why do you need to support this?</p>
<p>For a single op definition, it is okay to have a partially initialized fragment whether that is in cpp or in python. So someone should be able to import your cpp extension and later import the python module that registers the python kernel impl.</p>
Will profiler.record_function be affected by the asynchronous execution?

ideaV: <p>In the <a href="https://pytorch.org/tutorials/beginner/profiler.html#performance-debugging-using-profiler" rel="noopener nofollow ugc">doc of pytorch profiler</a>, it suggests that you can wrap each line of your code into a context manager to find the bottleneck of your code.</p>
<p>However, I’m wondering whether the asynchronous nature of cuda will cause inaccurate measurement in this case? Should we call torch.cuda.synchronize() before entering and leaving the context?</p>

soulitzer: <blockquote>
<p>Should we call torch.cuda.synchronize() before entering and leaving the context?</p>
</blockquote>
<p>You should be fine just to execute the code as usual</p>
Dropout design choice

Siddhanth_Ramani: <p>Hi,<br>
Why does the dropout implementation in test time scale the outputs and not the weights?<br>
Wouldn’t scaling the weights give the same result as scaling the outputs but requires only a single time operation?</p>
<p>Ex. If dropout applied with “p”.<br>
And for a single neuron, o = x<em>w, where o is the output, x is the input, and w is the weight, then<br>
p</em>o = x<em>p</em>w.<br>
But the RHS multiplication is a single time op (better for inference model deployed).<br>
What was the design choice behind scaling outputs in model.eval mode?</p>
<p>Thank you,<br>
Siddhanth Ramani</p>

ptrblck: <aside class="quote no-group" data-username="Siddhanth_Ramani" data-post="1" data-topic="214984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddhanth_ramani/48/62072_2.png" class="avatar"> Siddhanth_Ramani:</div>
<blockquote>
<p>Why does the dropout implementation in test time scale the outputs and not the weights?<br>
Wouldn’t scaling the weights give the same result as scaling the outputs but requires only a single time operation?</p>
</blockquote>
</aside>
<p>You are thinking about <a href="https://paperswithcode.com/method/dropconnect"><code>DropConnect</code></a>, which drops the weights rather than the activations.</p>
<aside class="quote no-group" data-username="Siddhanth_Ramani" data-post="1" data-topic="214984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddhanth_ramani/48/62072_2.png" class="avatar"> Siddhanth_Ramani:</div>
<blockquote>
<p>But the RHS multiplication is a single time op (better for inference model deployed).</p>
</blockquote>
</aside>
<p>I don’t understand this claim, as <code>Dropout</code> is usually not used during inference and even if you want to use dropout during inference, the masking will still be random.</p>
Bfloat16 on nvidia V100 gpu

602389789: <p>Hello everyone!<br>
It is said that bfloat16 is only supported on GPUs with compute capability of at least 8.0, which means nvidia V100 should not support bfloat16.</p>
<p>But I have test the code below on a V100 machine and run successfully.</p>
<pre><code class="lang-auto">
a = torch.randn(3,3,dtype=torch.bfloat16,device="cuda")
b = torch.randn(3,3,dtype=torch.bfloat16,device="cuda")
c = torch.matmul(a,b)
print(c.dtype)
print(c.device)
</code></pre>
<p>and get the result</p>
<pre><code class="lang-auto">torch.bfloat16
cuda:0
</code></pre>
<p>but when I run print(torch.cuda.is_bf16_supported()), I got False</p>
<p>So what is situation here?</p>

ptrblck: <p>Creating tensors with <code>bfloat16</code> might be supported on older architectures, but the actual compute kernels would not be.</p>

602389789: <p>So does that mean although the type is bfloat16 but the computation is actually run in fp32 way in GPU?</p>

ptrblck: <p>Yes, older hardware which does not support <code>bfloat16</code> compute will emulate it via <code>float32</code> compute.</p>

602389789: <p>Thanks for you answer!<br>
And I have tried to do bfloat16 mixed precision on V100, the time cost is almost the same as the full fp32 training (even a little slower).</p>

too_many_watts: <p>Any source code pointers to where/how this emulation happens? I’m super curious about what’s going on under the hood.</p>

Qichao_Ying_CN: <p>hi! very strange. I am using DGX V100 and the following result</p>
<blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.cuda.is_bf16_supported())<br>
True<br>
Also I could train models using bf16 in Deepspeed</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p>I am using CUDA 12.4, PyTorch 2.5.0 and NVIDIA driver 550.127.08.</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/9f94710e48bfefc6a8e32af956e45d5a847c6467/torch/cuda/__init__.py#L132-L159"><code>is_bf16_supported()</code></a> checks is the used CUDA toolkit used to build PyTorch supported BF16 as well as if the compute capability of the device is Ampere+. However, even if these checks fail, the test will try to allocate a <code>BF16</code> tensor and returns the support based on the success of the tensor creation.<br>
IMO, creating a tensor alone is not sufficient to claim <code>BF16</code> is supported as math operations could fail (and will in the case of Volta devices).</p>
Triton kernel to efficiently dequantize int4

vgoklani: <p>We would like to quantize LLAMA-3.3 70B using int4 quantization, analogous to what was done in the gpt-fast repo.</p>
<p>Is there a fast triton based de-quantize code sample that we could use for making the forward pass implementation fast?</p>
<p>Are there any code examples for de-quantizing using triton that we could use for experimenting?</p>
<p>Or would torch.compile magically handle this for us?</p>
<p>Thanks!</p>
Process Pool freezes

ACitronella: <p>The code below just freezes, and I don’t really know why. However, one of these small edits makes the code run just fine.</p>
<ul>
<li>remove “weights=torchvision.models.DenseNet121_Weights.DEFAULT”</li>
<li>reduce the size of the returned tensor (i.e. 3, 224, 50 is a maximum)</li>
</ul>
<pre><code class="lang-auto">from tqdm import tqdm
import torch
import torchvision
from torch.multiprocessing import Pool

def load_item(idx):
    print(idx)
    return torch.ones(3, 224, 224, dtype=torch.float32)
    
if __name__ == "__main__":
    model = torchvision.models.densenet121(weights=torchvision.models.DenseNet121_Weights.DEFAULT)
    with Pool(8) as p:
        dataset = p.map(load_item, tqdm(range(1000)))
</code></pre>
<p>The system I use</p>
<ul>
<li>Ubuntu 20.04.4 LTS with 252 GB of RAM</li>
<li>python 3.8.18</li>
<li>torch 2.2.1+cu118</li>
<li>torchvision 0.17.1+cu118</li>
</ul>
<p>I cannot replicate the bug on Windows.</p>
<p>Is there any particular reason why this code froze?</p>
Building a Model for Multi-Output Embedding Generation: Seeking Advice and Insights

eyalbenbarouch1997: <p>Hi everyone,</p>
<p>I’m currently working on an deep learning project and would love to get your thoughts and advice on a few key aspects. Here’s an overview of what I’m doing:</p>
<p>I’m developing a model that takes a single high-dimensional embedding vector (256 dimensions) as input and generates three distinct high-dimensional embedding vectors (each 256 dimensions) as output (right now as a concatenated 768 vector output). The goal is to enable the model to learn meaningful representations for multi-output embeddings based on the input data. This is a supervised learning task, and I have a dataset with several hundred thousand input-output pairs (input to 3 outputs) to work with.</p>
<h4><a name="p-462630-model-architecture-1" class="anchor" href="#p-462630-model-architecture-1"></a><strong>Model Architecture</strong>:</h4>
<ul>
<li>What types of architectures would you recommend for multi-output embedding generation?</li>
<li>Are there specific design patterns or loss functions you’d suggest to ensure each output embedding is unique and well-represented?</li>
</ul>
<p>I’ve experimented with a few approaches so far:</p>
<ol>
<li><strong>Straightforward Neural Network</strong>: A simple fully connected network has been surprisingly effective and performs well overall.</li>
<li><strong>Transformer Decoder</strong>: I’ve also tried a Transformer decoder-based architecture, which works decently but isn’t as efficient as I’d like.</li>
<li><strong>GAN Architecture</strong>: Since this task involves generating embeddings, I’ve been considering a GAN-based approach. Do you think this could be effective for high-dimensional embedding generation? Any advice on GAN design for this type of task?</li>
</ol>
<p>Additionally, I’d prefer not to fine-tune LLM due to resource constraints. However, I might explore fine-tuning a smaller 1B-parameter model. If you have insights on:</p>
<ul>
<li>How to modify or fine-tune such a model to directly output embeddings instead of text.</li>
</ul>
<h4><a name="p-462630-loss-functions-2" class="anchor" href="#p-462630-loss-functions-2"></a><strong>Loss Functions</strong>:</h4>
<ul>
<li>I’ve experimented with the following:
<ol>
<li><strong>Cosine Similarity Loss</strong>: It works well for aligning the embeddings with the expected direction.</li>
<li><strong>MSE + Cosine Similarity Loss</strong>: This combination seems to strike a good balance, optimizing both magnitude and directional alignment.</li>
</ol>
</li>
</ul>
<p>Are there other commonly used or innovative loss functions for high-dimensional vector generation tasks?<br>
For example, any techniques that emphasize diversity between outputs or enforce specific constraints on embedding properties?</p>
<p>Thank You!</p>
Updating masks in the Pruning

Shashank_Priyadarshi: <p>I am pruning a model where I am pruning the model to make it 40% sparse. So to do this I am doing iteratively where my pruning_percent = 40/Num of batches and I am pruning the model every mini-batch by the pruning_percent. For example if I just have 2 mini-batches then I am pruning 20% weights in the first mini batch and rest of the 20% in the next mini batch.</p>
<p>So to achieve this I first pruned the model with mask as all ones so that I can have weight_orig and weight_mask. And then I am running this iterative pruning with my pruning logic and based on the threshold I am pruning the weights and updating the layer.weight_mask = mask[layer]. This pruned the model to 40% but when I am running my evaluation method on this pruned model then I get NaN outputs.</p>
<p>Is it because I am somewhere messing up with the grads while pruning?</p>
<p>Any thoughts on this?</p>

Shashank_Priyadarshi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any thoughts on this?</p>
Move ExportedProgram to CPU

Dheeraj_Peri: <p>Can we have a utility which moves the <code>ExportedProgram</code> from GPU to CPU ? My usecase is I want to run <code>exp_program = torch.export.export(model, inputs)</code> on GPU (since it is faster) and later move the exp_program to CPU to preserve the GPU memory for other tasks (TensorRT compilation). Something like <code>exp_program.to("cpu")</code> would help. If there’s another way to do this currently, please share. Thank you</p>

soulitzer: <p>Isn’t exporting your model something you only do once, and then run the exported model many times. If that is the case, maybe its just better to trace on CPU even if it is slower?</p>
<p>How much slower is the exporting on CPU? Taking much longer would unexpected to me since you’re tracing with real tensors either way.</p>

Dheeraj_Peri: <p>I have a script which initializes a Pytorch model, exports it and then converts to TensorRT. In my case, I have to run this script and export it multiple times for debugging purposes. This could in theory be split into two scripts but that’s not very convenient.<br>
How much slower depends on the model. In my case of Flux 12B model, I see a considerable difference. Having <code>.to</code> utility on ExportedPrograms to move then to CPU and release the GPU memory would be helpful in general.</p>

bdhirsh: <p>Is there a reason you expect the export process faster to be faster on the gpu than the cpu?</p>
<p>Export doesn’t actually do any compute (it traces your program into a graph with symbolic execution, but doesn’t need to actually run any kernels). So I would be surprised if export ran faster when your model is on the gpu.</p>
<p>One reason that adding an ExportedProgram.to() API is a bit sketchy is that we don’t promise to trace your model in a device-generic way. For example, given a function like this:</p>
<p>def f(x):<br>
y = x.to(device=“cuda”)<br>
y.mul_(2)<br>
return x + 1</p>
<p>This function will end up mutating its input if you pass in a cuda tensor, but it won’t if you pass in a cpu tensor (.to is a “maybe aliasing” operation, since it returns the input directly if no conversion is needed). Export will end up specializing on the aliasing behavior when it traces with the particular input tensors you passed in.</p>
<p>One thing to call out is that if you want to export a model for GPU but you can’t do it from a GPU-enabled machine, you can instantiate your model with cuda FakeTensors</p>
Dynamic shapes and PyTorch

Mark_Fanter: <p>Hello,</p>
<p>I’m curious about how training is done on datasets with dynamic shapes. For instance, in point cloud segmentation tasks you might have different sizes for each point cloud.</p>
<p>I’m immediately confused on how people handle these shapes, as even stacking two tensors of unequal size results in an error. The Dataloader class similarly expects that each item in a batch will have an identical shape - so you cannot even <strong>load</strong> such a dataloader.</p>
<p>From looking online, people suggest a workaround such as using collate_fn and padding the points, but this seems like a massive issue if your dataset has large deviations. For instance, what if one pointcloud has 1,000 vertices and another has 1 million? What if you have a batch size of 256, with all having ~1,000 points and the final one having 1 million? Suddenly, you need to clear space and compute gradients with 256 million vertices, not to mention the additional extracted features.</p>
<p>Another ‘solution’ would be to use a batch size of 1, but this results in other similar issues. Perhaps using a batch size of 1 and accumulating gradients is the solution? This still results in significantly reduced performance.</p>
<p>The main question: I would like to know how one generally deals with a dataset which has varying, dynamic shapes to classify. Any information would be great.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

soulitzer: <p>You might be interested in NestedTensors <a href="https://discuss.pytorch.org/t/how-to-apply-vmap-on-a-heterogeneous-tensor/214109/2" class="inline-onebox">How to apply vmap on a heterogeneous tensor - #2 by soulitzer</a> which supports representing a tensor with varying shapes. (e.g., you can have your collate_fn create a nested tensor). The nested tensor behaves as if you had a batch of padded data, but without requiring any extra memory or compute.</p>

KFrank: <p>Hi Mark!</p>
<aside class="quote no-group" data-username="Mark_Fanter" data-post="1" data-topic="214929" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mark_fanter/48/74876_2.png" class="avatar"> Mark_Fanter:</div>
<blockquote>
<p>I would like to know how one generally deals with a dataset which has varying, dynamic shapes to classify.</p>
</blockquote>
</aside>
<p>The approach I generally take is to group together samples of approximately the same<br>
size and then pad the smaller samples up to the size of the largest sample in that<br>
group.  (If padding somehow confuses or breaks your network, then you would have<br>
to group together samples of equal size.)  It will depend on the nature of your dataset,<br>
but for many use cases it is possible to know the sizes of your samples in advance<br>
of batching.</p>
<p>Best.</p>
<p>K. Frank</p>