akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error when load network on GPU libTorch 2.5.1

username: weitaoliu
cooked: <p>Hello all,</p>
<p>I have an error when loading the network on GPU. I link the libTorch 2.5.1 with CUDA 12.4 to my codes. It works fine on my local machine (with GTX 1070) and a testing machine (with RTX 4070 Ti). However, when I move the codes to a computing node with A100, the solver does not work and throws an error when loading the network. The error looks like the following:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  _ivalue_ INTERNAL ASSERT FAILED at "XXPath_To_Codes/ThirdParty/libtorchCUDA/include/torch/csrc/jit/api/object.h":38, please report a bug to PyTorch. 
Exception raised from _ivalue at XXPath_to_Codes/ThirdParty/libtorchCUDA/include/torch/csrc/jit/api/object.h:38 (most recent call first):
</code></pre>
<p>I set up the same environment (cuda driver) on the computing node and have no idea how to address this issue. Do you have any suggestions?</p>
<p>If I choose to load on the CPU, then there is no problem.</p>
<p>Note that the A100 is split into 7 instances. I don’t know if that could be the issue.</p>
<p>Thanks for your time.</p>
<p>Best, Weitao.</p>
[Tensor Parallel] what will happen if some modules have no parallel plan?

username: FindDefinition
cooked: <p>I’m working on a large model TP training. the prep part of this model runs fast and contains small number of parameters, so I don’t want to design plans for them to decrease maintain cost.</p>
<ol>
<li>If a linear layer have no parallel plan (not in transformer blocks) and have replicate input, each device inside same TP group will have same weight gradient, do they get all-reduced across TP group?</li>
<li>If a linear layer have no parallel plan and have <strong>Shard</strong> input (local tensor), each device inside same TP group will have <strong>different</strong> weight gradient, do they get all-reduced across TP group?</li>
<li>If a RMSNorm layer have <strong>Sequence Parallel</strong> plan and have <strong>Shard</strong> input, its weight becomes Replicate, when do they get all-reduced across TP group?</li>
<li>optimizer will raise error mixed DTensor/Tensor if a model contains both local tensor (no parallel plan) and DTensor, we have two option for this, the first one is create two param groups and group them by type of tensor, the second one is create plans for all layers. which one is best in practice?</li>
</ol>
Add install instruction for Intel GPU xpu

username: silveroxides
cooked: <p>I suggest adding the install commands for installing torch on intel xpu on the front page considering the upcoming launch. Taking a look at their own site it is not much different from the Pytorch Get Started page and would be useful if such an instruction was in place on the official page before the launch of the new SKUs in 10 days. <a href="https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu&amp;version=v2.3.110%2bxpu&amp;os=windows&amp;package=pip" rel="noopener nofollow ugc">https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu&amp;version=v2.3.110%2bxpu&amp;os=windows&amp;package=pip</a></p>

username: Aknw_Fen
cooked: <p>that’s so cool, i hope they add it, many users have intel core processors aside from xpu devices, and will likely make a diff with avx512 instructions used, isn’t it?</p>
Installation error requirement torch

username: Arioner
cooked: <p>I’m trying to install PyTorch on my PC and I get this error messages.<br>
(cv) C:\Users\arion&gt;pip3 install torch torchvision torchaudio<br>
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch</p>
<p>My PC:<br>
CPU:AMD Phenom 2 1100T 6 core<br>
Ram:16GB 1600mhz DD3<br>
GPU: Asus ROG GTX1050TI</p>

username: ptrblck
cooked: <p>If you are using Python 3.13 on Windows, downgrade it to 3.12 as 3.13 binaries are not available on Windows yet.</p>
Training multiple independent models on one GPU/multiple GPUs using joblib

username: stonez
cooked: <p>Suppose we want to train 50 models independently, even if you have access to an online gpu clustering service you can probably only submit say10 tasks at one time. I want to figure out if it is possible to put all 50 models to multiprocessing training in one single script and train all of them concurrently. They are all independent models so there is no information exchange issue.</p>
<p>I was reading some posts discussing this topic, and seems some believe this is overall impossible, because only cpu do multiprocessing, while gpu can only train one model at one time. I still try two approaches, but they both seem to behave weirdly. The two approaches are both trying to train 50 models on gpus cuda:0 to cuda:3</p>
<ol>
<li>The first method I try is the build-in torch.multiprocessing. Suppose you have a console input args.num_gpu recording the current available number of gpus,</li>
</ol>
<pre><code class="lang-auto">    num_gpu = args.num_gpu
    id_list  # &lt;- a list of 50 elements ,each record the id for the specific model we want to train
    DEVICE = 'cuda:{}'
    
    processes = []

    mp.set_start_method("spawn")

    for pid, id_info in enumerate(id_list):
        gpu_id = pid % num_gpu
        process = mp.Process(target=run, args=(cfg, args, id_info, torch.device(DEVICE.format(gpu_id))))
        process.start()
        processes.append(process)
</code></pre>
<p>The method run here takes the assigned gpu and the model information, initializes the model, optimizer, loss function, datasets, dataloader, etc. and trains the model in a number of epochs.</p>
<p>What I observe is that some models are training, some models’ training process never get launched, and some models forward passing processes have the error of two tensors are on two devices, cuda:0 and cuda:X where X can be any gpu I assign in this task. The whole training behavior is messy and unpredictable.</p>
<ol start="2">
<li>The next method is joblib,</li>
</ol>
<pre><code class="lang-auto">    with parallel_backend('loky', n_jobs=-1):
        parallel = Parallel(n_jobs=-1)
        print(f"Number of jobs running: {parallel.n_jobs}")
        n_jobs = parallel.n_jobs

        parallel(
            delayed(run)(cfg, args, id_pair, torch.device(DEVICE.format(pid % num_gpu)))
            for pid, id_info in enumerate(id_list)
        )
</code></pre>
<p>If I set the number of gpu to be 1, it only trains one model. If I increase the number of gpu, then it trains that number of models simultaneously, but then during the forward pass I once again get an error says the tensors are on two different devices, and the two reported devices have some randomness, like the first time it reports they are on cuda:0 and cuda:1, the next time I rerun the script it says they are on cuda:0 and cuda:3 .</p>
<p>So my question is, does this whole idea of training a large amount of model simultaneously ever make sense? Or just as what I saw from the previous discussion that pytorch &amp; gpu does not actually support it?</p>
How can I implement operations on queue with supported MIL ops?

username: JimW
cooked: <p>I am trying to implement something like this:</p>
<blockquote>
<p>input_list = [torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8)]<br>
diff_tensor = [nn.KLDivLoss()(torch.tensor(input_list[-1]).log(), torch.tensor(input_list[i])).item() for i in range(5)]</p>
</blockquote>
<p>But obviously, the above implementation won’t be converted to coreML. So I tried to modify it to this:</p>
<blockquote>
<p>input_list = [torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8), torch.rand(1, 8)]<br>
input_tensor = torch.concat(input_list)<br>
cal_matrix =  torch.tensor([[-1, 0, 0, 0, 1],<br>
[0, -1, 0, 0, 1],<br>
[0, 0, -1, 0, 1],<br>
[0, 0, 0, -1, 1],<br>
[0, 0, 0, 0, 0]])<br>
diff_tensor = torch.matmul(cal_matrix.type(torch.FloatTensor), input_tensor)</p>
</blockquote>
<p>The new version can calculate the diff between adjacent elements as subtraction. However, the original algorithm requires a KLDiv operation, instead of a simple subtraction.</p>
<p>Could someone provide some guidance how I can modify to make the new version work exactly the same as the old version while making it convertible to coreML?</p>
Second order derivative with torch.autograd.function

username: bpfrd
cooked: <p>Hi,</p>
<p>I have problems understanding this tutorial: <a href="https://pytorch.org/tutorials/intermediate/custom_function_double_backward_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Double Backward with Custom Functions — PyTorch Tutorials 2.5.0+cu124 documentation</a></p>
<p>could someone simplify this a bit?<br>
how do you know a function support double backward or not?</p>
<p>best</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/bpfrd">@bpfrd</a>,</p>
<p>A good example for defining a DoubleBackward can be found here: <a href="https://discuss.pytorch.org/t/how-to-customize-the-double-backward/17846/5" class="inline-onebox">How to customize the double backward? - #5 by amensch</a></p>
<p>and this thread shows an example I wrote explaining a custom <code>torch.autograd.Function</code> object here: <a href="https://discuss.pytorch.org/t/customizing-torch-autograd-function/143161/4" class="inline-onebox">Customizing torch.autograd.Function - #4 by AlphaBetaGamma96</a></p>
<p>In short, you’ll need to manually define a derivative of the output of your function with respect to all inputs. Then define how to combine the <code>grad_output</code> (the gradient of the loss with respect to the output of your function) with your manually defined derivative (gradient of output with respect to the input) via the chain rule to get the gradient of the loss with respect to your inputs, which is what autograd is looking for.</p>
<p>In the case of the second-order derivative (or DoubleBackward), you just need to repeat this process of defining the derivative of the first Backward call with respect to its inputs, and how to combined them with the <code>grad_output</code> term of the second derivative.</p>
<p>The lingo’s a bit confusing, but do look at the examples I’ve listed above and see if the examples explain this process in more detail.</p>
Mismatch on multiclass model

username: vojo_0001
cooked: <p>Got a multiclass model</p>
<ul>
<li>.csv file</li>
<li>columns 0 thru 7 are features</li>
<li>column 8 is category
<ul>
<li>4 categories in that 9th columm</li>
<li>columns 1,3,5,7 distinquish categories</li>
<li>columns 0,2,4,6  feature attributes</li>
</ul>
</li>
<li>y_train is 1 by 8 tensor</li>
<li>pred_y seems to be 5 by 4 tensor  (returns a 5X4 Tensor)</li>
<li>Accuracy says dimension mismatch btween y_train and pred_y</li>
</ul>
<p>How do I translate  either one enable so I can compare  results???</p>
<p>Note, the model output is the expected 1 by 4 array and I can take max to figure out which category is most likely.  But I want to use accuracy since accuracy is NOT 1-loss</p>

username: vojo_0001
cooked: <p>mismatch   y_pred vs y_train!!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/8543695ddeb3df0835af5cc5c6500a6fb5a077e9.gif" data-download-href="https://discuss.pytorch.org/uploads/default/8543695ddeb3df0835af5cc5c6500a6fb5a077e9" title="Screenshot 2024-12-02 175530"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/8543695ddeb3df0835af5cc5c6500a6fb5a077e9.gif" alt="Screenshot 2024-12-02 175530" data-base62-sha1="j0TVNv1YjE70rgoerscyreKFMY9" width="690" height="247" data-dominant-color="ECECFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-12-02 175530</span><span class="informations">905×325 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Check what <code>y_pred</code> represents and transform it to class predictions if possible.<br>
E.g. if it represents logits or log probabilities, you can use <code>argmax</code> to create a prediction containing class indices.</p>
PyTorch Profiler self cpu time vs cpu time

username: m7md_hka
cooked: <p>Hi, I’m looking for explanation for this sentence “Note the difference between self cpu time and cpu time - operators can call other operators, self cpu time excludes time spent in children operator calls, while total cpu time includes it.” in in this topic: <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 2.5.0+cu124 documentation</a></p>
<p>What is the children operators calls inside that layer? and how can I see them?</p>

username: ptrblck
cooked: <p>Functions can call into other functions (or operators) as seen in the example profile:</p>
<pre data-code-wrap="python"><code class="lang-python">#                   model_inference       5.509ms      57.503ms      57.503ms             1
#                      aten::conv2d     231.000us      31.931ms       1.597ms            20
#                 aten::convolution     250.000us      31.700ms       1.585ms            20
#                aten::_convolution     336.000us      31.450ms       1.573ms            20
#          aten::mkldnn_convolution      30.838ms      31.114ms       1.556ms            20
</code></pre>
<p>Here <code>model_inference</code> calls into <code>aten::conv2d</code> which calls into <code>aten::convolution</code>, which …<br>
The time spent in <code>model_inference</code> in total (including all calls into all operators) is 57ms while the time spent in the <code>model_inference</code> body (without following any other calls) is only 5.5ms.</p>
Calculating Training Loss in DDP

username: Alec-Stashevsky
cooked: <p>I have the following basic average loss calculation in my training loop:</p>
<pre><code class="lang-auto">def train_one_epoch(model, criterion, optimizer, train_loader):
    model.train()
    running_loss = 0
    for i, data in enumerate(train_loader):
        optimizer.zero_grad()
        data = data.to(device='cuda')
        out = model(data)
        loss = criterion(out, data.labels)
        loss.backward()
        optimizer.step()
        # Do I need to reduce/gather from all distribution processes?
        running_loss += loss.detach()
    return running_loss / (i + 1)
</code></pre>
<p>My question is about the <code>running_loss</code> reported in DDP vs Single-GPU training. Do I need to apply an <code>all_gather</code> or <code>all_reduce</code> operation when running this training loop in DDP?</p>

username: H-Huang
cooked: <p>Hi <a class="mention" href="/u/alec-stashevsky">@Alec-Stashevsky</a>, DDP keeps loss local and averages the gradients. How are you using this running loss? If it is just for reporting the loss on each node then this seems okay, each rank will have report a different running loss.</p>

username: Alec-Stashevsky
cooked: <p><a class="mention" href="/u/h-huang">@H-Huang</a> thanks for reply! I am actually using it to report training loss across all ranks at the end of each training epoch. In that case it sounds like I would need to use some type of <code>reduce</code> or <code>gather</code> operation to sync across all devices? I only log these things on the rank 0 process. But currently, I am using the default loss reduction (<code>reduction='mean'</code>) and that doesn’t produce a tensor. Unsure about that as well.</p>

username: Alec-Stashevsky
cooked: <p><a class="mention" href="/u/h-huang">@H-Huang</a> any update? <a class="mention" href="/u/bravedistribution">@BraveDistribution</a></p>

username: cruzas2
cooked: <p>Hello. I would also be interested in case there are any updates.<br>
In particular, in my case, I am using NLLLoss.</p>

username: basilique
cooked: <p>Did you ever work out how to do this?</p>

username: fduwjj
cooked: <p>Does using DDP hooks mentioned here: <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 2.5 documentation</a> help?</p>

username: basilique
cooked: <p>Thank you, Hugo. Do you know where I could find a code example that uses DDP communication hooks to compute the local loss equivalent for training (or validation)?</p>

username: fduwjj
cooked: <p>I think the loss you get in DDP (before calling backward) is just the local loss?</p>

username: basilique
cooked: <p>I am looking for the local-equivalent (without using DDP) loss (an overall loss across all processes) rather than any individual local loss.</p>

username: fegin
cooked: <p>You can just have an allreduce or allgather to manually aggregate the loss as you prefer. This doesn’t have to couple with DDP.</p>

username: basilique
cooked: <p>yes, tanx, actually, I managed to compute the local equivalent losses.</p>
Join us for the LLM Hackathon in Ecology in Bielefeld, Germany, or Online!

username: Jennifer_DSouza
cooked: <p>Greetings! Might be of interest in this group. Together with a team of researchers, we’re organizing a Hackathon called “EcoHack: AI &amp; LLM Hackathon for Applications in Evidence-based Ecological Research &amp; Practice”. If anyone wishes to participate here is the event page with further information <a href="https://www.uni-bielefeld.de/einrichtungen/zif/events/#/event/8023" class="inline-onebox" rel="noopener nofollow ugc">Events - Universität Bielefeld</a> . Kind regards.</p>
Achieving sth like _foreach_clone?

username: sytrus-pytorch
cooked: <p>Hi there! I have been exploring the multi-tensor operators (<code>torch._foreach_*</code>) for implementing a custom optimizer. And I am wondering why there isn’t sth like <code>_foreach_clone</code> to copy the list of tensors. Right now I am achieving this with <code>_foreach_add(..., 0)</code> but this feels more like a hack. Is there sth obvious I am missing here? Or maybe I should just go with list comprehension instead?</p>
Checkpoint() doesn't work with use_reentrant=False

username: kwohlfahrt
cooked: <p><a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">The documentation</a> for <code>torch.utils.checkpoint</code> gives many reasons to prefer the <code>use_reentrant=False</code> version. However, it doesn’t say when <code>use_reentrant=True</code> is required.</p>
<p>I am trying to migrate a model to <code>use_reentrant=False</code>, but see errors like the example below. The same model runs successfully with <code>use_reentrant=True</code>, but I’m not sure what causes the incompatibility. I haven’t yet been able to reduce this in a minimal example, and ss far as I know, we aren’t doing anything weird with the backwards pass.</p>
<pre><code class="lang-auto">tensor at position 160:
saved metadata: {'shape': torch.Size([512, 128]), 'dtype': torch.bfloat16, 'device': device(type='cuda', index=0)}
recomputed metadata: {'shape': torch.Size([128, 128]), 'dtype': torch.bfloat16, 'device': device(type='cuda', index=0)}
</code></pre>
<p>What might cause <code>checkpoint(..., use_reentrant=False)</code> to fail when <code>checkpoint(..., use_reentrant=True)</code> works?</p>

username: kwohlfahrt
cooked: <p>I’ve now found a minimal reproduction for this, and it seems to be an interaction between mixed-precision and checkpointing. I’ve reported the issue upstream: <a href="https://github.com/pytorch/pytorch/issues/141896" class="inline-onebox" rel="noopener nofollow ugc">CheckpointError with checkpoint(..., use_reentrant=False) &amp; autocast() · Issue #141896 · pytorch/pytorch · GitHub</a></p>
<p>For context, this is sufficient to reproduce the issue:</p>
<pre data-code-wrap="python3"><code class="lang-python3">import torch
from torch.utils.checkpoint import checkpoint


class PairStack(torch.nn.Module):
    def __init__(self, size: int):
        super().__init__()
        self.block = torch.nn.Sequential(
            torch.nn.LayerNorm(size),
            torch.nn.Linear(size, size)
        )

    def forward(self, x):
        # Issue also occurs if we unconditionally checkpoint()
        if torch.is_grad_enabled():
            return checkpoint(self.block, x, use_reentrant=False, debug=True)
        else:
            return self.block(x)


class Mod(torch.nn.Module):
    def __init__(self, size: int):
        super().__init__()
        self.stack = PairStack(size)
        self.linear = torch.nn.Linear(size, size)

    def forward(self, x):
        with torch.set_grad_enabled(False):
            x = self.stack(x)
        x = self.stack(x)
        return self.linear(x)


def main():
    device = torch.device("cpu")
    size = 64

    m = Mod(size=size).to(device)
    x = torch.linspace(0, 1, 2 * 3 * size).reshape(2, 3, size).to(device)

    with torch.autocast(device.type, dtype=torch.bfloat16):
        output = m(x)

    loss = output.sum()
    loss.backward()

if __name__ == "__main__":
    main()
</code></pre>
<p>Does anyone have advice on where to start fixing this?</p>
Copy value between cuda tensors in non contiguous memory

username: zizhao.mo
cooked: <p>Hi,</p>
<p>I try to illustrate my problem more detailed with my example code as follows:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    device = 'cuda:2'
    torch.cuda.set_device(device)

    stream = torch.cuda.Stream()
    tensor_a = torch.zeros((100, 10000, 10000), device=device, dtype=torch.float16)
    tensor_b = torch.ones((100, 10000, 10000), device=device, dtype=torch.float16)

    a_indices = torch.as_tensor([1, 3, 7, 9, 11, 13, 17, 19, 21, 25], device=device, dtype=torch.int)
    b_indices = torch.as_tensor([0, 1, 2, 3, 4, 5, 6, 7, 9, 11], device=device, dtype=torch.int)
    
    with torch.cuda.stream(stream):
        # tensor_a[:, a_indices, :] = tensor_b[:, b_indices, :]
        tensor_a[:, a_indices, :].copy_(tensor_b[:, b_indices, :], non_blocking=True)
</code></pre>
<p>In the above code, I want to copy some data from tensor_b to tensor_a. However, this process is time-consuming. My objective is to reduce or hide this overhead, but either wrapping them in a stream or using the non-blocking flag does not take effect. I think the reason is that I am fetching a non-contiguous memory.</p>
<p>My question is, is it possible to make this assignment process asynchronous, which will not block the main process and will not block the subsequent operation in CUDA? If Pytorch cannot do this, is there any workaround with libtorch?</p>
<p>Looking forward to your response.</p>
<p>BR</p>

username: ptrblck
cooked: <p>Did you profile your code to verify the operation is not executed asynchronously? If so, could you show the <code>nsys</code> timeline?</p>
Is it possible to keep a chunk of continuous gpu memory (e.g. 20G) in DDP mode for gradient synchronization？

username: Lutao
cooked: <p>My training code will produce a large amount of memory fragments. Therefore, in DDP mode, it usually reports that it can’t allocate 15.00GB memory in backward stage.</p>
<p>I set gradient_as_bucket_view=True and it works well. However, when I use gradient_accumulation, this method can’t work.</p>
<p>So I want to create a chunk of continuous gpu memory for gradient backward. Are there any methods that can achieve this goal?</p>

username: ptrblck
cooked: <p>Could you describe why gradient accumulation fails?</p>

username: Lutao
cooked: <p>I guess that when setting gradient_as_bucket_view=True, maybe the backward operation conflicts with the function of gradient_as_bucket_view, because gradient accumulation needs multiple times of backward.</p>

username: fduwjj
cooked: <p>IIUC, <code>gradient_as_bucket_view </code> is trying to solve the fragmentation. Is there any reason why you don’t want to use <code>gradient_as_bucket_view </code>?</p>
<p>cc: <a class="mention" href="/u/fegin">@fegin</a></p>

username: Lutao
cooked: <p>Sorry for the spelling error.<br>
I mean that, when I set gradient_as_bucket_view=True, the multiple backward operations may make the gradient_as_bucket_view lose efficacy.</p>
<p>There are three experiments.</p>
<ol>
<li>The GPU memory is unstable when gradient_as_bucket_view=False.</li>
<li>If I don’t use gradient_accumulation, the GPU memory is stable when gradient_as_bucket_view=True.</li>
<li>But if I use gradient_accumulation, the GPU memory is unstable when gradient_as_bucket_view=True.</li>
</ol>
<p>With <code>gradient accumulation</code> , there will be at least 2 copies of the gradients. Maybe this is the reason why the gradient_as_bucket_view does not work.</p>

username: Lutao
cooked: <p>Thank you all for your help. I have solved this problem. The <code>gradient accumulation</code> operation occupies some extra GPU memory. After slightly reducing the memory consumption, it works.</p>
How can I use ExecuTorch to deploy a model to Infineon TC3xxx MicroController?

username: AlexLuya
cooked: <p>Hello,<br>
Is it possible?<br>
If yes,any example is highly appreciated.</p>

username: digantdesai
cooked: <p>I am not very familiar with this series of 32b MCUs. ET runtime is very portable and does not assume availability of an operating system, or filesystem etc.</p>
<p>FWIW, we have a <a href="https://pytorch.org/executorch/stable/executorch-arm-delegate-tutorial.html#non-delegated-workflow" rel="noopener nofollow ugc">tutorial</a> to cross compile and run ET on Cortex-M processors.</p>
Rnn moel can not converge while using nn.DataParallel

username: sugar_17
cooked: <p>my environment is win10, with torch 1.2<br>
rnn model cannot converge while using nn.DataParallel<br>
but it work well while not using nn.DataParallel</p>
<p>I using the api as the same as the following example, is there any problem with my using method in Win10, or could you give me some correct example, thanks a lot.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/spencerpomme/f77ac78f7451666be2b0230564519226" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/spencerpomme/f77ac78f7451666be2b0230564519226" target="_blank" rel="nofollow noopener">https://gist.github.com/spencerpomme/f77ac78f7451666be2b0230564519226</a></h4>
<h5>char_rnn.py</h5>
<pre><code class="Python">'''
Character-Level LSTM in PyTorch

In this code, I'll construct a character-level LSTM with PyTorch. The network will train
character by character on some text, then generate new text character by character.
This model will be able to generate new text based on the text from any provided book!

This network is based off of Udacity RNN mini project and which is in turn based off of Andrej Karpathy's
post on RNNs and implementation in Torch.
</code></pre>
This file has been truncated. <a href="https://gist.github.com/spencerpomme/f77ac78f7451666be2b0230564519226" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Xia_Yandi
cooked: <p>I encountered the same problem with PyTorch 1.4.0. At first I thought it couldn’t converge. But after a [very] * 100 long time, it started to converge [very] * 100 slowly.</p>
<p>And then I turned to DistributedDataParallel, and it worked. There may be a bug in LSTM + DP, I guess.</p>
<p>Check out how to use DDP here: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a></p>

username: Menki_Guo
cooked: <p>I had same issue, I think it is because I put optimizer(model) before model=DP(model). This either cause model not converge or error</p>
Environment suitability for new contibutor

username: ANISH_TULSYAN
cooked: <p>Dear All,</p>
<p>I am a new contributor to Pytorch. I want to know if my Environment is suitable or not for contributing to Pytorch. My environment details are :<br>
Windows : 11 Pro<br>
Version: 24H2<br>
CPU: I5 , GPU: Basic Intel GPU .<br>
CUDA: No Support<br>
Python: 3.12.6<br>
GCC : 8.1.0<br>
Virtual Environment : venv(Python’s default virtual environement module)<br>
torch : 2.5.1<br>
torchaudio : 2.5.1<br>
torchvision : 0.20.1<br>
pip : 24.3.1</p>
<p>I want to know if this environment is suitable for building pytorch from source in my local system.</p>
<p>Thanks in Advance,<br>
Anish Tulsyan</p>
Cuda FatBinary format error

username: jack00
cooked: <p>Hi,<br>
As far as I know, pytorch cuda implement will call cuda runtime API.<br>
I install cuda10.2 and torch-1.8.1+cu102-cp39-cp39 in a centos7 host, and it works well.<br>
I hijacked pytorch cuda function __cudaRegisterFatBinary by LD_PRELOAD, and convert the args(void <em>fatCubin) to __fatBinC_Wrapper_t</em>, finally I extracted the elf from it. But the elf.e_ident start with “A2 7F 45 4C 46 …”. However, a valid elf.e_ident should start with “7F 45 4C 46 …”. May I ask where I went wrong?</p>
<pre><code class="lang-auto">extern "C" __host__ void **__cudaRegisterFatBinary(void *fatCubin) {
    __fatBinC_Wrapper_t *bin = (__fatBinC_Wrapper_t *)fatCubin;
    char *data = (char *)bin-&gt;data;
    NvFatCubin *pFatCubin = (NvFatCubin *)data;
    Elf64_Ehdr *eh = &amp;(pFatCubin-&gt;elf);
    for (int i = 0; i &lt; 16; ++i) {
           printf("%X ", eh-&gt;e_ident[i]);
    }
}
</code></pre>

username: jack00
cooked: <p>I figured it out, CUDA compressed the fatBin.</p>
Build pytorch from source with cuda

username: jack00
cooked: <p>I built pytorch from source, infos:<br>
* centos7<br>
* pytorch v1.8.1<br>
* cuda 10.2<br>
* cudnn 8.4.0<br>
But after installation, libtorch_cuda.so is quite different from the official whl. The version I compiled need “libcusparse.so.10, libcurand.so.10, libcusolver.so.10, libcudnn.so.8, libcufft.so.10, libcublas.so.10”. How can I get a libtorch_cuda.so like the official version?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc03fb435c3d71c4efad6d5402cae5962252e0fa.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc03fb435c3d71c4efad6d5402cae5962252e0fa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc03fb435c3d71c4efad6d5402cae5962252e0fa.png" alt="image" data-base62-sha1="qPgih3Ve1K32h4ooZHwChX1tP4K" width="690" height="339" data-dominant-color="333333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">713×351 8.07 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1714da4f2cf979f9d149c593323ad7a6b3a9603c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1714da4f2cf979f9d149c593323ad7a6b3a9603c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1714da4f2cf979f9d149c593323ad7a6b3a9603c.png" alt="image" data-base62-sha1="3ibEPWMd57WhwHQAFEdVa5F1HaY" width="690" height="255" data-dominant-color="333333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">701×260 7.05 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jack00
cooked: <p>I resolved it by set option(CAFFE2_STATIC_LINK_CUDA “Statically link CUDA libraries” ON). In addition, cudnn version should &lt; 8.3.0 when compiling statically.</p>
DDP no support for sparse tensor

username: rira.18ki
cooked: <p>I would like to use PyTorch DDP to speed up additional training of openAI’s whisper.<br>
However, when wrapping the whisper model into DDP, I get a “RuntimeError: No support for sparse tensors” error and cannot proceed.<br>
Can you please tell me how to solve this problem?</p>
<p>I believe this error occurs because whisper is a sparse transformer.<br>
Does not PyTorch DDP support this transformer?</p>

username: H-Huang
cooked: <p>I’m not familiar with whisper, which part of the model is using sparse tensors? DDP will allreduce the gradients of each rank after iteration and so if part of the model is sparse then the gradients will also be sparse and they cannot be allreduced.</p>
<p>I am assuming you are using nccl, nccl does not support sparse collectives. You can try using the <code>gloo</code> backend which supports sparse allreduce but be aware it may not be as performant as nccl.</p>

username: rira.18ki
cooked: <p>Thanks for your advice.<br>
I also looked for sparse tensors in the whisper model, but I was able to find.<br>
So, I try to do “gloo”.</p>

username: hotbaby
cooked: <pre><code class="lang-auto">class Whisper(nn.Module):
    def __init__(self, dims: ModelDimensions):
        super().__init__()
        ...
        self.register_buffer("alignment_heads", all_heads.to_sparse(), persistent=False)
</code></pre>
<p><code>alignment_heads</code> buffer is sparse tensor.</p>
<p>Modify alignment_heads to dense tensor may solve this problem.</p>
<pre><code class="lang-auto">alignment_heads_dense = model.get_buffer("alignment_heads").to_dense()
model.register_buffer("alignment_heads", alignment_heads_dense, persistent=False)
</code></pre>

username: hotbaby
cooked: <p><code>alignment_heads</code> buffer is sparse tensor.</p>

username: xunji_yao
cooked: <p>I met the same issue when my model contained whisper, I solved it following your solutions.<br>
Thanks</p>
Advice on using executorch on iOS

username: ericbomgardner
cooked: <p>I’m trying to use executorch to run a Llama 3.2 quantized 3B QAT+LoRA model in an iOS app. The demo app works with the <code>.pte</code> file I generated for that model, but then when I went to try to add executorch into my app, it seems like a lot of the code that the demo app uses isn’t available through the executorch Swift package – the demo app uses a bunch of stuff in <code>executorch/extension/llm</code>, and as far as I can tell, the package just exposes <code>executorch/extension/module/</code>, <code>executorch/extension/tensor/</code>, and <code>executorch/runtime/*</code> (but I could be wrong, I’m not super-familiar with C++!).</p>
<p>Is there a way you’d recommend to use that code from <code>executorch/extension/llm</code> in my app? Or is that outside of the scope of the executorch library?</p>
<p>So far I’ve tried:</p>
<ul>
<li>copying the files directly into my app (license-permitting, of course), but that seemed brittle since it wouldn’t update as those files update, plus I ran into issues with the CMake build script</li>
<li>getting away with only using the code in module.h like the SwiftPM example (<a href="https://pytorch.org/executorch/main/_static/img/swiftpm_xcode.mp4" rel="noopener nofollow ugc">https://pytorch.org/executorch/main/_static/img/swiftpm_xcode.mp4</a>) does, but for my use case it seems like I’d need most of the text-specific stuff (tokenizer, text decoder, text prefiller, text token generator) so that didn’t work</li>
</ul>
<p>Thanks!</p>

username: shoumikhin
cooked: <p>Hi Eric,</p>
<p>Sorry for a late reply, I just noticed your question.</p>
<p>At the moment, I’m afraid you need to copy the corresponding files from <code>extension/llm</code> and setup your project to build them, including the 3rd party dependencies, like RE2 or Abseil.</p>
<p>The demo app and the <a href="https://github.com/pytorch/executorch/tree/main/extension/benchmark/apple/Benchmark" rel="noopener nofollow ugc">benchmark app</a> are examples of such setup, where we build certain files from <code>extension/llm</code> directly with Xcode, and build the rest of 3rd party dependencies with Cmake using a custom run script build phase in Xcode. Let me know which issues you run into with the cmake invocation and I’ll try to help there. Note that we also leverage <a href="https://pytorch.org/executorch/main/apple-runtime.html#linkage" rel="noopener nofollow ugc"><code>.xcconfig</code></a> files for more convenient build settings tuning.</p>
<p>I know that setup it less than ideal, that’s why we’re actively working on providing the LLaMA runner API as a standalone SwiftPM package, so that eventually the customers would just integrate it in their Xcode projects natively and skip all the Cmake hassle.</p>
<p>Thanks.</p>
On-device ExecuTorch LLM API

username: Alex_Zhou1
cooked: <p>Hi all,</p>
<p>So I want to make a mobile game where the NPCs are basically chatbots. To do so I need to quantize LLMs and deploy them onto mobiles. I have been reading ExecuTorch tutorials but not completely, and since my focus is on computer vision, this is a new field to me really.</p>
<p>To my understanding, I only need LLM APIs to get this idea to work, as opposed to compiling a chatbot app. I was wondering if this is possible? Many thanks in advance.</p>

username: larryliu0820
cooked: <p>Hi <a class="mention" href="/u/alex_zhou1">@Alex_Zhou1</a> thanks for your interest in ExecuTorch!</p>
<p>The high level flow of deploying LLM models to mobile application is:</p>
<ul>
<li>Use <code>extension/llm</code> APIs (WIP see more in this RFC: <a href="https://github.com/pytorch/executorch/issues/6558" class="inline-onebox" rel="noopener nofollow ugc">RFC: extension/llm API design · Issue #6558 · pytorch/executorch · GitHub</a>) to quantize, optimize and eventually export to ExecuTorch file format <code>.pte</code>.</li>
<li>Leverage the existing JNI layer (<a href="https://pytorch.org/executorch/0.4/llm/llama-demo-android.html" class="inline-onebox" rel="noopener nofollow ugc">ExecuTorch Llama Android Demo App — ExecuTorch 0.4 documentation</a>) or iOS examples (<a href="https://pytorch.org/executorch/0.4/llm/llama-demo-ios.html" class="inline-onebox" rel="noopener nofollow ugc">ExecuTorch Llama iOS Demo App — ExecuTorch 0.4 documentation</a>) to build a native runner. A C++ Tokenizer implementation is also provided under <code>extension/llm/tokenizer</code>.</li>
<li>Integrate the runner into your mobile app. The app is responsible to handle I/O (e.g., feeding input into the runner, getting back generated tokens from callback) and artifacts (.pte and tokenizer artifact) management.</li>
</ul>
<p>For more questions please create an issue in <a href="https://github.com/pytorch/executorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/executorch · GitHub</a></p>
Where to find documentation of ProcessGroup/DeviceMesh methods?

username: anonymous_user_83259
cooked: <p>A seemingly simple question.</p>
<p>I lost some debugging time to <code>group.allreduce()</code>, not realizing it <a href="https://github.com/pytorch/pytorch/blob/96be048f060c02678647a60026eda4c7f4fc62a0/torch/_C/_distributed_c10d.pyi#L340" rel="noopener nofollow ugc">returned an async Work handle</a>, unlike global <code>torch.distributed.*</code> methods which sync by default.</p>
<p>To avoid similar mistakes, I searched for documentation of its methods, but failed to find it.</p>

username: paulge
cooked: <p>They should be part of the <code>torch.distributed</code> docs <a href="https://pytorch.org/docs/stable/distributed.html#" rel="noopener nofollow ugc">here</a>. It definitely contains the docs for <code>DeviceMesh</code> and the other distributed communications API</p>

username: anonymous_user_83259
cooked: <p>Are the methods of DeviceMesh listed on that page? What about ProcessGroup’s?</p>

username: paulge
cooked: <p>If you click on ‘[Source]’, the source code with all methods of <code>DeviceMesh</code> will unfold. Alternatively have a look at GitHub <a href="https://github.com/pytorch/pytorch/blob/5deca07c0dcf1482eba99bf93b805cf1cc41ad6c/torch/distributed/device_mesh.py#L379" rel="noopener nofollow ugc">DeviceMesh class</a><br>
The functions regarding <code>ProcessGroup</code> are under the ‘Collective functions’ section. Or here in the <code>.pyi</code> interface here <a href="https://github.com/pytorch/pytorch/blob/5deca07c0dcf1482eba99bf93b805cf1cc41ad6c/torch/_C/_distributed_c10d.pyi#L298" rel="noopener nofollow ugc">ProcessGroup class</a></p>

username: anonymous_user_83259
cooked: <p>I check the source code of DeviceMesh often in lieu of documentation. I seek documentation.</p>
<p>The link to the ProcessGroup class points to the same file I linked in the root thread post. I shan’t engage more.</p>

username: fduwjj
cooked: <p>cc: <a class="mention" href="/u/irisz">@irisz</a> I think we do have documents for DeviceMesh already?</p>
Difference between ProcessGroup and Backend classes

username: nathanbrown-Arm
cooked: <p>Hello,</p>
<p>I am curious about the difference between the ProcessGroup and Backend C++ classes. In the docs on <a href="https://pytorch.org/docs/stable/distributed.html#third-party-backends" rel="noopener nofollow ugc">third-party backends</a>, it mentions that the backends should inherit from <code>c10d::ProcessGroup</code>. However, on the page for <a href="https://pytorch.org/tutorials/intermediate/process_group_cpp_extension_tutorial.html" rel="noopener nofollow ugc">Customizing ProcessGroup Backends</a>, it says to inherit from the <code>Backend</code> class. I also saw that the <a href="https://github.com/pytorch/pytorch/pull/136343/commits" rel="noopener nofollow ugc">PR</a> for a newly created XCCL backend <a href="https://github.com/pytorch/pytorch/blob/652da01588ecb64e354707bc52496c28c01f07ce/torch/csrc/distributed/c10d/ProcessGroupXCCL.hpp#L76" rel="noopener nofollow ugc">also inherits</a> from the <code>Backend</code> class. From all this, I have a few questions:</p>
<ul>
<li>What is the difference between these classes?</li>
<li>How do these classes interact?</li>
<li>What is the future plan for these classes? Will ProcessGroup replace the Backend class at any point?</li>
</ul>
<p>Thank you,<br>
Nathan</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="nathanbrown-Arm" data-post="1" data-topic="213483">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/9dc877/48.png" class="avatar"> nathanbrown-Arm:</div>
<blockquote>
<p>mentions that the backends should inherit from <code>c10d::ProcessGroup</code>. Howev</p>
</blockquote>
</aside>
<p>Hi, long story short. We split backend from ProcessGroup in 2023. So customized backend should inherit Backend class. The ProcessGroup is mostly for Python and BC compatible. When user create a ProcessGroup class in python by calling <code>init_process_group</code>, we will create backend class for different devices (CPU, GPU, e.g.), so when you pass in a tensor on certain device, certain backend will be triggered.</p>
How to train RNN without a teacher?

username: lanskoyk
cooked: <p>I have a project that models evolution, especially since I don’t have the right answers for backpropagation. But I need the neural network to remember the past and understand sequences, RNN(GRU, LSTM) and Transformer would be suitable for this (they require many times more computing resources and a training sample), but I couldn’t even find Reinforcement Learning for them, let alone NEAT or GA, like MLP. Now I just don’t know what to do, maybe I need to take MLP and give tens of neurons of the input layer to memory, but I don’t think this solves the problem.</p>
Soft Actor-Critic Algorithm does not manage to solve slightly more complicated tasks

username: b-gtr
cooked: <p>I started with reinforcement learning a relatively short time ago and tried to program several algorithms myself. My current project was to write the Soft Actor-Critic algorithm and then use it for autonomous driving in Carla.</p>
<p>My code manages to solve simple tasks, but when I try it on Carla, I still achieve poor performance, even though I base my rewards on master’s theses and these achieve better performance. Do I have a programming error somewhere in my code or a mathematical error that it does not learn. It would be very kind if someone would look at my code and give feedback.</p>
<p>My Code is on Github with following link: <a href="https://github.com/b-gtr/Soft-Actor-Critic" rel="noopener nofollow ugc">b-gtr/Soft-Actor-Critic: Soft-Actor-Critic with image preprocessing</a></p>
How to use PyTorch lightning?

username: blackpearl006
cooked: <pre><code class="lang-auto">for epoch in range(config['total_epochs']):
    model.train()
    train_loss = 0.0
    train_total = 0.0
    train_correct = 0.0

    for inputs, label_info in dataloader:
        combined_loss, reg_loss, class_loss = 0.0, 0.0, 0.0
        inputs = inputs.to(DEVICE)
        labels = label_info['label']
        labels = labels.to(DEVICE)
        merge_mask = label_info['Merge_transformation'].to(DEVICE)
        optimizer.zero_grad()
        outputs_reg, outputs_class = model(inputs)
        
        reg_loss = criterion_reg(outputs_reg, labels)
        reg_loss = reg_loss * (1 - merge_mask)

        class_loss = criterion_class(outputs_class, labels)
        class_loss = class_loss * merge_mask

        reg_loss = reg_loss.sum() / (1 - merge_mask).sum() if (1 - merge_mask).sum() &gt; 0 else reg_loss.sum()
        class_loss = class_loss.sum() / merge_mask.sum() if merge_mask.sum() &gt; 0 else class_loss.sum()
</code></pre>
<p>I want to useFabric/ PyTorch lightning to implement DDP, But how to send both labels and merge_mask to GPU ?<br>
The fabric sample code :</p>
<pre><code class="lang-auto">  model.train()
  for epoch in range(20):
      for batch in dataloader:
          input, target = batch
-         input, target = input.to(device), target.to(device)
          optimizer.zero_grad()
          output = model(input, target)
          loss = torch.nn.functional.nll_loss(output, target.view(-1))
-         loss.backward()
+         fabric.backward(loss)
          optimizer.step()
</code></pre>
<p>is not helping me enough <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=12" title=":frowning_face:" class="emoji" alt=":frowning_face:" loading="lazy" width="20" height="20"></p>

username: blackpearl006
cooked: <p>I have already tried to use fabric and it throws the below error</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu!
  File "/home/neelamlab/ninad/MAE/basic_fabric.py", line 636, in &lt;module&gt;
    reg_loss = criterion_reg(outputs_reg, labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu!
</code></pre>