akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pybind11v3 integration

pya: <p>According to <a href="https://pybind11.readthedocs.io/en/stable/upgrade.html" class="inline-onebox" rel="noopener nofollow ugc">Upgrade guide - pybind11 documentation</a> ,</p>
<p>“We recommend migrating to pybind11 v3.0 promptly, while keeping initial changes to a minimum. Most projects can upgrade simply by updating the pybind11 version, without altering existing binding code.”</p>
<p>I am wondering if I replace the torch/include/pybind11 directory with pybind11v3 and recompile torch, do I have to pay attention to any other torch-specific aspects?</p>
How to handle step size for grid_sample

Speido: <p>Suppose you have a model/layer that outputs coordinates that are then sampled like this:</p>
<pre><code class="lang-auto">coordinate = somelayer() #output is tensor with shape [4,128,128,2]

weights = torch.rand( (4, 3, 64, 64) )

result = F.grid_sample(weights, coordinate, "bilinear")
result = result.mean(0)

#loss calculation with result, backward and optimizer step
</code></pre>
<p>Now, the gradients of grid_sample are calculated within a “pixel” and are obviously invalid for steps larger than 1/64, however the optimizer does bigger steps wrongly assuming continuous behaviour. So the new guessed coordinate is now pointing to completely unrelated data (e.g. 2 pixels to the right) which will get “blurred” in the next optimization step to minimize the loss.</p>
<p>Any ideas how I can enforce a limit on the model/optimizer step with respect to the output of the layer?</p>
<p>The problem accumulates with deeper architectures. I am getting decent results at absurdly low learning rates but it is not a real solution.</p>
<p>Clipping of layer gradients does not seem to ensure that the coordinate delta is within resolution step size.</p>
Help 'NoneType' object has no attribute 'shape'

kdxcj: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/018eae771de1f32886c63e9a627adb04bed356d7.png" data-download-href="https://discuss.pytorch.org/uploads/default/018eae771de1f32886c63e9a627adb04bed356d7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/018eae771de1f32886c63e9a627adb04bed356d7_2_689x219.png" alt="image" data-base62-sha1="dMaC2evGUsNgSYKmG7nAabyevZ" width="689" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/018eae771de1f32886c63e9a627adb04bed356d7_2_689x219.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/018eae771de1f32886c63e9a627adb04bed356d7_2_1033x328.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/1/018eae771de1f32886c63e9a627adb04bed356d7.png 2x" data-dominant-color="1D1D1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1209×385 31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I printed “model._<em>class</em>_“, “model.config“, “inputs_id’s shape”, “attention mask shape“, “eos_id“ and “pad_id”. I didn’t find any problem. I don’t know how to fix it.</p>

Arunprakash-A: <p>Where are you calling <code>.shape</code> in your code?</p>
Guiding Image Generation in Diffusion Models with PyTorch

sid-ls: <p>I just open-sourced an introductory-friendly PyTorch <a href="https://github.com/sidhantls/minimal-casteer" rel="noopener nofollow ugc">implementation</a> for controlling image generation in diffusion models. It’s built on <strong>CASteer</strong>, which lets us apply pre-computed steering vectors to guide models toward generating images with specific attributes—like <em>“anime”</em>, <em>“happiness”</em>, or even a mix of both!</p>
<p>The code also shows how to use <strong>PyTorch hooks</strong> to easily extract activations from diffusion models, and then reapply transformations to those activations during inference.</p>
<p>The project is fully open-source—I’d love to hear your feedback and thoughts!</p>
<p><a href="https://github.com/sidhantls/minimal-casteer" rel="noopener nofollow ugc">Repo</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc_2_690x176.jpeg" alt="image" data-base62-sha1="7awNCbiNCC2Y8fP3QpY7VAg0rFO" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc_2_690x176.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc_2_1035x264.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/323fe8fb8f2e38da6ed4f9556d3323002d1aa8cc_2_1380x352.jpeg 2x" data-dominant-color="AFAFAE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×491 47.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.cuda.device_count() does not return correct number

Jasmine: <p>I am running a script on a slurm cluster with 6 nodes, each with 2 A40 gpus.</p>
<pre><code class="lang-auto">world_size = int(os.environ["WORLD_SIZE"])
rank = int(os.environ["SLURM_PROCID"])
gpus_per_node = int(os.environ["SLURM_GPUS_ON_NODE"])
print(f"Hello from rank {rank} of {world_size} on {platform.node()} where there are {gpus_per_node} allocated GPUs per node.", flush=True)
print(gpus_per_node, torch.cuda.device_count(), flush=True)
assert gpus_per_node == torch.cuda.device_count()
</code></pre>
<p>I sometimes (not always) get an `AssertionError’, and it seems to be related to torch.cuda.device_count() returns 1.</p>
<p>I try to print(os.environ[‘CUDA_VISIBLE_DEVICES’], flush=True), it returns ‘0,1‘ when torch.cuda.device_count() returns 1.</p>
<p>After browsing around, I still cannot figure out why this happens. I assume this is not because MIG, because MIG is not supported in A40?</p>

ptrblck: <p>I don’t know how it why MIG would be related but you can check if it’s enabled in <code>nvidia-smi</code> and disable it if needed.</p>
Using class weights with loss function for train/val/test splits

Mole_Turner: <p>I’m investigating a classification problem where the dataset is unbalanced. Because of the data pipeline (which I don’t really want to go back and totally rewrite) the balance of classes is different between the train, validation, and test sets.</p>
<p>For training I’ve calculated the class weights using <code>sklearn.utils.compute_class_weights()</code>, the output of which can be passed to <code>nn.LossFunc(weight=class_weights)</code>. What I’m unsure about is whether I then need to initialise a different loss function instance for the validation loss, and further more whether I should include class weights for the validation set? My feeling is that, I do need to have a separate instance for the validation set because I don’t want them weighted based on the training set class distribution, but I shouldn’t use any class weights because the validation set should be a proxy for the test set……within which I wouldn’t know the class distribution.</p>

Tanmay_Pani: <p>If your train-valid split was stratified, then you shouldn’t have to redo the class weights during validation, as the relative abundances of classes in training set and validation set are similar to the full dataset (Which is atleast what I do for my classification tasks). But if they are not stratified, then the answer depends on the source of the imbalance.</p>
<p>For example, in binary classification, if one of the classes is naturally rarer than the other and the total dataset (train + valid) represents the abundance of the rarer class in the “wild“ then using class-weights calculated from the full dataset makes sense (to me). If one of the classes is rarer because of say, experimental inefficiencies, then you have to tread carefully, and use other methods to address the imbalance like downsampling the more abundant class or upsampling the rarer class.</p>

Mole_Turner: <p>In this instance, I’ve adapted an existing dataset that is already separated into train/test/val splits.</p>
<p>The raw versions of these are balanced, however, as they are audio data of differing lengths, I’ve separated all examples into equal length chunks with each chunk now being a data example. As each raw full length example of different length results in a different number of equal length chunks, this has resulted in the sets becoming imbalanced.</p>

Tanmay_Pani: <p>So if I understand correct, the source of your imbalance is purely stochastic. In that case I suggest making the train set balanced by upsampling/downsampling the chunks (idk how easy upsampling audio data is, never worked with it, but downsampling should be easy), and not using class weights.</p>
Pytorch mobile for RAG applications

Wimboj: <p>Looking to build a medical based offline RAG application for personal use on android or ios, would Pytorch mobile be feasible to use for this project</p>

python: <p>PyTorch Mobile can be a feasible option for a lightweight RAG setup, especially for personal use. The main thing to consider is model size and inference speed; mobile devices have limited resources, so you’ll likely need to quantize or distill your models. Also, keep in mind that offline RAG means you’ll need both the retriever and generator components to run locally, which can get a bit heavy. You need to look into using smaller models like DistilBERT or TinyBERT for retrieval and something compact like a distilled language model for generation. It’s doable, but it will take some careful optimization. If you’re exploring professional support for this kind of mobile AI integration, <a href="https://ideamaker.agency/mobile-app-development-services/" rel="noopener nofollow ugc">Idea Maker</a> has experience building efficient, on-device AI solutions.</p>

Cagatay_Bilgin: <p>PyTorch team has been building ExecuTorch <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a> which replaced PyTorch mobile. It has better performance, support, features and has been deployed inside Meta on multiple products. We deprecated PyTorch mobile.</p>

James_Whitmore: <p>Great insights into using PyTorch Mobile for RAG setups on mobile devices! Optimizing for model size and inference speed is key, especially when working offline. It’s interesting to consider smaller models like DistilBERT or TinyBERT for retrieval, and exploring tools like Idea Maker for professional support can definitely make the optimization process smoother. A well-optimized, mobile-friendly setup could offer impressive performance with the right adjustments.</p>
<aside class="quote no-group" data-username="python" data-post="2" data-topic="200342">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ee7513/48.png" class="avatar"> python:</div>
<blockquote>
<p>PyTorch Mobile can be a feasible option for a lightweight RAG setup, especially for personal use. The main thing to consider is model size and inference speed; mobile devices have limited resources, so you’ll likely need to quantize or distill your models. Also, keep in mind that offline RAG means you’ll need both the retriever and generator components to run locally, which can get a bit heavy. You need to look into using smaller models like DistilBERT or TinyBERT for retrieval and something compact like a distilled language model for generation. It’s doable, but it will take some careful optimization. If you’re exploring professional support for this kind of mobile AI integration, <a href="https://ideamaker.agency/mobile-app-development-services/" rel="noopener nofollow ugc">Idea Maker</a> has experience building efficient, on-device AI solutions.</p>
</blockquote>
</aside>
Flash attention with padding mask or nested tensors

SlavPsh: <p>Hi everyone,<br>
I’m trying to find out how to use flash attention for large sequences of variable length in training. Flash attention currently doesn’t support (padding) masks.<br>
People suggested nested tensors but those seem to only work in evaluation with flash attention. Then there’s a possibility to manually set key/query/value elements to -inf or 0, imitating padding. Are there any other options for flash attention for variable length sequences?</p>

Kshitiz_Khandelwal: <p>Were you able to figure it out? Also facing the same issue</p>

Arunprakash-A: <p>Could you share the code snippet where you are calling the <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">scaled_dot_product_attention</a>? Or are you using the default <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">MultiheadAttention</a>?</p>
How do you structure a stateful LSTM with PyTorch?

zezimabig: <p>Dear PyTorch community,</p>
<p>I apologise for the uncategorizing of this particular question as I could not find a forecasting category.</p>
<p>Basically, I would like to find out if anyone has ever tried to define a stateful LSTM model with PyTorch? If so, please do share your wisdom with the community as this has been an ongoing debate online (from what I observed in several forums). If there was a link that shows that the issue had been resolved already, please do feel free to share it!</p>
<p>Below is the code I have at the moment:</p>
<pre><code class="lang-auto">class MultivariateLSTM(nn.Module):
    
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(MultivariateLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size = input_size, hidden_size = hidden_size, num_layers = num_layers, batch_first = True, dropout = 0.2)
        self.fc1 = nn.Linear(hidden_size, 16)
        self.fc2 = nn.Linear(16, output_size)
        self.relu = nn.ReLU()
        
    def forward(self, x):
        out, (h, c) = self.lstm(x)
        out = self.relu(out)
        out = self.fc1(out[:, -1, :])
        out = self.relu(out)
        out = self.fc2(out)
        
        return out
</code></pre>
<p>Could you also share how will the training loop look like (i.e., initialisation of hidden and cell states, passing the current (h, c) to the next loop, and etc.)?</p>
<p>I very much appreciate any help that comes by!</p>

torchvin123: <p>I think you can refer to this link here</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/confusion-regarding-pytorch-lstms-compared-to-keras-stateful-lstm/44502/4" class="inline-onebox">Confusion regarding PyTorch LSTMs compared to Keras stateful LSTM - #4 by deividbotina</a></li>
</ol>

zezimabig: <p>Thank you for the link! I will test it and see if it works.</p>

zezimabig: <p>I tried with and without the passing of hidden and cell states, and it seemed that the results (MSE, MAE, etc.) had little to no change. It seems that the LSTM unit itself handles this in the shadows without explicitly showing its operations with these terms.</p>

Pytorch-Noob: <p>it’s ok, most simple use-cases does not show difference between stateful and stateless implementations.</p>
<p>Source (the guy in video tells this; so he goes on to show stateless implementation!):</p>
<p>            <iframe src="https://www.youtube.com/embed/8A6TEjG2DNw?feature=oembed&amp;wmode=opaque&amp;list=PLzJ0cKlK0v0PvIOQ5bRd4lhTzbdh6_Y8W" width="480" height="360" frameborder="0" allowfullscreen="" class="youtube-onebox" seamless="seamless" sandbox="allow-same-origin allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-presentation"></iframe>
</p>
Switching between multi-processing and main process

catosphere: <p>Hi all,</p>
<p>I am fairly new to multi-processing and could not find info on running parts of a python code on a single main process as part of a distributed training e.g. single node and 8 GPUs.</p>
<p>I launch as follow:<br>
OMP_NUM_THREADS=12 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nnodes=1 --nproc_per_node=8 my_python_script.py --my_parser_args …</p>
<p>Distributed training works well, for context I use HuggingFace trainer with FairScale --sharded_ddp zero_dp_2, however it seems that the whole script is executed as many times as there are devices, in this case 8 times. This can be seen as e.g. all prints are replicated.</p>
<p>To my understanding, ideally the whole part of the script before training (which doesnt require GPUs) should only run as a single main process e.g. importing data, setting up hyper-parameters and so on. And once the training starts, it should be replicated to as many processes as there are used GPUs.</p>
<p>What is the PyTorch way to locally disable multi-processing please ?<br>
In HuggingFace, there is an option “main_process_first” which lets a block of code be run with the main process, e.g. dataset pre-processing with tokenizer.</p>
<p>Thanks !</p>

ptrblck: <aside class="quote no-group" data-username="catosphere" data-post="1" data-topic="150202">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/catosphere/48/3716_2.png" class="avatar"> catosphere:</div>
<blockquote>
<p>To my understanding, ideally the whole part of the script before training (which doesnt require GPUs) should only run as a single main process e.g. importing data, setting up hyper-parameters and so on. And once the training starts, it should be replicated to as many processes as there are used GPUs.</p>
</blockquote>
</aside>
<p>No, most likely this wouldn’t be ideal as you might be running into the Python GIL, which would then slow down your entire code. The single process - multiple devices approach is used in e.g. <code>nn.DataParallel</code> and is thus not recommended.<br>
Since your workflow already seems to work fine, you should get the best performance using a single process per device.</p>

catosphere: <p>Thanks for your answer, then I will continue this way if it seems desired that all operations including pre-processing and setup stages are replicated as many times as there are GPU devices used in distributed training.</p>

Hanqi_Xiao: <p>My processing happens to break if I do it in multiple processes at once due to a library that assumes single process.</p>
<p>Any workaround possible?</p>
<p>Thanks.</p>
Creating Multiple Cnn sequential models on the same dataset

pushkar_bhardwaj: <p>So i have been working on a recyclable classes image dataset which has 6 different classes. I am trying to do a CNN image classification on it, which worked fine when i ran a single model. But now I’m trying to do ensembling on it with trying to run atleast 15 models with different parameters and the getting the best model out of them based on the performance metrics. But i am not sure how to do it, also i have been trying to get which class performed the worst out of the six.</p>
<p>Can somebody please guide me, how can i approach this problem?</p>
Reducing compiled call overhead

crizz: <p>Hello, I have been optimizing a Soft Actor Critic implementation of mine using torch compile, and I feel like I am hitting a limit in how much I can reduce cuda calls overhead. It looks like this is because the PyTorch code involved in launching the optimized code itself is quite complex in itself, and, as such, a long time passes between my Python function call and when my CUDA Graphs are being launched. I attach here a screenshot from Nsight as a reference.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8_2_690x487.png" alt="image" data-base62-sha1="eTiWUV0mIsRRfp3Tju24kqqM6AE" width="690" height="487" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8_2_690x487.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8_2_1035x730.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/8/685ee2f0ecdaaaad8c5d3ce7669a897be7b454d8.png 2x" data-dominant-color="DAE7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1379×975 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the image it can be seen how a long time passes between when I call the compiled function (approximately at the start of the _compute_actor_and_alpha_loss range) and when the CudaTrees run gets called and when finally the actual cudagraph is launched. I suspect this is because of time spent on the CPU running Python code. Is this normal behaviour? Can I do something to reduce this apart from merging things in one single cuda graph as much as possible? Are there any plans by PyTorch to reduce this kind of inefficiencies? In use cases like mine this slows down things quite a lot. Thanks!</p>
FSDP Mesh_dim_names in init_device_mesh not work

Ruibin_LI: <p>I use fsdp to init my model. The torch version is 2.7.1</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">device_mesh = init_device_mesh(
                "cuda",
                mesh_shape=(1, 8),
                mesh_dim_names=("replicate","shard"),
            )
            model = FSDP(
                model,
                auto_wrap_policy=size_based_auto_wrap_policy,
                device_id=device,
                # sharding_strategy=ShardingStrategy.FULL_SHARD,
                mixed_precision=MixedPrecision(
                    param_dtype=torch.float,
                    reduce_dtype=torch.float,
                    buffer_dtype=torch.float,
                ),
                sync_module_states=True,
                limit_all_gathers=True,
                use_orig_params=True,
                device_mesh=device_mesh,
            )
</code></pre>
<p>In my understanding, when I define the shard dimension as 8 and the replicate dimension as 1, the model should be fully sharded. However, this results in a <code>no_shard</code> FSDP configuration, which is confusing.</p>
<p>I also tried swapping the <code>mesh_dim_names</code> from <code>("replicate", "shard")</code> to <code>("shard", "replicate")</code>, but again, the model ends up with no sharding. This suggests that <code>mesh_dim_names</code> may not be having the intended effect. It appears that FSDP always treats the first dimension as the sharding dimension and the second as the replication dimension, regardless of the assigned names.</p>
<p>If I cannot explicitly assign which dimension corresponds to sharding or replication by name, how can I reliably control which dimension is used for what purpose? This becomes even more confusing when considering other parallelism dimensions such as tensor parallelism (TP) or sequence parallelism (SP), especially since I cannot directly assign a size to the TP dimension in the same way.</p>
FSDP model with non-standard gradient computation

mseeger: <p>I have implemented a method for gradient computation in transformer models with long contexts, where sparse attention with KV-caching is built in. It works on a single GPU, so does DDP, but to use larger models and full fine-tuning, I’d like to apply FSDP.</p>
<p>I am running into the problem that the way gradients are computed does not have standard form, in the sense of</p>
<pre><code class="lang-auto">outputs = model(inputs)
loss = loss_func(outputs, targets)
loss.backward()
</code></pre>
<p>I am using some fairly complex concepts, like nested activation checkpointing and autograd saved tensor hooks, and I just cannot get it into standard form.</p>
<p>I’d still like to use FSDP, but all I found so far is top-level implementations that do some magic wrapping, under the assumption that gradients are computed in the standard form. I am able to write code for what FSDP units would do locally. Especially, I’d like to supply custom code for the backward local part.</p>
<p>Are there examples for how FSDP would be applied to such a non-standard scenario, where all the necessary communication is handled, but the local forward and backward computations on the shards can be specified as such, without backward having to be the standard one represented by <code>autograd</code> when <code>loss.backward()</code> is called?</p>

mseeger: <p>Maybe a different way of asking this: It is all fine to provide an easy-to-use FSDP implementation that assumes gradient computation be done in standard way, and do lots of magic behind the scenes. But is there a basic low-level way to run FSDP, where code can be supplied for the FSDP units, both for forward, and especially for backward?</p>
<p>I am not even that interested in communication happening in parallel with computation, if that helps.</p>

mseeger: <p>Somebody pointed me to this one:</p>
<pre><code class="lang-auto">https://github.com/facebookresearch/capi/blob/main/fsdp.py#L184

https://arxiv.org/html/2411.00284v1
</code></pre>
<p>This seems what I need.</p>
Senior AI/LLM Engineer (Part-Time, Remote – US Only)

James_Hayashi: <p><strong>About Us</strong><br>
We are a <strong>CELBI PRO TECH</strong>, a profitable AI software company with successful deployments in Asia. We specialize in Large Language Model (LLM) applications and enterprise AI automation solutions. As we expand into the US market, we’re looking for an experienced engineer who can both contribute technically and help us adapt our proven products for American enterprises.</p>
<p><strong>The Role</strong><br>
We are seeking a <strong>part-time Senior AI/LLM Engineer (15 hours/week)</strong> to support our US business expansion. This role blends technical development with occasional client-facing work. You will collaborate with our core R&amp;D team abroad while representing our technology to US customers.</p>
<p><strong>Key Responsibilities</strong></p>
<ul>
<li>
<p>Develop and adapt AI/LLM-powered solutions for the US market (Python, LangChain, OpenAI API, etc.)</p>
</li>
<li>
<p>Support demos and technical discussions with prospective clients</p>
</li>
<li>
<p>Provide market insight and feedback to our dev team</p>
</li>
<li>
<p>Contribute to building integrations, PoCs, and customizations for clients</p>
</li>
</ul>
<p><strong>Requirements</strong></p>
<ul>
<li>
<p>US-based (citizenship or work authorization required)</p>
</li>
<li>
<p>3+ years in AI/ML development, including LLM applications</p>
</li>
<li>
<p>Strong Python and experience with frameworks such as LangChain, HuggingFace, OpenAI API</p>
</li>
<li>
<p>Ability to communicate technical concepts to non-technical clients</p>
</li>
<li>
<p>Entrepreneurial mindset and interest in helping shape US strategy</p>
</li>
</ul>
<p><strong>Compensation</strong></p>
<ul>
<li>
<p>Part-time: ~15 hours/week</p>
</li>
<li>
<p>$1,000–$3,000 monthly (based on experience)</p>
</li>
<li>
<p>Additional hours at $50–$100/hour</p>
</li>
<li>
<p>Remote and flexible schedule</p>
</li>
</ul>
<p><strong>Why Join Us</strong></p>
<ul>
<li>
<p>Work on real products already proven in Asian markets</p>
</li>
<li>
<p>Contribute at an early stage of US expansion — high growth potential</p>
</li>
<li>
<p>Flexible part-time structure with opportunity to scale into a larger role</p>
</li>
</ul>
<p><strong>How to Apply</strong><br>
Please submit your resume and a short description of your AI experience via this <a href="https://forms.gle/gQTDxjtVGe9ipPQBA" rel="noopener nofollow ugc">form</a> : <a href="https://forms.gle/gQTDxjtVGe9ipPQBA" rel="noopener nofollow ugc">https://forms.gle/gQTDxjtVGe9ipPQBA</a>.</p>
CUDA toolkit 13.0 uninstalled but torch.utils.cpp_extension gives CUDA_HOME of v13

alicemabille: <p>I installed CUDA toolkit v12.9, then uninstalled CUDA toolkit v13.0, deleted <code>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0</code>, checked that no environment variables had this version, <code>nvcc –-version</code> gives</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:29:17_Pacific_Daylight_Time_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0
</code></pre>
<p>and <code>torch.__version__</code> gives <code>‘2.8.0+cu126’</code>in my vitrual env. But when calling <code>pip install ./causal-conv1d/ --no-build-isolation</code>in the same virtual env, I find that the <code>CUDA_HOME</code>variable imported from <code>torch.utils.cpp_extension</code>is <code>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0</code>.</p>
<p>Any idea on what could be the problem ? Any workaround ? If possible I would not want to change the code in every package that I use that relies on torch.utils.cpp_extension to get CUDA version.</p>

ptrblck: <aside class="quote no-group" data-username="alicemabille" data-post="1" data-topic="222694">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alicemabille/48/76264_2.png" class="avatar"> alicemabille:</div>
<blockquote>
<p>I find that the <code>CUDA_HOME</code>variable imported from <code>torch.utils.cpp_extension</code>is <code>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0</code>.</p>
</blockquote>
</aside>
<p>Check the <a href="https://github.com/pytorch/pytorch/blob/379ebdaf5e2fbf387de297f673254d005e583b31/torch/utils/cpp_extension.py#L95-L119">code</a> to see how the <code>cuda_home</code> variable is found.</p>
<p>As an explicit workaround you could set the <code>CUDA_HOME</code> env variable yourself pointing to the CUDA toolkit you want to use.</p>
[MPS Crash] Assertion failed in _getLSTMGradKernelDAGObject using LSTM on macOS with MPS backend

HarishAi: <p>Running an Bidirectional LSTM model on macOS with MPS backend crashes with the following error during training/backpropagation:<br>
Assertion failed: (shape4.size() &gt;= 3), function _getLSTMGradKernelDAGObject, file GPURNNOps.mm, line 2417.<br>
zsh: abort      python3 translation_model.py<br>
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown</p>

HarishAi: <p>Hello Jack,</p>
<p>PyTorch version: Version: 2.7.1</p>
<p>Mac OS version: 15.5 (24F74)</p>
<p>I am feeding the input as dimension (Sequence ,Batch, length) example: (N,1,27), I am not using the batch first<br>
Model definition:<br>
self.input_size=input_size<br>
self.hidden_size=hidden_size<br>
self.output_size=output_size<br>
self.encoder=nn.LSTM(self.input_size,self.hidden_size,2,bidirectional=True)<br>
self.u=nn.Linear(self.hidden_size<em>2,self.hidden_size)<br>
self.w1=nn.Linear(self.hidden_size,self.hidden_size)<br>
self.w2=nn.Linear(self.hidden_size,self.hidden_size)<br>
self.w3=nn.Linear(self.hidden_size,self.hidden_size)<br>
self.w4=nn.Linear(self.hidden_size,self.hidden_size)<br>
self.w=nn.Linear(self.hidden_size,self.hidden_size)<br>
self.atten=nn.Linear(self.hidden_size,1)<br>
self.output2catin=nn.Linear(self.output_size,self.hidden_size</em>2)<br>
self.decoder=nn.LSTM(self.hidden_size*4,self.hidden_size,4)<br>
self.hidden_output=nn.Linear(self.hidden_size,self.output_size)<br>
self.softmax=nn.LogSoftmax(dim=2)</p>
<p>yes the model works fine on CPU, even in GPU we can inference model, only the training procedure is killed.</p>

HarishAi: <p>Hello Jack,</p>
<p>I am mean model fails while training with the GPU, do you have idea why its failing, if so kindly help with the root cause?</p>
<p>Thanks,<br>
Harish.</p>

Vinnie: <p>Hi all,</p>
<p>Getting this exact same error when training a RecurrentPPO model on mps, which also uses a LSTM PyTorch model underneath. Running on the M3 Ultra.</p>
<p>Just updated to MacOS 15.6 and still the same error. Running Torch version 2.7.1</p>

ptrblck: <p>Could you create an issue on GitHub describing this use case in case the most recent nightly binary also fails?</p>

Vinnie: <p><a class="mention" href="/u/harishai">@HarishAi</a> , will you raise that issue?</p>

Vinnie: <p>Having exact same problem with the latest nightly build (2.9.0.dev20250801).</p>
<p>Did you already raise that issue <a class="mention" href="/u/harishai">@HarishAi</a> ?</p>

HarishAi: <p><a class="mention" href="/u/vinnie">@Vinnie</a> , Yes, I have raised it.</p>

Vinnie: <p>I am seeing no activity on the GitHub issue. How are they normally being picked up?</p>
How to pass options to env.reset within a data collector

icarosadero: <p>If you create a `GymEnv` you can pass keyword arguments to the reset method in the usual way:</p>
<pre><code class="lang-auto">env.reset(options={"custom_arg":"custom_value"}) 
</code></pre>
<p>I would like to know how to do the same for environments instantiated by data collectors like <code>SyncDataCollector</code>, would it be `<strong>create_env_kwargs</strong>`?</p>
How can we reliably figured out what the parameters of a generated PTX kernel are with torch.compile()?

anneouyang: <p>When we run torch.compile() on torch.matmul(A, B), we get a PTX function with signature:<br>
``<br>
<code>.visible .entry triton_(</code><br>
<code>.param .u64 triton__param_0,</code><br>
<code>.param .u64 triton__param_1,</code><br>
<code>.param .u64 triton__param_2,</code><br>
<code>.param .u64 triton__param_3,</code><br>
<code>.param .u32 triton__param_4,</code><br>
<code>.param .u32 triton__param_5</code><br>
<code>)</code><br>
```<br>
I want to be able to run this PTX function as a standalone with my own launch code in CUDA. In general, how do we know what parameters are being passed in?</p>
TorchRec EmbeddingBagCollection

acmilannesta: <p>Could TorchRec EmbeddingBagCollection (EBC) support no pooling?</p>
<p>Bascially I want to apply transformer encoder layer on sequence features (B*S*D), but EBC gives will only return B*D. I want to return the same shape of input, same as nn.Embedding</p>

ptrblck: <p>From the <a href="https://docs.pytorch.org/torchrec/modules-api-reference.html">docs</a>:</p>
<ul>
<li>
<p><code>EmbeddingBagCollection</code> is a collection of <code>torch.nn.EmbeddingBag</code></p>
</li>
<li>
<p><code>EmbeddingCollection</code> is a collection of <code>torch.nn.Embedding</code></p>
</li>
</ul>
<p>Maybe <code>EmbeddingCollection</code> would work for your use case?</p>
Torch distributed in different streams

Peiyuan_Zhou_CN: <p>Hi, experts</p>
<ol>
<li>We found 2 strange things when using torch distributed:<br>
OP always run in a separate stream (not current stream), but the <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L3614" rel="noopener nofollow ugc">code</a> shows it chooses default stream when <code>async_op</code> is false.</li>
<li>2.Sometimes we found 2 OPs can even run in 2 separate streams (neither in current stream).</li>
</ol>
<p>Really appreciate your help!</p>
Compiling forward inside of __init__() for a torch.nn.Module model

PyDevC: <p>while looking to some issues i found this code</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch._inductor.config as config
config.cpp_wrapper = True
torch._logging.set_logs(output_code=True)

class SimpleMLP(torch.nn.Module):
  def __init__(self):
    super().__init__()
    self.fc1 = torch.nn.Linear(28 * 28, 128)
    self.fc2 = torch.nn.Linear(128, 10)
    self.forward = torch.compile(self.forward)

  def forward(self, x):
    x = x.view(x.size(0), 28*28)
    x = torch.nn.functional.relu(self.fc1(x))
    x = self.fc2(x)
    return x

mlp = SimpleMLP().cuda()
</code></pre>
<p>here <code>torch.compile</code> is called inside of <code>__init__.py</code>. How is this method different from compiling the model with</p>
<pre data-code-wrap="python"><code class="lang-python">model = SimpleMLP().cuda
model.compile()
</code></pre>

vmoens: <p>I would bet that it’s just people not knowing about the built-in method (or it precedes that feature of non.Module)</p>
Support for Overlapping AllGather and ReduceScatter in FSDP

nariaki3551: <p>Hi all,</p>
<p>We’ve been exploring overlapping AllGather and ReduceScatter in FSDP’s backward pass to reduce communication time. The concept is based on <a href="https://sc24.supercomputing.org/proceedings/tech_paper/tech_paper_pages/pap545.html" rel="noopener nofollow ugc">this SC24 paper</a>, which demonstrates performance benefits from executing in-network multicast and reduction concurrently.</p>
<h2><a name="p-474463-challenge-1" class="anchor" href="#p-474463-challenge-1"></a>Challenge</h2>
<p>Although NVIDIA SHARP supports both multicast and reduction, we observed <strong>no speedup</strong> when running both collectives concurrently via NCCL + SHARP. This suggests SHARP cannot pipelined internally collectives.</p>
<p>To overcome this, we ran the two collectives via different communication paths:</p>
<ul>
<li>ReduceScatter: NCCL + SHARP (switch-based reduction)</li>
<li>AllGather: <a href="https://github.com/nariaki3551/muliticast-based-allgather/tree/dev/zero_copy" rel="noopener nofollow ugc">UCC/spin TL</a>, which uses InfiniBand <strong>UD multicast</strong> via low-level verbs. It is a prototype based on what the <a href="https://sc24.supercomputing.org/proceedings/tech_paper/tech_paper_pages/pap545.html" rel="noopener nofollow ugc">SC24 paper</a> team created with zero-copy functionality added for speed-up.</li>
</ul>
<p>In addition, we use the <a href="https://github.com/pytorch/pytorch/pull/153215#issuecomment-2864674866" rel="noopener nofollow ugc">patch for FSDP</a> to allow overlap these collectives in the FSDP1 runtime.</p>
<h2><a name="p-474463-experiments-2" class="anchor" href="#p-474463-experiments-2"></a>Experiments</h2>
<p>We trained a simple 3B model consisting of Linear layers using FSDP on 10 GPUs (V100) with Connect-6 connected via a NVIDIA Quantum IB switch. We got the <strong>1.07×</strong> speedup of FSDP1’s backward pass.</p>
<ul>
<li>Average Backward pass time
<ul>
<li>No overlap: 4.040s</li>
<li>With overlap (ReduceScatter + AllGather): 3.738s</li>
</ul>
</li>
</ul>
<h2><a name="p-474463-discussion-3" class="anchor" href="#p-474463-discussion-3"></a>Discussion</h2>
<p>I believe we should <strong>advance the discussion around supporting AllGather–ReduceScatter overlap in FSDP</strong> , starting with proposals like: <a href="https://github.com/pytorch/pytorch/pull/153215#issuecomment-2864674866" rel="noopener nofollow ugc">patch for FSDP</a> .</p>
<p>This pattern could benefit large-scale training on SHARP or multicast-enabled networks.<br>
I’ll continue evaluating it on more practical models, and would love to hear your thoughts.</p>
<p>Thanks!</p>

ptrblck: <p>I believe FSDP2 is replacing FSDP, if I’m not mistaken. In this case I would advice against spending time trying to improve legacy utils. and would suggest focusing on FSDP2.</p>

nariaki3551: <p>Thanks! Yes, we believe this method applies to FSDP2 as well. We’re currently implementing and testing it there.</p>
Confusion About LibTorch Binary Naming and ABI on "Get Started" Page

R90: <p>Hey all,</p>
<p>I’m confused by the LibTorch binary naming and ABI labeling on the PyTorch “Get Started” page (<a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a>). When selecting <strong>Linux</strong> and <strong>LibTorch</strong> with <strong>CUDA 12.6</strong>, the page provides the following:</p>
<ul>
<li>
<p><strong>Text</strong>: “Download here (cxx11 ABI)”</p>
</li>
<li>
<p><strong>URL</strong>: <code>https://download.pytorch.org/libtorch/cu126/libtorch-shared-with-deps-2.8.0%2Bcu126.zip</code></p>
</li>
</ul>
<p>The flavor text indicates this is a <strong>C++11 ABI</strong> binary, but the filename does <strong>not</strong> include “cxx11-abi,” unlike other binaries, such as:</p>
<ul>
<li><code>https://download.pytorch.org/libtorch/test/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip</code> (linked <a href="https://dev-discuss.pytorch.org/t/pytorch-2-5-release-branch-cut-for-pytorch-core-is-completed/2452" rel="noopener nofollow ugc">here</a> for example)</li>
</ul>
<p>This inconsistency is confusing. I believe in older PyTorch versions, the <code>libtorch-shared-with-deps</code> filename typically indicated a <strong>Pre-cxx11 ABI</strong> binary, while <code>libtorch-cxx11-abi-shared-with-deps</code> was used for the C++11 ABI. Has the naming convention changed, or is the flavor text on the “Get Started” page incorrect? Is the <code>libtorch-shared-with-deps-2.8.0+cu126.zip</code> binary truly compiled with the C++11 ABI?</p>
<p>I’m using [GCC 9.3, and RL9] and would appreciate clarification on:</p>
<ol>
<li>
<p>Whether <code>libtorch-shared-with-deps-2.8.0+cu126.zip</code> is indeed a C++11 ABI binary.</p>
</li>
<li>
<p>Any recommended steps to confirm the ABI of a LibTorch binary (e.g., via <code>nm</code> or a test program).</p>
</li>
</ol>
<p>Thanks for any insights or confirmation!</p>

ptrblck: <p>All of our builds should use the cxx11 ABI now after dropping legacy CUDA 11.8 builds and the file name of our <code>libtorch</code> builds should not indicate any ABI tag anymore. You can check it by printing symbols and filtering for e.g. <code>std::string</code> which should indicate if cxx11 ABI is used.</p>
Jupyter notebook repository for self-learning

vdw: <p>I hope this fits the posting guidelines here: For self-learners in the wider area of AI/ML/DL/NLP but also data science in general, we are building a <a href="https://github.com/chrisvdweth/selene/tree/master" rel="noopener nofollow ugc">repository of Jupyter notebooks</a>. The repository can be seen as an open, large-scale, interactive textbook – well, hopefully becoming large in the future. The original purpose of these notebooks was to serve as supplementary learning materials for students in my university courses, but now I just share them with everyone.</p>
<p>All topics related to deep learning and neural networks use PyTorch for code examples or full-fledged tutorials. Here are just some current examples:</p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/transformers_basic_architecture.ipynb" rel="noopener nofollow ugc">Transformers</a> and <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/attention_mha_basics.ipynb" rel="noopener nofollow ugc">Attention</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/recurrent_neural_networks_basics.ipynb" rel="noopener nofollow ugc">Recurrent Neural Networks – An Introduction</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/batch_processing_sequence_tasks.ipynb" rel="noopener nofollow ugc">Working with Batches for Sequence Tasks</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/handwritten_digit_recognition_ann.ipynb" rel="noopener nofollow ugc">Handwritten Digit Recognition with ANNs</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/text_classification_rnn.ipynb" rel="noopener nofollow ugc">Text Classification with RNNs</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/llm_training_data_preparation_batching.ipynb" rel="noopener nofollow ugc">Data Batching for Training LLMs</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/masking_sequence_models.ipynb" rel="noopener nofollow ugc">Masking in Sequence Models</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/mixture_of_experts_basics.ipynb" rel="noopener nofollow ugc">Mixture of Experts (MoE)</a></p>
<p>* <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/text_token_indexing.ipynb" rel="noopener nofollow ugc">Token Indexing with Vocabularies</a></p>
<p>Right now, the repository is more targeted toward beginners, with a strong focus on detailed explanations and illustrations. However, we have more advanced topics planned and already in the pipeline. So, yeah, you’re welcome to check it out. We actually created a Discord server for announcement, support, and feedback, but I feel posting the link here might violate some guidelines :).</p>
Heatmap normalization

marcleon: <p>Hi,</p>
<p>Does anybody know how to normalize the score map that represents the heatmap of an anomaly detector and is used to localize the anomalies?</p>
<p>What I mean is that the heatmap should not be normalized using the min and the max of the individual heatmap. It should be normalized using a global min and max, for example. I’ve tried to normalize using the min and the max score from the training dataset, but the result is not the best one. Does anybody know how to solve this?</p>
<p>Thanks!</p>

mycul: <p>My immediate answer is that you need to robust normalization. Essentially this means that you clip all of the values within a certain quantile range such as [0.05-.95] (you can play with this threshold) and then you take the max and min of that range. I only suggest this because sometimes heatmaps can have a few spikes that make for poor visualization of the image</p>
<p>However, I can’t entirely be sure how that differs from what you’re doing. Could you provide some code or images of the heatmaps to give us a better idea of what might be happening?</p>
In NVIDIA container environments, PyTorch's NCCL allreduce operation exhibits extremely poor performance

ginjia_lee: <p>hello，pytorch team, have you noticed issue mentioned in <a href="https://forums.developer.nvidia.com/t/torch-allreduce-with-low-performance-on-cuda12-8-compatibility/342624" class="inline-onebox" rel="noopener nofollow ugc">Torch allreduce with low performance on cuda12.8 compatibility - GPU-Accelerated Libraries - NVIDIA Developer Forums</a> , where nccl-test works fine but torch allreduce not?</p>

ginjia_lee: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a>  could you help on this issue? sounds like a cuda toolkit + pytorch issue, thank you.</p>

ptrblck: <aside class="quote no-group" data-username="ginjia_lee" data-post="2" data-topic="222505">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ginjia_lee/48/78005_2.png" class="avatar"> ginjia_lee:</div>
<blockquote>
<p>sounds like a cuda toolkit + pytorch issue, thank you.</p>
</blockquote>
</aside>
<p>I see this was cross-posted a few times already and based on the additional information it currently sounds like a setup issue to me.</p>
<p>I would recommend keeping the discussion in one place to avoid cross-posting different debug steps and outputs. In this case, let’s follow up in the bug you have created.</p>

ginjia_lee: <p>thank you for your reply. let’s follow in the bug report.</p>
Get FlexAttention generated code

Da1sy_petals: <p>I have looked into source code a bit and knew that FlexAttention works by generating triton code (correct me if i am wrong).<br>
Is it possible to get the generated code in some ways? I would like to reuse or manually modify that code.</p>
<p>Thanks a lot.</p>

marksaroufim: <p>It might be easiest for you to look at the jinja templates here <a href="https://github.com/pytorch/pytorch/tree/main/torch/%5C_inductor/kernel/flex/templates">https://github.com/pytorch/pytorch/tree/main/torch/\_inductor/kernel/flex/templates</a></p>
<p>Otherwise the classic torch.compile trick also works if you compile the kernel</p>
<pre><code class="lang-auto">os.environ["TORCH_COMPILE_DEBUG"] = "1"

</code></pre>