akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Android build: Native libraries not 16 KB page-aligned, Play Store rejecting apps

Prahlad: <p>APK is not compatible with 16 KB devices.<br>
Some libraries have LOAD segments not aligned at 16 KB boundaries:</p>
<p>base/lib/arm64-v8a/libpytorch_jni.so<br>
base/lib/arm64-v8a/libpytorch_vision_jni.so</p>
<p>base/lib/x86_64/libfbjni.so<br>
base/lib/x86_64/libpytorch_jni.so<br>
base/lib/x86_64/libpytorch_vision_jni.so</p>
<p><strong>Google requires 16 KB alignment starting Nov 1, 2025</strong>.</p>
<p>Requesting you to <strong>update Android build pipeline</strong> (CMake/NDK flags) to produce aligned <code>.so</code> files.</p>

cbilgin: <p>Please use ExecuTorch. We are no longer investing in PyTorch Mobile. You’ll get much more support along with all sorts of perf, memory, sizewins.</p>
<ul>
<li>ExecuTorch in Meta’s app <a href="https://engineering.fb.com/2025/07/28/android/executorch-on-device-ml-meta-family-of-apps/" class="inline-onebox" rel="noopener nofollow ugc">Accelerating on-device ML on Meta’s family of apps with ExecuTorch - Engineering at Meta</a></li>
<li>ExecuTorch in Reality Lab products: <a href="https://www.facebook.com/zuck/posts/pfbid0GchXk9XnAVgxRPfgCqLeZFivQGpcf6kzWWKZgGGxMkVEwJyQfmRa739Lwu5nt2ozl" class="inline-onebox" rel="noopener nofollow ugc">Mark Zuckerberg</a></li>
<li>VB post from a few years ago: <a href="https://venturebeat.com/ai/pytorch-executorch-extends-open-source-ai-for-new-quests-at-the-edge" rel="noopener nofollow ugc">https://venturebeat.com/ai/pytorch-executorch-extends-open-source-ai-for-new-quests-at-the-edge</a></li>
</ul>

Prahlad: <p>But still, the latest version of ExecuTorch is not aligned at 16 KB page boundaries.</p>
<p>// <a href="https://mvnrepository.com/artifact/org.pytorch/executorch-android" rel="noopener nofollow ugc">https://mvnrepository.com/artifact/org.pytorch/executorch-android</a><br>
implementation ‘org.pytorch:executorch-android:1.0.0-rc1’</p>
<p>When we can expect the newer version?</p>
<p>Can we use it in production app?</p>

olokobayusuf: <p>Do you still need this?</p>
Export model to pure C

Peter_Featherstone: <p>Is there a way to export a pytorch model to pure C code ?</p>
<p>Can you use <code>torch.export()</code> to go from Export IR to LLVM/MLIR IR to pure C, maybe using EmitC ?</p>
<p>My alternative approach is to export my model to ONNX, then load into Tinygrad, then compile to C code, which TinyGrad does anyway as it is essentially a compiler. However i would have thought that pytorch could do this natively with <code>torch.export()</code>.</p>
<p>Motivation is to create a self contained C library without any dependencies on executorch or onnxruntime.</p>

GerryMan: <p>Direct C code export isn’t supported but <code>torch.export</code> plus AOTInductor can produce a self contained C compatible library without python or onnx dependencies.</p>

olokobayusuf: <p><a class="mention" href="/u/peter_featherstone">@Peter_Featherstone</a> we’re building this exact thing at <a href="https://docs.muna.ai/predictors/ai" rel="noopener nofollow ugc">Muna</a>.  We ingest plain Python/PyTorch code, transpile it to C++, then cross-compile it for different platforms. The compiled binaries are fully self-contained, hardware accelerated, and can be used from multiple languages in as little as two lines of code.</p>
<p><a href="https://muna.ai/slack" rel="noopener nofollow ugc">Hop on our Slack</a> and I’ll be happy to give you a walkthrough.</p>
Pytorch cross entropy loss and softmax

Mia_B: <p>Theoretically, my model architecture should have a dense layer with a softmax activation at the output. However, in practice, and since I’m using the cross entropy loss from Pytorch, which already uses softmax, I do not add a softmax activation to my model. My question is, should I add a softmax activation exclusively for the evaluation mode? Especially since the model now has been trained to output values between [0, 1] without the need for one. Also, in theory, can I say that my model ends with a Softmax activation? If not, is there a way to use pytorch’s categorical cross entropy loss without the softmax activation?</p>

Mia_B: <p>Thank you for your response.<br>
Now let’s suppose one would still want the Softmax activation to figure in the model’s definition. Would that somehow be possible using Pytorch?</p>

valerian.rey: <p>You can have a Softmax layer in the model definition, and then in your training function apply a log to the output and use the NLLLoss. This will be equivalent to no last layer + CrossEntropyLoss. However, it’s not recommended because of numerical stability issues when doing the softmax and the log separately, so don’t do that. See the note in <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.Softmax.html#softmax" rel="noopener nofollow ugc">Softmax</a>.</p>
<p>Another valid option is to have a LogSoftmax layer at the end and use NLLLoss (again, equivalent to no last layer + CrossEntropyLoss). This one is numerically stable.</p>

Mia_B: <p>Thank you! I thought of something similar but had no idea about the stability issues related to using softmax seperately.</p>
How to complie libtorch in low-version GLIBC

cx_yang: <p>Libtorch needs GLIBC 2.29 or newer for cxx11 ABI version. However,the computing cluster only supports GLIBC2.17. I can’t uograde the GLIBC version without root privileges. I also tried compling a new version of GLIBC locally and adding it to the environment variables,but this caused a serious crash</p>

cx_yang: <p>I successfully solved this problem and recorded the solution process. If you encounter similar problems, you can refer to this tutorial <a href="https://zhuanlan.zhihu.com/p/1958514567463301363" rel="noopener nofollow ugc">https://zhuanlan.zhihu.com/p/1958514567463301363</a></p>
Torchtext download from whl gives old version

Daniel_Cardenas: <p>Following instructions from: <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a><br>
pip3 install torchtext --index-url <a href="https://download.pytorch.org/whl/cu124" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu124</a><br>
Get torchtext version 0.6<br>
Should that be version 0.18?<br>
If the reason is that torchtext is deprecated, then should that pip install option be removed rather than installing an archaic version?</p>

ptrblck: <p>Where did you find this install command, as the <code>Start Locally</code> install matrix shows the install command for <code>torch torchvision torchaudio</code>, so unclear which option should be removed.<br>
<code>pip install torchtext</code> installs properly <code>torchtext==0.18</code> for me.</p>

magla2901: <p>Same problem here:</p>
<p>pip install torchtext==0.18.0</p>
<p>returns “ERROR: Could not find a version that satisfies the requirement torchtext==0.18.0 (from versions: 0.1.1, 0.2.0, 0.2.1, 0.2.3, 0.3.1, 0.4.0, 0.5.0, 0.6.0)”</p>
<p>(Win 11, python 3.13, torch 2.8)</p>

ptrblck: <p><code>torchtext==0.18.0</code> was released in April 2024 before Python 3.13 was released in Oct 2024, so you would need to downgrade Python. You can find the available wheels <a href="https://pypi.org/project/torchtext/#files">here</a>.</p>

vdw: <p>Since torchtext is not longer maintained, I would refrain from using it.</p>
I am running the below code, which is wrong, but still the torch run command runs without any errors? How do I debug this?

Vishnu_Kadam: <pre><code class="lang-auto">
def main( save_every, total_epochs, batch_size,nlat, nlon,ds,mean,std,snapshot_path = ‘snapshot.pth’):
init_process_group(backend=“nccl”)
device = int(os.environ[“LOCAL_RANK”])
rank = int(os.environ[“RANK”])
channels = 6
model = SFNO(operator_type=‘driscoll-healy’,img_size=(nlat, nlon),num_layers=8,scale_factor=3,embed_dim=384,pos_embed=None,use_mlp=True,normalization_layer=“none”,in_chans=channels,out_chans=channels,)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
model = model.to(device)
model = DDP(model, device_ids=[device])
dataset_train = ERA5Dataset(ds.sel(time=slice('2018-06-30', '2018-12-31')), input_steps=1, output_steps=1)
sampler = DistributedSampler(dataset_train, shuffle=True)
loader = DataLoader(
        dataset_train,
        batch_size=256,
        sampler=sampler,
        num_workers=NUM_WORKERS,
        pin_memory=True,
        drop_last=True,
        prefetch_factor = 150
    )
batch_size = 256  # dynamic scaling
print(f"\n--- Phase 3: Training  ---")
criterion = LatWeightedLoss(ds, p=2, device=device)

for initial_input, y_true in loader:
    initial_input = initial_input.squeeze(1).to(device)  # (B, C, H, W)
    y_true = y_true.to(device)                           # (B, C, autoSteps, H, W)

    current_input = initial_input
    loss = 0.0

    pred = model(current_input)                        # (B, C, H, W)
    lss = criterion(pred, torch.rand(3,3,3,3)) 
        
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    print(f"Step {step}, AR={autoSteps}, Loss={loss.item():.6f}")



destroy_process_group()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b779f74fb89b82d7dc588266ef009ae7152e48f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/b779f74fb89b82d7dc588266ef009ae7152e48f1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b779f74fb89b82d7dc588266ef009ae7152e48f1.png" alt="image" data-base62-sha1="qb6GFkpS6j7eRixf2hv0VoMIbol" width="523" height="313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">523×313 75.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the output.</p>

ptrblck: <p>Could you describe what exactly is wrong and what the code is expected to print?</p>

Vishnu_Kadam: <pre><code class="lang-auto"> lss = criterion(pred, torch.rand(3,3,3,3))
</code></pre>
<p>there is a size mismatch at this line. So i should get the RuntimeError. Pred is (Batch, 6, 32, 64)</p>

ptrblck: <p>I don’t know how <code>LatWeightedLoss</code> is defined, but please feel free to post a minimal and executable code snippet reproducing the issue in case you are stuck in solving the shape mismatch error.</p>
Better model than CNN and Attension on image object detection?

Way_New: <p>There are some images and corresponding annotations. Under some transforms on image the labels are the same.<br>
How to design a good model with good accuracy and fast speed?<br>
The current model is CNN and Attesion, training by gradient decent.<br>
I have some experiences on using UNets with Conv(kernel=3,padding=1), Maxpool(kernel=2,stride=2) and upsampling fusion, it’s better than one conv and one Mamba linear state space layer and not much slow.</p>
I made a Candy AI Clone, but I'm stuck with a weird problem

Nitin_Pandey: <p>Hey everyone,<br>
I’m working in Triple Minds and i have a project So, I’ve been working on a <strong>Candy AI Clone</strong> project for the past few days — basically building my own version of Candy AI with custom data and character responses. Everything’s going great so far, but I’m stuck on one issue.</p>
<p>Whenever I try to <strong>save the chat history and reload it</strong>, the AI starts repeating old messages instead of continuing the conversation naturally. I’m using a <strong>Python + FastAPI backend</strong> with <strong>OpenAI API integration</strong> and a <strong>React frontend</strong>.</p>
<p>Here’s what I’ve tried so far:</p>
<ul>
<li>
<p>Added unique session IDs for every chat</p>
</li>
<li>
<p>Stored conversation in MongoDB</p>
</li>
<li>
<p>Tried clearing context after every 10 messages</p>
</li>
<li>
<p>Still, it loops back to the same responses when reloaded</p>
</li>
</ul>
<p>Has anyone faced a similar issue while making a Candy AI or any chatbot clone?<br>
Would love some advice or code suggestions to handle this properly.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/folded_hands.png?v=14" title=":folded_hands:" class="emoji" alt=":folded_hands:" loading="lazy" width="20" height="20"></p>
Struggling to pick the right batch size

GerryMan: <p>Training a CNN on image data keeps running into GPU memory issues when using bigger batch sizes but going smaller makes the training super slow and kind of unstable.</p>

Thomas201Tang: <p>Hello!</p>
<p>When dealing with CNN training where large batch sizes cause GPU memory issues but small sizes cause slowness and instability, the primary fix is to decouple the effective batch size from VRAM usage. Implement Gradient Accumulation, which allows you to process several small batches before performing <a href="https://www.myfordbenefits.com.co" rel="noopener nofollow ugc">MyFordBenefits</a> one large weight update, simulating a bigger batch size. Additionally, utilize Mixed Precision Training (FP16) to halve the memory footprint, further increasing the capacity of your GPU and improving stability without sacrificing speed.</p>

GerryMan: <p>Did you personally see any slowdown or memory quirks when using gradient accumulation with mixed precision??</p>
TS2EPConverter and `dynamic_shapes`

olokobayusuf: <p>We are working with pipelines where we have to convert pre-existing <code>torch.jit</code> models to downstream AI serving formats (e.g. ONNX, ExecuTorch). The current <code>torch.export.export</code> API does not support these, so instead we use <code>torch._export.converter.TS2EPConverter</code>.</p>
<p>We need to add support for dynamic shapes; and internally, the converter computes dynamic shapes. So it seems like this could be as simple as exposing <code>dynamic_shapes</code> in the converter’s API.</p>
May I ask when CUDA 13.0 is expected to be released

tao_Zhao: <p>The current CUDA version of the computer is 13.0, and Pytorch 12.9 cannot be used</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used as our Pytorch binaries ship with all needed CUDA runtime dependencies. You would need to properly install an NVIDIA driver before installing our binaries.</p>
<p>I’m any case, the CUDA 13 nightly binaries are WIP already.</p>

tao_Zhao: <p>Thanks for your reply</p>

John_Beneke: <p>Any new developments on when PyTorch will be ready for CUDA 13?</p>

ptrblck: <p>Nightly binaries are already available.</p>

btakita: <p>Hi, I’m looking on the <a href="https://download.pytorch.org/whl/nightly/cu130/torch" rel="noopener nofollow ugc">nightly page</a> for torch cu130. I don’t see v2.8.0 however.</p>
<p>Will there be a release for torch 2.8.0+cu130?</p>
<p>Unfortunately I have a development issue b/c I’m running CachyOS (Arch Linux). The rolling distro upgraded us to CUDA 13.0. So I’m having issues with some libraries that expect a matching CUDA version…which are stuck on 12.8. I could upgrade to the next version, but some of the builds aren’t available yet…<a href="https://github.com/pytorch/pytorch/pull/163239" rel="noopener nofollow ugc">like xformers</a>…Though hopefully it will be out soon!</p>

ptrblck: <p>No, PyTorch 2.8.0 won’t be re-released. Our nightly binaries as well as the upcoming 2.9.0 release support CUDA 13.</p>

btakita: <p>FWIW, It looks like there’s an Arch package for torch 2.8.0 built with CUDA 13. But I don’t see it anywhere in PIP.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://archlinux.org/packages/extra/x86_64/python-pytorch-cuda/">
  <header class="source">
      <img src="https://archlinux.org/static/favicon.png" class="site-icon" alt="" width="31" height="32">

      <a href="https://archlinux.org/packages/extra/x86_64/python-pytorch-cuda/" target="_blank" rel="noopener nofollow ugc">archlinux.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://archlinux.org/packages/extra/x86_64/python-pytorch-cuda/" target="_blank" rel="noopener nofollow ugc">Arch Linux - python-pytorch-cuda 2.8.0-6 (x86_64)</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Mm() implementation in pytorch for CPU

subhojitdas: <p>I am trying to find out the detailed implementation of matrix multiplication in PyTorch. Where can I find the source code ( CPU implementation ) for PyTorch’s implementation of matrix multiplication?</p>
<p>I dig down till aten/src/ATen/native/LinearAlgebra.cpp and found the <code>static Tensor _matmul_impl</code> , however after that I am not able to get to the dispatch of <code>at::mm()</code> or <code>bmm()</code> .</p>
<p>I see the question was answered <a href="https://discuss.pytorch.org/t/matrix-multiplication-source-code/71071">here</a> in 2020 , however the link provided in the answer is no longer valid as pytorch architecture evolved in the last 5 years.</p>
<p>Thanks,</p>
<p>Subhojit</p>
Nanonets_OCR on 2 * A100 GPU

Ahmed225: <p>Hi everybody,</p>
<p>I am using Nanonets_ocr from huggingface to perform some ocr task. The model is quite good at extracting data but the problem is that, it is taking too long to extract the data (almost 2 minutes).</p>
<p>This is the application set up:</p>
<p>The model is running on 2* A100 nvidia GPU server, and I have exposed a flask api where user can upload one image and call the api, in the back end the model extract specific data and return it to the front end developper.</p>
<p>Can somebody helps me to achieve better accuracy ?</p>

ptrblck: <aside class="quote no-group" data-username="Ahmed225" data-post="1" data-topic="223443">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ce7236/48.png" class="avatar"> Ahmed225:</div>
<blockquote>
<p>but the problem is that, it is taking too long to extract the data (almost 2 minutes</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Ahmed225" data-post="1" data-topic="223443">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ce7236/48.png" class="avatar"> Ahmed225:</div>
<blockquote>
<p>Can somebody helps me to achieve better accuracy ?</p>
</blockquote>
</aside>
<p>Is the issue the long extraction time or the model accuracy? In the former case, did you profile your use case already to isolate the bottleneck?</p>

Ahmed225: <p>The issue is the long extraction time. The model is quite accurate to extract data without errors. But it is taking so much time to do so</p>
Is a B580 or RTX 5050 better for pytorch

Paradox_Phantom: <p>Im looking to get a new GPU for pytorch but can only fit (price range) between a B580 and a 5050 (RX 9060 8gb is also a option). Im thinking the b580 is the best due to its 12gb of vram but im not sure how well they do in pytorch or if they even work.</p>

marksaroufim: <p>At that VRAM budget you’re likely better off just using google colab with a nice 16 GB of VRAM</p>

Paradox_Phantom: <p>But if u had to go with either two, which is the least bad?<br>
Want to make a tool to use in real time to detect objects (more simple ones, so smaller model)</p>
I have GPU, but `torch.cuda.available()=False`

Zoe_Zhao: <p>I submitted my job by SLURM to a public compute resource. But there is an error:RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device(‘cpu’) to map your storages to the CPU.</p>
<p>But I’ve already applied the GPU:</p>
<p><strong>CUDA_VISIBLE_DEVICES</strong>=0<br>
<strong>LD_LIBRARY_PATH</strong>: /cvmfs/restricted.computecanada.ca/easybuild/software/2023/x86-64-v3/Core/cudacompat/12.9/lib</p>
<p>=== NVIDIA-SMI ===<br>
Wed Oct  1 17:38:56 2025<br>
±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 12.6     |<br>
±----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:5D:00.0 Off |                    0 |<br>
| N/A   28C    P0             68W /  700W |       0MiB /  81559MiB |      0%      Default |<br>
|                                         |                        |             Disabled |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------------------+<br>
| Processes:                                                                              |<br>
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |<br>
|        ID   ID                                                               Usage      |<br>
|=========================================================================================|<br>
|  No running processes found                                                             |<br>
±----------------------------------------------------------------------------------------+</p>
<p>And I print the pytorch info:</p>
<p>torch.version.cuda 12.6<br>
torch.backends.cudnn.version() 90501<br>
torch.cuda.device_count() 0<br>
torch.cuda.is_available() <strong>False</strong></p>
<p>Is there something wrong?</p>

ptrblck: <p>Are you able to run any plain CUDA application in this environment, e.g. the CUDA samples?</p>

Zoe_Zhao: <p>Oh,I cannot pass the CUDA samples!</p>

ptrblck: <p>In this case it seems the environment itself is not set up correctly and you might need to check if the drivers, compat packages, etc. are properly installed.</p>
Vision and Torch update compatibility matrix

AMD-Python: <p>I am trying to install <code>torch</code> 2.5.0 for CPU-only use on Python 3.13. I am facing some issues  <code>torchvision</code> as I attempted to install versions from 0.19 to 0.22, but I still got a hard failure. I would like to <strong>avoid</strong> building<code>torchvision</code> from scratch. As it depends on many dependencies that are, in my case, fixed (e.g., NumPy 2.1.3 and SciPy 1.14.1). Something like:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>torch</th>
<th>vision</th>
<th>python</th>
<th>numpy</th>
</tr>
</thead>
<tbody>
<tr>
<td>2.5.0</td>
<td>…</td>
<td>&gt;=3.10 &amp; &lt;=3.13</td>
<td>…</td>
</tr>
<tr>
<td>2.4.0</td>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
</div><p>My question: Is there a compatibility matrix for Torch, Vision, and Python versions that is up-to-date (<a href="https://discuss.pytorch.org/t/pytorch-torchvision-on-jetson-agx-xavier-incompatibility/174048">2023</a> and <a href="https://blog.csdn.net/m0_45871473/article/details/134582240" rel="noopener nofollow ugc">2023b</a>) and backed up by the Torch community?</p>
<p>Thanks.</p>
Defining loss that maximizes separation

maxspiri: <p>I am devising a loss function that seeks to maximize separation between each point and its positive/negative samples, kind of like triplet loss. My loss is currently defined as the difference of squares between the distance from a sample and its nearest neighbor with the same label and its nearest neighbor with a different label. However, it typically begins with this value being positive, but instead of going to negative infinity, the loss approaches zero which makes the positive distance equal to the negative distance, instead of the positive distance approaching zero and the negative distance approaching infinity. What could I do to make the loss work this way?</p>
The implementation of AdamW in torch is different from the original paper?

nansea: <p>When I was looking up to the implementation of AdamW in torch, I found it was different from the original paper. In the original paper, the weight decay item was multiplied by lr_scheduler multiplier while in torch it was multiplied by lr. I also check the source code of the paper and I am pretty sure it is different from the torch implementation. Why? Am I misunderstanding something?</p>
<p>original paper:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41ddc0fecb24ffb6aa6232a929c729811ec6527f.png" data-download-href="https://discuss.pytorch.org/uploads/default/41ddc0fecb24ffb6aa6232a929c729811ec6527f" title="paper"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ddc0fecb24ffb6aa6232a929c729811ec6527f_2_690x334.png" alt="paper" data-base62-sha1="9oG8KmluZirLbjdNT4oYuIhAVYz" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ddc0fecb24ffb6aa6232a929c729811ec6527f_2_690x334.png, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41ddc0fecb24ffb6aa6232a929c729811ec6527f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41ddc0fecb24ffb6aa6232a929c729811ec6527f.png 2x" data-dominant-color="EDECEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">paper</span><span class="informations">881×427 73.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Pytorch implementation: <a href="https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html#torch.optim.AdamW" rel="noopener nofollow ugc">AdamW — PyTorch 2.7 documentation</a></p>

veritas: <p>Hello. This is actually a well-known issue.</p>
<p>See: <a href="https://fabian-sp.github.io/posts/2024/02/decoupling" class="inline-onebox" rel="noopener nofollow ugc">How to jointly tune learning rate and weight decay for AdamW - Fabian Schaipp</a></p>
<p>I would also love to see this bug fixed.</p>
Ost Training Quantization fails on SPAN model with type_as

MarioAvolio: <p>Hi all,</p>
<p>I’m trying to run post training quantization (PTQ) on a super-resolution model called <strong>SPAN</strong>, but I’m running into an error when I forward a quantized input.</p>
<p>I wrapped the model like this:</p>
<pre><code class="lang-auto">@ARCH_REGISTRY.register()
class QuantizedSPAN(SPAN):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)  # original SPAN
        self.quant = torch.ao.quantization.QuantStub()
        self.dequant = torch.ao.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        print(f"quantized x --&gt; \n {x} \n  Type: {x.dtype}\n")

        x = super().forward(x)
        x = self.dequant(x)
        return x

</code></pre>
<p>The parent <code>SPAN.forward()</code> starts with:</p>
<pre><code class="lang-auto">def forward(self, x):
    self.mean = self.mean.type_as(x)  # &lt;-- this line crashes
    x = (x - self.mean) * self.img_range
    ...

</code></pre>
<p>Before quantization, the input looks like this (<code>torch.float32</code>):</p>
<pre><code class="lang-auto">tensor([[[[0.1137, 0.1137, 0.1137,  ..., 0.1098, 0.1059, 0.1059],
          ...
]]], dtype=torch.float32)

</code></pre>
<p>After quantization, it becomes (<code>torch.quint8</code>):</p>
<pre><code class="lang-auto">tensor([[[[0.1124, 0.1124, 0.1124,  ..., 0.1092, 0.1059, 0.1059],
          ...
]]], device='cuda:0', size=(1, 3, 40, 800), dtype=torch.quint8,
       quantization_scheme=torch.per_tensor_affine, scale=0.003210579277947545,
       zero_point=0)

</code></pre>
<p>And then I get this error:</p>
<pre><code class="lang-auto">File "SPAN_arch.py", line 263, in forward
    x = super().forward(x)
File "SPAN_arch.py", line 232, in forward
    self.mean = self.mean.type_as(x)
RuntimeError: empty_strided not supported on quantized tensors yet
see https://github.com/pytorch/pytorch/issues/74540

</code></pre>
<hr>
<h3><a name="p-475840-my-questions-1" class="anchor" href="#p-475840-my-questions-1"></a>My questions:</h3>
<ol>
<li>
<p>Is there a recommended way to handle situations where <code>.type_as(x)</code> is called but the tensor is quantized (<code>torch.quint8</code>)?</p>
</li>
<li>
<p>Should I just keep <code>self.mean</code> in float32 and avoid casting it to the quantized dtype?</p>
</li>
<li>
<p>Is this a known limitation of PTQ in PyTorch, especially on CUDA?</p>
</li>
</ol>
<p>Any advice would be appreciated</p>
Get ROCm 7.0 correctly installed and able to detect my RX 9070 XT (gfx1201, Chip ID 0x7550) to be able to use Stablediffusion on ComfyUI for text to video or text to image and image to video on a functioning workflow/workflows

Ozzie_Murray: <p>I want to Install a PyTorch wheel built for ROCm 7.0 that matches my Python ABI (cp310) so ComfyUI / AnimateDiff can use my AMD GPU (Radeon RX 9070 XT, gfx1201) for either text to video, or text to image then image to video. What we tried                                                                                                                     (brief list of failed attempts)</p>
<ul>
<li>
<p>Probed the PyTorch ROCm index with curl/grep; index returned HTTP 403 Access Denied so no wheels could be listed or fetched.</p>
</li>
<li>
<p>Let pip try to auto-select a wheel from the ROCm index; pip either found nothing or installed incompatible builds (CUDA or cp312) and/or produced huge terminal output that closed the shell.</p>
</li>
<li>
<p>Created and switched to a Python 3.10 conda env (comfyui-rocm-py310) to match expected ROCm wheel ABI, then attempted pip install from rocm index — index blocking prevented success.</p>
</li>
<li>
<p>Ran mamba/conda installs; solver downloaded ~3GB but installed CUDA-targeted PyTorch (cuda12.4) into the env instead of a ROCm build. Removed those CUDA packages repeatedly.</p>
</li>
<li>
<p>Tried to list cp310/cp311 wheels via scripted parsing; returned nothing because the index is inaccessible.</p>
</li>
<li>
<p>Attempted browser access to the ROCm index from the same machine; browser also failed with Access Denied.</p>
</li>
<li>
<p>Repeated uninstall/cleanup cycles to remove wrong CUDA artifacts; ended with a clean env but still no ROCm wheel available.</p>
</li>
</ul>
<h3><a name="p-475830-current-status-key-facts-1" class="anchor" href="#p-475830-current-status-key-facts-1"></a>Current status (key facts)</h3>
<ul>
<li>
<p>My GPU is detected by ROCm: AMD Radeon RX 9070 XT — gfx1201, Chip ID 0x7550.</p>
</li>
<li>
<p>comfyui-rocm-py310 exists and is cleaned of previous CUDA/pytorch packages.</p>
</li>
<li>
<p>The PyTorch ROCm index (<a href="https://download.pytorch.org/whl/rocm7.0/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/rocm7.0/</a>) is blocked (HTTP 403) from your network, preventing pip from fetching +rocm wheels automatically.</p>
</li>
</ul>
PyTorch Mastery: Stop Watching Tutorials, Start Building Neural Networks! 🤯

vuhung: <p><a href="https://github.com/vuhung16au/pytorch-mastery" rel="noopener nofollow ugc">PyTorch Mastery</a>: Stop Watching Tutorials, Start Building Neural Networks! <img src="https://discuss.pytorch.org/images/emoji/apple/exploding_head.png?v=14" title=":exploding_head:" class="emoji" alt=":exploding_head:" loading="lazy" width="20" height="20"></p>
<p>Ever feel lost jumping between different PyTorch guides, struggling to connect the fundamentals to a full, deployable project? You’re not alone. The deep learning journey has a steep learning curve, but I’ve built something to flatten it out!</p>
<p>The reason I created this repo is: I couldn’t run most of sample code on PyTorch website!</p>
<p>This is my PyTorch study notes, with focus on NLP and transition from TensorFlow.</p>
<p>I’m stoked to share my new repository: PyTorch Mastery! This isn’t just a collection of notebooks; it’s a structured, hands-on roadmap designed to take you from PyTorch fundamentals to tackling real-world problems.</p>
<p>Here’s what you’ll find inside this ripper of a resource:</p>
<p>PyTorch Fundamentals: We start with the basics—tensors, data types, and GPU setup—so you know exactly what’s under the hood.</p>
<p>The Deep Learning Workflow: A clear, repeatable outline for approaching any deep learning problem, making project initiation a breeze.</p>
<p>Practical Applications: Hands-on code for Neural Network Classification and Computer Vision projects.</p>
<p>Modular Code: Learn how to structure your projects into industry-standard, modular Python scripts (how the pros do it, fair dinkum!).</p>
<p>The focus is code, code, code and experiment, experiment, experiment. By the time you’re done, you won’t just know what PyTorch is; you’ll know how to build and structure production-ready models.</p>
<p>Ready to level up your deep learning skills? Dive into the code and start your mastery journey today!</p>
<aside class="onebox githubrepo" data-onebox-src="https://github.com/vuhung16au/pytorch-mastery">
  <header class="source">

      <a href="https://github.com/vuhung16au/pytorch-mastery" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/36806a85059e41fcd43568fa5580d24c/vuhung16au/pytorch-mastery" class="thumbnail">

  <h3><a href="https://github.com/vuhung16au/pytorch-mastery" target="_blank" rel="noopener nofollow ugc">GitHub - vuhung16au/pytorch-mastery: PyTorch Learning Journey</a></h3>

    <p><span class="github-repo-description">PyTorch Learning Journey</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
TorchCurves: differentiable curve modules (B-spline/Legendre Poly)

Alex_Shtof: <p>I’ve released a small library for parametric curves for PyTorch that are differentiable: you can backprop to the curve’s inputs and to its parameters. At this stage, I have B-Spline curves (efficiently, exploiting sparsity!) and Legendre Polynomials.</p>
<p>Link: <a href="https://github.com/alexshtf/torchcurves" class="inline-onebox" rel="noopener nofollow ugc">GitHub - alexshtf/torchcurves: Parametric differentiable curves with PyTorch for KANs, continuous embeddings, or shape-restricted models</a></p>
<p>Applications include:</p>
<ul>
<li>Continuous embeddings for embedding-based models (i.e. factorization machines, transformers, etc)</li>
<li>KANs. You don’t have to use B-Splines. You can, in fact, use any well-approximating basis for the learned activations.</li>
<li>Shape-restricted models, i.e. modeling the probability of winning an auction given auction features x and a bid b. You have a neural network c(x) that predicts the coefficients of a function of b. If you force the coefficient vector to be non-decreasing, then if used with a B-Spline you will get a non-decreasing probability, which is the right inductive bias.</li>
</ul>
<p>I hope some of you will find it useful!</p>
GPU Memory Leak when using MixStyle Layer

Vedant_Dalimkar: <p>I am using the <a href="https://arxiv.org/abs/2104.02008" rel="noopener nofollow ugc">MixStyle</a> methodology for domain adaptation and it involves using a custom layer which is inserted after every encoder stage. However, it is causing VRAM to grow linearly, which causes OOM error. No memory leak occurs on disabling the layer. Any idea on why this is happening?</p>
<p>This is the layer for reference -</p>
<pre><code class="lang-auto">import random
import torch
import torch.nn as nn


class MixStyle(nn.Module):
    """MixStyle.
    Reference:
      Zhou et al. Domain Generalization with MixStyle. ICLR 2021.
    """

    def __init__(self, p=0.5, alpha=0.1, eps=1e-6, mix='random'):
        """
        Args:
          p (float): probability of using MixStyle.
          alpha (float): parameter of the Beta distribution.
          eps (float): scaling parameter to avoid numerical issues.
          mix (str): how to mix.
        """
        super().__init__()
        self.p = p
        self.beta = torch.distributions.Beta(alpha, alpha)
        self.eps = eps
        self.alpha = alpha
        self.mix = mix
        self._activated = True

    def __repr__(self):
        return f'MixStyle(p={self.p}, alpha={self.alpha}, eps={self.eps}, mix={self.mix})'

    def set_activation_status(self, status=True):
        self._activated = status

    def update_mix_method(self, mix='random'):
        self.mix = mix

    def forward(self, x):
        if not self.training or not self._activated:
            return x

        if random.random() &gt; self.p:
            return x

        B = x.size(0)

        mu = x.mean(dim=[2, 3], keepdim=True)
        var = x.var(dim=[2, 3], keepdim=True)
        sig = (var + self.eps).sqrt()
        mu, sig = mu.detach(), sig.detach()
        x_normed = (x-mu) / sig

        lmda = self.beta.sample((B, 1, 1, 1))
        lmda = lmda.to(x.device)

        if self.mix == 'random':
            # random shuffle
            perm = torch.randperm(B)

        elif self.mix == 'crossdomain':
            # split into two halves and swap the order
            perm = torch.arange(B - 1, -1, -1) # inverse index
            perm_b, perm_a = perm.chunk(2)
            perm_b = perm_b[torch.randperm(B // 2)]
            perm_a = perm_a[torch.randperm(B // 2)]
            perm = torch.cat([perm_b, perm_a], 0)

        else:
            raise NotImplementedError

        mu2, sig2 = mu[perm], sig[perm]
        mu_mix = mu*lmda + mu2 * (1-lmda)
        sig_mix = sig*lmda + sig2 * (1-lmda)

        return x_normed*sig_mix + mu_mix
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
How to use quantized weights for manual implementation of the model in FPGA?

lucasmazz: <p>Hi! I’m starting to study about the implementation of quantized models in FPGA. Although I’m still learning, I would like to know if I can use PyTorch quantization for this.</p>
<p>I mean, I’d like to train a simple CNN model using PyTorch, quantize it to integers, and save the quantized weights and biases to a file, so I can load it later into the same CNN implemented manually on the FPGA.</p>
<p>So I believe I will need to implement Post-Training Static Quantization in the trained model, as shown on <a href="https://pytorch.org/blog/quantization-in-practice/#post-training-static-quantization-ptq" rel="noopener nofollow ugc">this page</a>, but I’m not quite sure what to do with the weights and biases after the PyTorch quantization process. That way, when I try to check the weights of the layers after quantization, they still appear to be float, but now they also have scale and zero_point values. How can I use this information to effectively have the weights and biases as integer values for future manual implementation of the model in FPGA?</p>
<p>Also, I would really appreciate it if anyone has any tips or suggestions for this kind of hardware implementation.</p>
<p>Thank you very much.</p>

Sairam954: <p><a class="mention" href="/u/lucasmazz">@lucasmazz</a>, here is a discussion on how you can perform post-static quantization on pre-trained models.  <a>https://discuss.pytorch.org/t/i-want-to-quantize-my-trained-model-model-pth-model-size-188mb/151065/2?u=sairam954</a></p>
<p>PyTorch also provides int8 pre-trained models, available models information here : [<a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.14 documentation</a>](<a href="https://Pre-Trained" rel="noopener nofollow ugc">https://Pre-Trained</a> Models)</p>
<p>You can access the int8 values of quantized tensors by using the int_repr() method.</p>
<p>I don’t have an answer for how you would implement it on FPGA. I am working on similar requirement, if it works, I will share more about it.</p>
<p>Thanks</p>

daitruong: <p>Hi <a class="mention" href="/u/lucasmazz">@lucasmazz</a>, Currently, I am also starting to study about he implementation of quantized models in FPGA. Could you please share your research on this topic? thanks.</p>
Explain Adam state when using DDP

sh0416: <p>Is the Adam’s state dict work correct when we use DDP???</p>
<p>I think there is no sync for the optimizer. Do I have to make code for this manually?</p>
<p>Thanks,</p>

ptrblck: <p>DDP should synchronize the gradients in the backward using AllReduce operations, such that each optimizer should use the same model parameters as well as gradients, thus also creating the same internal states.<br>
I don’t think there is a need to synchronize the optimizer’s state, but <a class="mention" href="/u/mrshenli">@mrshenli</a> might correct me, if I’m wrong.</p>

sh0416: <p>I just found out that <code>LayerNorm</code> with learnable parameter destroy the synchronization between the weights.<br>
<code>SyncBatchNorm</code> is a solution for <code>BatchNorm</code>, but <code>LayerNorm</code> doesn’t have <code>SyncLayerNorm</code>. I just give up using <code>DistributedDataParallel</code> because it may not improve the performance if I add more synchroize point for <code>LayerNorm</code>. In my case, almost all submodule have <code>LayerNorm</code>.<br>
By the way, I really appreciate your help and thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

mrshenli: <p>Yep, I agree with <a class="mention" href="/u/ptrblck">@ptrblck</a>’s comment above.</p>

osivaz61: <p>Did you find a way to sync LayerNorm.</p>

shendiaomo: <p>If we load from an existing checkpoint, there may be some bugs that cause different ranks to have different optimizer states. For example, assume we are resuming from another training job that’s dumping checkpoints quickly, rank 0 of our new job load from a checkpoint at step 10, while rank 1 load from the next checkpoint at step 11</p>

ptrblck: <p>The DDP initialization will broadcast the <code>state_dict</code> from rank0 to all participating ranks. So as long as you load the proper checkpoint on rank0 you should be fine and no additional synchronization would be needed.</p>
torch.cuda.set_per_process_memory_fraction() doesn’t see shared memory

elismasilva: <p>torch.cuda.set_per_process_memory_fraction() doesn’t see shared memory. I have a GPU with 8GB of video memory. I’m running a model that consumes my 8GB and overflows another 8GB into Windows’ shared video memory. When I set torch.cuda.set_per_process_memory_fraction(fraction=1.0, device=0), the process limits the total to just the GPU’s 8GB of video memory. I wanted to simulate, for example, telling PyTorch that I have, for example, 13GB available, also considering shared memory. I want to set a limit to know how far my model can load. For example, sometimes the model shows that it consumed 16.9GB considering shared memory. But I wanted to set it to 16GB to see if it would run because I have the impression that it doesn’t actually need the 16.9GB, but I can’t run this simulation.</p>

ptrblck: <p>PyTorch does not enable or use Windows’ “shared video memory” feature as it’s a driver feature. It thus also does not have any control when and if it will be used.</p>

elismasilva: <p>So you’re telling me that when PyTorch applies set_per_process_memory_fraction it’s only seeing the 8GB of my GPU. I tried using max_split_size_mb to set a limit, but that doesn’t stop the driver from allocating shared memory, even though I set it to something. It seems to only be related to the cache. So, there’s no way to tell PyTorch that you can use 8GB of the GPU and only 4GB of shared memory, right?</p>

ptrblck: <aside class="quote no-group" data-username="elismasilva" data-post="4" data-topic="223336">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elismasilva/48/78308_2.png" class="avatar"> elismasilva:</div>
<blockquote>
<p>when PyTorch applies set_per_process_memory_fraction it’s only seeing the 8GB of my GPU</p>
</blockquote>
</aside>
<p>Yes, this option limits the allowed allocated memory by checking the total memory via <code>cudaMemGetInfo</code> as seen <a href="https://github.com/pytorch/pytorch/blob/0ebfa3d7d29e9278d7c41c9b349bfc53268268b4/c10/cuda/CUDACachingAllocator.cpp#L2019-L2026">here</a>. PyTorch does not have any knowledge about Windows driver features which offload GPU memory to the host behind your back without opting-in.</p>
ModuleNotFoundError: No module named 'torch._six'----please help me

Merhemet_Ehmedov: <p>Torch version: 2.0.0+cu118<br>
Cuda version: 11.8<br>
Device name: NVIDIA GeForce RTX 3050 Ti Laptop GPU</p>

ptrblck: <p>Unfortunately, you didn’t share any information where this issue is raised from so I assume from a 3rd party library. If so, note that <code>torch._six</code> was removed in <a href="https://github.com/pytorch/pytorch/pull/94709">this commit</a> and is not supported anymore since also Python 2.x is dead.</p>

nebulaanish: <p>Had the same issue. The issue in my case was from a later version of timm. I then downgraded it to use <code>timm==0.6.13</code>.<br>
So, there’s no one works for all solution, but you will have to see where the error originated from and fix the version of the package that it occurs from.</p>

Bjorn_Lindqvist: <p>For me, the source of this error was an old timm install. <code>pip install --force-reinstall timm</code>fixed it.</p>
Resume training from exportedProgram

ruijin: <p>Is there a way to resume training from pytorch’s exportedProgram? I have a use case where I want to load a module from an IR and then resume training after exporting my module from elsewhere using torch.export. I assume that that’s what export_for_training() is meant to do, but I can’t call train, or keep track of my parameters when I do this conversion.</p>
<p>Here’s a prototype within a single script that’s failing, is an export_for_training() IR already in “training mode”, so I don’t need to call train?</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor

training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

train_dataloader = DataLoader(training_data, batch_size=64)
test_dataloader = DataLoader(test_data, batch_size=64)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork()
model = torch.export.export_for_training(model, (torch.rand(64,1,28,28),)).module()
parameters = list(model.parameters()) #this is empty for some reason?
model.train() #this can't be called because it isn't implemented, am I supposed to be able to train?
</code></pre>