akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot run torchvision.transforms.functional.perspective on GPU

Alberto_Gomez: <p>I am trying to perform a perspective transform, as folllows:</p>
<pre><code class="lang-auto">        B, C, H, W = reference_image.shape

        # Define source points (original corners of the target image)
        src_points = torch.tensor([
            [0, 0],  # Top-left
            [W - 1, 0],  # Top-right
            [W - 1, H - 1],  # Bottom-right
            [0, H - 1]  # Bottom-left
        ], dtype=torch.float32, device=reference_image.device)

        src_points = src_points.unsqueeze(0).repeat(B, 1, 1)  # (B, 4, 2)

        warped_images = []
        for i in range(B):
            warped = TF.perspective(
                target_image[i],
                src_points[i],
                predicted_points[i],
                interpolation=TF.InterpolationMode.BILINEAR,
                fill=0,
            )
</code></pre>
<p>where all target_image, src_points, predicted_points are definitely tensord in the GPUY (device is cuda). However when I run this, I get the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2021.3.2\plugins\python\helpers-pro\pydevd_asyncio\pydevd_asyncio_utils.py", line 117, in _exec_async_code
    result = func()
             ^^^^^^
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "C:\Users\AlbertoGomez\.conda\envs\torch\Lib\site-packages\torchvision\transforms\functional.py", line 741, in perspective
    coeffs = _get_perspective_coeffs(startpoints, endpoints)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\AlbertoGomez\.conda\envs\torch\Lib\site-packages\torchvision\transforms\functional.py", line 701, in _get_perspective_coeffs
    res = torch.linalg.lstsq(a_matrix, b_matrix, driver="gels").solution.to(torch.float32)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument b in method wrapper_CUDA_out_linalg_lstsq_out)
</code></pre>
<p>It seems that internally, the perspective transform crerates tensors in the cpu and there is no way to move them to cuda. For the record, I have verified that both torch and torchvision are installed with cuda support (cuda 12.8). My torch versions are:</p>
<p>torch.<strong>version</strong><br>
‘2.7.0+cu128’<br>
torchvision.<strong>version</strong><br>
‘0.22.0+cu128’</p>
<p>Anybody knows what I am doing wrong or how can I run this on GPU? Thanks!</p>

ptrblck: <p>Thanks you for reporting this issue! I can reproduce it and have created a GitHub issue <a href="https://github.com/pytorch/vision/issues/9076">here</a>.</p>
Convert exported model from TRAINING dialect to Aten dialect

Himalayjor: <p>I am trying to understand why exported_program from torch.export.export is having TRAINING dialect.</p>
<pre><code class="lang-auto">model.eval()
 with torch.no_grad():
     exported_model = torch.export.export(model, args)
     print(exported_model.dialect)
</code></pre>
<p>Model has no TRAINING related operations.<br>
These are all the operations model has:</p>
<ol>
<li>placeholder</li>
<li>call_function</li>
</ol>
<p>Target for all the call_function is all Aten operations.How can I convert this exported program to Aten dialect ?</p>

zhxchen17: <p>Hi <a class="mention" href="/u/himalayjor">@Himalayjor</a>. Does exported_model.run_decompositions() work for you?</p>
How to modify a torch ExportedProgram with custom operator by torch.export?

biggerfish: <p>I train a model with a custom gpu layer implementation, let’s call it as CustomLookupModule. Now I want to export the model for serving in C++. But I’ve met a few issues.</p>
<ol>
<li>The export method needs to run IR decomposition for unsupported operators, how can I export a module as a whole, like an standalone operator called CustomLookupModule?</li>
<li>during serving/inference, the module has different implementation from training. So I need to modify the ExportedProgram to replace the operator into custom implementation. How can I do it?</li>
<li>How to I make AOT compilation support custom operator for ExportedProgram?</li>
</ol>
<p>Thanks.</p>
DDP with imbalanced loss values

jian_zhang: <p>Hi,<br>
I’m trying to understand if variations in local loss values across GPUs during DDP training pose a problem.</p>
<p>For instance, when using two GPUs with the same batch size, but observing substantial differences in their respective local losses, does the DDP gradient averaging process lead to the GPU with the smaller local loss receiving an inflated gradient?</p>

Bjorn_Lindqvist: <p>I don’t think that should happen since the local models are synced after each backward pass. See the explanation here: <a href="https://www.youtube.com/watch?v=Cvdhwx-OBBo" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=Cvdhwx-OBBo</a> Gradients are averaged so each GPU is weighted equally.</p>

python: <p>The model parameters stay in sync because after each backward pass, all-reduce sums the gradients and divides by the world size. Variations in local loss are normal (due to different data shards), but they don’t break the averaging or update process.</p>
Seeking Guidance on Model Accuracy Evaluation

ZeeJay: <p>Hi,<br>
I seek your guidance in evaluating the accuracy of my models.<br>
<strong>Background</strong></p>
<p>I have pretrained two distinct models:</p>
<ol>
<li><strong>PatchTST</strong> (trained from scratch)</li>
<li><strong>DenseNet-121</strong> (using contrastive learning)</li>
</ol>
<p>Subsequently, I fine-tuned both models using <strong>BitFit</strong> and <strong>quantization techniques</strong>. However, I observed that the <strong>Mean Absolute Error (MAE)</strong> did not increase as expected—these techniques do reduce the accuracy (MAE in my case - MAE should have been increased).<br>
The bitfit code that I applied is as follows:</p>
<blockquote>
<p>for name, param in model.named_parameters():<br>
if ‘bias’ not in name:<br>
param.requires_grad = False  # Freeze non-bias parameters<br>
else:<br>
print(f"Tuning: {name}")  # Confirm bias parameters are trainable</p>
</blockquote>
<p>and quantization code is as follows:</p>
<pre><code>learn.load(weight_path)

# Move model to CPU (important!)
learn.model.to('cpu')

# Set to eval mode before quantizing (required)
learn.model.eval()

original_size = get_model_size(learn.model)
print(f"Original model size: {original_size:.2f} MB")
print(f"Trainable parameters of original model: {count_parameters(learn.model)}")

# Apply quantization
learn.model = torch.quantization.quantize_dynamic(
  learn.model, {torch.nn.Linear}, dtype=torch.qint8
)
</code></pre>
<p>Could you help me identify potential reasons why the MAE did not decrease, is my implementation correct?</p>
Does pytorch work properly with Intel Arc b580 and docker on AMD Ryzen cpu?

vc1cv1: <p>Hi All,</p>
<p>I’m trying to get some validation if it works with this hardware combination when using docker.</p>
<p>System info:<br>
Amd Ryzen 5600G<br>
Intel Arc B580<br>
Ubuntu 24.10 with passthru from Proxmox<br>
Resizeable rombar is set fine<br>
Docker</p>
<p>Comfyui on ubuntu using pytorch in the same Ubuntu vm looks to run fine with some bugs here and there.</p>
<p>For some reason, I’ve never gotten the docker versions of the comfyui/stable diffusion working.</p>
<p>Going back to trying to get things in docker last night … this illegal instruction/core dump looks to come in in any docker image I try with pytorch/comfyui/sd/etc.  Ollama in docker works fine and sees my intel arc b580.</p>
<p>docker run --rm -it --privileged -v /dev/dri:/dev/dri --ipc=host intel/intel-extension-for-pytorch:2.7.10-xpu bash<br>
root@9c8494f7a126:/# python -c “import torch; import intel_extension_for_pytorch as ipex; print(torch.<strong>version</strong>); print(ipex.<strong>version</strong>); [print(f’[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];”<br>
[W517 00:16:02.736196261 OperatorEntry.cpp:154] Warning: Warning only once for all operators,  other operators may also be overridden.<br>
Overriding a previously registered kernel for the same operator and the same dispatch key<br>
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) → Tensor(a!)<br>
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6<br>
dispatch key: XPU<br>
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:37<br>
new kernel: registered at /build/intel-pytorch-extension/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:186 (function operator())<br>
Illegal instruction (core dumped)</p>
Pytorch dataloader is freezing

alifani: <p>I’m encountering a freeze during distributed data parallel training on 8 GPUs with multiple data loader workers. The training process runs for some time and then abruptly stops without any output, error messages, or noticeable resource exhaustion (GPU and CPU utilization appear normal). Any insights or troubleshooting suggestions would be greatly appreciated.</p>
<p>Thanks</p>

ptrblck: <p>Do you see this hang only if multiple workers are used?</p>

alifani: <p>After investigating the code further, I noticed that I’m currently sharding files by rank. When a rank finishes processing its data early, the entire training process seems to stall, likely because the remaining ranks are still waiting to complete their gradient updates.</p>
<p>Is this understanding correct?</p>
<p>I’m using TorchRec’s model parallelism in combination with an <code>IterableDataset</code> to stream data into memory.</p>
<p>Is there a supported way to combine data parallelism with model parallelism in TorchRec?<br>
Or is it expected that I manually manage data sharding and ensure balanced record distribution across ranks?</p>

ptrblck: <aside class="quote no-group" data-username="alifani" data-post="3" data-topic="219939">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/aca169/48.png" class="avatar"> alifani:</div>
<blockquote>
<p>When a rank finishes processing its data early, the entire training process seems to stall, likely because the remaining ranks are still waiting to complete their gradient updates.</p>
</blockquote>
</aside>
<p>Yes, DDP would require the same number of batches on all ranks, which is often achieved with the <code>DistributedSampler</code>. Since you are using an <code>IterableDataset</code> I would assume you need to make sure the same number of batches are provided for all ranks or you could also skip (some) synchronizations via the <code>no_sync()</code> context manager if possible (e.g. this might allow you to run a few gradient accumulation steps on ranks with more batches).</p>
<aside class="quote no-group" data-username="alifani" data-post="3" data-topic="219939">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/aca169/48.png" class="avatar"> alifani:</div>
<blockquote>
<p>Is there a supported way to combine data parallelism with model parallelism in TorchRec?</p>
</blockquote>
</aside>
<p>I’m not familiar enough with <code>TochRec</code> to know the best approach for data and model parallel strategies.</p>
Unable to load my model to GPU

4d42: <p>Good evenning everybody.</p>
<p>I am trying to load my model, a transformer, to my GPU, however my GPU rises the following error:<br>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!</p>
<p>The example I am trying to run came 100% from this site:</p>
<p><a href="https://www.datacamp.com/tutorial/building-a-transformer-with-py-torch" rel="noopener nofollow ugc">https://www.datacamp.com/tutorial/building-a-transformer-with-py-torch</a></p>
<p>I tried to use, before the training cell:<br>
device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’<br>
print(f"Using device: {device}")</p>
<p>transformer.to(device)<br>
src_data.to(device)<br>
tgt_data.to(device)</p>
<p>However the error continues. When a try nvidia-smi on terminal it looks like my model was loaded to GPU since a lot of memory becomes occupied, the same occurs with variables, however the error persists.<br>
I am become crazy with this error, I will be very happy with any help.</p>
<p>Best regards.</p>

vdw: <p>From what I can tell, seems the example code does not move the masks to the device.</p>
<p><code>src_mask</code> and <code>tgt_mask</code> are tensor that have to be moved to the same device as the model and the training data. Maybe you can try</p>
<pre><code class="lang-auto">def generate_mask(self, src, tgt, DEVICE):
    ...
    return src_mask.to(DEVICE), tgt_mask.to(DEVICE)
</code></pre>
<p>or something like that for a quick test.</p>

4d42: <p>Thank you. I will try soon.</p>
Which web server framework

Ahmet: <p>hi,<br>
which web server framework is the most suitable for libtorch development ?,</p>
<p>regards</p>

albanD: <p>Hey!</p>
<p>Why do you need to do C++ development there? For large language models, projects like <a href="https://github.com/vllm-project/vllm" class="inline-onebox">GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs</a> work pretty well from python !</p>
Who's Running LibTorch 2.X on Mobile?

Cyrill: <p>Hi everyone,</p>
<p>I’m reaching out to see if anyone in the community has managed to get LibTorch 2.X up and running on iOS or Android. I’m currently working on a project that could greatly benefit from this, because currenly our researchers have to install an ancient version of pytroch (1.11), for the scripted to run on pytorch mobile.</p>
<p>It’s borderline comedic that the latest PyTorch Android API (2.1.0) popped up on <a href="https://mvnrepository.com/artifact/org.pytorch/pytorch_android_lite/2.1.0" rel="noopener nofollow ugc">MVNRepository</a> just a month ago, yet for iOS, we’re seemingly stuck in the dark ages with Cocoapods still at version 1.14.</p>
<p>Can anyone share insights on compiling Libtorch 2.0 for iOS, or getting it uploaded to Cocapods?</p>
<p>If you’ve successfully implemented LibTorch 2.X on either iOS or Android, I would love to hear about your experience, any challenges you faced, and how you overcame them.</p>

Cyrill: <p>On a second note, I might just consider using <a href="https://pytorch.org/executorch/stable/index.html" rel="noopener nofollow ugc">ExecuTorch</a> for sustainable, long term working solution that supports Pytroch 2.0</p>

Cyrill: <p><a href="https://github.com/CocoaPods/Specs/blob/master/Specs/c/c/3/LibTorch-Lite/2.1.0/LibTorch-Lite.podspec.json" rel="noopener nofollow ugc">LibTorch-Lite-2.1.0</a> seems to exist. It was just not appearing in search on <a href="https://cocoapods.org/" rel="noopener nofollow ugc">https://cocoapods.org/</a></p>

Cagatay_Bilgin: <p>Yes, ExecuTorch now replaced the older stack! <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a>. If you have any questions or want to stay in the loop, join our Discord server at <a href="https://discord.gg/Wyn9PWqP" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Edge</a>.</p>
PyTorch Model on Mobile

Manuel_Rucci: <p>I retrained a model based on deespeech2 to transcribe audio to test by following this <a href="https://www.assemblyai.com/blog/end-to-end-speech-recognition-pytorch" rel="noopener nofollow ugc">blog post</a>.  Then I followed the  <a href="https://pytorch.org/mobile/home/" rel="noopener nofollow ugc">PyTorch mobile documentation</a> because I would like to use the model on a mobile android/ios device.<br>
To be able to run the model I need to load and transform the wav audio 16000 Hz into a Mel spectrogram. I am doing this by using the transforms.MelSpectrogram function.  However right now I have this function inside the Dataset module.<br>
Is it correct to add this transformation inside the Model module before forward?<br>
By adding this transformation inside the model will create problems when I call torch.jit.script and torch.jit.save functions?</p>

ptrblck: <p>I think you can add the transformation inside the model as long as scripting the model doesn’t complain about unsupported methods. A quick check of <code>transforms.MelSpectrogram</code> shows that PyTorch methods seem to be used, so I guess it would work.<br>
If I’m not mistaken, <a class="mention" href="/u/tom">@tom</a> also used a similar workflow by adding transformations into the model to export it to his mobile in a <a href="https://lernapparat.de/pytorch-android/">blog post</a>.</p>

tom: <p>One thing that is a limitation here is the coverage of stft/fft in the backend.<br>
<a href="https://github.com/pytorch/pytorch/blob/ac04cc775bc1cc7e85e7bbc8a54744720fe7c068/aten/src/ATen/native/SpectralOps.cpp#L748" rel="noopener nofollow ugc">stft maps to the internal <code>_fft_r2c</code></a> and that in turn is implemented for <a href="https://github.com/pytorch/pytorch/blob/ac04cc775bc1cc7e85e7bbc8a54744720fe7c068/aten/src/ATen/native/native_functions.yaml#L2107" rel="noopener nofollow ugc">only through MKL on the CPU</a>.</p>
<p>I have an upcoming tutorial (in a week or two) for keyword recognition on ARM (but Raspberry rather than mobile) where I use the M5 model from the tutorial to avoid the need for spectrogram. It also has a few wrinkles.<br>
For Linux on ARM you can always get numpy’s fft (like librosa does) but that might not be as straightforward on mobile.</p>
<p>Best regards</p>
<p>Thomas</p>

Cagatay_Bilgin: <p>For people landing on this page from a web search, we deprecated the older stack and have been using ExecuTorch for a few years now <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a>. If you have any questions or want to stay in the loop, join our Discord server at <a href="https://discord.gg/Wyn9PWqP" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Edge</a>.</p>
Am I unfreezing these blocks correctly

Abas1: <p>So what im trying to do is essentially unfreeze the last portion of resnet50, densent121 and inceptionv3. I’m trying to unfreeze the last 30% of the network and I do this by unfreezing specific blocks or modules. The FC layer is already set to <code>requires_grad = True</code> In another function hence this function is only called to unfreeze specific blocks. However i’m not sure if im doing it correctly</p>
<pre><code class="lang-auto">def unfreeze_blocks(model, model_name):
  """
  ARGS

  model- represents our current model instance
  model_name  - represents the name of the model we are using

  we are unfreezing only the last 30% of each architecture approximately 

  for resnet last 30% is layer 4 which is 3 residual blocks + the last block of layer 3 

  for densenet 30% of the total dense layers is 17 which is inside the last dense block(dense block 4). each dense layer has 2 conv layers


  for inception v3 we will unfreeze the last 3 modules 
  """

  if model_name == 'resnet50':
    for param in model.layer4.parameters():
            param.requires_grad = True
        
    # Unfreeze the last block of layer3
    if hasattr(model, 'layer3') and len(model.layer3) &gt; 5:
        for param in model.layer3[5].parameters():
            param.requires_grad = True
      
      


  elif model_name == 'DenseNet121':
    for name, params in model.named_parameters():
      if 'denseblock4' in name:
        params.requires_grad = True

  elif model_name == 'InceptionV3':   
    for name, param in model.named_parameters():
      if any(module in name for module in ["Mixed_7a", "Mixed_7b", "Mixed_7c"]):
        param.requires_grad = True
</code></pre>

ptrblck: <p>The code looks alright, but I would double check if the number of trainable parameters is correct.<br>
E.g. for <code>resnet50</code> I see <code>layer4 + layer3[5] + fc</code> contain <code>41</code> of <code>161</code> parameters, so you are currently unfreezing ~25% instead of the desired 30%.</p>
Pytorch Slack Participation

Asperjasp: <p>Hello, I  am participating in a AI Research Seminar on Universidad Nacional de Colombia Bogotá HQ, taking approach on hands on Machine Learning and enhancing the knowledge of the Quantum team of the Medellin HQ, we would like to participate and share knowledge.</p>
<p>These are mainly led by</p>
<ul>
<li>Christian Eliecer Bello as Quantum Leader in Medellín</li>
<li>Robert Gomez as Machine Learning leader in Bogotá</li>
</ul>
<p>If it be possible to add the domain, or if there’s another way to do it, we would be grateful to hear it.</p>
<p>Thank you beforehand Pytorch Team and it’s community.</p>
<ul>
<li><span class="mention">@unal.edu.co</span></li>
</ul>
How to handle training of few layers with DDP

Guillaume_Jeanneret: <p>Hi all,</p>
<p>I have a conceptual question regarding how to best use DistributedDataParallel (DDP) in a setup involving two models.</p>
<p>Suppose I have two <code>nn.Module</code> instances: <code>model1</code> and <code>model2</code>. I want to train all parameters in <code>model2</code>, but only a subset of parameters in <code>model1</code>. The remaining parameters in <code>model1</code> should remain frozen. Please note that both models are used in sequentially to compute the loss function.</p>
<p>Naturally, I would wrap <code>model2</code> with DDP as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">model2 = DDP(model2, ...)
</code></pre>
<p>However, I am unsure how to ensure proper gradient synchronization across GPUs for the trainable parameters in <code>model1</code>. Clearly, if I don’t include <code>model1</code> in DDP, its parameters won’t be synchronized.</p>
<p>I see a couple of options:</p>
<ol>
<li>
<p>Wrap a <code>nn.ModuleList</code> containing only the trainable submodules from both model1 and model2, and pass that into DDP.</p>
</li>
<li>
<p>Wrap both model1 and model2 with DDP independently, making sure to set <code>requires_grad = False</code> for the frozen parameters in model1, as suggested in <a href="https://discuss.pytorch.org/t/how-does-distributeddataparallel-handle-parameters-whose-requires-grad-flag-is-false/90736">this post</a>.</p>
</li>
</ol>
<p>Are either of these approaches recommended or is there a better practice for this use case? I’d like to make sure I’m not missing something regarding how DDP handles parameter synchronization and gradient communication when mixing frozen and trainable parameters across multiple modules.</p>
<p>Thanks in advance!</p>

ptrblck: <p>I would stick to the linked post explaining that frozen parameters will be skipped in <code>DDP</code>’s communication. You would thus have to set the <code>.requires_grad</code> attributes of all frozen parameters to <code>False</code> first before creating the <code>DDP</code> instance.</p>

Guillaume_Jeanneret: <p>Thank you! I’ll mark your answer as the solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Torch.cuda.is_available() returns false for p106-100 GPU

Andrey_Brykin: <p>I installed torch with cuda 11.8 as described here <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>Is was installed successfully, but I am getting <code>torch.cude.is_available()</code> of <code>False</code> with warning:<br>
<code>UserWarning: CUDA initialization: The NVIDIA driver on your system is  too old (found version 10010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internal\CUDAFunctions.cpp:108.)</code></p>
<p>My environment:</p>
<p>`PyTorch version: 2.5.0+cu118<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.8<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Майкрософт Windows 10 Pro<br>
GCC version: (GCC) 9.3.0<br>
Clang version: Could not collect<br>
CMake version: version 3.12.2<br>
Libc version: N/A</p>
<p>Python version: 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.19045-SP0<br>
Is CUDA available: False<br>
CUDA runtime version: Could not collect<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: GPU 0: P106-100<br>
Nvidia driver version: 432.00<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture=9<br>
CurrentClockSpeed=3001<br>
DeviceID=CPU0<br>
Family=205<br>
L2CacheSize=1024<br>
L2CacheSpeed=<br>
Manufacturer=GenuineIntel<br>
MaxClockSpeed=3001<br>
Name=Intel(R) Core™ i5-4590S CPU @ 3.00GHz<br>
ProcessorType=3<br>
Revision=15363</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.26.3<br>
[pip3] torch==2.5.0+cu118<br>
[pip3] torchaudio==2.5.0+cu118<br>
[pip3] torchvision==0.20.0+cu118<br>
[conda] Could not collect`</p>
<p>I tried <code>pip install torch==1.8.1+cu101 torchvision==0.9.1+cu101 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>But getting error <code>ERROR: Could not find a version that satisfies the requirement torch==1.8.1+cu101 (from versions: 2.0.0, 2.0.0+cpu, 2.0.0+cu117, 2.0.0+cu118, 2.0.1, 2.0.1+cpu, 2.0.1+cu117, 2.0.1+cu118, 2.1.0, 2.1.0+cpu, 2.1.0+cu118, 2.1.0+cu121, 2.1.1, 2.1.1+cpu, 2.1.1+cu118, 2.1.1+cu121, 2.1.2, 2.1.2+cpu, 2.1.2+cu118, 2.1.2+cu121, 2.2.0, 2.2.0+cpu, 2.2.0+cu118, 2.2.0+cu121, 2.2.1, 2.2.1+cpu, 2.2.1+cu118, 2.2.1+cu121, 2.2.2, 2.2.2+cpu, 2.2.2+cu118, 2.2.2+cu121, 2.3.0, 2.3.0+cpu,  2.3.0+cu118, 2.3.0+cu121, 2.3.1, 2.3.1+cpu, 2.3.1+cu118, 2.3.1+cu121, 2.4.0, 2.4.1, 2.5.0) ERROR: No matching distribution found for torch==1.8.1+cu101</code>.</p>

Andrey_Brykin: <p>I upgraded nvidia driver and it works now</p>

ptrblck: <p>Thanks for the update and good to know it’s working now since this is what the error message also suggested.</p>

glebov00ivan: <p>I have the same thing, but the drivers and all the latest versions!</p>
Torch not compiled with CUDA enabled for ComfyUI

Jean-Luc_PIETRI: <p>Hello,<br>
Which Pytorch version is compatible with NVIDIA’s CUDA 12.7 to run comfyui ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da6fdda38a505e1bf0b8523550b9e3c7b4093cca.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/da6fdda38a505e1bf0b8523550b9e3c7b4093cca" title="Capture d’écran 2025-05-13 202148"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da6fdda38a505e1bf0b8523550b9e3c7b4093cca_2_690x348.jpeg" alt="Capture d’écran 2025-05-13 202148" data-base62-sha1="vanL6Ystvna4m49oAU5cJVBy0qu" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da6fdda38a505e1bf0b8523550b9e3c7b4093cca_2_690x348.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da6fdda38a505e1bf0b8523550b9e3c7b4093cca_2_1035x522.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da6fdda38a505e1bf0b8523550b9e3c7b4093cca_2_1380x696.jpeg 2x" data-dominant-color="2C2A29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2025-05-13 202148</span><span class="informations">1816×916 337 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thank you for your help</p>

ptrblck: <aside class="quote no-group" data-username="Jean-Luc_PIETRI" data-post="1" data-topic="219999">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jean-luc_pietri/48/76994_2.png" class="avatar"> Jean-Luc_PIETRI:</div>
<blockquote>
<p>Which Pytorch version is compatible with NVIDIA’s CUDA 12.7 to run comfyui ?</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used, so install a PyTorch binary with CUDA support from our <a href="https://pytorch.org/get-started/locally/">install matrix</a> by selecting the desired CUDA version.</p>
Torch.export stability and timeline

tolleybot: <p>I’ve been experimenting with the new torch.export API (version 2.7) and noticed the documentation currently labels it as a “prototype under active development” with potential for future breaking changes. Could you share any information on the expected timeline or roadmap for when torch.export will reach a stable, production-ready release? In particular, I’m interested in:<br>
•	Planned milestones or beta releases<br>
•	Any target PyTorch version for stability guarantees<br>
•	Guidance on how best to track progress (e.g. specific GitHub issues or project boards)</p>
<p>Thank you for any insight you can provide!</p>
What are the most effective and reliable ways to load minibatches efficiently from HDD for deep learning training?

AlanTuring: <p>I’m training a deep learning model using PyTorch on ~100k individual files (each 7–12 MB, .mat) stored on a standard HDD (I cannot have an SSD righit now so its not an option). Each file is one training example.</p>
<p>Current setup:</p>
<pre><code class="lang-auto">Batch size: 16

num_workers=16

prefetch_factor=2

pin_memory=True

persistent_workers=True

Dataset reads directly from disk, no decoding or preprocessing

Machine: 256 GB RAM, fast GPU (no GPU bottleneck)
</code></pre>
<p>Problem:</p>
<p>Despite applying all the standard PyTorch performance tricks, I/O remains the bottleneck. Each epoch takes over 1 hour, and GPU usage is poor due to slow data loading. Disabling shuffling helps slightly at first, but performance still degrades. Already tried — did NOT help:</p>
<pre><code class="lang-auto">pin_memory, prefetch_factor, persistent_workers, high num_workers

LMDB -&gt; I generated 1 enourmous file (but was even slower in my case, despite claims i read that this would improve due to the supposedly-better file system of lmdb)

Tried removing shuffling (no perf gain whatsoever)
</code></pre>
<p>Question:</p>
<p>What are effective, practical solutions to speed up minibatch loading from HDD in this scenario?</p>
<p>I’m open to:</p>
<ul>
<li>Packing files into a single large file (LMDB, HDF5, tar, memory-mapped arrays, etc.)</li>
<li>Caching strategies or chunk loading</li>
</ul>
<p>Anything that actually solves the issue in practice</p>
<p>Please suggest solutions that you’ve personally used successfully with large datasets stored on HDD.</p>
<p>What do people do in practice ? I dont understand how one can do better with an HDD … but maybe it is possible ?</p>
Unexpected DataLoader hanging when torch.set_num_threads in worker init

Sohrab_Andaz: <p>Torch’s <code>_worker_loop</code> defaults the num_threads to 1, but in my case I do a lot of copying data off disk using memory mapped tensors, so it’s useful to have num_threads&gt;1. If i try to force the number of threads to be greater than 1</p>
<pre><code class="lang-auto">def worker_init(worker_id):
    torch.set_num_threads(96 // 4) 
</code></pre>
<p>My DataLoader does not return a batch. Is there any way around this? Is there any way to configure the number of threads used by worker_fn of a torch Dataloader. I’ve included code to repro below</p>
<pre><code class="lang-auto">import torch
import time
from tqdm import tqdm
from tensordict import TensorDict
from torch.utils.data import Dataset, DataLoader, BatchSampler, SequentialSampler

write_new = True
if write_new:
    B = 512_000
    D = 128 
    flat_fields = {
        f"feat_{i}": torch.randn(B, D) for i in range(20)
    }
    flat_fields.update({
        f"cat_{i}": torch.randint(0, 100, (B,)) for i in range(5)
    })
    flat_fields.update({
        f"mask_{i}": torch.randint(0, 2, (B,), dtype=torch.bool) for i in range(5)
    })

    nested1 = TensorDict({
        "subfeat_1": torch.randn(B, D),
        "subfeat_2": torch.randn(B, D)
    }, batch_size=[B])
    
    nested2 = TensorDict({
        "subcat_1": torch.randint(0, 10, (B,)),
        "submask_1": torch.randint(0, 2, (B,), dtype=torch.bool)
    }, batch_size=[B])
   
    td = TensorDict(flat_fields, batch_size=[B])
    td["nested1"] = nested1
    td["nested2"] = nested2
    
    td.memmap_("/tmp/mmap_tensordict_large", num_threads=8)

td_mmap = TensorDict.load_memmap("/tmp/mmap_tensordict_large")
class MemmapTensorDictDataset(Dataset):
    def __init__(self, td):
        # torch.set_num_threads(1)
        self.td = td

    def __getitem__(self, idx):
        return self.td[idx]

    def __len__(self):
        return self.td.batch_size[0]

batch_size = 8_000
dataset = MemmapTensorDictDataset(td_mmap)
inner_sampler = SequentialSampler(range(len(dataset)))
sampler = BatchSampler(inner_sampler, batch_size=batch_size, drop_last=False)

def benchmark(loader, name, epochs=1):
    t0 = time.time()
    for _ in tqdm(range(epochs)):
        for _ in loader:
            break
    elapsed = time.time() - t0
    print(f"{name:15s}: {elapsed:.2f}s")

batch_size = 4096

loader0 = DataLoader(
    dataset,
    batch_size=batch_size,
    num_workers=0,
    pin_memory=True,
    sampler=sampler,
    collate_fn=lambda x: x[0],
)


def worker_init(worker_id):
    torch.set_num_threads(96//4)  

loader4 = DataLoader(
    dataset,
    batch_size=batch_size,
    num_workers=4,
    pin_memory=False,
    sampler=sampler,
    collate_fn=lambda x: x[0],
    prefetch_factor=2,
    persistent_workers=True,
    worker_init_fn=worker_init,
)

benchmark(loader0, "num_workers=0", epochs=10)
benchmark(loader4, "num_workers=4", epochs=10) # THIS WILL HANG when `worker_init` us used.. and if you don't use it's much slower, in part because the num_threads=1
</code></pre>