akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA Error: an illegal memory access was encountered with ProArt 4080 Super

locnhinho: <p>I had google and read various error of illegal memory access was encountered. But before I got this error, I was able to run just fine for weeks prior. All I am doing is a simple binary classification using Pytoruch and Cuda 12.6 (was 12.4 before) and encounter this error. I had contact ASUS support and they suspect this is an Pytorch issue. As such, I here looking for help.</p>
<p>Here is the output of the error with this fragment of code:</p>
<pre><code class="lang-auto">
num_epochs = 20

# We use the pre-defined number of epochs to determine how many iterations to train the network on
for epoch in range(num_epochs):
    print("Computing Epoch " + str(epoch) + "...")
    # Load in the data in batches using the train_dataloader object
    for i, (images, labels) in enumerate(train_dataloader):
        # Move tensors to the configured device
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))
</code></pre>
<p>Error Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebceee8b41e75cb564eac4b2e800421832adf53a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ebceee8b41e75cb564eac4b2e800421832adf53a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebceee8b41e75cb564eac4b2e800421832adf53a.png" alt="image" data-base62-sha1="xE3yp6HX6HDOcAVLYN6RULRRsam" width="690" height="206" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">974×292 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help would be appreciated. I can also provide the test.py if required as it is a basic binary classification with random generated image. And also the output from “compute-sanitizer python .\test.py”.</p>

locnhinho: <p>After further interaction with ASUS, it was determined that the card is defected. Had to RMA the card.</p>
Export PyTorch documentation into PDF form

Guanqun_Yang: <h2>Problem</h2>
<p>This need here may seem to be a little weird but I need the PDF document because network instability and frequent interruption. At the same time, the only PDF version of the doc I could find is <a href="https://buildmedia.readthedocs.org/media/pdf/pytorch-zh/latest/pytorch-zh.pdf" rel="nofollow noopener">0.11.5</a>, which is outdated.</p>
<p>There is a <code>doc</code> folder in <a href="https://github.com/pytorch/pytorch/tree/master/docs" rel="nofollow noopener">source code directory</a> on GitHub and there is a <code>Makefile</code> avaiable. Therefore, I downloaded the entire source repo and entered <code>doc</code> to generate the PDF doc. However, when I  <code>make latexpdf</code> in that folder, a series of errors occurred. After working around these error messages for hours, there is no luck. I also tried to first export the documentation to <code>.html</code> and then use <code>pandoc</code> to make the conversion. However, it did not work either.</p>
<p>I am wondering if anyone has ever tried to do this and succeeded? Thank you in advance!</p>

Guanqun_Yang: <p>I finally solved this problem and I will share the exported documents with everyone in this forum. Here is the link (<a href="https://drive.google.com/file/d/12ct7-bHK-ZD72SZ-jD6a3cAgRi-YQtXI/view?usp=sharing" rel="nofollow noopener">Google Drive</a>).</p>

Chen-Cai-OSU: <p>Hi Guanqun,</p>
<p>Thanks for the awesome pdf. I was wondering would you also like to share the pdf for the latest pytorch documents? Many thanks!</p>

Dimpy_Tanwar: <p>How did you do it ? covert and merge all?</p>

Geremia: <p><a class="mention" href="/u/dimpy_tanwar">@Dimpy_Tanwar</a><br>
<a href="https://pytorch.org/docs/" rel="noopener nofollow ugc">https://pytorch.org/docs/</a> is built with Sphinx. In the <code>pytorch</code> source <a href="https://github.com/pytorch/pytorch/tree/main/docs" rel="noopener nofollow ugc"><code>docs</code></a> dir, run:</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ pip install -r requirements.txt
$ make latexpdf
</code></pre>
<p>You can also make an EPUB with <code>make epub</code>.</p>

essington: <p>Thanks a lot for the pdf Guanqun !</p>
PyTorch with CUDA 12.6 and Ubuntu 24.04

goodquant: <p>Hi there.  I have the following specs:</p>
<ul>
<li>Ubuntu 24.04</li>
<li>Pytorch 2.4.1+cu124 (which uses CUDA 12.4)</li>
<li>CUDA 12.6</li>
<li>NVIDIA RTX 500 Ada GPU</li>
<li>NVIDIA-SMI 560.35.03.  Driver Version: 560.35.03</li>
</ul>
<p>When I run torch.cuda.is_available(), I get False.  I have a feeling this is to do with having CUDA 12.6 rather than 12.4 on my machine.  However, I cannot seem to get CUDA 12.4 on Ubuntu 24.04.  Any suggestions?</p>

ptrblck: <aside class="quote no-group" data-username="goodquant" data-post="1" data-topic="212002">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/3be4f8/48.png" class="avatar"> goodquant:</div>
<blockquote>
<p>I have a feeling this is to do with having CUDA 12.6 rather than 12.4 on my machine.</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used as the PyTorch binaries ship with their own CUDA dependencies. You would need to make sure the NVIDIA driver is properly installed and can communicate with the GPU.</p>

goodquant: <p>Thank you for your response.  I’m pretty sure the driver is installed correctly.  I’m still a bit lost with how to solve this problem.  Would you recommend installing PyTorch from source?  Would PyTorch work with CUDA 12.6?</p>

Yally_Simionovici: <p>Hi, was it resolved? I encountered a similar problem with trying to use pyannote.audio , cuda 12.4 ubuntu 24.04. I get an error of cudnn version mismatch although according to torch.backends.cudnn.version() It matches the required cudnn version required by pytorch</p>
BatchMemoryManager with Opacus in Lightning

IoutofAI: <p>I’ve implemented Opacus with my Lightning training script for an NLP application. I’m having issues with GPU out-of-memory errors that I’m not able to resolve. So, I’m looking to implement BatchMemoryManager to increase my batch size while preserving GPU memory. My question is:</p>
<p><strong>How do I implement BatchMemoryManager with Lightning? Is that supported currently?</strong></p>
<p>My current implementation follows the same skeleton as the Opacus Lightning tutorial here: <a href="https://github.com/pytorch/opacus/blob/main/examples/mnist_lightning.py" class="inline-onebox" rel="noopener nofollow ugc">opacus/examples/mnist_lightning.py at main · pytorch/opacus · GitHub</a></p>
<p>Thanks in advance for any help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

IoutofAI: <p>I’m thinking it could be implemented with a custom Lightning Loop to activate the BatchMemoryManager, as explained here: <a href="https://devblog.pytorchlightning.ai/train-anything-with-lightning-custom-loops-4be32314c961" class="inline-onebox" rel="noopener nofollow ugc">Train anything with Lightning custom Loops | by PyTorch Lightning team | PyTorch Lightning Developer Blog</a></p>
<p>Has anybody done this before?</p>

Ruchen-Liu: <p>Hi there, may I ask how did you eventually do this?</p>

Ruchen-Liu: <p>I think I have solved this with defining train_dataloader() inside the Lightning module and therefore I can access self.optimizers() with wrap_dataloader() from opacus.</p>

kdurmaz: <p>Did this solve the issue?</p>
Reshaping tensors while using model parallelism

Ryan_Zotti: <p>I’d like to adapt the DLRM model mentioned here [1] for a recommendations ranking use case that requires model parallelism (I have several TB of training data and large embedding tables). I was hoping to use the ListNet loss from here [2]. However, the original DLRM model uses pointwise loss, and the loss function expects a 1D tensor of shape [batch]. The ListNet loss function that I linked to expects a tensor of shape [batch,items], since each user has a variable number of retrieved items that the model must learn to optimally sort.</p>
<p>I can think of two approaches to adapt the code.</p>
<ol>
<li>
<p>Update the DLRM code so that all forward pass operations expect tensors of shape [batch,items,features] rather than [batch,features]. This feels risky because I would need to modify distributed modules like embedding bags and potentially key-jagged-tensors.</p>
</li>
<li>
<p>Include a user_id index mapping tensor from my batch to convert the 1D predictions and labels to the 2D format expected by my loss function. This feels less risky because I don’t need to update a bunch of built-in torchrec modules. I would only need to add potentially 10 lines of code between the output of the final forward pass and the loss calculation. However, I plan to use the model parallelism capability of torchrec, and my experience with distributed systems tells me that collecting distributed results often changes the final order of collected rows. I’m worried that the forward pass of the model might reorder records so that they no longer line up with my index mapping tensor that comes from the data loader and doesn’t get forward-passed through the model. I don’t plan to explicitly re-order rows in my forward passes, and I was hoping to use the DLRM design mostly as-is, so I likely have nothing to worry about. Nonetheless, I wanted to confirm, because if my concern is valid the model would learn complete noise. Lastly, would this approach be inefficient during training because I’d likely need to pull the predictions back to the CPU and then back again to the GPU for the reshaping operation?</p>
</li>
</ol>
<p>[1] <a href="https://github.com/pytorch/torchrec/blob/main/torchrec/models/dlrm.py" class="inline-onebox" rel="noopener nofollow ugc">torchrec/torchrec/models/dlrm.py at main · pytorch/torchrec · GitHub</a><br>
[2] <a href="https://github.com/allegro/allRank/blob/master/allrank/models/losses/listNet.py" rel="noopener nofollow ugc">https://github.com/allegro/allRank/blob/master/allrank/models/losses/listNet.py</a></p>
Custom Vectorized environment for torchrl

manavkoolz: <p>Hi, I am working on an RL project for which I made a custom (stateful) environment which handles everything on gpu tensors. I want to make this handle batches as well in a vectorized manner, but an struggling about the exact requirements for this and cannot find any good docs.</p>
<p>From what I understand, I will essentially have my tensors just have a leading batch dimension as well. My question is - what happens when one of the environments ends/terminates (maybe either bounds exceeded by agents which is failure or if the goal is achieved which is success) and the others are still going? What should step return?</p>
<p>Any example or docs for this would be greatly appreciated</p>

vmoens: <p>There’s a “_reset” entry that will be passed that will reflect which elements of the batch to reset.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/rl/main/reference/envs.html#partial-resets">
  <header class="source">

      <a href="https://pytorch.org/rl/main/reference/envs.html#partial-resets" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://pytorch.org/rl/main/reference/envs.html#partial-resets" target="_blank" rel="noopener nofollow ugc">torchrl.envs package — torchrl main documentation</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Hope that helps!</p>

manavkoolz: <p>Hi Vincent, thanks for your prompt response. I’m using the syncdatacollector class for using the environments. I have one more question:</p>
<p>Will the “_reset” key be properly populated depending on the “done” key for all batches and self.reset() called when doing rollouts with the collector or do I need to specify this somewhere?</p>
<p>If I have batch size 3, “_reset” key looks something like [[True], [False], [True]] if I want to reset the 1st and 3rd entry in the batch. I would want to get these from the done, probably. Do I populate this key in the tensordict returned by my _step() function?</p>

vmoens: <p>If the done states are set properly by your _step function, the _reset will be written where appropriate<br>
Then it’s up to your batched env to deal with these!</p>
Complicated Custom Loss Function

kryptic27: <p>Hi!</p>
<p>EDIT: I’m simplifying my question considerably from its previous version.</p>
<p>I want to implement a decoder-only transformer that will be trained in an unsupervised manner only. It will be used to generate, as one does in inference mode, a set of sentences. These sentences however should be generated probabilistically, and not in a greedy way (that is, the next word can be picked according to the probability distribution of the next token and not the maximum probability only). From Umar Jamil’s YouTube video on coding transformers, I believe I can create a function that will generate such sentences.</p>
<p>Now, my cost function is actually computed from computing statistics of these random sentences. I do not understand how torch tensors with require_grad=True work. How do I make sure that once I compute correlations from tensors representing the random sentences generated, that I can back propagate.</p>
<p>Here’s the code that generates the random sentences from the model.</p>
<pre><code>def generate_random_sentences(model, batch_size, vocab_size, max_len, device):

# Initialize the decoder input with random first word
decoder_input = torch.randint(low=0,high=vocab_size,size=(batch_size,1)).to(device) #(batch_size, 1)
while True:
    if decoder_input.size(1) == max_len:
        break

    # build mask for target
    decoder_mask = causal_mask(decoder_input.size(1)).unsqueeze(0).to(device) # (1, curr_seq_len, curr_seq_len) -&gt; (1, 1, curr_seq_len, curr_seq_len)

    # calculate output
    out = model.decode(decoder_input, decoder_mask) # (batch_size, seq_len = decoder_input.size(1), vocab_size)

    # get next token
    prob = model.project(out[:, -1]) # (batch_size, vocab_size)
    prob_cumul = prob.cumsum(dim = 1) # (batch_size, vocab_size)
    next_word = torch.searchsorted(prob_cumul,torch.rand(batch_size,1)) #(batch_size, 1)
    decoder_input = torch.cat(
        [decoder_input, next_word.to(device)], dim=1
    )

return decoder_input.squeeze(0) #(1, batch_size, max_len)
</code></pre>
DDP and multi-GPU related issue

shenwh94: <p>I am working on graph similarity prediction using the SimGNN model. Since SimGNN requires input as pairs of graphs, I cannot use PyTorch’s DataLoader to batch multiple graphs together efficiently. As a result, my GPU utilization is only around 10% per GPU, and I am using 4 GPUs for multi-GPU training.</p>
<p>To improve GPU utilization, I attempted to run multiple processes in parallel on each GPU, so that each GPU could train multiple pairs of graphs at the same time. However, I encountered the following issues:</p>
<ul>
<li>I tried using PyTorch’s multiprocessing, but due to the DDP (Distributed Data Parallel) environment, each process cannot properly communicate during backpropagation.</li>
<li>Running multiple processes on each GPU seems to conflict with PyTorch’s DDP, preventing inter-process communication across GPUs.</li>
</ul>
<p><strong>My Goal:</strong><br>
I want to launch multiple processes per GPU, so that each GPU can efficiently process multiple graph pairs in parallel and achieve higher utilization.</p>
<p>Are there any efficient ways to train multiple pairs of graphs in parallel on each GPU? Alternatively, are there other ways to improve GPU utilization in this scenario?</p>
Can't build 2.3.1 from source

Krytern: <p>Hello, I am trying to build specifically v2.3.1 from source. I am able to build the latest version of Pytorch fine but when I try this version specifically it errors, I tried googling the error but nothing I’ve tried seems to work, here is the error;</p>
<pre><code class="lang-auto">-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /home/ubuntu/miniconda3/envs/test/lib/libgomp.so
Building PyTorch for GPU arch: gfx1030
-- Found HIP: /opt/rocm-6.3.4 (found suitable version "6.3.42134-a9a80e791", minimum required is "1.0")
HIP VERSION: 6.3.42134-a9a80e791
-- Caffe2: Header version is: 6.3.4

***** ROCm version from rocm_version.h ****

ROCM_VERSION_DEV: 6.3.4
ROCM_VERSION_DEV_MAJOR: 6
ROCM_VERSION_DEV_MINOR: 3
ROCM_VERSION_DEV_PATCH: 4
ROCM_VERSION_DEV_INT:   60304
HIP_VERSION_MAJOR: 6
HIP_VERSION_MINOR: 3
TORCH_HIP_VERSION: 603

***** Library versions from dpkg *****

rocm-developer-tools VERSION: 6.3.4.60304-76~24.04
rocm-device-libs VERSION: 1.0.0.60304-76~24.04
hsa-rocr-dev VERSION: 1.14.0.60304-76~24.04

***** Library versions from cmake find_package *****

hip VERSION: 6.3.42134
hsa-runtime64 VERSION: 1.14.60304
amd_comgr VERSION: 2.8.0
rocrand VERSION: 3.2.0
hiprand VERSION: 2.11.1
rocblas VERSION: 4.3.0
CMake Error at /home/ubuntu/miniconda3/envs/test/share/cmake-3.31/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
  By not providing "Findhipblas-common.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "hipblas-common", but CMake did not find one.

  Could not find a package configuration file provided by "hipblas-common"
  with any of the following names:

    hipblas-commonConfig.cmake
    hipblas-common-config.cmake

  Add the installation prefix of "hipblas-common" to CMAKE_PREFIX_PATH or set
  "hipblas-common_DIR" to a directory containing one of the above files.  If
  "hipblas-common" provides a separate development package or SDK, be sure it
  has been installed.
Call Stack (most recent call first):
  /opt/rocm/lib/cmake/hipblas/hipblas-config.cmake:92 (find_dependency)
  cmake/public/LoadHIP.cmake:36 (find_package)
  cmake/public/LoadHIP.cmake:157 (find_package_and_print_version)
  cmake/Dependencies.cmake:1258 (include)
  CMakeLists.txt:754 (include)


-- Configuring incomplete, errors occurred!
</code></pre>

singh20anurag: <p>Have you verified hipblas-common <a href="https://rocm.docs.amd.com/projects/rocRAND/en/docs-5.5.1/installing.html" rel="noopener nofollow ugc">is installed on your system?</a></p>

Krytern: <p>Yes. I wouldn’t be able to build the current version of Pytorch otherwise though I just checked with the link you’ve sent and it is definitely installed.</p>

Krytern: <p>I can’t be the only person having this issue on a fresh OS install. Really no one knows how to fix this?</p>
PyTorch 2.6.0 installation on 2019 MacBook Intel

niMtAndoX: <p>I need to install PyTorch 2.6.0, but i can only install versions up to 2.2.2.<br>
Is there a workaround to get PyTorch 2.6 on an Intel based Mac?</p>

ptrblck: <aside class="quote" data-post="2" data-topic="204546">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-no-macosx-x86-64-build-after-torch-2-2-2-cp39-none-macosx-10-9-x86-64-whl/204546/2">Why no macosx x86_64 build after torch-2.2.2-cp39-none-macosx_10_9_x86_64.whl</a> <a class="badge-category__wrapper " href="/c/mac-os-x/32"><span data-category-id="32" style="--category-badge-color: #283890; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category "><span class="badge-category__name">Mac OS X</span></span></a>
  </div>
  <blockquote>
    PyTorch macOS x86 builds were deprecated as explained <a href="https://dev-discuss.pytorch.org/t/pytorch-macos-x86-builds-deprecation-starting-january-2024/1690">here</a> and <a href="https://github.com/pytorch/pytorch/issues/114602">here</a>. torch==2.2.x was the last release for it.
  </blockquote>
</aside>
How works dist.ProcessGroupGloo?

HariSeldon11988: <p>Hello together,</p>
<p>I have the following setup: I have three different computenodes each with two processes (rank 0 and rank 1).</p>
<p>So on each node two ranks are launched. I have some code creating some gloo Processgroups with:</p>
<pre><code class="lang-auto">global_pg = dist.ProcessGroupGloo(
prefix_store, {id of node}, {global_world_size}, some timeout
)
</code></pre>
<p>This means:</p>
<pre><code class="lang-auto">
node 0:
rank 0: dist.ProcessGroupGloo(prefix_store, 0, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 0, 3}

node 1:
rank 0: dist.ProcessGroupGloo(prefix_store, 1, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 1, 3}

node 2:
rank 0: dist.ProcessGroupGloo(prefix_store, 2, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 2, 3}
</code></pre>
<p>I do not understand what is happening there. I get two processgroups on each node, with different ids (id(glogal_pg)), what makes sense. For both processgroups I use the same parameters for <code>{id of node}</code> and <code>{global_world_size}</code> on a node.</p>
<p>Which ranks are now connected to which process group? Can someone help me with this?<br>
After having a look at the code I would expect, that I overall get two processgroups. The first is connecting all ranks 0 from all nodes, and the second one all ranks 1 from the nodes. (not sure)<br>
Is this correct and how recognizes e.g. node1 rank1 that it belongs to node0 rank1 and not node0 rank0?</p>
Integration / anti-differentiation in PyTorch

rasbt: <p>I know that PyTorch is primarily an auto-diff library, but I was wondering if there are also ways to integrate efficiently? Unfortunately, I couldn’t find anything when searching (probably due to the overloading of the term “integrate” in engineering contexts).</p>
<p>For simple things, I could probably come up with something like</p>
<pre><code class="lang-auto">def integrate(f, a, b, points):
    x = torch.linspace(a, b, points)
    area = torch.sum(f(x))*(b-a)/points
    return area
</code></pre>
<p>but I was wondering if there are more efficient ways to do it, maybe something similar to scipy.integrate?</p>

tom: <p>So there aren’t efficient ways to integrate similar to autodiff, so basically, you need (more refined) formulas like yours.<br>
I haven’t seen implementations of the advanced quadrature rules in PyTorch.</p>
<p>It’s not quite what you want, but the Neural ODE code has ODE solvers which would give you some adaptivity even if they try to solve a much harder problem and are not as efficient for the integration problem:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/rtqichen/torchdiffeq/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/3375899?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/rtqichen/torchdiffeq/" target="_blank" rel="nofollow noopener">rtqichen/torchdiffeq</a></h3>

<p>Differentiable ODE solvers with full GPU support and O(1)-memory backpropagation. - rtqichen/torchdiffeq</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Best regards</p>
<p>Thomas</p>

rasbt: <p>Thanks a lot, that’s very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Saleh: <p>Use torchquad package</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://torchquad.readthedocs.io/en/main/tutorial.html">
  <header class="source">

      <a href="https://torchquad.readthedocs.io/en/main/tutorial.html" target="_blank" rel="noopener nofollow ugc">torchquad.readthedocs.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://torchquad.readthedocs.io/en/main/tutorial.html" target="_blank" rel="noopener nofollow ugc">Tutorial — torchquad 0.4.1 documentation</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
CPU RAM Leak when using Huggingface BERT model for inference

Luca_Guarro: <p>I am attempting to use my fine-tuned DistilBERT model to extract the embedding of the ‘[CLS]’ token. For every row in my dataset I want to extract this feature and return the result into an array.</p>
<p>However, my code seems to be suffering from a memory leak. I have roughly ~14K rows in my dataset and by the time the code has finished executing, Google Colab has either crashed or reports that I have used almost all 25GB of RAM!</p>
<p>Each embedding is a tensor with 768 elements. So for 14K elements, the returned array should be on the order of 20-30 MBs.</p>
<p>Here is my function that is failing from the memory leak:</p>
<pre><code class="lang-auto">def getPooledOutputs(model, encoded_dataset, batch_size = 128):
  pooled_outputs = []
  print("total number of iters ", len(encoded_dataset['input_ids'])//batch_size + 1)

  for i in range(len(encoded_dataset['input_ids'])//batch_size + 1):
    up_to = i*batch_size + batch_size

    if len(encoded_dataset['input_ids']) &lt; up_to:
      up_to = len(encoded_dataset['input_ids'])

    input_ids = th.LongTensor(encoded_dataset['input_ids'][i*batch_size:up_to])
    attention_mask = th.LongTensor(encoded_dataset['attention_mask'][i*batch_size:up_to])
    with torch.no_grad():
      embeddings = model.forward(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True)['hidden_states'][-1][:,0] # Pooled output
      pooled_outputs.extend(embeddings)

  return pooled_outputs
</code></pre>

eqy: <p>Can you tell if memory usage is growing with each iteration? (e.g., if you forcibly reduce the number of iterations, does the memory usage go down?)</p>

Luca_Guarro: <p>I’ve tried using a fourth of my dataset and I did not see a concerning amount of RAM usage. It is only towards the very end of the loop using the entire dataset does my RAM usage skyrocket. This is solely from monitoring what the Google Colab environment displays to me.</p>
<p>This image can give you an idea:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd361ec3b11d06bb7d28b6f1ae438747d8653125.png" alt="RAM" data-base62-sha1="thnPFvyZEZfL0PUG6rALqhdI0Zf" width="124" height="102"></p>

shakeel604: <p>How did you solve this issue? <a class="mention" href="/u/luca_guarro">@Luca_Guarro</a></p>
How to avoid casting DTensor to Tensor before calling a custom operator (a CUDA kernel)

terunofuji: <p>I am doing tensor parallel training with DTensor. In my training code, I have a custom operator gmm, which is linked to a CUDA kernel in the backend. Currently, I am casting DTensor to Tensor before the operation and casting it back afterward, which I find to be the bottleneck in the code. Is there a way I can somehow pass DTensor directly to my custom op and have my custom op interpret the tensor dimension (local dimension) correctly? Doing so naively would lead to a dimension bug; the DTensor dimension is interpreted as the global dimension, not the local one.</p>
<pre><code class="lang-auto">        # Casting to local tensor
        Ds_a_dtensor = DTensor.from_local(a_out, device_mesh=self.device_mesh, placements=[Shard(1)])
        Ds_a_gathered = Ds_a_dtensor.redistribute(device_mesh=self.device_mesh, placements=[Replicate()])
        a_b = Ds_a_gathered.to_local().contiguous()
        b_b = torch.cat([module.weight.to_local().T.contiguous() for module in self.b], dim=0).contiguous()
        batch_sizes_b = torch.tensor(self.local_rank, device='cpu', dtype=torch.long)
        
        # Perform custom op
        b_out = ops.gmm(a_b, b_b, batch_sizes_b)

        # Cast back to DTensor
        Ds_b_dtensor = DTensor.from_local(b_out, device_mesh=self.device_mesh, placements=[Shard(1)])
        Ds_b_gathered = Ds_b_dtensor.redistribute(device_mesh=self.device_mesh, placements=[Replicate()])
</code></pre>

tianyu: <p>Thanks for the question.<br>
It looks the <code>register_sharding</code> API could help you</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/docs/stable/distributed.tensor.html#torch.distributed.tensor.experimental.register_sharding">
  <header class="source">

      <a href="https://pytorch.org/docs/stable/distributed.tensor.html#torch.distributed.tensor.experimental.register_sharding" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://pytorch.org/docs/stable/distributed.tensor.html#torch.distributed.tensor.experimental.register_sharding" target="_blank" rel="noopener nofollow ugc">torch.distributed.tensor — PyTorch 2.6 documentation</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Arm has published tutorials for PyTorch on Arm64

mhall119: <p>We just launched a new web page that curates various tutorials and guides, including a section for PyTorch on Arm-based servers and cloud deployments.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.arm.com/markets/computing-infrastructure/arm-cloud-migration#c-c2d64d88-aefc-4994-9679-58a7b2897461">
  <header class="source">
      <img src="https://www.arm.com/shared/common/img/favicon/favicon-32x32.png?v=1.0.9203.13822" class="site-icon" width="32" height="32">

      <a href="https://www.arm.com/markets/computing-infrastructure/arm-cloud-migration#c-c2d64d88-aefc-4994-9679-58a7b2897461" target="_blank" rel="noopener nofollow ugc">Arm | The Architecture for the Digital World</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://www.arm.com/-/media/global/solutions/infrastructure/arm-cloud-migration/ATM_social_1080x1080.png?rev=3548097d5711412cb44b8c67cb2def2d&amp;revision=3548097d-5711-412c-b44b-8c67cb2def2d" class="thumbnail onebox-avatar">

<h3><a href="https://www.arm.com/markets/computing-infrastructure/arm-cloud-migration#c-c2d64d88-aefc-4994-9679-58a7b2897461" target="_blank" rel="noopener nofollow ugc">Arm Cloud Migration</a></h3>

  <p>Easily transition to Arm-based cloud with step-by-step guides, tutorials, community support, and direct access to Arm experts. Get started today!</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If there are additional topics anyone would like to see added, or if you have one you want to contribute, let me know in the comments!</p>
Why does Batch.from_data_list add edge_index, edge_attr, and pos to my Torus objects even when not set?

Tommaso_Guarniera: <p>I’m using PyTorch Geometric and have a custom Torus class (a subclass of Data) that conditionally initializes attributes such as edge_index, edge_attr, and pos only when no_fixed is False. For example:</p>
<pre><code class="lang-auto">class Torus(Data):
    def __init__(self, N: int = 64, L: float = 64.0, M: int = 64,
                 x: torch.Tensor = None, y: torch.Tensor = None,
                 eight_neighbors: bool = False, device=None, no_fixed: bool = False):
        if no_fixed:
            super().__init__(x=x, y=y)
        else:
            pos = grid_pos(N, L, device)
            edge_index, edge_attr = torus_edges(N, L, pos, eight_neighbors, device)
            super().__init__(x=x, y=y, edge_index=edge_index, edge_attr=edge_attr, pos=pos)
        self.N = torch.tensor(N, dtype=torch.long).unsqueeze(0)
        self.L = torch.tensor(L, dtype=torch.float).unsqueeze(0)
        self.M = torch.tensor(M, dtype=torch.long).unsqueeze(0)
        self.eight_neighbors = torch.tensor(eight_neighbors, dtype=torch.int).unsqueeze(0)
</code></pre>
<p>When I create individual Torus objects with no_fixed=True, they do not have the edge_index, edge_attr, or pos attributes. However, if I batch them using:</p>
<p>batch = Batch.from_data_list(torus_list)</p>
<p>the resulting batch unexpectedly contains edge_index, edge_attr, and pos (with shapes like [128^2, …] etc).</p>
<p>Why does the batching process add these attributes to the batch even though none of the individual objects have them? Is this expected behavior due to how PyG’s Batch class works?</p>
<p>Any insights would be appreciated!</p>
Pytorch_geometric Linear layer return different results

shahar: <p>Hi all,<br>
I am using ubuntu 22.04 with cpu, python version 3.10, torch version 2.0.1, and torch_geometric version 2.3.1</p>
<p>I am running a Linear layer in a loop and collect the results, and if I am running the loop enough, I get different results.<br>
I printed the max diff, to show this is not a numeric stability issue, but I get totally different results.<br>
Any reason why?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea1df01cd4284170c6c91d2a995637b0c9d5a371.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea1df01cd4284170c6c91d2a995637b0c9d5a371" title="{7E8A1522-8A74-42E9-AF25-93D7829F8257}"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea1df01cd4284170c6c91d2a995637b0c9d5a371.png" alt="{7E8A1522-8A74-42E9-AF25-93D7829F8257}" data-base62-sha1="xp5RVBLwejFKkw29qczhMJis2hH" width="516" height="250"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">{7E8A1522-8A74-42E9-AF25-93D7829F8257}</span><span class="informations">516×250 32.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If you are seeing the same issue using the latest stable or nightly releases, I would recommend creating a GitHub issue for it.</p>

shahar: <p>I updated the versions to torch 2.4.1 and torch_geometric 2.6.1.<br>
Can’t update to later versions because of other constrains.<br>
Anyways - it feels like a “not that old version”, and the problem remains.<br>
I opened a bug here: <a href="https://github.com/pyg-team/pytorch_geometric/issues/10159" class="inline-onebox" rel="noopener nofollow ugc">nn.dense.Linear return different results · Issue #10159 · pyg-team/pytorch_geometric · GitHub</a></p>
<p>BTW, it is a good aprotunity to thank all your support <a class="mention" href="/u/ptrblck">@ptrblck</a> . You are doing amazing &lt;3</p>
Can Torch support training on multiple GPUs which have different memory size?

salidw: <p>Few years ago, I remembered that I had trained my model on 2 RTX 3060, and 1 RTX 2060. There was an error raised up.</p>
<p>best regards,<br>
SW.</p>

ptrblck: <p>Yes, PyTorch supports multi-GPU training using different GPUs, but you would have to make sure you are not running out of memory etc. The training will also most likely be bottlenecked by the slowest GPU.</p>

salidw: <p>Great and thanks. Let me try.</p>

rav2n: <p>A.</p>
<blockquote>
<p>bottlenecked by the slowest GPU</p>
</blockquote>
<p>Is it still the case?<br>
Caz, slower GPU can be tasked with lower params and batch size<br>
in relation to the faster GPU right?</p>
<p>The same holds for unequal vrams as well right?</p>
<p>B.</p>
<blockquote>
<p>make sure you are not running out of memory etc</p>
</blockquote>
<p>Well, it’d make sense for the PyTorch module (I don’t know, FSDP) to take care of all these right?<br>
What is the current scenario? Does PyTorch handle these by itself?</p>
<p>C.<br>
How about cpu offloading?<br>
For large models requiring high vram, but in a setup with a single gpu and limited vram,<br>
(given batchsize of 1, so that I get a better picture of the bare minimum weights, activation, opt. states, gradients vram requirements) is PyTorch currently able to divide the model and cpu offload automatically so that even if the training takes longer, it still works (leaving aside dynamic batchsize etc, and wanting to know this base case).</p>
<p>D. How does all the above questions apply for 3 scenarios:<br>
a. 1 nvidia gpu only<br>
b. 1 latest amd gpu only<br>
c. a &amp; b - ie, 2 in total (nvidia and amd)</p>
<p>Please give as much detail as possible.<br>
Would also welcome pointers to learning resources and others implementations, real world stuff, etc.</p>
<p>PS: WAITING EAGERLY</p>

ptrblck: <aside class="quote no-group" data-username="rav2n" data-post="4" data-topic="190839">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rav2n/48/76447_2.png" class="avatar"> rav2n:</div>
<blockquote>
<p>Caz, slower GPU can be tasked with lower params and batch size<br>
in relation to the faster GPU right?</p>
</blockquote>
</aside>
<p>Of course a proper balancing is possible if you implement it. Since these scenarios are uncommon (and can hardly be tested on available machines for leases) standard utils. might not support them.</p>
Model inference consumes memory, every time model(input) is called

shakeel604: <p><strong>CPU memory gets accumulated everytime I call model for inference</strong></p>
<pre><code class="lang-auto">def model_inference(
    model,
    device,
    input_ids
):
    input_ids = input_ids.to(device)

    with torch.no_grad():
        outputs = model(input_ids = input_ids, output_hidden_states = True)
    
    torch.cuda.empty_cache()
   
    gc.collect()

     
    #print("+"*5, psutil.virtual_memory().available/(1024 ** 3))

    return outputs
    
</code></pre>
<pre><code class="lang-auto">## Memory profiler of this function is below: If you can see that at line
    138, the program is using 5967 MB of memory. And it add 316 MB memory
    when it calls lines 140-141. But then it never frees this memory.
    And this is the memory which keeps on getting accumulated 
    and results in memory consumption 
    Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   132   5967.3 MiB   5967.3 MiB           1   @profile
   133                                         def model_inference(
   134                                             model,
   135                                             device,
   136                                             input_ids
   137                                         ):
   138   5967.3 MiB      0.0 MiB           1       input_ids = input_ids.to(device)
   139                                         
   140   6284.1 MiB      0.0 MiB           2       with torch.no_grad():
   141   6284.1 MiB    316.8 MiB           1           outputs = model(input_ids = input_ids, output_hidden_states = True)
   142  
   153   6284.1 MiB      0.0 MiB           1       torch.cuda.empty_cache()
 
   157   6284.1 MiB      0.0 MiB           1       gc.collect()
   158                                         
   159                                              
   160                                         
   161                                         
   162                                            
   163                                             #print("+"*5, psutil.virtual_memory().available/(1024 ** 3))
   164                                         
   165   6284.1 MiB      0.0 MiB           1       return outputs
</code></pre>
<p>I have this function in my code when I used inference on lines <code> outputs = model(input_ids = input_ids, output_hidden_states = True)</code>, it keeps on consuming 300-600 MBs of memory from the allocated available memory. Even though if I use garbage collection, it still adds up this memory, every time I call this model_inference function and then runs out of memory. I am passing a sample at a time</p>
<p>Can you please suggest me what should be done here ?</p>
Correct usage of VGG16 Convolution Layers for Edge Detection Problem

rmsValue9232: <p>Hello,<br>
I am building an Edge Detector Based on the Conv2d layers of the VGG16 for feature extraction, and I wanted to ask how to correctly load part of the model architecture, say I want only the convolution layers block and do not want the fully connected layers at the end?</p>
<p>Some background:<br>
I am building an Edge Detection model, in which I want to use the convolution layers of the VGG16 for feature extraction, then pass it through the decoder consisting of two ConvTranspose2d layers and finally upsampling to bring it to correct dimensions as the target edge mask.</p>
<p>I am currently using this architecture:</p>
<pre><code class="lang-auto">vgg16 = models.vgg16(pretrained=False)

features = list(vgg16.features.children())[:-1]  # Remove the last max-pooling layer

vgg16_no_maxpool = torch.nn.Sequential(*features)

decoder = torch.nn.Sequential(
            torch.nn.ConvTranspose2d(512, 256, kernel_size=3, stride=2, padding=1, output_padding=1),
            torch.nn.ReLU(),
            torch.nn.ConvTranspose2d(256, 128, kernel_size=3, stride=2, padding=1, output_padding=1),
            torch.nn.ReLU(),
            torch.nn.Conv2d(128, 1, kernel_size=1),  # Directly output after 128 channels
            torch.nn.Sigmoid()
        )

#Upsample to match input size (224x224)
upsampler = torch.nn.Upsample(size=(224, 224), mode='bilinear', align_corners=True)

class VGG16EdgeDetector(pl.LightningModule):
    def __init__(self, learning_rate=0.001):
        super(VGG16EdgeDetector, self).__init__()
        self.learning_rate = learning_rate
        
        # Define the layers of the CNN
        self.vgg16_no_maxpool = vgg16_no_maxpool

        # Define the decoder
        self.decoder = decoder

        # Define the upsampling layer
        self.upsampler = upsampler


        # Define the loss function
        self.loss_fn = class_balanced_loss

        self.save_hyperparameters()
</code></pre>
<p>However, upon using this, I am getting a checkered-esque output prediction mask:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dfa50b681cc753d992a31544e25a672c97a8047.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8dfa50b681cc753d992a31544e25a672c97a8047" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dfa50b681cc753d992a31544e25a672c97a8047.jpeg" alt="image" data-base62-sha1="kfZCwuW7DG723xWUWu2Tfk8DHaD" width="652" height="482"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">652×482 75.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here I am loading the image and ground truth edge mask from test dataloader (using Lightning Pytorch), so I believe the dataloading has been done correctly.</p>
<p>Thanks.</p>
How does pytorch triton work with dynamic shapes?

0xmchadha: <p>I have been trying to understand the codegen in torch.compile how pytorch operators get converted to triton in the presence of dynamic shapes. I understand torchinductor creates python callables which are pointwise operations which gets converted to triton. It uses sympy to express symbols where concrete shapes are not known. Does that mean the entire triton MLIR kernel is created with tensor types that are not know? Are these concrete shape values then passed by the runtime as CUDA kernel arguments?</p>
There is not libgomp-a34b3233.so.1 in torch/lib when I build the torch based on the V2.4.0 with USE_CUDA=0

edwinli: <p>I build the torch based on the source codes, but not libgomp in the torch/lib, here is the list in the lib directory. Could anyone help clarify how to build torch with use_cuda=0, or what’s the parameter/variable I need to set? Thanks in advance.<br>
-rw-r–r--  1 root root    296928 Mar 31 12:58 libbackend_with_compiler.so<br>
-rw-r–r--  1 root root   1130600 Mar 31 11:34 libc10.so<br>
-rw-r–r--  1 root root    253640 Mar 31 12:58 libjitbackend_test.so<br>
-rw-r–r--  1 root root     45712 Mar 31 13:29 libshm.so<br>
-rw-r–r--  1 root root     15464 Mar 31 12:58 libtorch.so<br>
-rw-r–r--  1 root root 304274880 Mar 31 12:58 libtorch_cpu.so<br>
-rw-r–r--  1 root root     15680 Mar 31 11:34 libtorch_global_deps.so<br>
-rw-r–r--  1 root root  26323856 Mar 31 13:30 libtorch_python.so<br>
-rw-r–r--  1 root root    967016 Mar 31 12:59 libtorchbind_test.so</p>
What's a good replacement for torchtext?

vdw: <p>In my courses, I share prepared Jupyter notebooks with my students. All notebooks that handle textual data currently still use <code>torchtext</code> (e.g., <a href="https://github.com/chrisvdweth/nus-cs5246/blob/master/notebooks/Named%20Entity%20Recognition%20(RNN%20Data%20Preparation).ipynb" rel="noopener nofollow ugc">this one</a> and <a href="https://github.com/chrisvdweth/nus-cs5246/blob/master/notebooks/Neural%20Networks%20(Transformer%20-%20Machine%20Translation.ipynb)" rel="noopener nofollow ugc">this one</a>. Now that that <code>torchtext</code> is no longer maintained/developed, I would like to “refresh” the notebooks to remove any use of <code>torchtext</code>.</p>
<p>What would be the recommended best practice? I basically used <code>torchtext</code> only for building the vocabulary, and then transforming tokens/words to their respective indices, and vice versa</p>
<p>Before using <code>torchtext</code>, I actually used my own <a href="https://github.com/chrisvdweth/ml-toolkit/tree/master/pytorch/utils/data/text" rel="noopener nofollow ugc">implementation</a> for that, so I know how to do it. But I would prefer to stick to popular libraries/tools/etc. and best practices to streamline my notebooks and keep the code clean.</p>
<p>Any suggestions are welcome!</p>

Aaron_M: <p>Realize this is an old post, but seeing no responses: I have also been looking for a replacement with no luck. My current workflow is to use <a href="https://spacy.io/api" rel="noopener nofollow ugc">SpaCy</a> which handles most of the plumbing tasks (e.g., tokeninzation) and then handling the conversion to tensors on my own in PyTorch Lightning DataModules.</p>
<p>Would love to hear about better alternatives.</p>

vdw: <p>I only ever used <code>torchtext</code> to create vocabularies and then convert list of tokens to list of indices. Any preprocessing such as tokenization (well, basic word-level tokenization), lemmatization, etc. I always do with spaCy as well.</p>
<p>I therefore simply do it myself now with a custom class which is simple, short, and easy to extend if needed. Here is a <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/text_token_indexing.ipynb" rel="noopener nofollow ugc">notebook</a> explaining the individual steps (it’s an education notebook) and here is the <code>Vocabulary</code> class is in <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/src/text/vectorizing/vocab.py" rel="noopener nofollow ugc">here</a>.</p>