akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Recompiles on loss_kwargs changes

canergen: <p>I have a VAE, for which the forward method takes a kl_weight as input. We warm up this kl, if I pass this changing kl_weight to the forward method of a compiled pytorch module, it gets extremely slow, whereas when I compute the loss by changing the kl_weight outside of the compiled model and pass the changed loss to pytorch lightning I get a significant speedup by compilation. Am I doing something wrong? For reference, this is the model I’m trying to optimize: <a href="https://github.com/scverse/scvi-tools/compare/main...can_compile" class="inline-onebox" rel="noopener nofollow ugc">Comparing main...can_compile · scverse/scvi-tools · GitHub</a></p>

canergen: <p>I figured it out. Providing kl_weight as torch.tensor instead of float works without recompiling.</p>
Efficient way to calculate gradient norms/gradient penalty

import-antigravity: <p>Currently, I have the following function implemented for getting the gradient norm for algorithms like SGA and Consensus Optimization:</p>
<pre><code class="lang-auto">def hamiltonian_loss(d_x: Tensor, d_z: Tensor, gan: GAN):
    """
    Calculates the "hamiltonian loss" as described in "The Mechanics of n-Player Differentiable Games" (Balduzzi et al
     2018), i.e. the loss function defined by 1/2 of the 2-norm of the overall gradient vector.

    @param d_x: Discriminator evaluated at real samples
    @param d_z: Discriminator evaluated at generated samples
    @param gan: GAN module
    @return: Hamiltonian loss
    """

    # Get gradient
    g_loss_ = gan.generator_loss(d_z)
    d_loss_ = gan.discriminator_loss(d_x, d_z)

    g_grad = grad(g_loss_, gan.G.parameters(), retain_graph=True, create_graph=True)
    d_grad = grad(d_loss_, gan.D.parameters(), retain_graph=True, create_graph=True)

    gradient_vec = tr.cat([tr.flatten(w) for w in g_grad] + [tr.flatten(w) for w in d_grad])

    norm = tr.pow(tr.linalg.norm(gradient_vec), 2) / 2
    return norm
</code></pre>
<p>I know this involves a second backpropagation pass and is going to come with a performance hit, but currently evaluation is very, very slow (over a second per iteration).</p>
<p>I think the issue has to do with the list comprehension as it is the only part that is not pure pytorch code. I’ve looked at the documentation for the higher-level API beta, but it doesn’t seem like that’s what I’m looking for.</p>
<p>Is there a way to do this more efficiently?</p>

MoH-assan: <p>Any luck with this? <a class="mention" href="/u/import-antigravity">@import-antigravity</a></p>
Gradient scaling with multiple scalers

Sam_Lerman: <p>I have multiple scalers that operate on losses which might overlap in the computation graph. Should I call <code>scaler1.scale(loss)</code> and <code>scaler2.scale(loss)</code>? I’m not sure how to do this. The different scalers correspond to different optimizers.</p>
<p>Maybe a minimal example (not tested):</p>
<pre><code class="lang-auto">scaler0 = torch.cuda.amp.GradScaler()
scaler1 = torch.cuda.amp.GradScaler()

for epoch in epochs:
    for input, target in data:
        optimizer0.zero_grad()
        optimizer1.zero_grad()
        with autocast(device_type='cuda', dtype=torch.float16):
            output0 = model0(input)
            output1 = model1(input)
            loss = loss_fn(2 * output0 + 3 * output1, target)

        # This is the part I don't know how to do
        scaler0.scale(loss).backward(retain_graph=True)
        scaler1.scale(loss).backward()

        scaler0.step(optimizer0)
        scaler1.step(optimizer1)

        scaler0.update()
        scaler1.update()
</code></pre>

ptrblck: <p>You could use a single <code>GradScaler</code> as seen in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers">Working with Multiple models, Losses, and Optimizers</a> section of the docs.</p>

cuijianaaa: <p>But will there be a problem if I use two scalers for some special case?</p>

ptrblck: <p>Could you describe what your special case would be?</p>

cuijianaaa: <p>Thank you for your reply. In my project, there is a large model with multiple subtasks, each with a separate optimizer to optimize a portion of the model’s weights. I hope that the resources related to each subtask are maintained separately, such as optimizer, LR scheduler, scaler, etc. Each subtask may use different optimizers and training strategies. If I want to use one overall scaler, it will break the existing code design. Therefore, I would like to know if using two scalars will result in unexpected behavior</p>

ptrblck: <p>No, using multiple scalers is supported and an example is given in the linked docs.</p>
Error using F.interpolate for large 3D input

miaomiao: <p>I get the error <code>RuntimeError: Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false</code> when I try to upsample a 3D input. Code to reproduce the error</p>
<pre><code class="lang-auto">x = torch.randn(1, 256, 16, 512, 512).to("cuda")
out = F.interpolate(x, scale_factor=(2, 1, 1))
</code></pre>
<p>It seems the reason is that the output shape would be too large. But won’t have this error with 2D input. Example, the below code would be fine. Is this something expected?</p>
<pre><code class="lang-auto">x = torch.randn(1, 256, 512*16, 512).to("cuda")
out = F.interpolate(x, scale_factor=(1, 2))
</code></pre>
How to store non-aligned data?

Roy-Kid: <p>I try to deal with the chemical dataset, which has a lot of small molecules with different atoms. How to use torchdict to store those non-aligned data, for example, coordinates. What should I set the batch_size?</p>
<p>for example, some data looks like this:</p>
<pre><code class="lang-auto">ndims = 3
data = {
    'xyz': torch.concatenate([torch.rand(10, ndims), torch.rand(7, ndims)...]),
    'energy': torch.tensor([1, 2, ...]),
    'batch_ud': torch.tensor([0] * 10 + [1] * 7)
}
</code></pre>
<p>Thanks a lot!</p>

SuperKenVery: <p>Even if you stored them in a batch, how would you consume this batch in a model? I don’t think non-aligned data is supported anywhere in pytorch.</p>
PyTorch Geometric: Is there a way to use DeepGCNLayer with HeteroConv?

rusty326: <p>This is a sample of my HeteroData</p>
<p>HeteroData(<br>
object={ x=[21, 2048] },<br>
attribute={ x=[60, 2048] },<br>
(object, relationships_a, object)={<br>
edge_index=[2, 12],<br>
edge_attr=[12, 2048],<br>
},<br>
(object, relationships_b, object)={<br>
edge_index=[2, 24],<br>
edge_attr=[24, 2048],<br>
}<br>
)</p>
<p>Is there a way to use the HeteroConv wrapper to define different operators to process (object, relationships_a, object) and (object, relationships_b, object) and then apply a DeepGCNLayer layer on top of it to use residual connection functionality?</p>
"Cleaning" dataset before DataLoader creation can significantly improve performance

mpgl: <p>Hi,</p>
<p>I recently discovered by chance that if first extract the data and target tensors from a dataset and then create a DataLoader using a custom TensorDataset, i get huge performance improvement (up to 2-3 times faster training time).</p>
<p>Here is some code:</p>
<pre><code class="lang-auto">def clean_dataset(dataset: torch.utils.data.Dataset,
                  max_devide: bool = True,
                  ) -&gt; torch.utils.data.TensorDataset:
    """
    Extracts data and targets from the dataset and normalizes the data. It can also significantly
    improve simulation performance.
    """
    X, y = dataset.data.unsqueeze(1).float(), dataset.targets.long()
    if max_devide:
        X /= X.max()
    return torch.utils.data.TensorDataset(X, y)

# Download mnist training data
training_data = datasets.MNIST(
    root="../datasets",
    train=True,
    download=True,
    transform=ToTensor(),
)

# Download mnist test data
test_data = datasets.MNIST(
    root="../datasets",
    train=False,
    download=True,
    transform=ToTensor(),
)

# Clean datasets (If I uncomment the next 2 lines, I get huge performance improvements)

# training_data = clean_dataset(training_data)
# test_data = clean_dataset(test_data)

# Create DataLoaders
train_loader = DataLoader(training_data, batch_size=BATCH_SIZE, shuffle=True)
test_loader = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True)

# AND THEN PERFORM SOME STANDARD CLASSIFICATION EXPERIMENT USING A SIMPLE FEED-FORWARD NETWORK
</code></pre>
<p>Given the above, I have 2 questions:</p>
<ol>
<li>Why is training performance so much better when I create the Data Loaders using the above trick?</li>
<li>Are there any unexpected side-effects of this approach? Should I take any special precautions? Classification accuracy does not seem to change, only training speed.</li>
</ol>
<p>Thank you in advance for your help!</p>
<p>Michalis</p>

ptrblck: <p>By default <code>MNIST</code> is loading each sample as a <code>PIL.Image</code> and then applies all transformations on it returning the tensor (assuming <code>ToTensor</code> was used). You are skipping the image to tensor transformation and are thus seeing a performance improvement.</p>
Functional.cross_entropy with one-hot vector

Jacopo_Rizzi: <p>I’m using torch.nn.functional.cross_entropy with one-hot vectors as target when I realized that it wants class probabilities or class indices as target, not one-hot vector.<br>
But my thought is, isn’t it the same? I mean, if I use one-hot vector as FloatTensor is like setting 100% probability on a class and 0% on the others?<br>
Indeed, with this simple example:</p>
<pre><code class="lang-auto">logits = torch.zeros([2,2], dtype=torch.float32)
logits[0] = torch.tensor([-0.1, 21.0])
logits[1] = torch.tensor([10.1, -2.0])
</code></pre>
<p>I get the same result with both approaches:</p>
<pre><code class="lang-auto"># One-hot to FloatTensor
t = torch.tensor([[0.0,1.0],[1.0,0.0]])
print(F.cross_entropy(logit,t))
# Class indices
t = torch.tensor([1,0], dtype=torch.int64)
print(F.cross_entropy(logit,t))
</code></pre>
<p>Can I have some problem with the first approach? Is it correct?<br>
Thank you!</p>

KFrank: <p>Hi Jacopo!</p>
<aside class="quote no-group" data-username="Jacopo_Rizzi" data-post="1" data-topic="207831" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jacopo_rizzi/48/72025_2.png" class="avatar"> Jacopo_Rizzi:</div>
<blockquote>
<p>I get the same result with both approaches:</p>
<pre><code class="lang-auto"># One-hot to FloatTensor
t = torch.tensor([[0.0,1.0],[1.0,0.0]])
print(F.cross_entropy(logit,t))
# Class indices
t = torch.tensor([1,0], dtype=torch.int64)
print(F.cross_entropy(logit,t))
</code></pre>
<p>Can I have some problem with the first approach? Is it correct?</p>
</blockquote>
</aside>
<p>You understand this correctly, and there is nothing wrong with the<br>
first approach.  (To be clear, <code>cross_entropy()</code> accepts targets that<br>
are either floating-point “probabilistic” targets – including floating-point<br>
one-hot encoded targets – <em>with</em> a class dimension, or integer class<br>
labels, <em>without</em> a class dimension.  If you one-hot encode your integer<br>
class labels – and convert to floating-point – the two versions will<br>
agree.)</p>
<p>However, using integer class labels is modestly more efficient (even<br>
if you start with one-hot encoded labels and convert them to integer<br>
class labels), so there’s no reason to use one-hot encoded labels<br>
with <code>cross_entropy()</code> (nor with <code>CrossEntropyLoss</code>).</p>
<p>Best.</p>
<p>K. Frank</p>
How to make custom c++ class torch compilable

iamyangchen: <p>This <a href="https://docs.google.com/document/d/1_W62p8WJOQQUzPsJYa7s701JXt0qf2OfLub2sbkHOaU/edit#heading=h.ptttacy8y1u9" rel="noopener nofollow ugc">doc</a> discussed about how to make custom operators work with torch.compile. Is it possible to make custom c++ classes compatible with torch.compile?</p>
TransE algorithm with PyTorch BigGraph(PBG)

kbayes: <p>Hi,</p>
<p>I am looking for examples on defining the config file for utilizing TransE algorithm for training on my graph. This official page from PBG is not very helpful: <a href="https://torchbiggraph.readthedocs.io/en/latest/related.html" class="inline-onebox" rel="noopener nofollow ugc">Related works — PyTorch-BigGraph 1.dev documentation</a>.<br>
Any help here would be very much appreciated</p>
Obtaining & Assigning values with tensors based on multiple indexes

krymes: <p><span class="hashtag-raw">#1</span> Problems come up when obtaining values with two index vector from a tensor.<br>
For example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.4.0'
&gt;&gt;&gt; x=torch.randn(2,3,4,5)
&gt;&gt;&gt; a=[0]
&gt;&gt;&gt; b=[0,1]
&gt;&gt;&gt; x[a,:,b,:].shape
torch.Size([2, 3, 5])
&gt;&gt;&gt; x[a][:,:,b,:].shape
torch.Size([1, 3, 2, 5])
</code></pre>
<p>Two index at different dimension <code>x[a,:,b,:]</code> doesn’t return the expected shape of tensor as <code>x[a][:,:,b,:]</code>. Furthermore:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a=[0,1]
&gt;&gt;&gt; b=[0,1,2]
&gt;&gt;&gt; x[a,:,b,:]
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [2], [3]
</code></pre>
<p>The mechanism of multiple index quite confuses me.</p>
<p><span class="hashtag-raw">#2</span> Even though I can obtain the expected shape of tensor from <code>x[a][:,:,b,:]</code>, however, assigning values to it is no use. Specifically:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a=[0]
&gt;&gt;&gt; b=[0,1]
&gt;&gt;&gt; y=torch.zeros(1,3,2,5)
&gt;&gt;&gt; x[a][:,:,b,:]=y
&gt;&gt;&gt; x[a][:,:,b,:]
tensor([[[[-0.4322, -0.2585, -1.2516, -0.6877, -0.2141],
          [ 1.9328, -1.1326,  2.1536, -0.1765,  0.1218]],

         [[-0.0773,  1.2094, -0.5099,  0.7909, -0.1155],
          [-0.1638,  1.0042,  0.2794,  0.2649,  1.0520]],

         [[ 0.5747,  0.7239,  0.9941, -1.0764, -0.3551],
          [-0.3900,  0.7646, -0.1625,  0.5727, -1.6328]]]])
</code></pre>
<p>The values in tensor <code>x</code> doesn’t become <code>y</code>. I guess it’s because the <code>x[a][:,:,b,:]</code> return a new view instead of reference.<br>
So how could I assign <code>y</code> to <code>x</code> correctly? <img src="https://discuss.pytorch.org/images/emoji/apple/melting_face.png?v=12" title=":melting_face:" class="emoji" alt=":melting_face:" loading="lazy" width="20" height="20"></p>
Transformer training unstability - CUDA error

afarbo: <p>Hi, I just started working on a new workstation with a 4080 super and I’m training a transformer with optuna to find the best hyperparameters.</p>
<h2><a name="during-the-training-something-strange-happens-for-first-epoch-everything-is-ok-but-during-the-second-one-i-get-this-error-1" class="anchor" href="#during-the-training-something-strange-happens-for-first-epoch-everything-is-ok-but-during-the-second-one-i-get-this-error-1"></a>During the training, something strange happens:<br>
for first epoch everything is ok but during the second one I get this error</h2>
<p>Traceback (most recent call last):</p>
<p>Cell In[65], line 1<br>
study.optimize(objective, n_trials=50)</p>
<p>File ~.conda\envs\torch\lib\site-packages\optuna\study\study.py:451 in optimize<br>
_optimize(</p>
<p>File ~.conda\envs\torch\lib\site-packages\optuna\study_optimize.py:62 in _optimize<br>
_optimize_sequential(</p>
<p>File ~.conda\envs\torch\lib\site-packages\optuna\study_optimize.py:159 in _optimize_sequential<br>
frozen_trial = _run_trial(study, func, catch)</p>
<p>File ~.conda\envs\torch\lib\site-packages\optuna\study_optimize.py:247 in _run_trial<br>
raise func_err</p>
<p>File ~.conda\envs\torch\lib\site-packages\optuna\study_optimize.py:196 in _run_trial<br>
value_or_values = func(trial)</p>
<p>Cell In[62], line 145 in objective<br>
loss.backward()</p>
<p>File ~.conda\envs\torch\lib\site-packages\torch_tensor.py:522 in backward<br>
torch.autograd.backward(</p>
<p>File ~.conda\envs\torch\lib\site-packages\torch\autograd_<em>init</em>_.py:266 in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>
<hr>
<p>The strange part is that I’m running the same code on two other pcs (one with an A4000 and one with a GTX2070) and there are no problems at all.</p>
<p>In order to keep the gpus as free as possible, I clear the cache every now and then.<br>
I’m also pretty sure the memory shouldn’t be an issue because, before starting the optuna optimization, I tried the maximum model dimensions alongside the maximum batch size and there were no problems.</p>
<p>virtual environment specifics:<br>
python 3.10.14<br>
torch version 2.2.2<br>
cuda version 12.1</p>
<p>UPDATE:</p>
<p>Just to be sure I run this simple line and I got another error:<br>
t = torch.tensor([1,2], device=device)<br>
Traceback (most recent call last):</p>
<p>Cell In[18], line 1<br>
t = torch.tensor([1,2], device=device)</p>
<p>RuntimeError: CUDA error: unspecified launch failure<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>I restarted the environment and the line works again. It seems that at some point, during the training, moving tensors to the gpu causes some problems but I don’t understand the reasons.</p>
What is the current/future best practice for custom autograd functions (with triton parts)

JonasGeiping: <p>Hi everyone,</p>
<p>I am a bit unsure what the current best practice is for torch 2.4 and the future when it comes to custom autograd implementations. The syntax I’m used to is to wrap custom functions in torch.autograd.Function, as described here: <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.4 documentation</a>.<br>
But now, there is also the <code>torch.library</code> implementation with torch.library.register_autograd, described here: <a href="https://pytorch.org/tutorials/advanced/python_custom_ops.html#adding-training-support-for-crop" class="inline-onebox" rel="noopener nofollow ugc">Python Custom Operators — PyTorch Tutorials 2.4.0+cu121 documentation</a>.</p>
<p>The actual use-case I have are custom triton function, which work nicely with the old implementation for normal things, but, for example, don’t work well with fake tensors, like the ones used for flop counting in <code>torch.utils.flop_counter</code>.</p>
<p>Concrete questions:</p>
<ul>
<li>What is the best practice for optimal <code>torch.compile</code> support?</li>
<li>If the custom_op is only pytorch code + triton code, does it matter?</li>
<li>If I want a fake tensor stub, or register that my op works only on cuda, should I switch to <code>torch.library</code>?</li>
<li>If I use the <code>torch.library</code> custom_op, does that disable torch.compile fusion into the custom operator? (E.g. merging a reshape in the custom_op into a previous operation outside the custom_op)</li>
<li>How do I annotate support for <code>torch.amp</code> with <code>torch.library</code>, so the same thing that <code> @torch.amp.custom_fwd</code> does for <code>torch.autograd.Function</code> ?</li>
</ul>
<p>Thanks!</p>
Is there a torch scatter operation to put sliced tensors from CPU to GPU?

AKT_TARAFDER: <p>lets assume a tensor  x = [ [ 2, 2], [2, 2] ] in CPU<br>
is there a touch scatter operation that puts the divided data into 2 GPU</p>
<p>like x1 = [2,2] in GPU 0<br>
and x2 = [2,2] in GPU 1<br>
?<br>
torch dist seems to keep the full tensor in GPU 0 then scatter to other GPUs?</p>
Why is LibTorch from pip not build with CXX11 ABI

trns1997: <p>The opencv and torch problem continues to persist with the libtorch that comes torch installed via pip. I use conan as my package manager for most packages, so naturally i install opencv via conan. But when i try to link both torch installed via pip (cmake … -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=<code>python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'</code> …) and opencv from conan i get the usual error that everyone faces. I need to specify in my conan profile the cxx flag CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 which builds opencv without the cxx11 abi and then everything compiles as expected. I have seen many threads that suggest to use the libtorch cpp package from the site with abi <strong>libtorch-cxx11-abi-shared-with-deps</strong> (see <a href="https://discuss.pytorch.org/t/library-conflict-between-libtorch-and-opencv/64489/8" class="inline-onebox">Library conflict between libtorch and opencv? - #8 by lzl_jnu</a>), but i like the idea of using the libtorch that comes with pip, it would reduce duplicates of the package. I was wondering why the libtorch that comes torch pip is not compile with -D_GLIBCXX_USE_CXX11_ABI=1 ? I use opencv/4.8.1 (conan) and torch==2.4.0 (pip). My conan profile for the curious:</p>
<pre><code class="lang-auto">[settings]
os=Linux
arch=x86_64
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
compiler.cppstd=17
build_type=Release

[options]

[tool_requires]

[buildenv]
CC=/usr/bin/gcc-11
CXX=/usr/bin/g++-11
CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 # I need it for the moment to tell conan to install opencv withouth cxx abi
</code></pre>
<p>As you can see i am pretty up to date with gcc version and all. So just curious, is it a miss that the libtorch packaged with torch pip is not the one installed with abi, or is it because it would break something?</p>

ptrblck: <p>The wheels support <code>manylinux</code> based on CentOS7 as described <a href="https://github.com/pytorch/pytorch/issues/51039">here</a> and thus are build without the CXX11 ABI. Since CentOS7 is EOL already, future releases could relax the requirement and use the CXX11 ABI.</p>

trns1997: <p>Ah i see, seems logical. I guess i’ll wait. It seems to be coming soon enough. Thanks</p>
How does the * operator dispatch to the special op on cuda in backend code?

AllenYcn: <p>When I was learning about autograd mechanism, I saw the following code.</p>
<pre><code class="lang-auto">template &lt;typename T&gt;
Tensor mul_tensor_backward(Tensor grad, T other, ScalarType self_st) {
  auto out = grad * other.conj();
  return handle_r_to_c(self_st, std::move(out));
}
</code></pre>
<p>I know this is the derivative function of multiplication. But the code</p>
<pre><code class="lang-auto">auto out = grad * other.conj();
</code></pre>
<p>confuses me. If all tensors are cuda tensor, this code should be executed  on gpu. however, i dont find the overloads of operate *, so how do the pytorch dispatch it to the mul operator in cuda?</p>
Toggling off 'shuffle' of a dataloader

TinfoilHat0: <p>I’d like to toggle off the <code>shuffle</code> attribute of a dataloader after certain point in my code. Can I do so without creating the dataloader again?</p>

ptrblck: <p>The <code>shuffle</code> argument determined which <code>sampler</code> will be created (<code>RandomSampler</code> or <code>SequentialSampler</code> as seen <a href="https://github.com/pytorch/pytorch/blob/b843f4db0a26aae6536e6b971f73bcc5af21c90a/torch/utils/data/dataloader.py#L345-L349">here</a>) and is not directly used as an attribute in the <code>DataLoader</code>. For this reason I would not recommend to try to replace the internal <code>sampler</code>, but to recreate the <code>DataLoader</code> instead. If you are lazily loading the data in your <code>Dataset</code>, the initialization of the <code>DataLoader</code> should also be cheap.</p>

Rodrigo_Santos: <p>just faced this problem… very outdated, but I hope help someone like me now… hehehhe<br>
In my case, I was working in a function that receive only the dataloader. In fact, you can retrieve the dataset with “dataloader.dataset” and with this you can reset the dataloader,as <a class="mention" href="/u/ptrblck">@ptrblck</a> proposed, without access dataset directly, … see the example below:</p>
<pre><code class="lang-auto">import torch

my_dataloader =torch.utils.data.DataLoader(my_dataloader.datasaset,
batch_size=my_dataloader.batch_size,shuffle=False) 
</code></pre>
How to Access Triton Kernels from TorchInductor when running on CPU?

JoeLi12345: <p>When running torch.compile for a PyTorch model and accessing the generated TorchInductor python code for graph with C++/Triton kernels as indicated <a href="https://pytorch.org/tutorials/intermediate/inductor_debug_cpu.html" rel="noopener nofollow ugc">here</a>, there appears to be no way to access the generated Triton code when using a CPU-device.</p>
<p>It appears as though all of the kernels are written in C++ when running on CPU; however, when running on CUDA, the kernels are written in Triton instead. Does PyTorch / TorchInductor offer a way to access the generated Triton kernels when running on a CPU device?</p>
<p>I can take care of running Triton on a CPU backend, but I need to be able to access the Triton kernels that are generated from PyTorch first.</p>
Using a training function instead of a script

ZachNew: <p>Hi all, I’m working on a single node, distributed object detection codebase where we currently run a single training script for each training run using torchrun. The “classic”</p>
<pre><code class="lang-auto">torchrun --nproc-per-node /my/training/script.py
</code></pre>
<p>setup.</p>
<p>I’m wondering if it possible/practical to instead run this as part of a larger python service. What we’d like to be able to do is take the functionality that exists in script.py and move it into a function, call it <code>script()</code> and then have some service that can spin off a training subprocess and run a function that would kick off however many processes you have gpus and run that <code>scrip()</code> function instead of a script.py file.</p>
<p>This would allow us to more easily chain together training and evaluation processes and stack multiple different types of training scripts together if it made sense for our experiment.</p>

ZachNew: <p>Follow up, we were able to come up with a solution to this. The basic idea is to bypass the run function used by torchrun and directly use the elastic launcher. Below is the key bit we had to come up with:</p>
<pre><code class="lang-auto">def train(*args) -&gt; None:
    launch_config = LaunchConfig(
        min_nodes=1,
        max_nodes=1,
        nproc_per_node=8,
        run_id="none",
        role="default",
        rdzv_endpoint="127.0.0.1:29500",
        rdzv_backend="static",
        rdzv_configs={"rank": 0, "timeout": 900},
        rdzv_timeout=-1,
        max_restarts=0,
        monitor_interval=5,
        start_method="spawn",
        metrics_cfg={},
        local_addr=None,
    )
    launcher = elastic_launch(launch_config, train_subprocess)
    launcher(*args)
</code></pre>
<p>The training subprocess function (which I’m not showing above) is then our standard training loop. That function would have been a script if we were using torchrun.</p>
Why is there a torch.nn.Parameter class but not a torch.nn.Buffer class?

mtanti: <p>It seems strange that I need to use <code>self.register_buffer('name', tensor)</code> instead of <code>self.name = torch.nn.Buffer(tensor)</code>.</p>

JuanFMontesinos: <p>Because a nn.Parameter is trainable whereas a buffer is just a tensor conveniently stored.<br>
There also exist <code>self.register_parameter</code></p>
GDB for Pytorch

Vishesh_Rangwani: <p>I want to know how I can see the stack trace of PyTorch functions written in C++.</p>
<p>Can anyone help with how I can use GDB or any other debugger?</p>
<p>Any help would be appreciated.</p>

ptrblck: <p><code>gdb</code> should directly work and I would recommend to use a debug source build for an easier debugging.<br>
E.g. assuming you would like to debug the <code>adaptive_avg_pool2d</code> method, you could use:</p>
<pre><code class="lang-auto">gdb --args python script.py
...
(gdb) break cpu_adaptive_avg_pool
Function "cpu_adaptive_avg_pool" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (cpu_adaptive_avg_pool) pending.
(gdb) run
...
</code></pre>
<p>which allows you to step, continue, set other breakpoints etc.</p>

Lianghong_Huang: <p>Hi! Is it possible to make breakpoints in release source build? I can make breakpoints in a <code>script.py</code> and step into Pytorch source code written in python. But why I can’t set breakpoints in Pytorch source code directly and step into code written in c++?</p>
<p>Because for some reason I can’t use a debug build, and I want to get the c++ traces of how <code>rnn</code> works. For now I can step into <code>rnn.py</code> (but not make breakpoints in this file directly), and I think it will call some functions from <code>rnn.cpp</code>, but I have no idea how to verify it in gdb.</p>
<p>I would be grateful for your help.</p>

ptrblck: <p>Yes, <code>gdb</code> works with release code, too. Which function are you calling and where are you trying to set a breakpoint?</p>
PyTorch custom-op with None return type does not show up in torch-mlir

jkshtj: <p>Hi all,<br>
I’m trying to add a custom-op to PyTorch that looks as follows:</p>
<pre><code class="lang-auto">myLib.define("custom_op(Tensor t) -&gt; None")

@impl(myLib, "custom_op", "CompositeExplicitAutograd")
def custom_op_impl(t: torch.Tensor) -&gt; None:
    return
</code></pre>
<p>The above op never shows up when I lower my PyTorch module to torch-mlir. However, if I change the op signature to return a <code>torch.tensor</code> things things start working.</p>
<p><strong>Is it not possible to create a custom op with a <code>None</code> return type?</strong></p>
Cuda not working on NVIDIA Jetson

Amal1: <p>I’m trying to get cuda working on Nvidia Jetson AGX Xavier with PyTorch but torch.cuda.is_available() keeps returning false. I have installed Cuda 11.4 along with the standard Jetpack (v5.1.3-b29) installation. I have also installed Python 3.9, PyTorch 2.0.1 compiled with Cuda 11.8 and TorchVision 0.15.</p>
<p>I would appreciate any help I can get in solving this issue. Thanks in advance.</p>

ptrblck: <p>Did you install the PyTorch binaries from <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a> or somewhere else?</p>

Amal1: <p>No, for this current installation I installed from the <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">PyTorch website</a> wrt to this comment on a <a href="https://discuss.pytorch.org/t/which-pytorch-version-2-0-1-support-cuda-11-4/190446">old topic</a>.</p>
<p>I had previously installed PyTorch from the link you gave but instead of using cuda 11.4 that came standard with Jetpack 5.1.3. I flashed the Jetson without cuda and installed cuda 11.8 and 12.1 separately. It didn’t work in both cases. I will try installing PyTorch again from the link you gave with my current cuda installation.</p>

ptrblck: <p>The linked post does not mention the Jetson platform, so install the binaries from the link I’ve posted or run an L4T container.</p>

Amal1: <p>Installing PyTorch from the above link worked. torch.cuda.is_available() returns True. Thanks for the help!</p>
<p>Below are the current settings,<br>
Jetpack - v5.1.3-b29<br>
Cuda - 11.4<br>
Python - 3.8<br>
PyTorch - 2.1.0<br>
Torchvision - 0.16.1</p>
Supporting communication hook with Hybrid shard strategy in torch FSDP

samsja: <p>Hey all,</p>
<p>It seems that communication hooks are not compatible with the HYBRID_SHARD strategy in FSDP.</p>
<p>Anybody know if there is a plan to support in the future or if they are fundamental blocker to it ?</p>
<p>Thanks for the great work.</p>

agu: <p>We do not have immediate plans to support this, but I would be curious to learn more about your use case <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

samsja: <p>I see, basically what I want to do is to have more flexibility around the communication. I want to do advanced grad compression for instance (beyond the fp16/bf16 one) or another advanced mechanism.</p>
<p>It does not have to be a communication hook though, I am wondering what is best in general to change the behavior of the all reduce in FSDP. Would fsdp2 be more suited for this?</p>
<p>Thanks in advance</p>

agu: <p>We may be able to add a hook to FSDP2.</p>
<p>For your use case, are you mainly interested in customizing the all-reduce across replica groups?</p>

samsja: <p>I am mostly interested in being able to do quantization of the gradient for the all reduce and being able to do some all reduce operation only on a sub-part of the total world size (ideally using group mesh)</p>
<p>I am implementing multiple local sgd algorithms and still want to leverage fsdp within one node and do “normal” ddp between the other. A bit like hybrid shard but more granular</p>
Forward Hooks Not Working When `model.eval()` and `torch.no_grad()`

enerd: <p>For <code>TransformerEncoder</code> network, when I created a forward hook, it does not get called if there are <code>model.eval()</code> and <code>with torch.no_grad():</code></p>
<p>Here is a code to reproduce:</p>
<pre><code class="lang-auto">import torch
model = torch.nn.TransformerEncoder(encoder_layer=torch.nn.TransformerEncoderLayer(100, 4, 200, batch_first=True), num_layers=3).to('cuda')
input = torch.randn(2, 10, 100).to('cuda')

def createHook(name):
    print(f"Hook for {name} is set")
    
    def hook(model, input, output):
        print(f"Hook working")

    return hook

for i in range(len(model.layers)):
    model.layers[i].self_attn.register_forward_hook(createHook(f"t_layer_{i}"))


model.eval()
with torch.no_grad():
    pred = model(input) 
    pred = pred.cpu().detach().numpy()
</code></pre>
<p>It gives “Hook for * is set” as output, but it does not print “Hook working”, (and it does not actually call the hook).</p>
<p>I observed that when there is only <code>model.eval()</code>, it works. Also, when there is only <code>with torch.no_grad()</code>, it also works. Somehow using both of them make it not working. And also, I only observed this with <code>TransformerEncoder</code></p>
<p>Is this some kind of a bug? Or, am I missing something?</p>

ptrblck: <p>I guess the “faster transformer” pass is used as described <a href="https://pytorch.org/tutorials/beginner/bettertransformer_tutorial.html">here</a> which does not seem to support forward hooks. Could you create a GitHub issue for it, if none is created yet?</p>

enerd: <p>Thanks for the answer. It is interesting. I could not find any related issue, and created <a href="https://github.com/pytorch/pytorch/issues/132700" rel="noopener nofollow ugc">one</a></p>

enerd: <p>As one user suggested in the github issue, this problem can be overcomed by simply using <code>torch.backends.mha.set_fastpath_enabled(False)</code>. Apparently fastpath disables the hooks</p>
RAM leak in TorchScript

frolkin.igor.1: <p>Hi! I have several hundred of the simplest binary classifiers that I need to combine into one model, for this I create a class that I later script, it is actually a dictionary torch.nn.ModuleDict(), in which the corresponding classifier can be obtained by the value of the key.<br>
When I create an instance of this class, there are no problems, I script it and work with it. But when I save the scripted model (it weighs 1.5Mb), and then load it, then at this moment a memory leak begins, and it takes about 50Gb in ram during the loading process. The problem is observed only on linux (Ubuntu), there is no such problem on Mac OS, the model loads quickly and takes up almost no space in ram, as it should be.<br>
Please tell me, what could be the problem?</p>
<pre><code class="lang-auto">class ClassifierModel(torch.nn.Module):

    def __init__(self, inp, out, threshold):
        super(ClassifierModel, self).__init__()
        self.linear = torch.nn.Linear(inp, out)
        self.threshold = threshold

    def forward(self, x: torch.Tensor):
        outputs = torch.sigmoid(self.linear(x))
        return outputs

class Class(torch.nn.Module):
    
    def __init__(self, path_to_json_classifiers: str):
        super().__init__()
        with open(path_to_json_classifiers, 'r', encoding='utf-8') as file:
            json_classifiers = json.load(file)

        input_size = json_classifiers['input_size']
        output_size = json_classifiers['output_size']
        path_to_state_dict_dir = json_classifiers['path_to_state_dict_dir']

        self.models_dict = torch.nn.ModuleDict()
        for classifier in json_classifiers['classifiers']:
            if 'path_to_state_dict' in json_classifiers['classifiers'][classifier]:
                json_classifier = json_classifiers['classifiers'][classifier]
                model = ClassifierModel(input_size, output_size, float(json_classifier['threshold']))
                model.load_state_dict(torch.load(os.path.join(path_to_state_dict_dir, json_classifier['path_to_state_dict'])))
                model = torch.jit.script(model)
                self.models_dict[classifier] = model

    def forward(self, hash: str, embedding: torch.Tensor):
        for model_hash, model in self.models_dict.items():
            if model_hash == str(hash):
                output = model(embedding)
                output = 1 if output &gt; model.threshold else 0
                return output
        return -1


model = Class(path)
model = torch.jit.script(model)
torch.jit.save(model, path2)
</code></pre>
<p>I create an instance and everything is fine up to here, but then I load the newly saved model:</p>
<pre><code class="lang-auto">model = torch.jit.load(path2)
</code></pre>
<p>And a long boot process begins, which takes at least 30GB of RAM.</p>
<p>NAME=“Ubuntu” VERSION=“18.04.6 LTS (Bionic Beaver)” python 3.9.10 64-bit torch 1.13.1</p>

Piyush_Raikwar: <p>The same issue on almalinux9 with torch=2.2.0. Have you found a solution to this?</p>
Understanding of GPU memory accumulation

David_Yao: <p>I am facing an issue with GPU memory accumulating in each for loop but I am not very sure why. I have replicated the problem in this very simple example, with a few different variations.</p>
<pre><code class="lang-auto">import torch

a = torch.zeros(10000, 3).cuda().requires_grad_() + 500

def clip():

    c = a * 5.0

    
    c[:, 2] = torch.clip(c[:, 2], 0, 2)               # 1

    # d = torch.clip(c[:, 2], 0, 2)                     # 2

    # e = torch.clip(c[:, 2], 0, 2).detach()            # 3

    # c[:, 2] = torch.clip(c[:, 2], 0, 2).detach()      # 4

    # c[:, 2].clip_(0, 2)                               # 5



for i in range(1000):

    clip()

    torch.cuda.empty_cache()

    torch.cuda.reset_peak_memory_stats()
    # loss_photo computation here
    print(f"Peak memory usage: {torch.cuda.max_memory_allocated() / 1e6}MB")
</code></pre>
<p>Using option 1, the gpu memory accumulates across the for loop.</p>
<p>All the other options does not lead to gpu memory accumulations. I do not understand why this is happening? Since I have no reference to variable c after each function call, shouldnt it go out-of-scope and free the memory from the tensor that is created from torch.clip?</p>
<p>My hunch is that it has something to do with the computational graph, because <span class="hashtag-raw">#4</span> does not lead to memory accumulation, though I am not very sure of the actual explanation.</p>
<p>Will love to hear some insights or explanation as to what is going on here! Thank you so much!</p>

ptrblck: <blockquote>
<p>Since I have no reference to variable c after each function call, shouldnt it go out-of-scope and free the memory from the tensor that is created from torch.clip?</p>
</blockquote>
<p>No, since the first option assigns the output to <code>c</code> in an inplace operation and this accumulates the computation graph.</p>
Support for Gradient Computation on Int Tensors in Multi-GPU and Distributed Training

yanghaojin: <p>Hello, thank you for PyTorch, such a powerful tool!</p>
<p>Background Information: Our team is dedicated to extending the boundaries of PyTorch to lower bit depths. Therefore, we established the <a href="https://github.com/GreenBitAI/bitorch-engine/" rel="noopener nofollow ugc">BitTorch Engine</a> project, which includes components and GPU kernel implementations for 4, 2, and 1-bit scenarios. For low-bit components like the QLinear layer, we have implemented the forward and backward passes based on <code>autograd.function</code> (code example: [<a href="https://github.com/GreenBitAI/bitorch-engine/blob/main/bitorch_engine/layers/qlinear/binary/cutlass/layer.py#L20" rel="noopener nofollow ugc">link</a>]. Additionally, we have developed an optimizer called DiodeMix (bitorch-engine/blob/main/bitorch_engine/optim/diode_beta.py#L37) that can train these low-bit components (our paper “Enhancing Optimization Robustness in 1-bit Neural Networks through Stochastic Sign Descent” has been accepted by ECCV 2024).</p>
<p>The Problem: We are currently facing a limitation where PyTorch restricts gradient computation to tensor types of float and complex, and does not support Int types for gradient computation. In the BitTorch Engine, we have made some adaptations at the Python code level, allowing us to compute gradients for Int tensors on a single machine with a single GPU. However, when it comes to multi-GPU and distributed training, we are still constrained by this limitation.</p>
<p>We are hesitant to modify the underlying C++ code directly, but we plan to expand our current work to multi-GPU and distributed scenarios. We would greatly appreciate your help and advice on how to relax the constraint on gradient computation for Int tensors without affecting other functionalities. Thank you very much!</p>
Inference cost more time when input shape changed

246yf87953246: <p>I run the resnet18 many times, when input shape is fixed，such as (1, 3, 800, 800), it cost about 2~3ms on RTX4090. But when shape changed each time, the inference cost about 10+ms.</p>
<p>pytorch2.4, cuda11.8, cudnn8.7</p>

ptrblck: <p>Did you make sure to synchronize the code if host timers are used? If so, are you using <code>torch.compile</code> or <code>torch.backends.cudnn.benchmark = True</code>?</p>
Using torch::jit::load in Xcode for C++

Indrajit: <p>Hi,<br>
I have trained a model in PyTorch, traced it and saved it with traced_script_module.save(). Now I want to import the traced model in my XCode C++ code. I tried <span class="hashtag-raw">#include</span> &lt;torch/script.h&gt; and supplied the relevant path in Build Settings. I also download libtorch and added its path in library search paths. Yet I receive a linker error.<br>
Is there a step by step guide to load and evaluate traced PyTorch model in C++?</p>

jakkes12: <p>Having the same issue. Did you figure it out?</p>

BiggestZocker: <p>Try<br>
<code> #include &lt;torch/csrc/jit/serialization/import.h&gt;</code><br>
If that works, you may look here <a href="https://discuss.pytorch.org/t/cmake-list-for-libtorch-and-aten/204509/2" class="inline-onebox">Cmake List for Libtorch and ATen - #2 by BiggestZocker</a> what the problem was and how to avoid it.</p>

Indrajit: <p>Hi. Apologies for the late follow up. I was in other project. I tried including the header and I get the same linker error again. " undefined Symbols.<br>
Linker command failed with exit code 1 (use -v to see invocation) ".<br>
What is the solution to this?</p>
BatchNormalization vs. LayerNormalization

L_Z: <p>From my understanding, batch normalization means normalize over all batches for each channel each time where as layer normalization is being used to normalize over all channels. But I got confused why layer normalization is more being used in language model whereas batch is more in CNN. Because in my case, I wanna to extract correlated features between the channels of my dataset, I am not sure which type of normalization I should be using. From my understanding, it makes more sense to use layer normalization, since I can normalize over all channels instead of all batches which I am trying to avoid (do not want the model to be confused that there are certain relationship of the sequences in a batch).</p>
Setting the running mean and running var of unet segmentation model to None while in inference mode

s_c21: <p>I have a pre-trained unet segmentation model trained on a particular image dataset. The model can not segment well on a new dataset that comes from the same domain but differs in quality.</p>
<p>But when I set the <strong>moving_mean=None and moving_var=None in the batch norm layer</strong> of the model, it starts to give good segmentation results on this new dataset in evaluation/inference mode.</p>
<p>What does this mean? Can we use this modification of the model during deployment?</p>
SDPA backend routes requirement

Sohaib_Ahmed: <p>What are the input requirements that tells <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">SDPA</a> to use memory-efficient, flash-attention or math backend. I saw some repo to use <code>is_causal=True</code> instead of attetnion_mask in order to dispatch to flash-attention backend but still where are these requirements documented?<br>
Also Is SDPA flash-attention implementation is same as original <a href="https://github.com/Dao-AILab/flash-attention" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Dao-AILab/flash-attention: Fast and memory-efficient exact attention</a></p>
Help to create such a neural network using nn.ModuleList

mariamaincra09: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09340c448d4bcf904f346a4ce38cec274b491610.png" data-download-href="https://discuss.pytorch.org/uploads/default/09340c448d4bcf904f346a4ce38cec274b491610" title="2024-03-31_21-13-11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09340c448d4bcf904f346a4ce38cec274b491610_2_199x500.png" alt="2024-03-31_21-13-11" data-base62-sha1="1jpObiqxarH8OJ95dXLSDqmGLUQ" width="199" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09340c448d4bcf904f346a4ce38cec274b491610_2_199x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09340c448d4bcf904f346a4ce38cec274b491610.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09340c448d4bcf904f346a4ce38cec274b491610.png 2x" data-dominant-color="4B493F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2024-03-31_21-13-11</span><span class="informations">279×701 42.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t really understand how to write such a model using ModuleList. Please help me</p>

ptrblck: <p>The model definition does not seem to be complete. However, you can simply add the mentioned modules to the <code>nn.ModuleList</code> and iterate it. Something like this should work:</p>
<pre data-code-wrap="python"><code class="lang-python">modules = nn.ModuleList([
    nn.Linear(..., 19),
    nn.ReLU(),
    ...
])

for module in modules:
    x = module(x)
</code></pre>
Can I get the best of Torchscript and Torch Compile?

Jacob_Jensen: <p>We’ve implemented support in a <a href="https://developer.nvidia.com/triton-inference-server" rel="noopener nofollow ugc">triton inference container</a> for torchscript-serialized model. However, we’d like to be able to take advantage of torch.compile when training. I’m having a hard time finding docs that cover both - is there a to use torch.compile when training and then serialize? Thanks.</p>