akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Not backpropogating through neighbor examples in graph network backbone

_Sandeep: <p>I have a graph neural network which has an architure that is roughly:<br>
Backbone → Graph network.<br>
In order to compute an embedding for neighbor nodes, I have to pass them through the backbone, however I don’t want to update the backbone based on gradients from neighbors–only the targets. Is there a way to achieve this?</p>
<p>I think the most simple approach would be to run a forward on the backbone on neighbor nodes and then just pass in the embeddings, but that seems a bit clunky.</p>

soulitzer: <p>If you want to backward to a particular set of parameters, you can specify those parameters using the <code>inputs=</code> parameter <code>.backward()</code>.</p>

_Sandeep: <p>Yeah that works, but then I have to do more work in the backwards method (like backpropping through groups of params individually, which can get annoying and is hard to do with lightning).</p>
Masked_scatter_ problem

vulkomilev: <p>When I try to use this function <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.masked_scatter_.html#torch.Tensor.masked_scatter_" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.masked_scatter_ — PyTorch 2.1 documentation</a> like so</p>
<pre><code>  local_add = input[0:67, 0:32, 160:192]
    zeros = torch.zeros_like(input)
    zeros[0:67, 0:32, 160:192] = local_add

    mask = torch.zeros(67,32*10,32*10).bool()
    #mask[0:67, 0:32,160:192] = True
    #zeros = zeros.cuda()
    mask = mask.cuda()
    self.memory.memory_vb.masked_scatter_(mask,zeros)
</code></pre>
<p>I am getting an error :<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [67, 320, 320]], which is output 0 of MaskedScatterBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)</p>

ptrblck: <p>Try to replace the inplace <code>tensor.masked_scatter_</code> op with <code>torch.masked_scatter</code> and assign the result to your attribute.</p>
What is the output format of DETR?

Mahdi_Amrollahi: <p>I am going to fine-tune DETR on my dataset and I need to add some additional augmentation using albumentations. The question is, how should I prepare the targets format in <code>__getitem__</code>? Should it be in yolo, coco or pascal_voc format?<br>
The original dataset uses coco format like <code>[xmin, ymin, w, h]</code> but I saw in dataset format that it converts to normalized <code>[xmin,ymin,xmax,ymax] </code> like:</p>
<pre><code class="lang-python">boxes[:, 2:] += boxes[:, :2]
</code></pre>
<p>But is post processing, I see a function that converts <code>[cx,cy,w,h]</code> to <code>[xmin,ymin,xmax,ymax]</code>?</p>
<pre><code class="lang-python">def box_cxcywh_to_xyxy():
   ...
</code></pre>
<p>So, in which format should I prepare the target bboxes so that they compare to predicted bboxes?</p>
Weird variation of computation times across platforms/machines

Xavier31: <p>Hi. I am benchmarking a jit model that we want to integrate into our production pipeline. While this model has suitable performance among the different platforms that we use for developing/testing, it has pretty bad performance on the target machine (AWS G4dn.xlarge, GPU is Tesla T4)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20.png" data-download-href="https://discuss.pytorch.org/uploads/default/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20" title="Capture d’écran de 2023-10-23 17-15-28"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20_2_690x255.png" alt="Capture d’écran de 2023-10-23 17-15-28" data-base62-sha1="dNHGXpZTMlzARYcKvhySykUqKbe" width="690" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20_2_690x255.png, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60baa19ccf2c4b68c1b66b9bbb7b1468c7dc0d20.png 2x" data-dominant-color="24282C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran de 2023-10-23 17-15-28</span><span class="informations">823×305 20 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am testing the same model file in 3 configurations:</p>
<ul>
<li>C++ standalone : dummy C++ execution, load the model and some random inputs</li>
<li>C++ integrated : the model running in the complete application</li>
<li>Torchscript : dummy execution in python, with <code>torch.jit.load</code></li>
</ul>
<p>What I do not understand is that this bad performance happens only in the integrated version on the EC2.<br>
The C++ version are compiled locally on my PC and copied to the EC2<br>
Times are consistent across executions. Times are averaged on 100 iterations, excluding the 2 first, where optimization happens. And yes, I am timing after sending the result <code>.to(torch::kCPU)</code>, so this is not a CUDA synchronisation problem.<br>
I am using libtorch 2.0.0<br>
Any ideas as to where this might come from ? how can I debug this ?</p>
How exactly to use torch.compile()

roshan_maur: <p>Hi, I have recently started using Pytorch and I have stumbled into a problem. So i have a model that is already fine-tuned and I upgraded my torch version to 2.x. Now, I am looking to add torch.compile() in the codebase but I only want to add compile during the inference mode.</p>
<p>the model that is being used is already trained and in production and would just want to add the compile method during the inference mode to leverage the Pytorch 2.x magical superpowers. So, I wanna clear a couple of doubts here.</p>
<ol>
<li>is torch.compile meant to be used for training loops to reduce the training time or it can be used directly in inference mode?</li>
<li>autograd calculates gradients in the backward pass and during the inference mode, we dont calculate gradients so if we do use torch.compile during inference mode how does it work in the backend?</li>
</ol>
<p>Thanks for your help!</p>

soulitzer: <ol>
<li>torch compile can also run in inference mode</li>
<li>we won’t try to trace and compile the graph used for backward in the inference case then</li>
</ol>
Is the index_add_ function differentiable?

lizhaohu: <pre><code>    verts_normals = torch.zeros_like(cornea_vertex)
    vertices_faces = cornea_vertex[face_index]

    faces_normals = torch.cross(
        vertices_faces[:, 2] - vertices_faces[:, 1],
        vertices_faces[:, 0] - vertices_faces[:, 1],
        dim=-1,
    )
    unit_faces_normals = safe_normalize(faces_normals)
    verts_normals.index_add_(0, face_index[:, 0], unit_faces_normals)
    verts_normals.index_add_(0, face_index[:, 1], unit_faces_normals)
    verts_normals.index_add_(0, face_index[:, 2], unit_faces_normals)
</code></pre>
<p>The function outputs <code>verts_normals</code> , and <code>unit_faces_normals</code> is differentiable. Will the above operation prevent overall gradient backpropagation?</p>

soulitzer: <p>yes, index_add_ is differentiable</p>
Nested lists of Tensors to Tensor

sevagh: <p>Hello. I have a deeply nested list of tensors. It’s returned by an external library which was previously numpy based, which I modified to convert numpy to torch:</p>
<pre><code class="lang-auto">a = list(self.nsgt.forward((x,)))

print(type(a))
print(len(a))
print(type(a[0]))
print(len(a[0]))
print(type(a[0][0]))
print(len(a[0][0]))
print(type(a[0][0][0]))
print(len(a[0][0][0]))
print(a[0][0][0].device)
</code></pre>
<p>Results in:</p>
<pre><code class="lang-auto">class 'list'
59
&lt;class 'list'&gt;
2
&lt;class 'list'&gt;
126
&lt;class 'torch.Tensor'&gt;
304
cuda:0
</code></pre>
<p>I would like to convert this into a tensor with the following shape:</p>
<pre><code class="lang-auto">(59, 2, 126, 304)
</code></pre>
<p>Previously, when this was an np ndarray, achieving what I needed was trivial:</p>
<pre><code class="lang-auto">A = np.asarray(a)
print(A.shape)
print(A.dtype)
</code></pre>
<p>This would result in the correct ndarray:</p>
<pre><code class="lang-auto">(59, 2, 126, 304)
float32
</code></pre>
<p>In torch, I’m having trouble achieving the same with <code>torch.tensor</code> or <code>torch.stack</code>.</p>
<p>torch.tensor issues:</p>
<pre><code class="lang-auto">A = torch.tensor(a)
ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>torch.stack issue:</p>
<pre><code class="lang-auto">A = torch.stack((a))
TypeError: expected Tensor as element 0 in argument 0, but got list
</code></pre>

tom: <p>For PyTorch you would need to use nested stacks or a for loop</p>
<pre><code class="lang-auto">t = torch.stack([torch.stack([torch.stack(l2, dim=0) for l2 in l1], dim=0) for l1 in a], dim=0)
</code></pre>
<p>or somesuch (I might have gotten the brackets wrong). But this will copy the tensors several times, so an alternative is just allocate and copy in a loop:</p>
<pre><code class="lang-auto">A = torch.empty(len(a), len(a[0]), len(a[0][0]), a[0][0][0].size(0))
for i, aa in enumerate(a):
  assert len(aa) == A.size(1)
  for j, aaa in enumerate(aa):
    assert len(aaa) == A.size(2)
    for k, aaaa in enumerate(aaa):
      assert len(aaa) == A.size(3)
      A[i, j, k] = aaaa
</code></pre>
<p>This will have a lot of looping but only copies data once. Once you do something interesting with the data, you might not care as much about the Python overhead of this even if you heard that Python loops are slow. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Tomas</p>

sevagh: <p>Perfect. I ended up using the for loop solution.</p>
<p>I thought of doing something similar, but I had the impression that tensors were immutable.</p>

OverLordGoldDragon: <p>A performant solution here: <a href="https://stackoverflow.com/a/77351862/10133797" class="inline-onebox" rel="noopener nofollow ugc">python - converting list of tensors to tensors pytorch - Stack Overflow</a></p>
Libc10.a has the symbols but I sill get linker errors

Bilahari_Akkiraju: <p>pytorch2.1.0</p>
<p>Building static c++ libtorch and co libraries from source code using cmake inside a ubuntu20.04 based arm64 docker container. The libs build fine, but when I try  to link to my. simple application code, I get a bunch of undefined reference linker errors from libc10 related symbols. When I do a nm -C libc10.a | grep , I find it. Only thing is its mangled, is linker looking for unmangled exact symbol matches?. Appreciate any help. thanks</p>

Bilahari_Akkiraju: <p>Its the link order, I had to link to libtorch.a and libtorch_cpu.a and then link to libc10.a plus other static libs for my program to compiel and link correctly. The link order matters.</p>
<p>But now I am stuck with run time error, linking libtorch_cpu.a with --whole-archive linker option did not help to get past the run time error.</p>
How do you transform validation data in PyTorch?

Xuei: <p>When I was looking for tutorials on how to perform transformations, I noticed that some people do transformations like this: first, they resize, and then they center crop.</p>
<pre><code class="lang-auto">val_transform = transforms.Compose([
    transforms.Resize(size=(230)),
    transforms.CenterCrop(img_size),
    transforms.ToTensor()
])
</code></pre>
<p>Is it necessary to do it like this, or can we just resize?</p>
<pre><code class="lang-auto">train_transform = transforms.Compose([
    transforms.RandomResizedCrop(img_size, scale=(0.7, 1)),
    transforms.RandomHorizontalFlip(),  
    transforms.RandomVerticalFlip(),    
    transforms.RandomRotation(7),
    transforms.ToTensor()
])

val_transform = transforms.Compose([
    transforms.Resize(size=(img_size)),
    transforms.ToTensor()
])
</code></pre>
How to obtain the corresponding relationship between forward and backward from a profile file

lk137095576: <p>I profile a model with train model ()and get the profile file  xxx.json</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(enabled=True, use_cuda=False, record_shapes=False, profile_memory=False) as prof:
</code></pre>
<p>I get the file, in the backward part, there are many kernels, but I don’t know the corresponding forward operator for it.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d730e514ca6e9aa026eae137df0f67704bb55e59.png" data-download-href="https://discuss.pytorch.org/uploads/default/d730e514ca6e9aa026eae137df0f67704bb55e59" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d730e514ca6e9aa026eae137df0f67704bb55e59_2_690x300.png" alt="image" data-base62-sha1="uHFpB5Z2k0KC1bSWy0wYTnzSv69" width="690" height="300" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d730e514ca6e9aa026eae137df0f67704bb55e59_2_690x300.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d730e514ca6e9aa026eae137df0f67704bb55e59_2_1035x450.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d730e514ca6e9aa026eae137df0f67704bb55e59_2_1380x600.png 2x" data-dominant-color="EAE5ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2707×1179 279 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Alex Settle integrated sequence IDs into the profile as described in <a href="https://github.com/pytorch/pytorch/issues/66105">this RFE</a>, which landed in <a href="https://github.com/pytorch/pytorch/pull/70264">this PR</a>. He also followed up for AOTAutoGrad <a href="https://github.com/pytorch/pytorch/issues/102375">here</a>, which might be useful for your workflow.</p>

lk137095576: <p>Much thanks! !  that loosk like which I was finding, I will try.</p>
Help in understanding the dispatch structure of operators

lucasreis: <p>Hi!</p>
<p>I’m doing a project that focuses on runtime recompilation for approximate computing and I have a proof-of-concept for C++ front-ends using LLVMs JIT compilation. I’m looking into expanding my approach to target object recognition frameworks and I’ve come across lots of frameworks that are built around Pytorch.</p>
<p>In short, I’m looking into a way to, from an arbitrary application at runtime, recompile Pytorch operators with different levels of optimization. I’m looking into ways to bind my framework into these operators but I’m finding the structure of Pytorch’s dispatch pretty complex. <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" rel="noopener nofollow ugc">This excellent blog post</a> clarified some things for me, but I’m still having trouble understanding where the dispatch routines are located on the source.</p>
<p>Given this background, my questions are:</p>
<ul>
<li>
<p>Given an arbitrary operator (i.e. <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">Linear</a>), what is the path in the source code from the front-end Python API up to its kernel call in C++?</p>
</li>
<li>
<p>Is it doable to hack the source and simplify the dispatch structure (at C++ level) if I’m only interested in a specific device for a proof-of-concept? i.e. always jumping directly to the CPU version of <code>conv2d</code>?</p>
</li>
</ul>
<p>The reason for the second question is that my proof-of-concept requires executing directly from LLVM IR, which is trivial to produce from C++ source, but I have no idea how to target Python’s front-end to replicate the same conversion.</p>
<p>Any help would be appreciated, especially in deciphering the structure of Pytorch’s dispatch so I know where to start.</p>
<p>Thanks in advance!</p>
Why user should specify cond function for conditional control flow in PyTorch 2.0?

redradist: <p>Recently, I have read torch.export tutorial, and I am wondering, why if currently PyTorch 2.0 uses Frame Capture Feature of CPython, we still need to use special functions for exporting code (like from functorch.experimental.control_flow import cond) ??<br>
Isn’t it would be the same problem as was with torch.script ?<br>
We’ll duplicate code and decrease readability !!</p>
<p>Why torch.export cannot do it by itself using Frame Capture Feature of CPython ??</p>

pkubik: <p>I’m not sure what is the rationale in Pytorch, but TensorFlow went this route and it’s not as clean as one could expect. Whenever TF encounters a conditional, it looks whether it’s a tensor or python variable to treat it as either variable or “compilation constant” respectively. For tensors it generates a cond blocks in the DAG like with <code>torch.jit.script</code>, but for constants it just traces the selected branch like with <code>torch.jit.exoprt</code>.</p>
<p>Seems simple enough when you have just one function, but once you consider a whole program, where exported functions are calling other exported functions, it becomes quite difficult to track what behavior would be triggered in each situation and some corner cases get really tricky. In those cases, you could wish you had a mechanism to force function to be either traced or scripted.</p>
<p>tl;dr - it’s probably a design decision.</p>

redradist: <p>I think it has much simpler explanation: <code>torh.compile</code> is not fully support all the corner cases and this particular comer case was not handled</p>

pkubik: <p>Maybe, but it doesn’t look like a corner case to me. Conditional statements are rather basic constructs.</p>
Change loss for BCEWithLogitsLoss() and model return nans

Daria_Mishina: <p>Due to some issues with cuda it was need to change loss from BCELoss() to BCEWithLogitsLoss() for GAN. And all my preds and loss value became nan<br>
comment and uncomment sigmoid lay in Discriminator did not help(<br>
Could you be so kind to advice where is an issue?</p>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.model = nn.Sequential(<br>
nn.Linear(9, 256),<br>
nn.ReLU(),<br>
nn.Dropout(0.3),<br>
nn.Linear(256, 128),<br>
nn.ReLU(),<br>
nn.Dropout(0.3),<br>
nn.Linear(128, 64),<br>
nn.ReLU(),<br>
nn.Dropout(0.3),<br>
nn.Linear(64, 1),<br>
# nn.Sigmoid() <span class="hashtag-raw">#comment</span> this due to change loss<br>
)</p>
<pre><code>def forward(self, x):
    output = self.model(x)
    return output
</code></pre>
<p>discriminator = Discriminator().to(DEVICE)</p>
<p>class Generator(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.model = nn.Sequential(<br>
nn.Linear(9, 16),<br>
nn.ReLU(),<br>
nn.Linear(16, 32),<br>
nn.ReLU(),<br>
nn.Linear(32, 9))</p>
<pre><code>def forward(self, x):
    output = self.model(x)
    return output
</code></pre>
<p>generator = Generator().to(DEVICE)</p>
<p>lr = 0.001<br>
loss_function = nn.BCEWithLogitsLoss() <span class="hashtag-raw">#NEW</span></p>
<p>optimizer_discriminator = torch.optim.Adam(discriminator.parameters(), lr=lr) <span class="hashtag-raw">#optimizer</span><br>
optimizer_generator = torch.optim.Adam(generator.parameters(), lr=lr)</p>
<p>for n, (real_samples, _) in enumerate(tqdm(train_loader)):<br>
real_samples_labels = torch.ones((batch_size, 1))<br>
latent_space_samples = torch.randn((batch_size, 9))<br>
generated_samples = generator(latent_space_samples.cuda())<br>
generated_samples_labels = torch.zeros((batch_size, 1))<br>
all_samples = torch.cat((real_samples.cuda(), generated_samples))<br>
all_samples_labels = torch.cat(<br>
(real_samples_labels, generated_samples_labels))</p>
<pre><code>    discriminator.zero_grad()
    output_discriminator = discriminator(all_samples.cuda()) 
    loss_discriminator = loss_function(
        output_discriminator, all_samples_labels.cuda())
    loss_discriminator.backward()
    optimizer_discriminator.step()

    # Данные для обучения генератора
    latent_space_samples = torch.randn((batch_size, 9)) 

    generator.zero_grad()
    generated_samples = generator(latent_space_samples.cuda())
    output_discriminator_generated = discriminator(generated_samples.cuda())
    loss_generator = loss_function(
        output_discriminator_generated, real_samples_labels.cuda())
    loss_generator.backward()
    optimizer_generator.step()
</code></pre>
<p>print(f"Loss D.: {loss_discriminator.item()}“)<br>
print(f"Loss G.: {loss_generator.item()}”)</p>
<p>torch.save(discriminator, ‘/home/GAN/GAN_models/Discriminator4.pth’)<br>
print(f’Model saved.')</p>
Flash Attention in Transformer API

AleTL: <p>Hello,</p>
<p>I have a super small model that I’m using for benchmarking how fast a model with different tricks is:</p>
<pre><code class="lang-auto">import torch
import pytorch_lightning as pl
import torch.nn as nn
import time

class TransformerModel(pl.LightningModule):
    def __init__(self, dim_model, nheads, dim_feedforward, nlayers, batch_first, dropout):
        super(TransformerModel, self).__init__()
        self.encoder_layer = nn.TransformerEncoderLayer(
            d_model=dim_model, 
            nhead=nheads, 
            dim_feedforward=dim_feedforward, 
            batch_first=batch_first, 
            dropout=dropout, 
            layer_norm_eps=1e-12
        )
        self.encoder = nn.TransformerEncoder(encoder_layer=self.encoder_layer, num_layers=nlayers, enable_nested_tensor=False)

    def forward(self, x):
        # with torch.backends.cuda.sdp_kernel(
        #     enable_flash=True, enable_math=True, enable_mem_efficient=True
        # ): 
        x = self.encoder(x)
        return x

# Define a function to benchmark the forward pass time
def benchmark_forward(model, input_tensor, num_iterations=100):
    model.eval()
    total_time = 0.0
    with torch.no_grad():
        for _ in range(num_iterations):
            start_time = time.time()
            output = model(input_tensor)
            end_time = time.time()
            total_time += end_time - start_time

    average_time = total_time / num_iterations
    print(f"Average forward pass time: {average_time} seconds")


dim_model = 512
nheads = 8
dim_feedforward = 2048
nlayers = 6
batch_first = True
dropout = 0.1

# Initialize the model
model = TransformerModel(dim_model, nheads, dim_feedforward, nlayers, batch_first, dropout)

# Create a sample input tensor (adjust the shape and data as needed)
input_tensor = torch.randn(6, 1000, dim_model)  # Batch size of 32, sequence length of 10

# Benchmark the forward pass
benchmark_forward(model, input_tensor)
</code></pre>
<p>The point is that I want to use Flash Attention to make my model faster. However, in the documentation of Pytorch 2.0 it appears (<a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" class="inline-onebox" rel="noopener nofollow ugc">TransformerEncoderLayer — PyTorch 2.1 documentation</a>) that Flash Attention is used uniquely during inference, not at training time. Hence, my question is, how can I leverage Flash Attention using the Transformer API of Pytorch? Is it not possible?</p>
<p>They also highlight the benefit of using <code>enable_nested_tensor=True</code>, even though I found a bug (already reported) and cannot set it to True.</p>
<p>Finally, they also advocate for not using that much attention_mask but in the case of the causal mask. Hence, is it common not using attention mask for the PAD tokens?</p>
<p>Thank you so much!!</p>
Generator feed an image

jS5t3r: <p>Hi,</p>
<p>In diffusion models generators are used to generate samples. E.g. from OpenAI <a href="https://github.com/openai/consistency_models/blob/e32b69ee436d518377db86fb2127a3972d0d8716/cm/karras_diffusion.py#L381" rel="noopener nofollow ugc">https://github.com/openai/consistency_models/blob/e32b69ee436d518377db86fb2127a3972d0d8716/cm/karras_diffusion.py#L381</a></p>
<p>I am wondering if I can feed an image as initialization to this generator?<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator</a></p>
<p>Where can I find more details about this generator like the source code?</p>

ptrblck: <aside class="quote no-group" data-username="jS5t3r" data-post="1" data-topic="190507">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/js5t3r/48/40545_2.png" class="avatar"> jS5t3r:</div>
<blockquote>
<p>I am wondering if I can feed an image as initialization to this generator?</p>
</blockquote>
</aside>
<p>I don’t understand how this should work and you can read more about pseudorandom number generators e.g. <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">here</a>.</p>

J_Johnson: <p>As far as I know, the torch.Generator class is the equivalent of NumPy Generator, a class used for generating random numbers.</p>
<p><a href="https://numpy.org/doc/stable/reference/random/generator.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://numpy.org/doc/stable/reference/random/generator.html</a></p>
<p>Regarding initializing with an image, this is possible, but to get differentiated results, you may want to combine that image with some random noise. Here is one function you can use to do that:</p>
<pre><code class="lang-auto">def image_scale_rand(image: torch.tensor, scale):
    # image should be converted into a torch tensor already and normalized between 0 and 1
    # scale is a number between 0 and 1, where 0 is just the image and 1 is noise
    return image * (1 - scale) + torch.rand_like(image) * scale
</code></pre>
How can I use multiple MIG instances with PyTorch?

tunahanertekin: <p>Hi,</p>
<p>I’m trying to run distributed training on a cloud instance that has A100 80Gb NVIDIA GPUs in MIG mode. I have 7 MIG instance with type “1g.10gb”. I assigned multiple MIG instances to a container (using NVIDIA’s Kubernetes device plugin) and checked that if the instances are visible when running <code>nvidia-smi</code> and saw the correct MIG instances.</p>
<p>My goal is to run distributed training on multiple MIG instances using YOLOV7 (<a href="https://github.com/WongKinYiu/yolov7#training" rel="noopener nofollow ugc">this section</a>). The problem is that the functions under <code>torch.cuda</code> do not seem to detect MIG instances as seperated GPU devices. And when I run the training using one instance (I can pass <code>0</code> or UUID <code>MIG-XYZ</code> as <code>--device</code> argument), training starts however it throws errors when I try to pass <code>0,1</code> or <code>MIG-XYZ,MIG-ABC</code> as <code>--device</code> argument and cannot perform distributed training.</p>
<p><strong>Is there any configuration that I can run distributed training on multiple MIG instances?</strong> Any kind of help would be appreciated.</p>

ptrblck: <p>You cannot use multiple MIG slices in a distributed setup as described also <a href="https://discuss.pytorch.org/t/parallel-training-with-invidia-migs/159445/2">here</a>.</p>
What is the difference between BCELoss() and CrossEntropyLoss()?

yuanfenghuang767: <p>I set batch_size to 32, and the following are input and labels</p>
<pre><code class="lang-auto">torch.Size([32, 3, 256, 256])
tensor([0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1,1, 0, 1, 1, 1, 1, 0, 0])
</code></pre>
<pre><code class="lang-auto">from torchvision.models import Swin_B_Weights
net = models.swin_b(weights=Swin_B_Weights.DEFAULT)
num_ftrs = net.head.in_features
net.head = nn.Linear(num_ftrs,2)
criterion_train = nn.BCELoss()
</code></pre>
<p>I try to use SwinTransformer to complete a two-classification task. When I use BCELoss, I get a **ValueError: Using a target size (torch.Size([32])) that is different to the input size (torch.Size([32, 2])) is deprecated. Please ensure they have the same size.**But the program runs normally when I use CrossEntropyLoss.So I’m confused what should I do if I want to use BCELoss?</p>

J_Johnson: <p>For BCELoss, that is <em>binary</em> cross entropy, you only need 1 model output per target. This is because binary probabilities can simply be represented by one value from between 0 to 1.</p>
<p>Contrast that with, say, 10 possible labels in CIFAR10. There you need 10 outputs to see which the model is “guessing” as the correct answer.</p>
<p>By the way, for numerical stability during training, it is recommended to use the raw logits(i.e. no Sigmoid final activation) with BCEWithLogitsLoss. See here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" class="inline-onebox" rel="noopener nofollow ugc">BCEWithLogitsLoss — PyTorch 2.1 documentation</a></p>
Getting the Jacobian of transformer encoder outputs with respect to the inputs

Saksham_Singh: <p>Hi</p>
<p>Suppose I provide an input to a transformer encoder with batch_first = True,  I want the Jacobian of a row of the output with respect to a row of the input. Suppose the shape of the input tensor I, is (B, M, D) and the output tensor is O, where B is the batch size, how can I get the Jacobians of O[ :, i] with respect to I[ :, j]. The output should be B Jacobians, each of size D x D. Using indexing seems to break the computation graph and so I get a None grad, and using jacfwd to get a 6D tensor seems like it would be too slow. How can this be done efficiently?</p>

soulitzer: <p>Indexing shouldn’t break the computation graph, does something like the following work?</p>
<pre><code class="lang-auto">input = &lt;original input&gt;
def fn(sliced_input):
    inner_input = input.clone()[:, j].copy_(sliced_input)
   output = fn(inner_input)
   return output[:, i]

jacobian(fn)(input[:, j])
</code></pre>

Saksham_Singh: <p>You’re right, the computation graph doesn’t break, I’ve found the actual issue to be that while the input tensor has requires grad set to true, calling torch.autograd.grad with a slice of the input and allow_unused set to true, returns none, since using the slice passes a tensor with requires_grad set to false. I cannot create a copy of the slice and pass this slice to the encoder since the encoder needs the entire input to get a meaningful output. Is there a workaround for this issue?</p>

soulitzer: <p>That is a good question. There are two parts to this:</p>
<ol>
<li>You’ll want to have a wrapper function that accepts a slice and returns a slice, so that when you pass this wrapper to compute the jacobian, it would only iterate of that subset of inputs/outputs.</li>
<li>Additionally, you’d store the entire input tensor as a global and inplace copy the slice into the global inputs before using that global input tensor as the actual input to the encoder.<br>
(see my code example above for details)</li>
</ol>
Torch distributed and dataset pickle error

farakiko: <p>Hi, my dataset is a <code>tensorflow_dataset source</code> which unfortunately cannot be serialized (see image below). If I use “spawn” or “forkserver” as my torch distributed start method, I get the same error in the image when each process attempts to retrieve batches from the dataset.</p>
<p>If I use “fork” then the dataloading works but then I cannot send the tensors to CUDA (because “fork” is not CUDA supported).</p>
<p>Anyone knows how to run torch distributed and get around a dataset that cannot be serialized?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/062f7dff44e5dc1f271507d8d062e7279827ee4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/062f7dff44e5dc1f271507d8d062e7279827ee4e" title="zzz 10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/062f7dff44e5dc1f271507d8d062e7279827ee4e_2_690x182.png" alt="zzz 10" data-base62-sha1="SICddnlJtq2huY7vN9Ojl8kjRI" width="690" height="182" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/062f7dff44e5dc1f271507d8d062e7279827ee4e_2_690x182.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/062f7dff44e5dc1f271507d8d062e7279827ee4e_2_1035x273.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/062f7dff44e5dc1f271507d8d062e7279827ee4e_2_1380x364.png 2x" data-dominant-color="F5F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">zzz 10</span><span class="informations">1752×464 62 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

H-Huang: <p>replied in your other thread: <a href="https://discuss.pytorch.org/t/torch-distributed-and-num-workers-0-pickle-error/190357/6" class="inline-onebox">Torch distributed and num_workers&gt;0 pickle error - #6 by H-Huang</a></p>
Will PyTorch team some day provide the similiar API as PyTorch Lightning?

redradist: <p>Recently I’ve run into PyTorch Lightning and I am very excited about how it easier to create model, reduce boilerplate code and switch devices …</p>
<p>Will PyTorch Team consider to do something similar ?<br>
This feature I would be glad to participate in <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><a href="https://github.com/pytorch/tnt"><code>pytorch/tnt</code></a> might come close but would most likely still have differences. Also, there is <a href="https://github.com/pytorch/ignite"><code>pytorch/igite</code></a> also providing a higher level-API.</p>
<p>I’m unsure about any other plans, but what would be the benefit to redo the same work and providing the same higher-level API?</p>

redradist: <p>Unfortunately they are not so cool as PyTorch Lightning …<br>
PyTorch Lightning is much high level wrapper</p>

ptrblck: <p>And is well maintained and compatible, so I still don’t see a reason to reinvent it.</p>

redradist: <p>Oh, sometimes it make sense, for example:</p>
<ol>
<li>Remove dependency from third-party: maybe in future PyTorch Lighting become paid</li>
<li>Better maintainability</li>
<li>Maybe during reinventing wheel it will be done better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></li>
</ol>
<p>As for me I would like to see PyTorch Lighting under <a href="https://github.com/pytorch" rel="noopener nofollow ugc">PyTorch Organization</a> as part of PyTorch Foundation, but not make it part of <code>pytorch</code> repository because maybe in future somebody will implement much better high level API for PyTorch</p>
PyTorch site inconsistency

redradist: <p>Small inconsistency in PyTorch site. See displaying of TorchData and torchtext</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bf57fac7b6266fc47fa64689b5c12b5d2af6008.png" data-download-href="https://discuss.pytorch.org/uploads/default/0bf57fac7b6266fc47fa64689b5c12b5d2af6008" title="Screenshot 2023-10-23 at 16.22.44"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bf57fac7b6266fc47fa64689b5c12b5d2af6008_2_161x499.png" alt="Screenshot 2023-10-23 at 16.22.44" data-base62-sha1="1HNej4F6bXUPneOtxI8w57E6mBa" width="161" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bf57fac7b6266fc47fa64689b5c12b5d2af6008_2_161x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bf57fac7b6266fc47fa64689b5c12b5d2af6008_2_241x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bf57fac7b6266fc47fa64689b5c12b5d2af6008_2_322x998.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-10-23 at 16.22.44</span><span class="informations">500×1546 31.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think it would be more readable to have the same style for all libraries beginning with <code>torch</code>: TorchData, TorchText, TorchAudio and etc.</p>
Which pytorch version >2.0.1 support cuda 11.4

nithinexe: <p>I’m trying to finetune Mistral 7B, and I get this runtime error of unsupported Cuda drivers.<br>
RuntimeError: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver.<br>
Using torch 2.0.0 doesn’t support FSDP and requires &gt;=2.0.1<br>
torch.cuda.is_available() == False</p>

ptrblck: <p>I assume you installed a recent PyTorch binary shipping with CUDA 12.1 while your system uses an older driver which shipped with CUDA 11.4. If you don’t want to update the NVIDIA driver you could install the latest PyTorch release with CUDA 11.8 instead.</p>

nithinexe: <p>It works, I installed Pytorch==2.0.1</p>
FSDP comm kernel is blocked by cpu op

cos120: <p>hi,<br>
I use FSDP to train model with 1k gpus, I found some ranks are slow and make all ranks slow down.</p>
<p>I have some trace described as below, the view op in FSDP’s unshard is blocking the all gather kernel, it let others have to wait it.</p>
<p>But view op only change strides of tensor, it is very fast, how it happens? Does pytorch has other threads in main python progress?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15a2cc6b304ea9ae454111a923c7be229665f525.png" data-download-href="https://discuss.pytorch.org/uploads/default/15a2cc6b304ea9ae454111a923c7be229665f525" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a2cc6b304ea9ae454111a923c7be229665f525_2_431x500.png" alt="image" data-base62-sha1="35oOUehAKAOqCOF2FnJcYdko20Z" width="431" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a2cc6b304ea9ae454111a923c7be229665f525_2_431x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a2cc6b304ea9ae454111a923c7be229665f525_2_646x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a2cc6b304ea9ae454111a923c7be229665f525_2_862x1000.png 2x" data-dominant-color="DFDBDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1530×1772 260 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorch compatibile with NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2

Rmelis80: <p>Hello<br>
i’m trying to to run as pytorch-gpu the follow transfer learning code  (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>)<br>
in my Ubuntu 20.04 LTS with the the follow nvidia version (see image attached)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13854767d0684f683a7e7bb7eeb692bcb9b50dba.png" alt="nvidia" data-base62-sha1="2MGCu41gb92cAjKMgijIoz9wnpw" width="643" height="361"></p>
<p>but I encontered several incompatibilities :</p>
<p>First, the as is cuda version not load the dynamic library libcudnn.so.8. When I tried to install previous pythorch and associated cuda versions the torch library give me other errors.</p>
<p>Can you suggest me a proper associations between torch and my nvidia workstation?..Thanks in advance!</p>

ptrblck: <p>Driver 535.104.05 is compatible with every stable and nightly binary we are building.</p>
<aside class="quote no-group" data-username="Rmelis80" data-post="1" data-topic="190298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c68b51/48.png" class="avatar"> Rmelis80:</div>
<blockquote>
<p>First, the as is cuda version not load the dynamic library libcudnn.so.8.</p>
</blockquote>
</aside>
<p>I don’t understand this statement since the PyTorch binaries ship with their own CUDA dependencies (including cuDNN and NCCL etc.) and load these. Your locally installed CUDA toolkit won’t be used so if you want to use a custom setup you should build PyTorch from source.</p>

Rmelis80: <p>Hy,</p>
<p>Sorry for unclearity in my question ( I’m still a novice with torch…),  I’ll try to explain now better :</p>
<p>The error appears only  when  I run the training in the previous mentioned code ( the necessary libraries were all correctly loaded). Here the full msn :</p>
<p>“Could not load library libcudnn_cnn_train.so.8. Error: /usr/local/cuda/lib64/libcudnn_cnn_train.so.8: undefined symbol: _ZN5cudnn3cnn34layerNormFwd_execute_internal_implERKNS_7backend11VariantPackEP11CUstream_stRNS0_18LayerNormFwdParamsERKNS1_20NormForwardOperationEmb, version libcudnn_cnn_infer.so.8”</p>
<p>In particular, I dont’ understand about this error in particular ( maybe it come from the code?)<br>
" undefined symbol: _ZN5cudnn3cnn34layerNormFwd_execute_internal_implERKNS_7backend11VariantPackEP11CUstream_stRNS0_18LayerNormFwdParamsERKNS1_20NormForwardOperationEmb"</p>

ptrblck: <p>The system path points to a cuDNN installation in <code>/usr</code> so either remove this cuDNN version completely and let PyTorch use its own cuDNN libs or remove this path from <code>LD_LIBRARY_PATH</code> as a workaround.</p>

Rmelis80: <p>I see many thanks again now code runs properly</p>
Overlap backward and optimizer update

oliver_md: <p>Hi,<br>
Sorry if this has been asked before. I spent hours but don’t seem to find the answer I want.<br>
In my experiment, optimizer.step()(Adam for example) can take a lot of time and sometimes as much time as forward/backward. So it seems reasonable to me to parallelize backward and optimizer update. Is that doable with one gpu in PyTorch?(no ddp or fsdp) Would I get speedup due to parallelism or save memory because gradient can be released early? I came up with this post <a href="https://github.com/pytorch/torchrec/issues/616" class="inline-onebox" rel="noopener nofollow ugc">Unified Authoring of Overlapped Optimizers, and per parameter optimizer settings · Issue #616 · pytorch/torchrec · GitHub</a> before. It seems that DDP/FSDP does that automatically. are there blog talking about how much gain we get from using it in DDP/FSDP?<br>
Thank you!</p>

ptrblck: <p>It seems the feature landed in TorchRec, not DDP, based on the linked issue. For your standard model you could try to use the <code>foreach</code> implementation of the optimizer (if available) as it would fuse multiple of the small update kernels into one.</p>

soulitzer: <p>Yes, doing the optimizer step/accumulating as you do the backward you would avoid keeping all the gradients alive at once, and in regimes where the parameters dominate memory, e.g. when your batch size/seq len are small this could reduce memory usage.</p>
<p>Here’s a tutorial for that <a href="https://pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">How to save memory by fusing the optimizer step into the backward pass — PyTorch Tutorials 2.1.0+cu121 documentation</a></p>

oliver_md: <p>I see, that is the tutorial I was looking for. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:" loading="lazy" width="20" height="20">. Another followup question is how does it affect performance? I timed the script in google colab(Nvidia T4 gpu) and see no speedup. I even see some performance regression in other cases(different model + different GPU). Can having multiple optimizers hurt the performance? If performance is my major concern, are there ways to speedup the training by parallelizing backward and optimizer updates in PyTorch?</p>

soulitzer: <aside class="quote no-group quote-modified" data-username="oliver_md" data-post="4" data-topic="190435">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver_md/48/42031_2.png" class="avatar"> oliver_md:</div>
<blockquote>
<p>another followup question is how does it affect performance?</p>
</blockquote>
</aside>
<p>I’d say that this is mostly a memory-reducing technqiue. One way this hurts performance is that it you would no longer be getting the horizontal fusion from batching the optimizer operations. This would be mitigated if we were able to fuse the optimizer into the backward itself, but that may not be possible today.</p>
Reproducibility when calculating AUC ROC

loveis98: <p>I try to solve classification task with LSTM. For reprodusibility I set the seed values:</p>
<pre><code class="lang-auto">def seed_everything(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    np.random.seed(seed)  
    random.seed(seed) 
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True)
    os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>But after epoch I evaluate my model with the function below:</p>
<pre><code class="lang-auto">def eval_model(model, dataset_val, cat_features, numeric_features, val_ids=None, batch_size=32, device=None) -&gt; float:
    metric = BinaryAUROC(device=device) #https://pytorch.org/torcheval/stable/generated/torcheval.metrics.BinaryAUROC.html
    val_generator = batches_generator(dataset_val, cat_features, numeric_features, val_ids, batch_size=batch_size, shuffle=False,
                                      device=device, is_train=True)
    model.cuda()
    model.eval()

    for batch in tqdm(val_generator, desc='Evaluating model'):
        targets = torch.flatten(batch['label'].to(device))
        with torch.no_grad():
            output = torch.flatten(model(batch['features'], batch['group'])).to(device)
        metric.update(output, targets)
    return metric.compute().detach().cpu().numpy().flatten()[0]
</code></pre>
<p>And the error occurs:</p>
<p><code>RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.</code></p>
<p>How can I fix it? (and use <a href="https://pytorch.org/torcheval/stable/generated/torcheval.metrics.BinaryAUROC.html" rel="noopener nofollow ugc">metric</a> at the same time)</p>

loveis98: <p>Maybe there are comments?</p>