akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Warp 3D volume with grid_sample

Ecard9934: <p>I am currently struggling to use grid_sample the right way and need some help.<br>
My problem (simplified) is the following<br>
I have multiple 3D volume of the shape [3,3,3]. All values are 0, except for the value at [0,0,0], which is 1.</p>
<p>In total, I have 5 batches. The resulting shape is<br>
[5,3,3,3]<br>
According to <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.grid_sample — PyTorch 2.0 documentation</a> I add the channel with unsqueeze(1), resulting in the shape [5,1,3,3,3]</p>
<p>I also have a flow field of the shape [3,3,3,3], where the last dimension indicates the movement into x/y/z direction. Again, 5 batches, so the resulting shape is [5,3,3,3,3].</p>
<p>How can I use grid_sample in order to apply the flow and generate a warped image where the value 1 is moved into another field? What shape is expected for the grid?<br>
Right now, I don’t undestand how<br>
<code>warped = torch.nn.functional.grid_sample(base, flow, mode='bilinear', padding_mode='zeros', align_corners=True)</code> moves the values of base</p>
Use pytorch and reduce forloops of indexadd function

Ajith: <p>I want to reduce/remove the forloops used in customIndexAdd() that implements torch.index_add_() (it works only for dimension of -2 ) . Could anyone kindly help me with implementation of faster customIndexAdd()  currently it takes 35seconds to execute.</p>
<pre><code class="lang-auto">import torch
import numpy as np
import time

def customIndexAdd(x1, index, tensor):
    s1,s2,s3,s4 = tensor.shape
    output_tensor = x1
    for i in range(s1):
        for j in range(s2):
            for k in range(s3):
                output_tensor[i][j][index[k]] += tensor[i][j][k]
    return output_tensor

# Create an array of sequential numbers starting from 1
sequential_numbers = np.arange(1, 2* 2* 352798* 2 + 1)

# Reshape the array to match the desired tensor shape
tensor = sequential_numbers.reshape(2, 2, 352798, 2)
t = torch.tensor(tensor).int()

values = torch.arange(1, 352796 // 2 + 1)

repeated_values = torch.repeat_interleave(values, repeats=2)
final_values = torch.cat([torch.tensor([0]), repeated_values, torch.tensor([176399])])
index = final_values

x = torch.ones(2, 2, 176400, 2).int()
x.index_add_(-2, index, t)

x1 = torch.ones(2, 2, 176400, 2)

start = time.time()
out1 = customIndexAdd(x1, index, t)
end = time.time()
print(end - start)

print(torch.equal(x, out1))

</code></pre>

ptrblck: <p>I’m not sure I understand the use case completely, but why don’t you just use <code>index_add_</code>, which seems to work already?</p>

Ajith: <p>While converting from torch to onnx i am facing issue due to scatterElements node which raises from index_add_() . So i need to eliminate index_add_() and implement custom index_add_() . Detailed issues are here <a href="https://github.com/PINTO0309/onnx2tf/issues/447" class="inline-onebox" rel="noopener nofollow ugc">[Conv-TasNet] Facing issue in converting Conv-TasNet model · Issue #447 · PINTO0309/onnx2tf · GitHub</a> , <a href="https://github.com/pytorch/pytorch/issues/106973" class="inline-onebox" rel="noopener nofollow ugc">Facing error while using onnx from scatterelements · Issue #106973 · pytorch/pytorch · GitHub</a></p>
<p>Also for my usage dim=-2 is enough so i am implementing only for it</p>
<p>these are the lines i need to replace <a href="https://github.com/kaituoxu/Conv-TasNet/blob/master/src/utils.py#L44-L45" rel="noopener nofollow ugc">https://github.com/kaituoxu/Conv-TasNet/blob/master/src/utils.py#L44-L45</a> without scaterElements node</p>

Ajith: <p>This code takes 11 seconds but i need still faster</p>
<pre><code class="lang-auto">def customIndexAdd(x1, index, tensor):
    j = 0
    for i in range(len(index)):
        x1[:,:,index[i],:] += tensor[:,:,j,:]
        j += 1
    return x1
</code></pre>
How to change MaskRCNN loss function?

johnny69: <p>I’m using PyTorch’s <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.detection.maskrcnn_resnet50_fpn.html#torchvision.models.detection.maskrcnn_resnet50_fpn" rel="noopener nofollow ugc">MaskRCNN implementation</a>. I want to change the loss function to something custom. I tried using this:</p>
<pre><code class="lang-auto">model = maskrcnn_resnet50_fpn(weights=MaskRCNN_ResNet50_FPN_Weights.DEFAULT)
model.roi_heads.maskrcnn_loss = my_loss
</code></pre>
<p>But this doesn’t seem to work. When I include a print statement in <code>my_loss</code> it never gets called.</p>
<p>Does anyone know how to fix this?</p>

johnny69: <p>Fixed it. Instead of:</p>
<pre><code class="lang-auto">model.roi_heads.maskrcnn_loss = my_loss
</code></pre>
<p>I needed to use:</p>
<pre><code class="lang-auto">roi_heads.maskrcnn_loss = my_loss
</code></pre>
Could this neural network be a good classifier?

Mah_Neh: <p>I wonder if there is any silly mistake or anything you recommend to be changed, I’m trying to avoid fully connected layers. And the network needs to output an array of 4 numbers (predicts a class.)</p>
<pre><code class="lang-auto">import torch
from torch import nn


class Backbone(nn.Module):
    """
    Backbone (first part) of the spot rotated neural network
    It is just a set of convolutions.
    Important: The Head expects a flattened tensor.
    Args: None
    """

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=2, stride=1, padding=2)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=2, stride=1, padding=2)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=2, stride=1, padding=2)
        self.conv4 = nn.Conv2d(64, 128, kernel_size=2, stride=1, padding=2)
        self.conv5 = nn.Conv2d(128, 256, kernel_size=2, stride=1, padding=2)
        self.conv6 = nn.Conv2d(256, 512, kernel_size=2, stride=1, padding=2)
        self.relu1 = nn.ReLU()
        self.dropout1 = nn.Dropout(0.3)
        self.maxpool = nn.MaxPool2d(kernel_size=(3, 3))

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.conv1(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        x = self.conv2(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        x = self.conv3(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        x = self.conv4(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        x = self.conv5(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        x = self.conv6(x)
        x = self.relu1(x)
        x = self.maxpool(x)
        return x
</code></pre>
<pre><code class="lang-auto">class HeadV2(nn.Module):
    """
    Classification of the image rotation angle.
    Args:
        input_size: the size of the input image
    """

    def __init__(self):
        super().__init__()
        self.gap = nn.AdaptiveAvgPool2d((1, 1))
        self.fc2 = nn.Linear(512, 4)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.gap(x)
        x = x.view(x.size(0), -1)
        x = self.fc2(x)
        return x


class ClassifierV2(nn.Module):
    """
    A PyTorch neural network for image classification.

    Args:
        channels: Number of channels in the input image.
        height: Height of the input image in pixels.
        width: Width of the input image in pixels.
    """

    def __init__(self):
        super().__init__()
        self.backbone = Backbone()
        self.head = HeadV2()

    def forward(self, x):
        x = self.backbone(x)
        x = self.head(x)
        return x
</code></pre>
How to replace FCC with GlobalPooling

Mah_Neh: <p>I’ve written the network:</p>
<pre><code class="lang-auto">class ClassificationHead(nn.Module):
    """
    Classification of the image rotation angle.
    Args:
        input_size: the size of the input image
    """

    def __init__(self, input_size: int):
        super().__init__()
        self.fc1 = nn.Linear(input_size, 16)
        self.relu1 = nn.ReLU()
        self.dropout1 = nn.Dropout(0.1)
        self.fc2 = nn.Linear(16, 4)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.dropout1(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>But would like to replace the behaviour with a single GlobalPooling, I cant figure out how at all.</p>

ptrblck: <p>I assume you define <code>GlobalPooling</code> as a pooling operation returning an activation with a spatial size of <code>1x1</code>?<br>
If so, <code>nn.AdaptiveAvgPool2d(output_size=(1, 1))</code> should work:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 224, 224)
pool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
out = pool(x)
print(out.shape)
# torch.Size([2, 3, 1, 1])
</code></pre>

Mah_Neh: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="186978">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>as a pooling operation returning an activation with a spatial size of <code>1x1</code>?</p>
</blockquote>
</aside>
<p>I tried but was actually confused by the output. I guess just needs to squeeze it ?</p>

ptrblck: <p>If you want to remove the spatial dimensions with a size of <code>1</code>, then yes you can <code>squeeze()</code> the output.</p>

Mah_Neh: <p>Thank you. I’m still figuring out how to extract the maximum information without using FC layers but convs only.</p>
<p>I think the layer proposed in the post should work, but the channels dimension is still  uncontrolled I think, not sure how to go about making it <code>4</code> or the number of classes.</p>
<p>thinking of this currently:</p>
<pre><code class="lang-auto">class ClassificationHead(nn.Module):
    """
    Classification of the image rotation angle.
    Args:
        input_size: the size of the input image
    """

    def __init__(self, input_size: int):
        super().__init__()
        # self.fc1 = nn.Linear(input_size, 24)
        # self.relu1 = nn.ReLU()
        # self.dropout1 = nn.Dropout(0.1)
        self.flatten = nn.Flatten()
        self.gap = nn.AdaptiveAvgPool2d((1, 1))
        self.fc2 = nn.Linear(input_size, 4)
</code></pre>

ptrblck: <p>Using an <code>nn.Linear</code> layer sounds like a valid option or you could also apply an <code>nn.Conv2d</code> layer before the pooling layer.</p>
A way to get your networks to Mobile and the Web

Mah_Neh: <p>After digging a bit, i find this process best for finally compressing my models and getting them into a browser:</p>
<ol>
<li>Train using float32 type</li>
<li>Export the model to onnx (or to pth and later on to onnx.)</li>
<li><a href="https://onnxruntime.ai/docs/performance/model-optimizations/ort-format-models.html" rel="noopener nofollow ugc">Convert using onnx converter</a> from onnx to ort format (reduces the size to half.)</li>
</ol>
<p>I wonder if others are doing the same, or following different strategies.</p>
Gradient normalization

wsouza: <p>Hello,</p>
<p>I have a question about the gradient that is computed when we do loss.backward().<br>
Anyone know if that gradient is normalized?</p>
<p>Thank you.</p>

ptrblck: <p>No, PyTorch won’t normalize the gradient automatically behind your back.</p>

wsouza: <p>Thank you very much!</p>
Torch._dynamo.config.log_level

trusira: <p>Hi,<br>
I’m trying to generate some debug traces using torch._dynamo.config, as demo’d in <a href="https://www.youtube.com/live/1FSBurHpH_Q?feature=share&amp;t=1009" rel="noopener nofollow ugc">this</a> talk. Looks like the <code>log_level</code> attribute is no longer defined in <code>torch/_dynamo/config.py</code>.<br>
When I set <code>torch._dynamo.config.log_level = logging.DEBUG</code>, I get the following error.</p>
<pre><code class="lang-auto">File "/home/pytorch-cuda/my_scripts/model.py", line 13, in &lt;module&gt;
    dcfg.log_level = logging.DEBUG
  File "/home/pytorch-cuda/torch/_dynamo/config_utils.py", line 72, in __setattr__
    raise AttributeError(f"{self.__name__}.{name} does not exist")
AttributeError: torch._dynamo.config.log_level does not exist
</code></pre>
<p>Thanks!</p>

marksaroufim: <p>A lot changed for logging, this is what I’d suggest you poke around in <a href="https://pytorch.org/docs/main/logging.html" class="inline-onebox">torch._logging — PyTorch main documentation</a></p>

trusira: <p>Thanks for pointing me in the right direction. I found the new debug config API here(<a href="https://pytorch.org/docs/main/generated/torch._logging.set_logs.html#torch._logging.set_logs" rel="noopener nofollow ugc">https://pytorch.org/docs/main/generated/torch._logging.set_logs.html#torch._logging.set_logs</a>).</p>
How does autograd(f,model.parameters(),grad_outputs) work?

Arkaprava_Majumdar: <p>Hi, I am new to the autograd() as I have mostly been dependent on loss.backward().</p>
<p>x = torch.tensor([1.0,2.0],requires_grad = True)<br>
W = torch.tensor([[1.0,1.0],[1.5,1.2]],requires_grad = True)<br>
f1 = W@x<br>
print(torch.autograd.grad(f,W,grad_outputs = torch.tensor([1.,1.])))<br>
throws an error “One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.” Puting allow_unused = True, returns None.</p>
<p>Now, if I do this -<br>
lin = nn.Linear(2,2,bias = False)<br>
lin.weight = nn.Parameter(W)<br>
f = lin(x)<br>
print(torch.autograd.grad(f,lin.parameters(),grad_outputs = torch.tensor([1.,1.])))<br>
I get the correct output [[1.,2.],<br>
[1.,2.]]</p>
<p>So, my first question is what changes in the two cases?<br>
I initially thought that may be in the second case, the calculation is broken down to something like this -</p>
<p>x = torch.tensor([1.0,2.0],requires_grad = True)<br>
W = torch.tensor([[1.0,1.0],[1.5,1.2]])<br>
for i in range(W.shape[0]):<br>
temp_W = W[0].T<br>
temp_W.requires_grad = True<br>
autograd.grad(x,temp_W)<br>
but here, we don’t even need the grad_outputs. However, if I remove grad_outputs while calling autograd with lin.parameters(), it shows error “grad can be implicitly created only for scalar outputs”.</p>
<p>Can someone please explain what am I missing?</p>

ptrblck: <p>You are using <code>print(torch.autograd.grad(f,W,grad_outputs = torch.tensor([1.,1.])))</code> where <code>f</code> is undefined:</p>
<pre><code class="lang-python">x = torch.tensor([1.0,2.0],requires_grad = True)
W = torch.tensor([[1.0,1.0],[1.5,1.2]],requires_grad = True)
f1 = W@x
print(torch.autograd.grad(f,W,grad_outputs = torch.tensor([1.,1.])))
# NameError: name 'f' is not defined
</code></pre>
<p>Fixing it shows a valid gradient:</p>
<pre><code class="lang-python">x = torch.tensor([1.0,2.0],requires_grad = True)
W = torch.tensor([[1.0,1.0],[1.5,1.2]],requires_grad = True)
f1 = W@x
print(torch.autograd.grad(f1, W,grad_outputs = torch.tensor([1.,1.])))
# (tensor([[1., 2.],
#         [1., 2.]]),)
</code></pre>
How to reduce the size of a neural network

Mah_Neh: <p>I’ve read a couple of posts here and there but couldn’t extract much useful for my case.</p>
<p>We are trying to train a neural net that will run in the browser, so for that we use onnx to export the model after training.</p>
<p>However, the model has to be small.</p>
<p>There are two ideas I can see that could achieve this:</p>
<ol>
<li>Use half precision (still trying to work out how could we do it.)</li>
<li>Just remove layers (but  this affects the precision.)</li>
</ol>
<p>I’d welcome some pointers here as it is my first model, and i’m a bit stuck with the last steps.</p>

HELLORPG: <p>I’ve heard that someone may use <code>int8</code> to quantize their net while inference. This will greatly speed up~</p>

J_Johnson: <p>Quantization works all the way down to 4-bit currently.</p>
<p>You can also look at pruning your model: <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Pruning Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>

Mah_Neh: <p>there is so much to read <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
C compiler dependency on training performance

SminYu: <p>Hello there,<br>
I found that my model’s training performance depends on the C compiler (whether it is the Intel compiler or GCC).<br>
For my model (It uses Pytorch and Pytorch_geometric), the training loss metric is quite large and saturated early when using the Intel compiler, otherwise the model is trained well (even in different two machines but the same compiler).<br>
The difference is quite large (about 50%, usually not acceptable in the same model and hyperparameters).<br>
I wonder if it is natural behavior or my fault.</p>
<p>My environments are as below.<br>
Python 3.9.15<br>
Pytorch 1.12.1<br>
Pyg 2.2.0<br>
CUDA 10.2 / 11.6<br>
Intel compiler 19.1<br>
GCC 4.8.5 / 10.2<br>
(Two machines use different CUDA and GCC, otherwise same.)</p>

QFuser: <p>I think its naturally I mean its depends on the compiler flags and how the compiler do optimization  which can be very different in the end also include the OS .This also is true for Pytorch itself because it has also stuff which aren’t turned on necessarily(Mip, AVX certain libraries from 3th party) on top of the compiler stuff. You can improve  a lot if you have the time for it.</p>
Basic question about number of kernels used for torch.nn.Conv2d( 3, 32, 3)

seberino: <p>Consider torch.nn.Conv2d( 3, 64, 3) applied to a single 32 x 32 RGB (3 channel) image.</p>
<p>Q1: I believe that means there is a separate kernel for each of the 3 input channels? In other<br>
words, each R, G and B color has a different kernel for its own 32 x 32 grid.  So you<br>
have a triplet of kernels!?</p>
<p>Q2: Are the outputs for each of those three kernels somehow combined?  How?  I’m picture<br>
3 output grids that need to be combined into one final grid of the same size?</p>
<p>Q3: Does the 64 mean you have 64 different kernel triplets for a total of<br>
3 x 64 different kernels applied to the 32 x 32 grids?</p>
<p>Thanks,</p>
<p>Chris</p>

ptrblck: <ol>
<li>
<p>No, since you are using 64 different kernels, each using 3 input channels.</p>
</li>
<li>
<p>Each kernel create a single activation map, so your output channels will be 64.</p>
</li>
<li>
<p>You don’t have a triplet, but each kernel uses 3 input channels.</p>
</li>
</ol>

seberino: <p>Thank you very much.  So if I understand you correctly, after applying a kernel, the output will have one less number of dimensions?</p>
<p>For example, if you apply a kernel to an image with dimensions W x H x N, the output will have dimensions w &lt;= W and h &lt;= H?</p>
<p>(Assume N is the number of channels.)</p>

ptrblck: <p>No, the number of dimensions will be the same. The conv layer expects an input of <code>[batch_size, in_channels, height0, width0]</code> and creates and output of <code>[batch_size, out_channels, height1, width1]</code>, where the spatial dimension sizes are defined by the kernel size, the stride, dilation, padding, etc. as explained in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">docs</a>.</p>

seberino: <p>ptrblck</p>
<p>Thanks again.  If can please ask one more question…<br>
Using your notation, would it be correct to say the<br>
conv layer applies multiple kernels to the input<br>
where the number of kernels applied is out_channels?</p>
<p>Chris</p>

ptrblck: <p>Yes, this is the case in the default setup (not for depthwise or grouped convs) and you can verify it by applying each kernel separately:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 224, 224)
conv = nn.Conv2d(3, 10, 3, 1, 1, bias=False)

ref = conv(x)

# apply each kernel separately
out = []
for kernel in conv.weight:
    kernel = kernel.unsqueeze(0)
    print(kernel.shape)
    tmp = F.conv2d(x, kernel, bias=None, stride=1, padding=1)
    out.append(tmp)

out = torch.cat(out, dim=1)
print(out.shape)
# torch.Size([2, 10, 224, 224])

print((ref - out).abs().max())
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

seberino: <p>Wonderful!  Thanks so much.  This was really confusing me.</p>
<p>cs</p>
Pythorch: .__init__() takes 1 positional argument but 2 were given

MeredithSK: <p>Hi all,</p>
<p>i am now facing a error: .<strong>init</strong>() takes 1 positional argument but 2 were given</p>
<p>my model is defined:</p>
<p>bc = datasets.load_breast_cancer()<br>
X, y = bc.data, bc.target<br>
n_samples, n_features = X.shape</p>
<p>class LogisticRegression(nn.Module):</p>
<pre><code>def _init_(self, n_input_features):
    super(LogisticRegression, self)._init_()
    self.linear = nn.Linear(n_input_features, 1)
    
def forward(self, x):
    y_pred = torch.sigmoid(self.linear(x))
    return y_pred
</code></pre>
<p>model = LogisticRegression(n_features) —&gt; here is the problem…</p>

ptrblck: <p>It works for me:</p>
<pre><code class="lang-python">class LogisticRegression(nn.Module):
    def __init__(self, n_input_features):
        super(LogisticRegression, self).__init__()
        self.linear = nn.Linear(n_input_features, 1)
        
    def forward(self, x):
        y_pred = torch.sigmoid(self.linear(x))
        return y_pred

model = LogisticRegression(n_input_features=10)
</code></pre>
<p>after fixing the <code>__init__</code> function definition and call (note the double underscores).</p>

MeredithSK: <p>oha… that was such a stupid mistake. thanks a lot!!</p>
Does TORCHELASTIC_RESTART_COUNT get broadcasted to all workers?

f_g: <p>Hi,</p>
<p>I’m running a elastic training job with 4 workers. I have noticed that when one worker fails and causes all workers to restart, all workers restarted in process, but only the worker that failed had <code>TORCHELASTIC_RESTART_COUNT</code> incremented upon restart, while the other workers still has the restart count at 0.</p>
<p>I use this count to check when to load previous checkpoints. Should this count be reliable enough to do that during multi-worker jobs? If not what would be a better way to detect such restarts?</p>
<p>Thanks</p>
Select values from a 4D tensor based on 3D indices

power: <p>Hi there!<br>
I have a 3D tensor indices like [16,12,12] and a 4D tensor values like [16,12,128,64].(sizes may vary but dimensions are fixed)<br>
I wanted to keep values based on indices and zero out others.<br>
I can write it with for loops like the following but it’s slow and need to take advantage of torch tensors.</p>
<pre><code class="lang-auto">for i in range(0, values.size(0)):
  for j in range(0, values.size(1)):
      for k in range(0, values.size(2)):
        if k in indices[i][j]:
          q[i][j][k] = values[i][j][k]
        else:
          q[i][j][k] = torch.zeros(values.size(3))
</code></pre>
Slowdown when using multiple cuda streams due to small grid size

ladberg: <p>I have a bunch of <code>torch.mm</code>s that don’t depend on each other so I thought I’d split them up across two cuda streams to potentially speed them up as not all of them are big enough to saturate the GPU.</p>
<p>I’m running them with the following code (mms is a list of partial calls to <code>torch.mm</code>):</p>
<pre><code class="lang-auto">def parallel_run(mms, device):
    orig = torch.cuda.current_stream(device)
    s1 = torch.cuda.Stream(device)
    s2 = torch.cuda.Stream(device)
    s1.wait_stream(orig)
    s2.wait_stream(orig)

    with torch.cuda.stream(s1):
        for mm in mms[:len(mms)//2]:
            mm()
    with torch.cuda.stream(s2):
        for mm in mms[len(mms)//2:]:
            mm()

    orig.wait_stream(s1)
    orig.wait_stream(s2)
</code></pre>
<p>Everything works correctly and I can see the two halves are running in parallel according to Nsight <em>but</em> the second set of functions runs much slower than the first. Looking at Nsight, the grid sizes of the second set of kernel launches are all clamped at max 54 (half of 108, the max number of SMs). This causes a long tail of slow kernels they don’t utilize the full GPU even when the first set have finished, negating all performance gains.</p>
<p>The grid size can reach up to 324 when run in a single stream and for the kernels in the first stream.</p>
<p>I’m running torch 1.13.1, cuda 12, and using an A100.</p>
<p>Is there any way to avoid this? Thanks!</p>

ladberg: <p>I’ve found a decently good workaround that still allows me to parallelize over multiple streams: allocate only the smaller ops to the secondary stream as they wouldn’t have taken the full grid anyway.</p>
<p>Still would like a way to disable this behavior in torch though!</p>
Better solution of "backward through the graph a second time" problem?

WinKawaks: <p>Hi all, I have a specific need of my model. There is a learnable tensor – <em>a</em>, and I need some slices of it, each corresponding to a different loss function. The code is like this:</p>
<pre><code class="lang-auto">import torch

a = torch.randn(3, 3)
a.requires_grad = True
b = a[:, :2].contiguous()
c = a[:, 1:].contiguous()
d = a[:, [0, 2]].contiguous()

loss_f = torch.nn.MSELoss()
opt = torch.optim.Adam([a], lr=0.1)
for i in range(100):
    opt.zero_grad()
    loss = loss_f(b, torch.randn_like(b)) + loss_f(c, torch.randn_like(c)) + loss_f(d, torch.randn_like(d))
    loss.backward()
    print(loss.item())
    opt.step()
</code></pre>
<p>When I run this code, I get the error “backward through the graph a second time”. Then I find it is the tensor index causes this error. In my case, <em>b</em>, <em>c</em>, <em>d</em> don’t update their values. Since the second optimization, there’s no graph on <em>a</em>. So I change the code like this and solve the error:</p>
<pre><code class="lang-auto">import torch

a = torch.randn(3, 3)
a.requires_grad = True

loss_f = torch.nn.MSELoss()
opt = torch.optim.Adam([a], lr=0.1)
for i in range(100):
    opt.zero_grad()
    b = a[:, :2].contiguous()  # changed
    c = a[:, 1:].contiguous()  # changed
    d = a[:, [0, 2]].contiguous()  # changed
    loss = loss_f(b, torch.randn_like(b)) + loss_f(c, torch.randn_like(c)) + loss_f(d, torch.randn_like(d))
    loss.backward()
    print(loss.item())
    opt.step()
</code></pre>
<p>However, I make a lot of wrapping operations on <em>b</em>, <em>c</em>, <em>d</em> in my project. I don’t want to re-index them at each iteration. Are there any solutions to control <em>b</em>, <em>c</em>, <em>d</em> update as <em>a</em> without re-indexing them? Thank you!</p>
Seeking Advice on Choosing an LED Monitor for Enhanced PyTorch Workflow

Steve_Anderson: <p>Hey PyTorch community,</p>
<p>I hope you’re all doing well. I’m in the process of upgrading my workstation setup for more efficient PyTorch development and experimentation. One key component I’m looking to upgrade is my monitor. I believe a high-quality LED monitor could significantly enhance my coding and debugging experience.</p>
<p>I’d really appreciate your insights and recommendations on the following points:</p>
<p>Resolution and Size: What’s the ideal monitor size and resolution for PyTorch work? I’m torn between a larger screen with a 4K resolution for more workspace and a smaller one with a high refresh rate for smoother visualization.</p>
<p>Color Accuracy: Since accurate color representation is crucial for visualizing data and model outputs, are there any specific LED monitors that are known for their excellent color accuracy? Any personal experiences or reviews would be valuable.</p>
<p>Connectivity and Ports: I often work with multiple devices, so having versatile connectivity options is important. Which types of ports (HDMI, DisplayPort, USB-C, etc.) should I prioritize for seamless integration with my PyTorch development setup?</p>
<p>Panel Technology: Should I opt for an IPS panel, which generally offers better color reproduction and wider viewing angles, or would a TN panel suffice for my PyTorch-related tasks?</p>
<p>Budget Considerations: While I’m willing to invest in a high-quality monitor, I’d like to strike a balance between performance and cost-effectiveness. Are there any budget-friendly LED monitors that still offer a good PyTorch development experience?</p>
<p>Additional Features: Are there any other features or specs that you find particularly helpful when working with PyTorch? This could include things like adjustable stands, blue light filters, or even integrated hardware calibration tools.</p>
<p>If you have any personal recommendations or experiences to share, I’d love to hear about them. Your insights will play a vital role in helping me make an informed decision. Thanks in advance for your assistance!</p>
<p>Best regards,<br>
Steve</p>

ptrblck: <aside class="quote no-group" data-username="Steve_Anderson" data-post="1" data-topic="186952">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/steve_anderson/48/63413_2.png" class="avatar"> Steve_Anderson:</div>
<blockquote>
<p>If you have any personal recommendations or experiences to share, I’d love to hear about them.</p>
</blockquote>
</aside>
<p>Personally, I wouldn’t spend time thinking about a great monitor setup when 90% of my coding time is staring at <code>vim</code> in a black terminal or at another IDE, which is still only responsible to render text most of the time.</p>
Distributed package doesn't have NCCL built in

bahadir_kulavuz: <p>There is no error in my pytorch installation, but I am getting such an error, is it because I have only one gpu or is there a solution with a single graphics card?</p>

ptrblck: <p>You might be using Windows, which doesn’t support NCCL, or might have installed the CPU-only binaries. In any case, why do you want to use NCCL with a single GPU? What should the GPU communicate with?</p>

bahadir_kulavuz: <p>Yes, I am using windows. I tried to do segmentation work with 3D point cloud data, but I encountered this error. Cuda appears but ncll gives false value, I tried reinstalling but the result did not change.</p>

ptrblck: <p>That’s expected as already examined since Windows does not support NCCL. However, you still didn’t answer why you want to use NCCL in the first place with a single GPU?</p>

bahadir_kulavuz: <p>I’m trying to run a deep learning architecture that does 3d segmentation on windows without using Ubuntu, so I’m trying to use the graphic card.</p>

bahadir_kulavuz: <p>Thank you for your interest.</p>

ptrblck: <p>You don’t need NCCL to run single GPU workloads, since NCCL is used for multi-GPU communication.</p>
Why does torch.eigh function use multiple cpus?

Yuby: <p>Hello,</p>
<p>May I ask why torch.eigh uses multiple cpus? Some clues may exist in the torch function <a href="https://github.com/pytorch/pytorch/blob/85ffbedfb2a2bffda220e3fb73dc311dba5e7fed/aten/src/ATen/native/BatchLinearAlgebraKernel.cpp#L223-L230" rel="noopener nofollow ugc">apply_lapack_eigh</a> and the corresponding LAPACK function <a href="https://netlib.org/lapack/explore-html/d3/d88/group__real_s_yeigen_ga6b4d01c8952350ea557b90302ef9de4d.html" rel="noopener nofollow ugc">SSYEVD</a>. I suspect that somewhere multithreading or multiprocessing was called, which is why multiple CPUs were utilized. However, after reading these codes, I still haven’t found the specific command that calls multithreading or multiprocessing.</p>
<p>Thanks in advance</p>
Possible to replace einsum expr "btgi,gih->btgh" by any matmul call?

vadimkantorov: <p>I found this expression in <code>torch.einsum("btgi,gih-&gt;btgh", x, self.weight)</code> in <a href="https://github.com/Rikorose/DeepFilterNet/blob/7e4ef182511eabe9c3325292afce0feb5e612e4d/DeepFilterNet/df/modules.py#L774C9-L774C59" rel="noopener nofollow ugc">https://github.com/Rikorose/DeepFilterNet/blob/7e4ef182511eabe9c3325292afce0feb5e612e4d/DeepFilterNet/df/modules.py#L774C9-L774C59</a> to implement <code>GroupedLinear</code> layer.</p>
<p>Is it possible to somehow replace it by a broadcasted batched matmul call without a manual loop?</p>
<p><code>torch.matmul(x, self.weight[None, None, :, :, :])</code>?</p>
<p>Would it be faster than the einsum?</p>

vadimkantorov: <p>Hmm, seems <code>torch.matmul</code> broadcasting somehow does not work, although docs say it should be working…</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

x = torch.rand(10, 20, 30, 40)
weight = torch.rand(30, 40, 50)

print(torch.einsum("btgi,gih-&gt;btgh", x, weight).shape)
# torch.Size([10, 20, 30, 50])

print(torch.matmul(x, weight[None, None, ...]).shape)
# RuntimeError: The size of tensor a (20) must match the size of tensor b (30) at non-singleton dimension 2
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5052243546188a979e26e4b74aaf5d474b48b111.png" data-download-href="https://discuss.pytorch.org/uploads/default/5052243546188a979e26e4b74aaf5d474b48b111" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5052243546188a979e26e4b74aaf5d474b48b111_2_690x292.png" alt="image" data-base62-sha1="bsybsA76RFqlA58ER3pnwQr6DZL" width="690" height="292" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5052243546188a979e26e4b74aaf5d474b48b111_2_690x292.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5052243546188a979e26e4b74aaf5d474b48b111_2_1035x438.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5052243546188a979e26e4b74aaf5d474b48b111.png 2x" data-dominant-color="DFDFE0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1187×504 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vadimkantorov: <p><a class="mention" href="/u/lezcano">@Lezcano</a> answered in <a href="https://github.com/pytorch/pytorch/issues/76702#issuecomment-1689795689" class="inline-onebox" rel="noopener nofollow ugc">torch.matmul copies instead of broadcasting, while F.linear doesn't · Issue #76702 · pytorch/pytorch · GitHub</a> ! Basically, this is batched vector-matrix product, so batch dims are computed differently</p>
Training from a checkpoint (pytorch_lightning)

StellaJourney_Huang: <p>Hi,</p>
<p>When I tried to continue training from a checkpoint<br>
<code>python scalingup/train.py dataset_path=scalingup/wandb/run-20230808_082724-ioa7gmrt/files/ evaluation=bin_transport_test algo=diffusion_default algo.replay_buffer.batch_size=256 tags='[sim,transport,diffusion]' load_from_path=scalingup/wandb/run-20230808_205236-ib7bi2uq/files/checkpoints/last.ckpt</code>, the following error happened:</p>
<pre><code class="lang-auto">Error executing job with overrides: ['dataset_path=scalingup/wandb/run-20230808_082724-ioa7gmrt/files/', 'evaluation=bin_transport_test', 'algo=diffusion_default', 'algo.replay_buffer.batch_size=256', 'tags=[sim,transport,diffusion]', 'load_from_path=scalingup/wandb/run-20230808_205236-ib7bi2uq/files/checkpoints/last.ckpt']
Traceback (most recent call last):
File "/home/yan/Documents/scalingup/scalingup/train.py", line 99, in train
trainer, algo = setup_trainer(
File "/home/yan/Documents/scalingup/scalingup/train.py", line 46, in setup_trainer
algo = ScalingUpAlgo.load_from_checkpoint(
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1531, in load_from_checkpoint
loaded = _load_from_checkpoint(
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 90, in _load_from_checkpoint
return _load_state(cls, checkpoint, strict=strict, kwargs)
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 136, in _load_state
obj = cls(_cls_kwargs)
TypeError: Can't instantiate abstract class ScalingUpAlgo with abstract method get_policy
</code></pre>
<p>Is it a bug that can be fixed?</p>
<p>Best regards</p>

HELLORPG: <p>This may be a specific error of PyTorch-Lightning, you can try to get some help from their <a href="https://lightning.ai/forums/" rel="noopener nofollow ugc">official forums</a>.</p>
<p>Best regards.</p>

StellaJourney_Huang: <p>thanks! I’ll move there</p>
Where is the newer version of pytorch?

Ardeal: <p>Hi,</p>
<p>From the following link:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/locally/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The latest pytorch that we could download is version 2.0. My questions are:</p>
<ol>
<li>is there any newer version of pytorch?</li>
<li>how to update my pytorch to the newer version?</li>
</ol>

HELLORPG: <p>The newest stable version is <a href="https://pytorch.org/" rel="noopener nofollow ugc">here</a> (PyTorch 2.0.1).</p>
<p>If you want to get a newer version than this stable version, you can also find the nightly version on the above website.</p>
<p>For me, I always update my PyTorch version by removing the older version (<code>conda remove</code>) and then just <code>conda install</code> the newer version. But I guess there must be some more elegant ways.</p>

Ardeal: <p>Dear <a class="mention" href="/u/hellorpg">@HELLORPG</a><br>
Thank you for your reply!</p>
<p>Version 2.0.1 has not been update for a very long time. Does it mean that Pytorch will not be updated in future?</p>

HELLORPG: <p>I guess the next version should be PyTorch 2.1.0, which is a major version update. It always takes more time than from 2.0.0 to 2.0.1.</p>
<p>In fact, just like you, I also do not know the date for PyTorch 2.1 release. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="Ardeal" data-post="3" data-topic="186901">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>Version 2.0.1 has not been update for a very long time</p>
</blockquote>
</aside>
<p><code>2.0.1</code> was released in May, which I wouldn’t call a “very long time” ago. The next stable release is targeting October and you are free to use the nightly binaries in the meantime which are updated every night.</p>
Nvidia cuda version with Pytorch cuda version

Sirius_Zhang: <p>“My NVIDIA CUDA version is 11.3, and my NVIDIA driver version is 465.19.01. However, I have installed PyTorch 1.13.1 and CUDA 16.0, and surprisingly, they seem to be working together. I’m quite curious about this. Could someone provide an explanation for this unexpected compatibility?”</p>

sirius777coder: <p>That is weird, does anyone know it ?</p>

Ed_Moman: <p>When you have a time machine that gives you access to Cuda 16.0, everything is possible, my friend.</p>
<p>You would probably be better off buying lottery tickets or something…</p>

ptrblck: <p>CUDA 16.0 is not a thing as already mentioned.<br>
You should however note, that the PyTorch binaries ship with their own CUDA dependencies and you would thus only need to properly install an NVIDIA driver.<br>
Your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.</p>

sirius777coder: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="186831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You should however note, that the PyTorch binaries ship with their own CUDA dependencies and you would thus only need to properly install an NVIDIA driver.</p>
</blockquote>
</aside>
<p>But if my driver can only support CUDA 11.3, why it also works for 11.6?</p>

ptrblck: <p>CUDA11 introduced minor version compatibility as explained <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version">here</a>.</p>
How to resolve this error

Nirmai: <p>2023-08-18 18:45:28 Exception message: CUDA error: an illegal memory access was encountered<br>
2023-08-18 18:45:28 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
2023-08-18 18:45:28 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
2023-08-18 18:45:28 Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Strangely, the above error was resolved by increasing batch size. Why?<br>
I dont want to increase the batch size and solve this error. Is there any other way to resolve this</p>

ptrblck: <p>Rerun your code with <code>compute-sanitizer python script.py args</code> and post the output here.</p>

Nirmai: <p>CUDA_LAUNCH_BLOCKING=1 tried running with this, there is no error.</p>

ptrblck: <p>Did you also run it with <code>compute-sanitizer</code> as asked?</p>
How to sample transitions in vectorized envs for off-policy algos

kbtorcher: <p>How is minibatch sampling for vectorized envs (consider sequential processing of envs) with off policy RL implemented? Let’s say there are 5 envs. Every step the agent gets 5 transitions and pushes it to the buffer. While doing critic and policy updates, do we randomly sample transitions of batchsize N or do we randomly sample timesteps of batchsize N, thereby effectively giving us N*5 transitions?</p>

J_Johnson: <p>Ideally random, if they are stored in the memory buffer.</p>
<p>It really makes no difference to the model what order you are feeding it the data, unless your model contains a memory state. Then you would definitely need the data fed sequentially.</p>
nn.MultiheadAttention with float mask results in 'nan' for all attention scores

HELLORPG: <p>I try to use nn.MultiheadAttention with some float masks, on PyTorch 2.0.1, like the below demo code:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

my_attn = nn.MultiheadAttention(
    embed_dim=256,
    batch_first=True,
    num_heads=8
)
my_attn = my_attn.to("cuda")

x = torch.randn((1, 10, 256), dtype=torch.float, device="cuda")
bool_mask = torch.ones((10, 10), dtype=torch.bool, device="cuda")
bool_mask[:2, :2] = False
bool_mask[2:, 2:] = False

float_mask = torch.randn((10, 10), dtype=torch.float, device="cuda")

my_attn.eval()
with torch.no_grad():
    bool_mask_res, bool_mask_scores = my_attn(
        query=x, key=x, value=x, attn_mask=bool_mask
    )
    print(bool_mask_scores)
    
    float_mask_res, float_mask_scores = my_attn(
        query=x, key=x, value=x, attn_mask=float_mask
    )
    print(float_mask_scores)       # NOTE: the bug is here, the scores will be all 'nan'.
</code></pre>
<p>The result is that the <code>float_mask_scores</code> is filled with <code>nan</code>, which is obviously wrong. And I also get a UserWarning:</p>
<pre><code class="lang-auto">torch/nn/modules/activation.py:1160: UserWarning: Converting mask without torch.bool dtype to bool; this will negatively affect performance. Prefer to use a boolean mask directly. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343995026/work/aten/src/ATen/native/transformers/attention.cpp:150.)
  return torch._native_multi_head_attention(
</code></pre>
<p>It seems that the <code>float_mask</code> is regarded as a bool mask in the built-in code.<br>
I guess there is caused by <code>scaled_dot_product_attention()</code> because I can run this code correctly in PyTorch 1.13.1.</p>

HELLORPG: <p>This is a bug of PyTorch 2.0.1.</p>
<p>As discussed in:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/107084" rel="noopener nofollow ugc">[regression] nn.MultiheadAttention does not respect adding of floating point mask to attention for the fast path · Issue #107084 · pytorch/pytorch (github.com)</a></li>
<li><a href="https://github.com/pytorch/pytorch/pull/107641" rel="noopener nofollow ugc">Disable nn.MHA fastpath for floating point masks by mikaylagawarecki · Pull Request #107641 · pytorch/pytorch (github.com)</a></li>
</ul>
Error using running model from torch script

eraserpencil: <p>I have a python3.6 virtualenv created, install opencv-python 3.4.5.20, torch 1.1.0 and torch vision 0.3.0 from <a href="https://download.pytorch.org/whl/cu90/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/cu90/torch_stable.html</a></p>
<p>My goal is to have torchscript save a model that i can run in libtorch 1.3.0, created by checking out the github repo to v1.3.0 and following first suggestions of the instructions here <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst</a>)</p>
<p>I used torchscript to save a model and when i run it in c++, i got the following error.</p>
<p>terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  cuDNN error: CUDNN_STATUS_NOT_INITIALIZED<br>
The above operation failed in interpreter, with the following stack trace:<br>
at code/posenet_libtorch.py:111:11<br>
_98 = self.offset<br>
_99 = _98.weight<br>
_100 = _98.bias<br>
_101 = self.displacement_fwd<br>
_102 = _101.weight<br>
_103 = _101.bias<br>
_104 = self.displacement_bwd<br>
_105 = _104.weight<br>
_106 = _104.bias<br>
input0 = torch._convolution(input, _2, _3, [2, 2], [1, 1], [1, 1], False, [0, 0], 1, False, False, True)<br>
~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
input1 = torch.hardtanh(input0, 0., 6.)<br>
input2 = torch._convolution(input1, _6, _7, [1, 1], [1, 1], [1, 1], False, [0, 0], 32, False, False, True)<br>
input3 = torch.hardtanh(input2, 0., 6.)<br>
input4 = torch._convolution(input3, _9, _10, [1, 1], [0, 0], [1, 1], False, [0, 0], 1, False, False, True)<br>
input5 = torch.hardtanh(input4, 0., 6.)<br>
input6 = torch._convolution(input5, _13, _14, [2, 2], [1, 1], [1, 1], False, [0, 0], 64, False, False, True)<br>
input7 = torch.hardtanh(input6, 0., 6.)<br>
input8 = torch._convolution(input7, _16, _17, [1, 1], [0, 0], [1, 1], False, [0, 0], 1, False, False, True)<br>
input9 = torch.hardtanh(input8, 0., 6.)</p>

Jonathans575: <p>Dear, I meet this same error.<br>
Have you solved this problem?</p>