akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to implement fp16 quantization on CPU

username: Relissc
cooked: <p>In Torch.fx quantization using Pytorch, the float16 data type only supports running in GPU, but it seems that GPU cannot support int8 quantization through experiments. Therefore, it is hoped that the PETR model FP16+INT8 quantization can be implemented in GPU (X86) devices. Is this solution feasible?</p>

username: supriyar
cooked: <p>yes, its feasible but currently not a part of PyTorch. We are working on a prototype solution for int8 GPU dynamic quantization and will share an update shortly.<br>
cc <a class="mention" href="/u/cdhernandez">@cdhernandez</a></p>

username: Relissc
cooked: <p>Thank you very much for your answer, it has been very helpful to me. Can I use a custom method to quantify fp16 on the CPU? Or use other data types to quantify on the CPU, such as torch.bfloat16?</p>
Properly initializing triangular weight matrices

username: bigtuna
cooked: <p>I’m using a class meant to implement a linear layer with a triangular weight matrix defined here:</p>
<pre><code class="lang-auto">class TriLinear(nn.Linear):
    def __init__(self, in_features, out_features):
        super().__init__(in_features, out_features, bias = False)
        with torch.no_grad():
            self.weight.copy_(torch.tril(self.weight))
        self.weight.register_hook(lambda grad: grad * torch.tril(torch.ones_like(grad)))
        
    def forward(self, x):
        return F.linear(x, self.weight, self.bias)
</code></pre>
<p>I have to replace a few regular linear layers of an existing model with this class and compare the performance. However, in the larger model initialization there is the followeing code:</p>
<pre><code class="lang-auto">for p in self.parameters():
            if p.dim() &gt; 1:
                nn.init.xavier_uniform_(p) 
</code></pre>
<p>It seems that when this runs, the weight matrices of my TriLinear class are no longer triangular.  Is there a way to implement this initialization without interfering with the triangular weight matrices?</p>

username: KFrank
cooked: <p>Hi Big!</p>
<aside class="quote no-group" data-username="bigtuna" data-post="1" data-topic="185885" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/4af34b/48.png" class="avatar"> bigtuna:</div>
<blockquote>
<pre><code class="lang-auto">for p in self.parameters():
            if p.dim() &gt; 1:
                nn.init.xavier_uniform_(p) 
</code></pre>
<p>It seems that when this runs, the weight matrices of my TriLinear class are no longer triangular.</p>
</blockquote>
</aside>
<p>The most satisfactory approach would be to modify the xavier-initialization<br>
to iterate over modules:</p>
<pre data-code-wrap="python"><code class="lang-python">for  m in model.modules():
    if  instanceof (m, TriLinear):  continue
    # xavier-initialize the parameters in m
</code></pre>
<p>However for a simple tweak to your code that is <em>exceedingly unlikely</em><br>
to fail, try:</p>
<pre data-code-wrap="python"><code class="lang-python">for p in self.parameters():
            if  p.dim == 2  and  torch.equal (p, p.tril()):  continue
            if  p.dim() &gt; 1:
                nn.init.xavier_uniform_(p) 
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: bigtuna
cooked: <p>This is perfect, thank you.</p>
Ensuring numerical stability

username: Millon_Madhur_Das
cooked: <p>How does pytorch ensure numerical stability for sigmoid? Python can handle upto approx 1.7e+709. How is this handled? I tried to go through the source code but couldn’t find the implementation for sigmoid.</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/millon_madhur_das">@Millon_Madhur_Das</a>,</p>
<p>You can change the <code>torch.dtype</code> to <code>torch.float64</code> for all operations via  <code>torch.set_default_dtype(torch.float64)</code> (docs: <a href="https://pytorch.org/docs/stable/generated/torch.set_default_dtype.html" class="inline-onebox" rel="noopener nofollow ugc">torch.set_default_dtype — PyTorch 2.0 documentation</a>)</p>
<p>If you want to see the source code for <code>torch.Sigmoid</code>, you’ll have to look for it on the GitHub repo (I’m sure someone will know the path to the file).</p>
<p>As a side note, you can always represent your problem within the log-domain and that’ll have a much better range than doing it within the linear-domain.</p>
Trying to make a chatbot with C++

username: Just_Samm
cooked: <p>Hi there,<br>
I’m trying to make a chatbot for C++ with PyTorch however I’m having a hard time figuring out how to convert some code originally from a python PyTorch <a href="https://youtu.be/k1SzvvFtl4w" rel="noopener nofollow ugc">tutorial</a> I found <a href="https://youtu.be/k1SzvvFtl4w" rel="noopener nofollow ugc">here</a></p>
<p>More specifically my problem is with “data[‘input_size’], data[‘hidden_size’]”, etc. because I have created the torch::jit::module (Which from what I can tell is how you load the model) from a Visual Studio .pth resource file that was created in python.</p>
<p>I loaded the pth resource compiled into the actual executable like this btw:</p>
<pre><code class="lang-auto">#include &lt;resource.h&gt;
#include &lt;Windows.h&gt;
#include &lt;pytorch/torch/script.h&gt;
#include &lt;string&gt;
#include &lt;sstream&gt;

HMODULE GCM()
{
	HMODULE hModule = NULL;
	GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS, (LPCWSTR)GCM, &amp;hModule);
	return hModule;
}

torch::jit::Module Load_Compiled_PyTorch_Model(int resource)
{
	HRSRC hRes = FindResource(GCM(), MAKEINTRESOURCE(resource), L"PTH");
	std::stringstream result_stream;
	result_stream &lt;&lt; hRes;
	torch::jit::Module hFinal = torch::jit::load(result_stream);
	return hFinal;
}
</code></pre>
<p>Anyways I need to figure out the C++ equivalent way to do <code>data['input_size']</code> and so on<br>
And I’m also kinda wondering if the way I loaded the torch model as a visual studio resource will work or if I need to do that differently.</p>
<p>Thanks in advance.</p>
How to create a branch in ResNet with ImageNet pretrained weights?

username: smani
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/593a18a556c8ed1e9532fa075ded84addeb2ff90.png" data-download-href="https://discuss.pytorch.org/uploads/default/593a18a556c8ed1e9532fa075ded84addeb2ff90" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/593a18a556c8ed1e9532fa075ded84addeb2ff90.png" alt="image" data-base62-sha1="cJkXRC715j2AV0f7Yzv4MEo5RTi" width="531" height="500" data-dominant-color="DDEAF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">645×607 13.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to create a ResNet-18 model with two branches as shown in the above figure, where I replicate the original Layer 4 to create the branches. However, I don’t know how to initialize the weights of these two layers (Layer 4_1 and Layer 4_2) with ImageNet pretrained weights. Could you please help me with this.</p>

username: ptrblck
cooked: <p>You could manually load all desired parameters by directly accessing the modules.<br>
E.g. something like this would work to load the parameters into the two custom layers:</p>
<pre><code class="lang-python">model = models.resnet18()
sd = model.state_dict()

layer4_1 = nn.Conv2d(256, 512, 3, 2, 1, bias=False)

# access custom layers and load parameters manually
with torch.no_grad():    
    layer4_1.weight.copy_(sd["layer4.0.conv1.weight"])
</code></pre>
<p>Note that you won’t be able to load the entire <code>state_dict</code> since you’ve changed the model architecture. You could use <code>strict=False</code>, but I would highly recommend checking the mismatches or missing keys explicitly.</p>
GPU based HEVC encoding and decoding

username: Harsha-Musunuri
cooked: <p>Hello All, I built the GPU based video encoding and decoding ffmpeg using this blog.</p>
<p>I am specifically looking for - let’s say I have a video as a pytorch tensor - for example, of shape: 30x3x1920x1080 - 30 frames of a video of size 1920x1080. Can this ffmpeg build takes in pytorch tensor and do hevc compression?</p>
<p>If not this build, is there any other tool that works off of pytorch tensor directly?</p>
Deactivate a feature map

username: abask61
cooked: <p>There are lots of posts on feature visualization. But how do I deactivate a feature map (not filter)?</p>
<p>(layer4): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(downsample): Sequential(<br>
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(relu): ReLU(inplace=True)<br>
<strong>(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)</strong><br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
)<br>
)</p>
<p>For the layer4.conv2 I want to set a feature map (not weights or filters) to zero at the output channel during inference. For example, ith output channel of conv2 in layer 4. Is it even possible?</p>

username: ptrblck
cooked: <p>Multiply the forward activation in the <code>forward</code> method with a mask, which will zero out the desired channel.</p>

username: abask61
cooked: <p>I tried to do the following:</p>
<pre><code>def forward(self, x):
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.relu(x)
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)
    
   if self.usingfeaturedeactivation and self.feature_idx is not None:
        x[:,self.feature_idx,:,:] = 0
    
   x = self.avgpool(x)

    x = x.view(x.size(0), -1)
    x = self.fc(x)

    return x
</code></pre>
<p>Seems like it’s working. Thanks!</p>
How does pytorch handle backward pass in a multi-GPU setting? (DLRM use case)

username: christindbose
cooked: <p>Past works have often proposed remote GPU caching as a performance optimization. As an example, if data x originally stored on GPU0 is requested by GPU1, then x is cached in GPU1’s L1 or L2 cache (there are pros and cons if it’s cached in L1 or L2 depending on the workload).</p>
<p>While applying remote caching is trivial for inference, getting it to work for training seems to be more challenging. Primarily because we need to maintain coherence across all cached copies after the backward pass. I was curious to know how Pytorch actually handles all this under the hood. I’ve read that the gradients are calculated based on a computational graph.</p>
<p>Some questions I have:</p>
<ul>
<li>in case of model parallel training, does the computational graph also store which device contains the required weights? (I think it is yes)</li>
<li>what does pytorch do if data is remotely cached (there are 2 options here: invalidate all cached copies and only update the original data or update both original and cached copies)</li>
</ul>
<p>Any pointers to find the answers to these questions would be great. Thanks!</p>
<p>PS: I was specifically trying to find out how the gradient updates for embedding tables occur for DLRM <a href="https://github.com/facebookresearch/dlrm/issues/353" rel="noopener nofollow ugc">How does pytorch handle backward pass in a multi-GPU setting? · Issue #353 · facebookresearch/dlrm · GitHub</a></p>
Deactivate a filter in conv layer

username: abask61
cooked: <p>I want to deactivate one filter of a conv layer just before inference.</p>
<p>Here’s a partial Resnet model:</p>
<p>(layer4): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(downsample): Sequential(<br>
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
(relu): ReLU(inplace=True)<br>
<strong>(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)</strong><br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=1)<br>
)<br>
)</p>
<p>How do I deactivate a particular filter (ith among 512 after convolution) in layer4, (1) basicblock, conv2 ?</p>

username: ptrblck
cooked: <p>You could set its <code>weight</code> (and <code>bias</code> if needed) at the corresponding slice to zeros.<br>
E.g. something like this should work:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer4[1].conv2.weight[idx].zero_()
</code></pre>

username: abask61
cooked: <p>Thanks for the response! I used the following code:</p>
<p>model.layer4[1].conv2.weight[idx].zero_()<br>
print(model.layer4[1].conv2.weight[idx])</p>
<p>tensor([[[[-0.0002,  0.0002, -0.0038],<br>
[ 0.0055,  0.0055,  0.0024],<br>
[ 0.0059,  0.0046,  0.0024]],<br>
[[-0.0066, -0.0105, -0.0040],<br>
[-0.0052, -0.0091, -0.0021],<br>
[ 0.0044,  0.0012,  0.0072]],<br>
[[ 0.0007,  0.0044,  0.0023],<br>
[-0.0045, -0.0009, -0.0018],<br>
[-0.0079, -0.0044, -0.0058]],</p>
<p>So, it seems like it’s not working.</p>

username: ptrblck
cooked: <p>It works for me:</p>
<pre><code class="lang-python">model = torchvision.models.resnet18()
idx = 1
with torch.no_grad():
    model.layer4[1].conv2.weight[idx].zero_()

print(model.layer4[1].conv2.weight[idx])
# tensor([[[0., 0., 0.],
#          [0., 0., 0.],
#          [0., 0., 0.]],

#         [[0., 0., 0.],
#          [0., 0., 0.],
#          [0., 0., 0.]],
# ...
</code></pre>

username: abask61
cooked: <p>Thanks!!! It worked. My idx was an ndarray. I had to get the item().</p>
Detecting when create_graph == True

username: BeachHut
cooked: <p>In order to save memory, I would like to only save Tensors for backward when backward might be called. For first derivatives I can do this by checking whether the input has <code>requires_grad=True</code>. For second derivatives it seems that the best I could do would be to check whether <code>create_graph == True</code>. Is there a way to check in the <code>forward</code> method of an <code>autograd.Function</code> whether <code>create_graph == True</code>?</p>
<pre><code class="lang-python">class SquareForward(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        if x.requires_grad:  # Only save when backward might be called
            ctx.save_for_backward(x)
        return x**2

    @staticmethod
    def backward(ctx, grad_y):
        x,  = ctx.saved_tensors
        return SquareBackward.apply(grad_y, x)

class SquareBackward(torch.autograd.Function):
    @staticmethod
    def forward(ctx, grad_y, x):
        # &gt;&gt;&gt; Want to only save when double backward might be called &lt;&lt;&lt;
        ctx.save_for_backward(grad_y, x)
        return 2 * grad_y * x

    @staticmethod
    def backward(ctx, grad_grad_x):
        grad_y, x = ctx.saved_tensors
        return grad_grad_x * 2 * x, grad_grad_x * 2 * grad_y
</code></pre>
<p>I would like to be able to do something like <code>if grad_y.requires_grad</code> or <code>if ctx.create_graph</code> to determine whether to save Tensors in the <code>SquareBackward</code> <code>forward</code> method. This is especially important when the Tensors that I would like to save are a large number of intermediate results that I would not otherwise allocate memory to store simultaneously. The only solution at the moment seems to be to never store the intermediate results in the <code>SquareBackward</code> <code>forward</code> method and to instead recompute these intermediate results in the <code>SquareBackward</code> <code>backward</code> method when we are sure that double backward is being performed, but that recomputation would be wasteful.</p>
Dataloader CPU Core usage problem

username: Tobischi
cooked: <p>Hi there,</p>
<p>I am working with the cityscapes dataset and want to use a DataLoader with several workers to speed up the training process, but with num_workers&gt;0 only two CPU cores are used.</p>
<p>E.g. with num_workers=8, only 2 cores are used:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/cadc2e8fb9cb04a5bc14bc41b582328debc19c57.png" alt="cpu_usage_8workers" data-base62-sha1="sWAbz2kmkcw8TS6CMIj8gITNyL5" width="480" height="266"></p>
<p>While trying to solve the problem I found <a href="https://github.com/pytorch/pytorch/issues/8126" rel="noopener nofollow ugc">this</a> discussion and noticed that when I add</p>
<pre><code class="lang-auto">os.system("taskset -p 0xffffffffffffffff %d" % os.getpid())
</code></pre>
<p>at the beginning of my Datasets <strong>get_item</strong> function, the number of used cores equals num_workers and I notice a considerable speedup.</p>
<p>It seems like the affinity masks are changed in each worker’s first iteration and then stay unchanged:</p>
<pre><code class="lang-auto">pid 1280270's current affinity mask: 100000001
pid 1280270's new affinity mask: ffffffffffffffff
pid 1280333's current affinity mask: 100000001
pid 1280333's new affinity mask: ffffffffffffffff
pid 1280459's current affinity mask: 100000001
pid 1280459's new affinity mask: ffffffffffffffff
pid 1280396's current affinity mask: 100000001
pid 1280396's new affinity mask: ffffffffffffffff
pid 1280270's current affinity mask: ffffffffffffffff
pid 1280270's new affinity mask: ffffffffffffffff
pid 1280396's current affinity mask: ffffffffffffffff
pid 1280396's new affinity mask: ffffffffffffffff
pid 1280270's current affinity mask: ffffffffffffffff
pid 1280270's new affinity mask: ffffffffffffffff
pid 1280396's current affinity mask: ffffffffffffffff
pid 1280396's new affinity mask: ffffffffffffffff
pid 1280333's current affinity mask: ffffffffffffffff
pid 1280333's new affinity mask: ffffffffffffffff
pid 1280270's current affinity mask: ffffffffffffffff
pid 1280270's new affinity mask: ffffffffffffffff
pid 1280396's current affinity mask: ffffffffffffffff
pid 1280396's new affinity mask: ffffffffffffffff
</code></pre>
<p>I don’t have much experience with multiprocessing, but this behavior seems quite strange to me and I think there should be a better solution to solve the problem.</p>
<p>Does anyone have an idea what the problem could be and how it could be solved?</p>

username: AMonszpart
cooked: <p>Seems to be an issue with <code>llvm-openmp</code>: <a href="https://github.com/pytorch/pytorch/issues/101850#issuecomment-1557249911" class="inline-onebox" rel="noopener nofollow ugc">Unexpected modification to CPU affinity of Dataloader workers · Issue #101850 · pytorch/pytorch · GitHub</a><br>
<code>&lt;16</code> works for me, but haven’t tried <code>&lt;16.0.6</code>.</p>
What are the best practises to ensure reproducibility across GPUs/devices?

username: A_work_user
cooked: <p>Hi<br>
I have a NN training problem which produces radically different losses/errors depending on the GPU/device that is being used. This is despite setting the default dtype to float64.</p>
<p>Of course I have read the Pytorch documentation and set all the seeds as recommended.</p>
<p>But I read in <a href="https://discuss.pytorch.org/t/different-results-on-cuda-and-cpu/168440/6">this thread</a> that one should use <code>x = x/2</code> instead of <code>x/= 2</code>, which apparently solved the user’s problems with getting different results.</p>
<p>This isn’t documented anywhere, so I’m wondering what other things there are that I’m using that shouldn’t be used if I want to achieve very similar results on different GPUs and devices. Does anyone know any other tricks or things I need to look out for?</p>

username: ptrblck
cooked: <p>In place operations are not causing issues by themselves. Read through the linked post and you will see that the input data of the user was manipulated via the inplace division on the CPU but not using the GPU as the user created copies of the inputs when the GPU was used.</p>

username: A_work_user
cooked: <p>Could he have avoided the problem if he had written <code>X_test = X_test.to(device)</code> directly after he defined it? Trying to understand the cause of the issue</p>
<p>Edit: yes, it looks like this solves the issue in the sense that it produces consistent output, but it’s different to the OP’s solution’s answer.</p>
<p>But I am still a bit concerned that there are other use cases which causes these differences between GPU/CPU that haven’t been documented very well.</p>
LibTorch build information

username: dfalbel
cooked: <p>Hi,</p>
<p>Is the build script for LibTorch’s pre-built binaries available somwhere?<br>
I’m currently interested in the value of <code>TORCH_CUDA_ARCH_LIST</code> that it uses for the CUDA 11.7 builds as I’m getting those after upgrading to LibTorch 2.0:</p>
<pre><code class="lang-auto">CUDA kernel failed : no kernel image is available for execution on the device
</code></pre>
<p>Thanks you very much!</p>
Gradient checkpointing inside gradient checkpointing

username: patrickvonplaten
cooked: <p>We are building a library where many large components can be used within each other e.g. a Transformer model can be used inside a UNet layer.</p>
<p>To save memory for training we use gradient checkpointing in both components. When use the transformer model inside the unet layer, we essentially use gradient checkpointing inside a function on which gradient checkpointing was already applied.</p>
<p>Is this problematic? The training seems to work just fine. Here some pseudo code:</p>
<pre><code class="lang-py">class UNetModel(nn.Module):
    self.layers = [TransformerModel() for _ in range(10)]

def forward(self, x):
    for layer in self.layers:
        x = torch.utils.checkpoint.checkpoint(layer, x, ...)


class TransformerModel(nn.Module):
    self.layers = [nn.TransformerLayer() for _ in range(10)]

def forward(self, x):
    for layer in self.layers:
         x = torch.utils.checkpoint.checkpoint(layer, x, ...)
</code></pre>
<p>Here you can see that gradient checkpointing is used for every transformer layer even though the whole transformer model has already gradient checkpointing. The reason we do this is because we can each component (UNet and Transformer) to be as memory efficient as possible out of the box even when used with other components.</p>
<p>=&gt; Is it problematic to have a gradient checkpointing function inside another one?</p>
Cuda devices inaccessible suddenly

username: Smith_David
cooked: <p>my environmen setup as</p>
<pre><code class="lang-auto">NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0
torch                     1.13.0a0+git7c98e70
cuda                      11.7
os                         Ubuntu 20.04.6 LTS              Debian 5.10.158-2 (2022-12-13) x86_64
</code></pre>
<p>run the instruction: python -c “import torch; print(torch.cuda.is_available())”, console output</p>
<pre><code class="lang-auto">/output/workspace/pytorch/torch/cuda/__init__.py:88: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /output/workspace/pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() &gt; 0
...
</code></pre>
<p>a few days ago, that is normal but today suddenly unusable, very  weird.</p>

username: ptrblck
cooked: <p>The issue doesn’t seem to e PyTorch-related as it seems your driver cannot initialize the GPU. Check what has changed between the last working setup, e.g. an unplanned driver update etc., and try reinstall the driver if that’s the case.</p>
How to increase RTX 3090 GPU usage?

username: donkey95
cooked: <p>hello, when i infer YOLOv5 for detect some object shows high CPU usage<br>
(about 45~50% in task manager).<br>
but GPU usage very low (“gpu 3D” about 4% and “gpu cuda” about 35~40%. in task manager)</p>
<p>CPU : 11th Gen Intel(R) Core™ i7-11700K @ 3.60GHz 3.60 GHz<br>
GPU : RTX 3090 x 1<br>
CUDA : 11.1</p>
<p>Im using libtorch for load YOLOv5 model and " torch::cuda::is_available() " function returns “TRUE”</p>
<p>Question. ===========================================</p>
<ol>
<li>Is it normal?</li>
<li>What is the difference between “GPU 3D” and “GPU cuda”  in task manager?</li>
<li>Why only high GPU cuda(35~40%) &gt; GPU 3D(4%)</li>
<li>How to increase GPU usage and decrease cpu usage?</li>
</ol>
<p>5.Judging from the 35-40% gpu cuda usage, it doesn’t seem like the gpu isn’t being used. is true?<br>
6. When same code run different env shows “gpu 3D” usage about 70~75% and cpu usage about under 10% why its happend…?<br>
(Diffent ENV : RTX 3060 and same cpu)</p>
<p>please let me know … thank you!</p>
Does batched_nms in torch support backpropagation?

username: jhp
cooked: <p>Hi I have a question about the backpropagation of batched_nms in pytorch.</p>
<p>Some of the works have used nms operation inside the model forward pass and I wonder whether the computation graph is all connected between the input and output of nms.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Yes, this should be the case and you can verify it by passing an input with <code>requires_grad=True</code> and checking the <code>.grad_fn</code> of the output, which should show a valid backward function. I’m not at my workstation right now as I would quickly verify it.</p>
Nordvpn vs hidemyass

username: Modler
cooked: <p>Hi i want to use one of these for security reason so which one should i go for <a href="https://vpnprofy.com/nordvpn-vs-hidemyass/" rel="noopener nofollow ugc">Nordvpn</a> vs hidemyass</p>

username: ptrblck
cooked: <p>Your question doesn’t seem to be PyTorch-related so you might want to ask it in another discussion board.</p>
What is `sym_size` and how is it different from `size` in pytorch C++?

username: arunppsg
cooked: <p>Here is a sample code:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor input = torch::rand({2, 3});

  auto size = input.sizes();
  // What is sym_size and how is it different from size?
  auto sym_size = input.sym_sizes();

  std::cout  &lt;&lt; " size " &lt;&lt; size &lt;&lt; " sym size " &lt;&lt; sym_size &lt;&lt; std::endl;
  // prints size [2, 3] sym size [2, 3]
}
</code></pre>
<p>In the above snippet, what is <code>sym_size</code> and how is it different from <code>size</code>?</p>

username: ptrblck
cooked: <p><code>sym_size</code> should correspond to symbolic shape and is used during tracing. Unless you are working on the symbolic tracing this method might not be useful.</p>
How to apply Conv2D to [time_dim, batch_dim, C_out, H_out, W_out]?

username: Yuerno
cooked: <p>Hi all! I’m dealing with batch of images where each item in the batch is a time sequence of data. So essentially, imagine that each entry in the batch is a set of frames. I’d like to apply convolutions to the last three channels, but of course Conv2D expects 4 dimensions.</p>
<p>Is the correct way to approach this problem that I combine time_dim and batch_dim (thus convert the overall input to 4 dimensions), apply the conv layer, and then split back out to 5 dimensions?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Yuerno" data-post="1" data-topic="185831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ccd318/48.png" class="avatar"> Yuerno:</div>
<blockquote>
<p>I’d like to apply convolutions to the last three channels</p>
</blockquote>
</aside>
<p>I don’t fully understand this idea. Do you want to slice the channels or to apply the conv to three <em>dimensions</em>?<br>
In the latter case you could <code>permute</code> the input into the shape <code>[batch_size, channels, temp, height, width]</code> and use an <code>nn.Conv3d</code> layer.</p>

username: Yuerno
cooked: <p>Essentially I have a batch of sequences of images (where the image dimensions are the last three channels). Conv2D wants a 4 dimensional input but I have 5 dimensions. The time dimension isn’t a depth dimension so I don’t want a 3D convolution (i.e. these aren’t 3D images or point clouds, they’re just sequences of 2D images).</p>
<p>What I’m wondering is, can I combine the batch and time dimensions, then apply a Conv2D as normal, and then split those dimensions back out while preserving the original time sequence ordering?</p>

username: ptrblck
cooked: <p>Yes, of you want to treat each time step as a separate sample you can flatten the temporal dimension into the batch dimension. However, you are still mentioning channels while I assume you mean dimensions?</p>
Weighted average of model parameter with trainable scalar

username: bluewater
cooked: <p>Hi.<br>
I want to find a variable <em>c</em> in order to make weighted average between two models.</p>
<p>For models(modelA, modelB) which has the same structure, I want to find variable <em>c</em> that minimize loss of the averaged model (modelC)<br>
modelC = (1-c)*modelA + c*modelB</p>
<pre><code class="lang-auto">modelA = nn.Linear(2, 1)
modelB = nn.Linear(2, 1)
modelC = nn.Linear(2, 1)


c = nn.Parameter(torch.rand(1))
c.requires_grad = True
print('C:', c)

criterion = nn.CrossEntropyLoss()

for p in modelA.parameters():
    p.requires_grad = False
for p in modelB.parameters():
    p.requires_grad = False
for p in modelC.parameters():
    p.requires_grad = False

for name, param in modelC.named_parameters():
    param = c*modelA.get_parameter(name).data() + (1+c)*modelB.get_parameter(name).data()
    
x = torch.randn(1, 2)
y = modelC(x)
y_hat = torch.randn(1, 1)
loss = criterion(y, y_hat)
loss.backward()
print(c.grad)
</code></pre>
Activation maximization for ResNets

username: rinkujadhav2013
cooked: <p>Does anyone know of any codebase to visualise filters of a ResNet using activation maximisation? I found a few but they all seem to work only for VGG or Inception but not ResNets. By work I mean the visualisations are good for VGGs but not for ResNets. Is there some code I use to visualise ResNet filters?</p>
A question about the center parameter of torchaudio.MelSpectrogram

username: Alethia
cooked: <p>Hi,</p>
<p>I’m little confused about the <code>center</code> parameter of <code>MelSpectrogram</code> in torchaudio.</p>
<p>I read the document of <code>torchaudio.MelSpectrogram</code> at <a href="https://pytorch.org/audio/main/generated/torchaudio.transforms.MelSpectrogram.html" rel="noopener nofollow ugc">here</a></p>
<p>It says that if <code>center=True</code>, then the <em>t</em> th frame will be centered at time <em>t</em> x <em>hop_length</em> in the original audio.</p>
<p>Then I have the following exp:</p>
<pre data-code-wrap="python3"><code class="lang-plaintext">import torch
import torchaudio

mel_transform = torchaudio.transforms.MelSpectrogram(sample_rate=24000, n_fft=1200, hop_length=300, n_mels=80, center=True, norm='slaney', mel_scale='slaney')
audio = torch.randn(10, 24000)
mel = mel_transform(audio)
print(mel.shape)
</code></pre>
<p>I get <code>torch.Size([10, 80, 81])</code>.<br>
I think this is weird since the 81th frame, based on the document, should be center at 81 x 300 = 24300,  but the time step 24300 even doesn’t exist in the original audio.<br>
So I’m very curious about how this happened.</p>
Downgraded performance after upgraded to Windows 11

username: Aram_Nasser
cooked: <p>I had my machine learning model working perfectly on Windows 10.<br>
When I upgraded to Windows 11, the model became too slow even though Cuda is present. I have not changed anything in the code…</p>
<p>I reinstalled Cuda and Pytorch again! Cuda version is cuda_11.7 (installed for Windows 11) and Pytorch is installed via the official command: “conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia”<br>
Have I missed anything?! Where might be the issue?</p>

username: ptrblck
cooked: <p>You would need to profile the code e.g. via <code>Nsight Systems</code> to narrow down where the bottleneck is. Since apparently nothing changed besides the Windows version it’s hard to even speculate what might be causing the slowdown.</p>

username: Aram_Nasser
cooked: <p>Thank you for your reply</p>
<p>I have found the problem in another part of the code, in the “DataLoader(…num_workers=12)”. Wrongly setting the “num_workers” caused the delay, not Pytorch setup as I expected.</p>
<p>Sorry, and thank you for your time.</p>
How to call torch.distributed.nn.all_gather on each node independently?

username: JP_W
cooked: <p>When I train my work with multinode, the code below can gather all tensors from all_gpus.</p>
<pre><code class="lang-auto">def gather_tensors(tensor):
    """
    We find this function works well for single node, but not for multi-node
    So we want to modify this function to gathered for gpus on same node
    """
    gathered_tensors = torch.cat(torch.distributed.nn.all_gather(tensor), dim=0)
    return gathered_tensors
</code></pre>
<p>But now, I want to gather tensors on same node and implement a version as below:</p>
<pre><code class="lang-auto">    def init_group(self):
        world_size = dist.get_world_size()
        num_gpus_per_node = torch.cuda.device_count()
        groups = []
        for j in range(world_size//num_gpus_per_node):
            node_ranks = [j * num_gpus_per_node + i for i in range(num_gpus_per_node)]
            node_ranks = [rank for rank in node_ranks if rank &lt; world_size]
            group = dist.new_group(ranks=node_ranks)
            groups.append(group)
        return groups

def gather_tensors_on_same_node(tensor, group=group.WORLD):
    """
    Gather tensors from all GPUs on the same node. Assume each node have 8 gpus.
    """
    gathered_tensors = torch.cat(torch.distributed.nn.all_gather(tensor, group=group), dim=0)
    return gathered_tensors
</code></pre>
<p>and I call them with</p>
<pre><code class="lang-auto">        global_rank = dist.get_rank()
        group = self.groups[global_rank//torch.cuda.device_count()]
        text_latents = gather_tensors_on_same_node(text_latents, group=group)
        image_latents = gather_tensors_on_same_node(image_latents, group=group)
</code></pre>
<p>But it will always report error "  File “/opt/conda/envs/ptca/lib/python3.8/site-packages/torch/distributed/nn/functional.py”, line 294, in backward<br>
gx = torch.empty_like(grad_outputs[rank])<br>
IndexError: tuple index out of range" and access data wrongly. Do you have any suggestions?</p>

username: ziteng
cooked: <p>same problem here… do you guys have any <code>all_gather</code> alternatives which retain computation graph and can compute grad?</p>

username: David_Junhao_Zhang
cooked: <p>This also gets me in stuck</p>
Size mismatch for model.diffusion_model.output_blocks.8.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640])

username: Drn_Mozi
cooked: <p>im a beginner. when loading an sdxl 1.0 model into stable diffusion, i’m presented with this-</p>
<p>size mismatch for model.diffusion_model.output_blocks.8.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).</p>
<p>what do i input? and where do i input? Laymans terms please. Do i add anything to the command line args? or as a command somewhere. please help</p>
Function signatures to implement new backend

username: nlgranger
cooked: <p>The <a href="https://pytorch.org/tutorials/advanced/extend_dispatcher.html" rel="noopener nofollow ugc">documentation on implementing new backends</a> points to <code>RegistrationDeclaration.h</code>. I noticed that existing implementations register functions with slightly different signatures. For instance:</p>
<ul>
<li><code>Tensor view(const Tensor &amp; self, c10::SymIntArrayRef size);</code></li>
<li>Is declared <a href="https://github.com/pytorch/pytorch/blob/dd2c22f4bba49434b37c4a0b36f7feac29b4d891/aten/src/ATen/native/vulkan/ops/Shape.cpp#LL45C1-L45C62" rel="noopener nofollow ugc">here</a>  for vukan the backend with <code>inline Tensor view(const Tensor&amp; self_arg, IntArrayRef shape)</code>
</li>
</ul>
<p>Is this an oversight? Is the Dispatcher intelligent enough to distinguish implemented versions?</p>

username: arunppsg
cooked: <p><em>implemented version</em> refers to the backend right - cpu/cuda etc? In that case, dispatcher can distinguish across the implementations. It maintains a dispatch table and dispatches the operation to the corresponding backend.</p>
<p>This <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" class="inline-onebox" rel="noopener nofollow ugc">Let’s talk about the PyTorch dispatcher : ezyang’s blog</a> has details on how dispatcher works.</p>

username: nlgranger
cooked: <p>I read that blog post. It’s just that some implementations seem to assume more specialized types in the function arguments. AFAIK the dispatcher won’t know about that.</p>
Shouldn't LSTM and LSTMCell produce identical sequence of hidden states when both are fed one timestep at a time?

username: Miro_Mlynarik
cooked: <p>Hi,</p>
<p>I’m implementing Seq2Seq decoder and the paper says that LSTMCell should be used in a loop for each decoder step. However, I’m not convinced why LSTM itself could not be used. I’d suppose these two procedures should yield the same sequence of hidden states when looped over incrementally. Here’s the code which I thought would give the same results, but it didn’t.</p>
<p>I thought that the source of differences could be because some non-determinism in LSTM layer as is stated in documentation so I also set all possible random seeds, but looks like it’s not the problem.</p>
<pre><code class="lang-auto">import random
import torch
import numpy as np
from torch import nn

random.seed(0)
torch.manual_seed(100)
np.random.seed(0)

torch.use_deterministic_algorithms(True)

lstm = nn.LSTM(input_size=5, hidden_size=5, num_layers=1)
lstm_cell = nn.LSTMCell(5, 5)

h0 = torch.Tensor([0.1, 0.2, 0.12, -0.3, 0.1]).unsqueeze(0)
c0 = torch.Tensor([0.1, 0.2, 0.12, -0.3, 0.1]).unsqueeze(0)
x = torch.randn(8, 5)

with torch.no_grad():
    h, c = h0, c0
    for i in range(8):
        _, (h, c) = lstm(x[i].unsqueeze(0), (h, c))
        print(f"{h=} {c=}")

    print("\n")
    h, c = h0, c0
    for i in range(8):
        h, c = lstm_cell(x[i].unsqueeze(0), (h, c))
        print(f"{h=} {c=}")
</code></pre>

username: Paplham
cooked: <p>Hello Miro,<br>
Indeed <code>nn.LSTM</code> with a single layer contains only a single <code>nn.LSTMCell</code>. However, even though you fix the random seed, both modules get initialized with different weights.</p>
<p>If you set the weights of one of the modules to be identical to the weights of the other one, their outputs should become identical as well. Try modifying your code with:</p>
<pre><code class="lang-auto">lstm_cell.weight_hh = lstm.weight_hh_l0
lstm_cell.bias_hh = lstm.bias_hh_l0
lstm_cell.weight_ih = lstm.weight_ih_l0
lstm_cell.bias_ih = lstm.bias_ih_l0
</code></pre>

username: Miro_Mlynarik
cooked: <p>Okay, looks like you are right. The results are now identical. So it seems that those two layers are equivalent for “single-layer one-step-at-a-time decoding use case”, if we disregard the initialization differences. Thanks.</p>
Pruning channels of pretrained ResNet-50 model while preserving the unpruned channels' weight?

username: King4819
cooked: <p>I want to perform channel-wise pruning on pretrained ResNet-50 model, but I can’t figure out whether there is a method to preserve the unpruned channels’ weights?</p>

username: Paplham
cooked: <p>Hello,<br>
Could you explain your task in more detail?</p>
<p>I assume that you have a convolutional network and for example the first convolutional layer receives an RGB image and outputs 512 channels. Do you want to remove some of these output channels?</p>
<p>Note that these channels are likely also used in the subsequent layers. Therefore the corresponding weights would also need to be removed (or fed with some fixed input).</p>
<p>What is the goal of the pruning? Perhaps I could offer you a different solution to achieve your goal.</p>
<p>If you want to preserve weights of a model, you can save the state dictionary.</p>
<p>You can then modify these weights, e.g., reduce the output dimension of a convolution layer. But these modified weights could not be loaded into the original model. A new model would have to be defined specifically for the pruned weights</p>
<p>However, though I did not try this myself, intuition tells me that performance of the “pruned” model would be potentially much worse than that of the original model (depending on how many modifications are made).</p>

username: Paplham
cooked: <p>Try going through this <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="noopener nofollow ugc">tutorial</a>. It might answer your questions.</p>

username: King4819
cooked: <p>Sorry but my pruning method isn’t pytorch api, is my defined method.</p>
How to rotate a bounding box with `transforms.functional.rotate`?

username: johnny69
cooked: <p>I’m doing an object detection task with FasterRCNN. I want to add data augmentation by rotating the image and the bounding box. I am able to rotate an image (which is just a tensor of size <code>[3, 256, 256]</code>) just fine. However, the bounding box is of format <code>[x, y, x + w, y + h]</code>, and I am not able to rotate this with <code>transforms.functional.rotate</code>.</p>
<p>Any ideas how I can do this rotation? It’s important that I use <code>functional</code> rotations instead of just <code>transforms.Compose</code> so that I can use the <em>same</em> random transformation for both the image and the boxes, instead of the image being rotated 90 degrees and the bounding box being rotated 270 degrees, for example.</p>

username: InnovArul
cooked: <p>AFAIK, you have to formulate an appropriate rotation matrix (3x3) and multiply it with the homogeneous coordinates (Nx3).</p>
<p>An easier way would be to use libraries such as <code>albumentations</code>, <code>imgaug</code>.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://albumentations.ai/">
  <header class="source">
      <img src="https://albumentations.ai/assets/img/custom/albumentations_logo.png" class="site-icon" width="190" height="182">

      <a href="https://albumentations.ai/" target="_blank" rel="noopener">albumentations.ai</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/309;"><img src="https://albumentations.ai/assets/img/custom/albumentations_card.png" class="thumbnail" width="690" height="309"></div>

<h3><a href="https://albumentations.ai/" target="_blank" rel="noopener">Albumentations</a></h3>

  <p>Albumentations: fast and flexible image augmentations</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/aleju/imgaug">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/aleju/imgaug" target="_blank" rel="noopener">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b9f4959b2463b37864d32430672dc62b08d9ec477afac83b17a46a36d3662044/aleju/imgaug" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/aleju/imgaug" target="_blank" rel="noopener">GitHub - aleju/imgaug: Image augmentation for machine learning experiments.</a></h3>

  <p>Image augmentation for machine learning experiments. - GitHub - aleju/imgaug: Image augmentation for machine learning experiments.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: johnny69
cooked: <p>Cheers, used <code>albumentations</code> and it worked well.</p>
Can torch.sparse.mm support the multiplication of two bool vectors

username: nuoyi_ma
cooked: <p>I want to multiply two bool sparse matrices. If the two matrices become dense, it will take up a lot of space. How to multiply two bool sparse matrices?</p>

username: Paplham
cooked: <p>Hello,<br>
<code>torch.sparse.mm</code> currently does not support the multiplication of boolean matrices and will fail with</p>
<pre><code class="lang-auto">RuntimeError: "sparse_matmul" not implemented for 'Bool'
</code></pre>
<p>Therefore, operation such as:</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([[1., 0, 1], [0, 1, 0]], requires_grad=False, dtype=bool).to_sparse()
b = torch.tensor([[0, 1.], [1, 0], [0, 0]], requires_grad=False, dtype=bool).to_sparse()
y = torch.sparse.mm(a, b)
</code></pre>
<p>will fail.</p>
<p>However, you could work around this by casting your bool tensor to a different type, multiplying the tensors, and casting back to bool. For example:</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([[1., 0, 1], [0, 1, 0]], requires_grad=False, dtype=bool).to_sparse()
b = torch.tensor([[0, 1.], [1, 0], [0, 0]], requires_grad=False, dtype=bool).to_sparse()
y = torch.sparse.mm(a.type(torch.FloatTensor), b.type(torch.FloatTensor)).type(torch.BoolTensor)
</code></pre>
<p>Of course, this requires more memory than if the operation was supported for boolean matrices. It could, however, solve your issue if your matrices are sufficiently sparse.</p>
Interpreting GAN Discriminator loss

username: Rituraj_19_Dutta_100
cooked: <p>I’m using a discriminator to classify real and fake instances in terms of segmentation. The loss is Binary Cross Entropy. The implementation looks something like this:</p>
<pre><code class="lang-auto"># Adversarial ground truths
valid = Variable(Tensor(5, 1).fill_(1.0), requires_grad=False)
fake = Variable(Tensor(5, 1).fill_(0.0), requires_grad=False)

real_img = Variable(batch_labels.type(Tensor))   # Ground truth masks 
        
fake_img = Variable(output.type(Tensor)) # Predicted masks

bce = nn.BCELoss().cuda(GPU)

adv_loss = bce(discrimintor(fake_img),valid)   ## Measures Segmenter's ability to fool the discriminator
</code></pre>
<p>I came up with an approach where instead of passing the valid tensor in bce loss we use ground truth masks. Something like this:</p>
<p><code>adv_loss = bce(discrimintor(fake_img),discriminator(real_img))</code></p>
<p>This is the implementation I came up with the adversarial loss. I just wanted to verify if it’s the correct implementation for the approach.</p>

username: Paplham
cooked: <p>In GANs, the discriminator is typically trained to distinguish between <em>real</em> and <em>fake</em> samples.</p>
<p>It can indeed be trained for this using <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">binary cross entropy</a>.</p>
<p>However, I have a hard time understanding the rationale behind your implementation.</p>
<p>What is the goal? Are you attempting to implement standard GAN training? Or is your use case a special one?</p>
<p>Could you please explain the task and the desired behavior in detail? It is difficult to help otherwise.</p>