akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I get the binary format of a float type tensor?

jerly-hjzhou: <p>The problem I met is that how can I get a binary format like 101101010010(assume it’s IEEE754 format, although it’s not) from its corresponding float type value like 0.365(this does not correspond with the binary format before)</p>

ptrblck: <p>You could transform the tensor to a numpy array and use <code>np.binary_repr</code> on it.</p>

stas: <p><code>np.binary_repr</code> only works with integers and not floats. <a href="https://numpy.org/doc/stable/reference/generated/numpy.binary_repr.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.binary_repr — NumPy v1.22 Manual</a></p>

kocchop: <p>Just following up if any new ideas came forward to get the binary/hex format out of a torch float tensor.</p>
<p>I’ve found one by on <a href="https://github.com/KarenUllrich/pytorch-binary-converter" rel="noopener nofollow ugc">Github</a> but it seems like manually converting the data to get the binary. Just wondering if there is any better way. For example, if I can directly read the binary from the memory address or something.</p>
<p>Any response would be much appreciated. Thanks in advance!</p>

jthakurH: <aside class="quote no-group" data-username="jerly-hjzhou" data-post="1" data-topic="106979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerly-hjzhou/48/32653_2.png" class="avatar"> jerly-hjzhou:</div>
<blockquote>
<p>0.365</p>
</blockquote>
</aside>
<p>view tensor in int equivalent.<br>
for example torch.bfloat16–&gt; torch.int16<br>
result = torch.tensor([100], torch.bfloat16)<br>
hex(result.view(torch.int16).item())</p>
Summation of a tensor is giving -inf as output even though no element has -inf value in it

Sanjeet_Patil: <p>The size of tensor is [4,884736].</p>
<p>I am doing the summation with torch.sum(dim=1)</p>
<p>I have checked whether the tensor has float(‘-inf’) value with :-<br>
1] torch.where(tensor == float(‘-inf’), 1, 0) → count the non_zero → output is zero<br>
2] torch.where(tensor &lt; -7, 1, 0) → count the non_zero → output is zero. Thus specifying the no element is lesser than -7<br>
3] tensor[tensor == float(‘-inf’)] == 0. Still the result is -inf.</p>
<p>Kindly suggest.</p>

ptrblck: <p>Could you post the <code>min()</code> and <code>max()</code> values of this tensor you are trying to reduce?</p>

Sanjeet_Patil: <p>Here are the max() and min() value of the tensor obtained with torch.max() and torch.min()<br>
MAX_VALUE = -0.0002<br>
MIN_VALUE = -4.9502</p>

Sanjeet_Patil: <p>Thank you for replying.<br>
This issue can be closed.<br>
It was solved by converting the tensor, before summation, from torch.float16 to torch.float64.</p>
Softmax Output Values Do Not Sum to 1 in PyTorch

feltface: <p>I’m encountering an issue when applying the softmax function to a tensor in PyTorch, where the returned values do not sum to 1 as expected. Below is the code I am using:</p>
<pre><code class="lang-auto">import torch

torch.set_printoptions(precision=12, sci_mode=False)

a = torch.tensor([15.5438404083251953125000000, -7.4692978858947753906250000, -7.7074594497680664062500000])

soft = torch.nn.functional.softmax(a, dim=0)
# Softmax output
print(soft)
# Output:
# tensor([1.000000000000, 0.000000000101, 0.000000000080])
</code></pre>
<p>The softmax output gives a tensor where the values are very close to 0 and 1, but they do not sum exactly to 1. I understand this may be due to floating-point precision issues, but I’m not sure how to resolve this.</p>
<ol>
<li>How can I ensure the softmax output values sum to 1?</li>
<li>Is there a more stable or accurate way to apply softmax in this context to avoid precision issues?</li>
</ol>

Marlene495Hadley: <p>Hello!<br>
The issue with softmax not summing to 1 is due to floating-point precision. To resolve this, convert your tensor to double precision before applying softmax, use log_softmax for better stability, or manually normalize the softmax output by dividing it by its sum. These methods should help achieve more accurate results.<br>
Best Regards,<br>
<a href="https://www.marykayin-touch.com" rel="noopener nofollow ugc">Mary Kay InTouch</a></p>
Training Trigram Language model gives indexing error

Aabid_Karim: <p>I am training a Trigram model where I have created two dictionaries as below, to give index to each character.</p>
<pre><code class="lang-auto">stoi = {s: i+1 for i, s in enumerate(chars)}
stoi['.'] = 0
itos = {i:s for s,i in stoi.items()}
</code></pre>
<p>I have torch array of 3D. And I converted all the entries of array to probabilities as given below</p>
<pre><code class="lang-auto">P = (N+2).float()
P = P / P.sum(1, keepdims = True)
</code></pre>
<p>I didn’t voilate any broadcasting rules and the shape of P is 3D.<br>
Now I am using torch.multinomial to draw a sample from my probablity distribution using the code below.</p>
<pre><code class="lang-auto">g = torch.Generator().manual_seed(2147483647)

for i in range(10):
    out = []
    ix = 0  
    while True:
        p = P[ix]
        ix = torch.multinomial(p, num_samples=1, replacement=True, generator=g)
        out.append(itos[ix])  
        if ix == 0:  
            break
    print(''.join(out))

</code></pre>
<p>But the above code does not draw samples from probability distribution, and gives me the below output.</p>
<p>""KeyError                                  Traceback (most recent call last)<br>
 in &lt;cell line: 3&gt;()<br>
7         p = P[ix]<br>
8         ix = torch.multinomial(p, num_samples=1, replacement=True, generator=g)<br>
----&gt; 9         out.append(itos[ix])<br>
10         if ix == 0:<br>
11             break</p>
<p>KeyError: tensor([[10],<br>
[14],<br>
[18],<br>
[10],<br>
[ 1],<br>
[14],<br>
[ 0],<br>
[17],<br>
[ 5],<br>
[15],<br>
[ 0],<br>
[25],<br>
[ 0],<br>
[ 3],<br>
[ 1],<br>
[ 2],<br>
[ 5],<br>
[16],<br>
[ 5],<br>
[25],<br>
[ 8],<br>
[ 0],<br>
[17],<br>
[ 8],<br>
[24],<br>
[ 1],<br>
[25]])  “”</p>
<p>Can someone explain where there is an error in indexing, why itos dictionary doesn’t get proper index from probability distribution.</p>

ptrblck: <p>Could you post the size of <code>itos</code> as well as how <code>P</code> (or <code>N</code>) was created?</p>

Aabid_Karim: <p>There are total 26 alphabats and itos (integer to string) is a dictionary that stores string with its index, the code is below.</p>
<pre><code class="lang-auto">N = torch.zeros((27,27,27), dtype = torch.int32)
chars = sorted(list(set(''.join(words))))
stoi = {s: i+1 for i, s in enumerate(chars)}
stoi['.'] = 0
itos = {i:s for s,i in stoi.items()}
itos
</code></pre>
<p>Itos:<br>
{1: ‘a’,<br>
2: ‘b’,<br>
3: ‘c’,<br>
4: ‘d’,<br>
5: ‘e’,<br>
6: ‘f’,<br>
7: ‘g’,<br>
8: ‘h’,<br>
9: ‘i’,<br>
10: ‘j’,<br>
11: ‘k’,<br>
12: ‘l’,<br>
13: ‘m’,<br>
14: ‘n’,<br>
15: ‘o’,<br>
16: ‘p’,<br>
17: ‘q’,<br>
18: ‘r’,<br>
19: ‘s’,<br>
20: ‘t’,<br>
21: ‘u’,<br>
22: ‘v’,<br>
23: ‘w’,<br>
24: ‘x’,<br>
25: ‘y’,<br>
26: ‘z’,<br>
0: ‘.’}</p>
Fixed scale and zero point with FixedQParamsObserver

zoodle: <p>I’m new to pytorch, so likely doing something silly!  I’m trying to do a simple 2D convolution with quantized weights where the scale and zero point are set manually.  I assumed that the FixedQParamsObserver would be suitable for this.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.ao.quantization as quantization

class M(nn.Module):
	def __init__(self):
		super(self).__init__()
		self.quant = quantization.QuantStub()
		self.conv = nn.Conv2d(4, 12, 3, stride=1, padding=1, padding_mode='reflect', bias=False)
		self.dequant = quantization.DeQuantStub()
	def forward(self,x):
		x = self.quant(x)
		x = self.conv(x)
		x = self.dequant(x)
		return x
		
weight_fq = quantization.FakeQuantize.with_args(observer=quantization.FixedQParamsObserver,
												quant_min=-8,
												quant_max=7,
												dtype=torch.qint8,
												qscheme=torch.per_tensor_symmetric,
												scale=1/16,
												zero_point=0)

act_fq = quantization.FakeQuantize.with_args(observer=quantization.MinMaxObserver,
											 quant_min=0,
											 quant_max=255,
											 dtype=torch.quint8,
											 qscheme=torch.per_tensor_affine,
											 reduce_range=False) # NOT USED

model = M()
model.train()
model.qconfig = quantization.QConfig(activation=act_fq, weight=weight_fq)
model_prepared = quantization.prepare_qat(model)
</code></pre>
<p>However the last line gives me an error: TypeError: <strong>init</strong>() got an unexpected keyword argument ‘factory_kwargs’</p>
<p>What am I doing wrong?</p>

jerryzh168: <p>you need to use <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fake_quantize.py#L264" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization/fake_quantize.py at main · pytorch/pytorch · GitHub</a> for FixedQParams Observer I think</p>

Yang_Liu: <p>Have you solved it yet? Seems that not all the observers have the “factory_kwargs” parameters during the init process. Then error appears for FixedQParamsObserver.<br>
It can be avoided by adding the “factory_kwargs=None” in the <strong>init</strong> func for “class FixedQParamsObserver(ObserverBase)”, but I’m not sure if any other bug will appear.</p>
How to use Node splitting graph optimization?

JUNSUNGKIM99: <p>Hi !<br>
I read the document in glow docs. <a href="https://github.com/pytorch/glow/blob/master/docs/NodeSplitting.md" rel="noopener nofollow ugc">Node splitting</a><br>
However, node splitting cannot be used because it does not specify when, where, and how to use it. What should we do specifically to use this graph optimization technique?</p>
<p>Where should I put this option in to compile the DNN?<br>
Thanks</p>
How to know *real* GPU utilization?

fzyzcjy: <p>I know <code>nvidia-smi</code> can show GPU utilization, but it does not seem to reflect real utilization. For example, even if it says 100% utilization, it seems only to mean all cores are activated, but may only use a small portion of the theoretical FLOPs.</p>
<p>It would be great to have something to provide a number representing how full the cores are actually utilized.</p>
<p>I guess <code>nsight</code> may be useful, but not sure whether people use it for this usage.</p>
<p>Thanks in advance for any suggestions!</p>
Torch.fx.GraphModule with Torch.jit.script gives warning

c-earth: <p>I am working on a model that utilize GraphModule, and convert it to TorchScript. The model I trained gives reasonable results, but every time I initialize a new model the following warning always pop up.</p>
<p>UserWarning: The TorchScript type system doesn’t support instance-level annotations on empty non-base types in <code>__init__</code>. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in <code>torch.jit.Attribute</code>.</p>
<p>When I looked at <code>Torch.fx.GraphModule.__init__</code>, I found these lines</p>
<pre><code>    # Dictionary to store metadata
    self.meta: Dict[str, Any] = {}
    self._replace_hook = None
    self._create_node_hooks: List[Callable] = []
    self._erase_node_hooks: List[Callable] = []
</code></pre>
<p>which likely are the reasons for the warning. I tried deleting the annotations to</p>
<pre><code>    # Dictionary to store metadata
    self.meta = {}
    self._replace_hook = None
    self._create_node_hooks = []
    self._erase_node_hooks = []
</code></pre>
<p>and the warning disappear. I would like to know the reason behind the warning, i.e., why jit.script doesn’t want the annotations of empty collections in <code>__init__</code>, and what is the proper practice to handle this without modifying <code>Torch.fx.GraphModule.__init__</code>.</p>
<p>My model is based on python-3.11.9, and torch-2.4.1 with cuda-11.8.</p>
Compatibility issues between OpenCV and Torchvision

AlfonsoGiso: <p>I’m having some troubles with a project on artificial vision where I need to use, among the various, OpenCV and torchvision libraries. In particular I have these versions:</p>
<p>Opencv-python Version: 4.10.0.84<br>
Torchvision Version: 0.19.1<br>
Torch Version: 2.4.1</p>
<p>I also tried with a simple Python script to ensure that the problem was not of my code</p>
<pre><code class="lang-auto">import torch
import torchvision
import cv2
import numpy as np

# Crete a black image
img = np.zeros((100, 100, 3), dtype=np.uint8)

# Show the image
cv2.imshow('Test Window', img)
cv2.waitKey(0)
cv2.destroyAllWindows()
</code></pre>
<p>If I run this code with both torchvision and cv2, my script freezes, but if I try without torchvision no problems occur. The main problem seems also to be relative to the imshow() function.</p>

ptrblck: <p>I cannot reproduce the issue and see the expected behavior waiting for a key input and closing the window afterwards using the same versions:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/039f144e2cdd493db4e6645e060ebcf5f32e4b60.png" data-download-href="https://discuss.pytorch.org/uploads/default/039f144e2cdd493db4e6645e060ebcf5f32e4b60" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/039f144e2cdd493db4e6645e060ebcf5f32e4b60_2_690x229.png" alt="image" data-base62-sha1="w2g0OSlWG3WQiEajdKEI6IBvJC" width="690" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/039f144e2cdd493db4e6645e060ebcf5f32e4b60_2_690x229.png, https://discuss.pytorch.org/uploads/default/original/3X/0/3/039f144e2cdd493db4e6645e060ebcf5f32e4b60.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/3/039f144e2cdd493db4e6645e060ebcf5f32e4b60.png 2x" data-dominant-color="56364B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">942×313 37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Video analysis development help with tracking 40 yard dash times and ball throwing metrics

SteveKahle: <p>Hi PyTorch Community!</p>
<p>I want to understand if the two use cases below can be accomplished and, if not, to identify the gaps.</p>
<p>Does anyone have development experience in these areas?</p>
<p><strong>Running use case.</strong> A person uploads a video of someone running the 40-yard dash. We want the 10, 20, 30, and 40-yard times, along with the top speed and acceleration times in mph, calculated automatically through video analysis software.</p>
<p><strong>Ball use case</strong>: A person uploads a video of someone throwing a ball. We want the ball’s top speed and rotational spin rate to be calculated automatically through video analysis software.</p>
GPU memory spike with torchvision.transforms

edwardjjj: <p>Hi, guys. I’m currently a first year Ph.D student in robotics. In my current project I need to train an image based diffusion policy. During training, I find very high VRAM usage. Following is some training details:<br>
Param size: 90m<br>
Precision: float32<br>
Batch size: 512<br>
Image shape: (3, 224, 224) already on ‘cuda’ device</p>
<p>I tried to debug the memory usage by stepping through the training loop in a debugger. The weird thing I find is that, after passing through a few torchvision.transforms, the memory usage increased by ~5GB. The following is my transforms.</p>
<pre data-code-wrap="python"><code class="lang-python">transform_train = transforms.Compose(
           [
               transforms.Resize([240, 320], antialias=True),
               transforms.ColorJitter(
                   brightness=0.3, contrast=0.3, saturation=0.3, hue=0.3
               ),
               transforms.GaussianBlur(kernel_size=5, sigma=(0.01, 2.0)),
               transforms.CenterCrop((input_size[0], input_size[1] - 2 * margin)),
]
a = torch.randn([512, 3, 224,224]).to('cuda')
b = transform_train(a)
</code></pre>
<p>I’m assuming the transforms does not require gradient, so there shouldn’t be such a high memory usage. What am I missing here?</p>

jrog: <p>Tensor <code>a</code>, of shape <code>512, 3, 224,224</code> holds <code>117964800</code> values in 32 bits, meaning moving this tensor to GPU already takes around ~472 MB of VRAM.</p>
<p>As far as I know, each step of transforms may require additional GPU VRAM for calculations/caching, which after adding up, make it possible to take few GB of VRAM.</p>
<p>you can add at the end of the code that you provided</p>
<pre><code class="lang-auto">print(torch.cuda.memory_summary(device='cuda'))
</code></pre>
<p>and pay attention to <em>Allocated memory</em> and <em>GPU reserved memory</em>. In my case, the GPU reserved memory shows few times higher value then allocated, meaning CUDA has reserved a lot of additional VRAM for future potential use.</p>
OneCycleLR Learning Rate vs. Optimizer Learning Rate

moth: <p>I am using the <code>OneCycleLR</code> scheduler. It has a <code>max_lr</code> parameter which is the:</p>
<blockquote>
<p>Upper learning rate boundaries in the cycle for each parameter group.</p>
</blockquote>
<p>I also have an <code>Adam</code> optimizer. This optimizer also has a learning rate.</p>
<p>Will the optimizer’s learning rate be overwritten by the scheduler’s? How do they relate?</p>
<p>I guess the only point of specifying a learning rate in the optimizer is if you do not use any scheduler, in which the learning rate will be constant throughout the training stage, but I am not entirely sure.</p>

jonathanjsmith: <p>Creating the scheduler <code>OneCycleLR</code> will change the learning rate of the optimizer. So, there is not really a point of specifying the learning rate in the optimizer if you use a scheduler, but the optimizer will have a learning rate by default when its created, but it will just be changed if you pass it to a scheduler.</p>

sad_robot: <aside class="quote no-group" data-username="jonathanjsmith" data-post="2" data-topic="190229" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jonathanjsmith/48/64199_2.png" class="avatar"> jonathanjsmith:</div>
<blockquote>
<p>Creating the scheduler <code>OneCycleLR</code> will change the learning rate of the optimizer. So, there is not really a point of specifying the learning rate in the optimizer if you use a scheduler, but the optimizer will have a learning rate by default when its created, but it will just be changed if you pass it to a scheduler.</p>
</blockquote>
</aside>
<p>To play devil’s advocate it seems that in the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html" rel="noopener nofollow ugc">Pytorch docs</a> they show the optimizer being given a different learning rate (0.1) from the max_lr (0.01) in the OneCycleLR scheduler.</p>
<p>It seems strange they would specify optimizer LR at all much less to a different value if it did nothing…</p>
How to load data for hierarchical multi label classificationa

amy2: <p>I have a classification task where I have a parent class and 3 subclasses<br>
and each for subclasses takes 2 labels also “label1.1” and “label2.1”<br>
the prediction of the parent class depends on the prediction of the subclasses<br>
my question is how should I pass the images along with their labels to the model</p>
<p>As a first attempt  for all the images I created a csv file with this structure for each of the images</p>
<pre><code class="lang-auto">    file_id , label1 , label2
</code></pre>
<p>and then used dataloaders to pass it to a multioutput model<br>
but now I have to add a parent class.the first thought that came to my mind was to create another csv file with this structure</p>
<pre><code class="lang-auto">    parent_id , label_value 
</code></pre>
<p>But I didn’t know how to proceed ?</p>
Flex Attention Score_mod with learnable parameters

berkelium: <p>Hello!,</p>
<p>i want to use the new flex_attention function to implement a biased attention. At my case i have a graph transformer and want to modify the attention scores based on the shortest distances between nodes. I have an Embedding module that has a learned parameter for each distance. My question is:<br>
is it possible to call an external module at my case the Embedding module that has a learned parameter for each distance from the score_mod fuction or is it another way to do this?<br>
Thank you in Advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>here is a code to demonstrate:</p>
<pre><code class="lang-auto">class GraphAttnBias(nn.Module):
    def __init__(self,  cfg):
        super().__init__()
        self.attn_bias = nn.Embedding(num_embeddings=cfg.vocab_size, embedding_dim=cfg.num_heads)
 
def forward(self,idx):
        return self.attn_bias(idx)

class CausalFlexAttention(nn.Module):
    
    def __init__(self, cfg, embed_dim, num_heads, dropout, bias):
        super().__init__()
        assert embed_dim % num_heads == 0
        self.c_attn = nn.Linear(embed_dim, 3 * embed_dim, bias=bias)
        self.c_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.attn_dropout = nn.Dropout(dropout)
        self.resid_dropout = nn.Dropout(dropout)
        self.n_head = num_heads
        self.n_embd = embed_dim
        self.dropout = dropout
       
        # stuff for attention bias
        attn_bias = GraphAttnBias(cfg)
    
    def score_mod(score, batch, head, q_idx, k_idx):
    #here i want to call the embedding module with the distances like this 
    assuming distance has shape (Batch_length* seq_length * seq_length)

        return score + attn_bias(distance[batch,q_idx, k_idx])

    def causal_mask(b, h, q_idx, kv_idx):
        return q_idx &gt;= kv_idx
    
    def forward(self, x):
        B, T, C = x.size()
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        
        y = flex_attention(
            q,
            k,
            v,
            score_mod= CausalSelfAttention.score_mod,
            block_mask= CausalSelfAttention.casual_mask
        )

        y = y.transpose(1, 2).contiguous().view(B, T, C)
        y = self.resid_dropout(self.c_proj(y))
        return y
</code></pre>
How to reset buffers of a module in train mode forward pass?

kkew3: <p>I’m interested in computing <code>y</code> that depends on many inputs and doing backward on it, as in this snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">net = ...  # This is an nn.Module in train mode with many parameters.
y = 0
zs = []
# `list_of_inputs` contains many `x`s
for x in list_of_inputs:
    z = net(x)  # Assume each z is a scalar for simplicity.
    zs.append(z)
zs = torch.cat(zs)
# The log-sum-exp is exemplary.
y = torch.logsumexp(zs, 0)
y.backward()
</code></pre>
<p>This snippet could easily eat up VRAM as there will be <code>len(list_of_inputs)</code> copies of network parameters attached to the computation graph before finally doing backward on <code>y</code>.</p>
<p>My solution is to use the chain rule. Denote the model parameters with p. Then dy/dp = [sum over i] dy/dzi dzi/dp. Computing dy/dzi is cheap, since there are few parameters involved. After that, we may call backward on each zi, feeding it with dp/dzi. I conjecture the following snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">net = ...  # The large model, in train mode.
zs = []
for x in list_of_inputs:
    z = net(x)  # NOTE #1
    zs.append(z.detach())
zs = torch.cat(zs).requires_grad_()
y = torch.logsumexp(zs, 0)
y.backward()
zs_grad = torch.empty_like(zs).copy_(zs.grad)
for x, dz in zip(list_of_inputs, zs_grad):
    z = net(x)  # NOTE #2
    # Accumulate gradient at the model parameters,
    # freeing the attached network parameters.
    z.backward(gradient=dz)
</code></pre>
<p>The snippet should work well, unless submodules in <code>net</code> such as <code>BatchNorm</code> or <code>Dropout</code> exist. They will cause the <code>z</code> computed at <code>NOTE #1</code> to be different from <code>z</code> at <code>NOTE #2</code>, breaking the chain rule. Such a problem is exactly why this topic is queried. I speculate that the problem might be resolved by somehow “undoing” the model buffer state after computing dy/dz, which is where the title is from.</p>
<p>Are there any solutions to it? Alternative approaches to compute dy/dp without exploding VRAM are also welcome. Thank you so much!</p>
Compiling Yolov3 with glow

Romario: <p>Hello I am trying to compile <a href="https://github.com/onnx/models/tree/main/vision/object_detection_segmentation/yolov3" rel="noopener nofollow ugc">Yolov3</a> from an ONNX Model file which is currently not possible since the glow does not support Slicing operations with a stepsize not equal to one.</p>
<p>The Yolov3 Model uses a Slicing operation with a stepsize of -1 and a backwards iteration over the tensor which should be equivalent to a simple one dimensional flip operation.<br>
The <a href="https://github.com/onnx/onnx/blob/main/docs/Operators.md" rel="noopener nofollow ugc">ONNX Operator list</a> does not support a Flip operation but the glow ONNX model Loader does could one not simply change the ONNX nodes from a negative Slice to a Flip to be able to compile the model?</p>
<p>Otherwise as far as I have seen from the Documentation, open Issues and the Code Base is that extending the support for Slicing operations is not to be trivially achieved?</p>
Using quantized operators: Embedding bag C++ API

Abhiraj_Kanse: <p>Hi,<br>
I am trying to call the quantized C++ API of embedding bag operator <a href="https://github.com/pytorch/pytorch/blob/c0d2f991b14d50f8081d788d4a3dc6584ee15502/aten/src/ATen/native/quantized/library.cpp#L137C33-L137C78" rel="noopener nofollow ugc">[embedding bad 4 bit rowwise offsets]</a>  CPP API but i face issues in doing so.</p>
<p>when i try to call it through the at::native namespace it throws the below error while compiling:</p>
<blockquote>
<p>error: ‘embedding_bag_4bit_rowwise_offsets’ is not a member of ‘at::native’ 30 |   torch::Tensor emb = at::native::embedding_bag_4bit_rowwise_offsets(weight,input,offsets,false, 0, false, nullptr, nullptr, false);</p>
</blockquote>
<p>I see that the README.md <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native/quantized#readme" rel="noopener nofollow ugc">ATEN native quantization</a> here mentions at the end that the “** You should not need to use the registered kernels in C++. Although officially not supported , you can use the following**”.<br>
Does this mean that we cannot call the c++ api of the quantized embedding bag ?</p>
<p>Anyway I tried the method mentioned in there and it still throws error while compiling:</p>
<blockquote>
<p>error: ‘const class c10::OperatorHandle’ has no member named ‘call’<br>
7 |   return op.call&lt;torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor, bool, int, bool, torch::Tensor, torch::Tensor, bool&gt;(weight, indices, offsets, scale_grad_by_freq, mode, pruned_weights, per_sample_weights, compressed_indices_mapping, include_last_offset);</p>
</blockquote>
<p>I am basically confused if it is possible to use these quantized operators in CPP and if so how?<br>
(using in python seems pretty easy “torch.ops.quantized.&lt;operator_name&gt;”)</p>
How can I save weights/model in stratified K fold cross validation?

edgy_sharma: <p>I am dealing with IDRID disease grading which is a small &amp; imbalanced and also my architecture is complex, so I am using k-fold here. But I am not sure how and at what point should I save weights. Please help!</p>

Soumya_Kundu: <p>What is your final objective? To ensemble?</p>
<blockquote>
<p>also my architecture is complex, so I am using k-fold here</p>
</blockquote>
<p>Why so? You generally use k-fold to gauge robustness/generizability, not much to do with the complexity of the architecture right?</p>

edgy_sharma: <p>so i’ll have 5 files for 5 folds this way, how to load these weights during eval in code?</p>
Memory requirement for torch.matmul

Abhishek_Tyagi: <p>Hi Everyone,<br>
I have the following line of code in Pytorch:</p>
<pre><code class="lang-auto">W = torch.matmul(mask, V_scaled_diag)
</code></pre>
<p>Let us assume that the mask’s dimension is <code>2304x768</code>; similarly, V_scaled_diag’s dimension is <code>768x2304</code> and W is also of dimension <code>2304x768</code>. The memory required to do this computation should be:</p>
<pre><code class="lang-auto">Mem = 3x2304x768x4 Bytes (assuming Float32) + x Bytes
</code></pre>
<p>where <code>x Bytes</code> are required to store the gradients, intermediate tensors etc.</p>
<p>Given the size of matrices, is there an easy way to estimate what this extra usage (value of <code>x</code>) might be?</p>
<p>Also, is it possible to check the occupied memory by each tensor in the computation above?</p>
<p>Any inputs are appreciated.</p>

ptrblck: <p>The cublas workspace will use some memory and you could clear it via <code>torch._C._cuda_clearCublasWorkspaces()</code> if needed:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

print(torch.cuda.memory_allocated() / 1024**2)
# 0.0
print(torch.cuda.memory_reserved() / 1024**2)
# 0.0

a = 2304
b = 768
mask = torch.randn(a, b, device="cuda")
V_scaled_diag = torch.randn(b, a, device="cuda")
print(torch.cuda.memory_allocated() / 1024**2)
# 13.5
print(torch.cuda.memory_reserved() / 1024**2)
# 20.0
print("expected: {}MB".format((a*b*4 + b*a*4)/1024**2))
#expected: 13.5MB

W = torch.matmul(mask, V_scaled_diag)
print(torch.cuda.memory_allocated() / 1024**2)
# 41.875
print(torch.cuda.memory_reserved() / 1024**2)
# 62.0
actual = (a*b*4 + b*a*4 + a*a*4)/1024**2
print("expected: {}MB".format(actual))
# expected: 33.75MB
print("delta: {}MB".format(actual - torch.cuda.memory_allocated() / 1024**2))
# delta: -8.125MB

# release cublas workspace
torch._C._cuda_clearCublasWorkspaces()
print(torch.cuda.memory_allocated() / 1024**2)
# 33.75
print(torch.cuda.memory_reserved() / 1024**2)
# 62.0
print("expected: {}MB".format((a*b*4 + b*a*4 + a*a*4)/1024**2))
# expected: 33.75MB
</code></pre>

Abhishek_Tyagi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I would like to understand better how Pytorch allocates memory to intermediate or backprop tensors so that I can justify one implementation compared to another. Do you have any recommendations?</p>

ptrblck: <p>Intermediate activations are stored if a differentiable computation graph is created and if these activations are needed for the gradient calculation during the backward call. The <code>derivatives.yaml</code> file defines which tensors are needed.<br>
Thus it depends a bit on your actual use case and a small example would be great to have.</p>
Extracting embeddings from log probabilities

Debodeep_Banerjee: <p>Hi, I am solving an image captioning related issue and eventually I have extract the embeddings of the tokens. One possible way is to extract the embeddings using the tokens. But I cannot do do that because in my case the whole process needs to be differentiable and the tokens are not differentiable. Instead, what I am currently doing is the following -<br>
embeddings= self.model.encoder_decoder.model.tgt_embed[0].lut<br>
embedded_sentence_pred = torch.matmul(seq_logps, embeddings.weight)<br>
I am not entirely sure if I am doing it correctly. Can you provide any insight?</p>
Opacus for 3D Segmentation

Ziva1011: <p>Dear Opacus community,</p>
<p>I’ve been looking into 3D segmentation models for medical imaging. I use several architectures, half of them from the Monai library.</p>
<p>When I try to run Opacus in combination with these architectures I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/vol/aimspace/users/viulapir/Documents/dp-thesis/test2.py", line 473, in main
    training_losses, validation_losses, lr_rates = trainer.run_trainer()
  File "/vol/aimspace/users/viulapir/Documents/dp-thesis/trainer.py", line 48, in run_trainer
    self._train()
  File "/vol/aimspace/users/viulapir/Documents/dp-thesis/trainer.py", line 106, in _train
    self.optimizer.step()  # update the parameters
  File "/u/home/viulapir/.conda/envs/torchsegmentation2/lib/python3.10/site-packages/opacus/optimizers/optimizer.py", line 553, in step
    if self.pre_step():
  File "/u/home/viulapir/.conda/envs/torchsegmentation2/lib/python3.10/site-packages/opacus/optimizers/optimizer.py", line 536, in pre_step
    if self.grad_samples is None or len(self.grad_samples) == 0:
  File "/u/home/viulapir/.conda/envs/torchsegmentation2/lib/python3.10/site-packages/opacus/optimizers/optimizer.py", line 342, in grad_samples
    ret.append(self._get_flat_grad_sample(p))
  File "/u/home/viulapir/.conda/envs/torchsegmentation2/lib/python3.10/site-packages/opacus/optimizers/optimizer.py", line 279, in _get_flat_grad_sample
    raise ValueError(
ValueError: Per sample gradient is not initialized. Not updated in backward pass?
</code></pre>
<p>The model that generated this error above was the Vnet (but this happens with other architectures too).</p>
<p>I now that Monai uses batch normalization layers so I used ModuleValidator.fix(model) to convert them to group normalization layers. I also tried to initialize the weights using torch.init but it produced the same error</p>
<p>Simplified version of the code is as below.</p>
<pre><code class="lang-auto"> model= VNet( spatial_dims=3, in_channels=1, out_channels=3)
model = ModuleValidator.fix(model)
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

        privacy_engine = PrivacyEngine(accountant="gdp")
      
        
        model, optimizer, train_dl = privacy_engine.make_private_with_epsilon(
            module=model,
            optimizer=optimizer,
            data_loader=train_dl,
            target_epsilon=8,
            target_delta=1e-2,
            epochs=epochs,
            max_grad_norm=1
        )
        train_dl.collate_fn = wrap_collate_with_empty(
            collate_fn=list_data_collate,
            sample_empty_shapes={x: train_dl.dataset[0][x].shape for x in ["img", "seg"]},
            dtypes={x: train_dl.dataset[0][x].dtype for x in ["img", "seg"]},
        )
</code></pre>
<p>Does anyone have an idea of what i am missing here? Or is opacus not compatible with these models and I should raise a feature request on Github?</p>
<p>Thank you!</p>
Why no_shard strategy is deprecated in FSDP

YUCHAO_GU: <p>In the code located at <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/_init_utils.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/fsdp/_init_utils.py at main · pytorch/pytorch · GitHub</a> Line 443, when I use the no-shard strategy in FSDP to switch to DDP, it displays a warning. I’m curious about why the NO-SHARD option is deprecated.</p>
<p>"elif sharding_strategy == ShardingStrategy.NO_SHARD:<br>
warnings.warn(<br>
“The <code>NO_SHARD</code> sharding strategy is deprecated. If having issues, "<br>
“please use DistributedDataParallel instead.”,<br>
# Level 1 is here, level 2 is from <code>FullyShardedDataParallel</code>, and<br>
# level 3 is from the true caller<br>
stacklevel=3,<br>
)”</p>

irisz: <p>If you are looking to use “NO_SHARD”, would you want to use DDP instead? 'NO_SHARD` is deprecating, because DDP offers the same option.</p>
How to implement keras.layers.core.Lambda in pytorch

11112: <h1>add a x -&gt; x^2 layer</h1>
<p>model.add(Lambda(lambda x: x ** 2))</p>

fmassa: <p>That is one of the cases where it’s better not to use <code>nn.Sequential</code>, and inherit from model yourself and perform the operations that you want.<br>
For example</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def forward(self, input):
        return input ** 2 + 1
model = MyModel()
</code></pre>
<p>But if you want an equivalent to a <code>Lambda</code> layer, you can write it very easily in pytorch</p>
<pre><code class="lang-python">class LambdaLayer(nn.Module):
    def __init__(self, lambd):
        super(LambdaLayer, self).__init__()
        self.lambd = lambd
    def forward(self, x):
        return self.lambd(x)
</code></pre>
<p>And now you can use it as you would in keras</p>
<pre><code class="lang-python">model.add(Lambda(lambda x: x ** 2))
</code></pre>

11112: <p>Your method is valid，thank you very much</p>

vivy: <pre data-code-wrap="python"><code class="lang-python">from torchvision import transforms
import torch

LamdaLayer = torch.jit.script(torch.nn.Sequential(
      transforms.Lambda(lambda x: &lt;expr&gt; ))
)
</code></pre>
Looking for a lightweight GPU-compatible Docker base image for deploying a PyTorch pretrained model on GCP

Manish_Thakur: <p>I am trying to deploy a pretrained PyTorch model on Google Cloud Platform (GCP). The model works fine on a CPU, but when I add GPU support to the service, the GPU memory is not utilized. Upon investigation, I found that I am using a <code>python:3.10</code> base image, and I realize that I should be using a CUDA-compatible base image for GPU support.</p>
<p>Currently, I am installing PyTorch and torchvision using pip, but the <code>requirements.txt</code> I received from the pretrained model has the following comment:</p>
<p><code>torch==1.13.1+cu116  # Kept as comment for version reference</code></p>
<p>However, when building the service, I encounter an error saying that this specific version is not available.</p>
<p>Can you suggest a lightweight GPU-compatible base image that includes PyTorch and torchvision versions with GPU support? I am looking for a base image that will work with CUDA on GCP.</p>
Clang and fbgemm

elsampsa: <p>Fighting to get pytorch (2.4) to compile in ROCm…</p>
<p>Clang compiler doesn’t like the fbgemm submodule:</p>
<pre><code class="lang-auto">/home/sampsa/amd/torch-compile/pytorch-amd/third_party/fbgemm/src/FbgemmI8Spmdm.cc:188:41: error: variable length arrays in C++ are a Clang extension [-Werror,-Wvla-cxx-extension]
  188 |       alignas(64) uint8_t A_temp_buffer[K * 32];
</code></pre>
<p>I tried by adding <code>-Wno-vla-extension -fms-extensions</code> to the compiler command, but that did not help.</p>
<p>Should I just disable fbgemm then?  I already disabled MKLDNN as there was no .-way to make it work with the clang-based hipcc compiler and OpenMP.</p>

elsampsa: <p>Now I’m not even sure if that was just a warning message… there was also problems in finding “.d” files.  After disabling fbgemm, I am getting</p>
<pre><code class="lang-auto">error: error opening 'caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/DLConvertor.cpp.o.d': No such file or directory
7 warnings and 3 errors generated when compiling for gfx906
</code></pre>
<p>At least that last error message disappeared when I switched to cmake version 3.26.  Cmake 3.30+ &amp; seem to create a broken build system with missing “.d” (build record) files.  Quite diabolic.</p>
<p>To summarize: I disabled fbgemm and downgraded to cmake 3.26.  Not sure if the first one was necessary…  Problem solved.</p>
Bfloat16 type, cuda, pytorch or gpu issue

robofar123: <p>Hello. When I try to run following code snippet:</p>
<pre><code class="lang-auto">self.metric3d = torch.hub.load('yvanyin/metric3d', 'metric3d_vit_small', pretrain=True).to(self.device)
pred_depth, confidence, output_dict = self.metric3d.inference({'input': rgb})
</code></pre>
<p>I get following error:</p>
<pre><code class="lang-auto">raise RuntimeError('Current CUDA Device does not support bfloat16. Please switch dtype to float16.')
RuntimeError: Current CUDA Device does not support bfloat16. Please switch dtype to float16.
</code></pre>
<p>RGB image <code>dtype</code> is <code>torch.float32</code>. Is this error related to <code>dtype</code> of image, or that simply my graphics card does not support type <code>bfloat16</code>?</p>
<p><code>Please switch dtype to float16</code> =&gt; does this mean to switch <code>dtype</code> or RGB to <code>torch.float16</code> in my code or it refers to code of your project?</p>
<p>If I cast my RGB to <code>torch.float16</code> then error is:</p>
<pre><code class="lang-auto">RuntimeError: Input type (c10::Half) and bias type (float) should be the same
</code></pre>
<p>My graphics card is: GeForce RTX 2080 SUPER</p>
<p>So I would like to know how to solve issue and whether this issue is related to the:</p>
<ol>
<li>pytorch and specific version</li>
<li>cuda and specific version</li>
<li>gpu device itself (i.e. that gpu device simply does not support this type)</li>
<li>something else</li>
</ol>

ptrblck: <p>The error is raised from <code>autocast</code> using <code>bfloat16</code> as it’s lower <code>dtype</code>, which is not supported on your GPU as <code>bfloat16</code> operations were added for Ampere+ GPUs.<br>
Thus <a href="https://github.com/pytorch/pytorch/blob/c35b9535319b9ba0b0a0c759759bfce2ff387e01/torch/cuda/__init__.py#L132"><code>is_bf16_supported</code></a> fails and raises the error from <a href="https://github.com/pytorch/pytorch/blob/c35b9535319b9ba0b0a0c759759bfce2ff387e01/torch/amp/autocast_mode.py#L323">here</a> (assuming you are using a recent PyTorch release).</p>
<p>Based on the code you’ve provided it seems an <code>autocast</code> context is used inside the model itself and you would need to narrow down where it’s used and switch to <code>float16</code>.</p>
Script(s) produce Segmentation Fault (Core Dumped) randomly

Bocaj_Horseman: <p>I have been getting segmentation faults randomly for a while now. It happens across all my PyTorchs scripts and at random times. Sometimes it is within a few minutes of starting, sometimes it doesn’t fault for a couple of hours. I am also unable to reproduce it. Here’s some relevant system information:</p>
<ul>
<li>Intel i9-7920X</li>
<li>4x Nvidia TitanXP (CUDA 12.6, driver version 560.28.03)</li>
<li>torch 2.4.0+cu124</li>
<li>Ubuntu 22.04.4 LTS</li>
</ul>
<p><code>gdb</code> stacktrace:</p>
<pre><code class="lang-auto">Thread 1 "pt_main_thread" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004e0800 in _Py_Dealloc (op=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.8.19/Objects/object.c:2215
#2  _Py_DECREF (op=&lt;optimized out&gt;, lineno=430, filename=0x5f56e0 "/croot/python-split_1710964293155/work/Objects/frameobject.c")
    at /usr/local/src/conda/python-3.8.19/Include/object.h:478
#3  frame_dealloc (f=0x19b2f510) at /usr/local/src/conda/python-3.8.19/Objects/frameobject.c:430
#4  0x00000000004d6f77 in _Py_Dealloc (op=0x19b2f510) at /usr/local/src/conda/python-3.8.19/Objects/object.c:2215
#5  _Py_DECREF (op=0x19b2f510, lineno=4314, filename=0x5f8e50 "/croot/python-split_1710964293155/work/Python/ceval.c") at /usr/local/src/conda/python-3.8.19/Include/object.h:478
#6  _PyEval_EvalCodeWithName (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, 
    kwargs=0x7fffffffa790, kwcount=&lt;optimized out&gt;, kwstep=1, defs=0x0, defcount=&lt;optimized out&gt;, kwdefs=0x0, closure=0x0, name=0x7fffa75498b0, qualname=0x7fffa754c530)
    at /usr/local/src/conda/python-3.8.19/Python/ceval.c:4314
#7  0x00000000004e80cc in _PyFunction_Vectorcall (func=&lt;optimized out&gt;, stack=0x7fffffffa780, nargsf=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;)
    at /usr/local/src/conda/python-3.8.19/Objects/call.c:436
#8  0x00000000004f4ff4 in _PyObject_Vectorcall (kwnames=&lt;optimized out&gt;, nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;, callable=&lt;optimized out&gt;)
</code></pre>
<p>Let me know if I should provide anything else</p>
DP with opacus: Want to understanding the functions

Chandan_Tankala: <p>Hello, I am using opacus for creating a DP - Hidden Markov Model. I was able to run the code using two methods of opacus, namely <em>make_private()</em> and <em>make_private_with_epsilon()</em>. I was able to print the epsilon values at each epoch of the training process using the function <em>privacy_engine.get_epsilon()</em>, and I observed that the rate at which epsilon increases over the training epochs is much higher in <em>make_private()</em> than in <em>make_private_with_epsilon()</em>.</p>
<p>So I am trying to understand why that is the case and, broadly, what’s the difference between these two functions, and when should one use one function over the other.</p>
<p>My understanding is that the <em>make_private_with_epsilon()</em> function has parameters <em>target_epsilon()</em> and <em>num_epochs</em> and so I believe this function adds variable amount of noise at each step to achieve the target epsilon value. On the other hand, the <em>make_private()</em> function doesn’t have the parameter <em>num_epochs</em>, but has parameters <em>noise_multiplier</em> and <em>max_grad_norm</em>. In the documentation of this function <em>noise_multiplier</em> is defined as the ratio of the standard deviation to the l2 sensitivity of the gradients. I believe the l2 sensitivity of the gradients can change over the training epochs and hence I think the standard deviation of the noise added at each epoch can also change.</p>
<p>In summary, my understanding is that the amount of noise added in both of these methods could be different at each training epoch. However, one function produces a higher rate of change of epsilon over training epochs and I don’t understand why that is the case. Any thoughts? I appreciate your time.</p>

Soumya_Kundu: <p>My general understanding is the one uses <code>privacy_engine.get_epsilon()</code> when your privacy bounds are strict. Although, I am yet to empirically test your claims.</p>