akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Something like MetricCollection in TorchEval?

Viktor_Todosijevic: <p>Hey,</p>
<p>I was wondering if there’s something like <a href="https://torchmetrics.readthedocs.io/en/latest/pages/overview.html#metriccollection" rel="noopener nofollow ugc">MetricCollection</a> from TorchMetrics in TorchEval? Like an object where I’d put a bunch of metrics and then call update(…) or compute() on once and it calls those methods for every metric in it.</p>
Any function that in-paints (interpolates) an image?

Masaaki-75: <p>I am wondering if there is any torch function that interpolates an image in an in-painting way.</p>
<p>I find that the function <code>torch.nn.functional.interpolate</code> only expands or squeezes the image, but cannot fill the values within a masked image.</p>
<p>This may sound confusing, so I made some code and diagrams to make it clear:</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
import torch
import torch.nn.functional as F
from PIL import Image
def gen_banded_mask(img, num_bands=10, band_width=10):
    # accept [H, W, C]
    height, weight = img.shape[:2]
    masked_indices = []
    
    mask = np.ones_like(img, dtype=img.dtype)
    rand_hs = np.random.choice(height, num_bands, replace=False)
    for h in rand_hs:
        h0 = max(h - band_width, 0)
        h1 = min(h + band_width, height)
        masked_indices.extend([*range(h0, h1)])
        mask[h0:h1, ...] = 0
    
    unmasked_indices = [_ for _ in range(height) if _ not in masked_indices]
    return mask, sorted(unmasked_indices)
        

img_path = '../bin/sample-rbg.png'
img = np.asarray(Image.open(img_path).convert('RGB')) / 255.0
mask, unmasked_hs = gen_banded_mask(img, num_bands=20)
img_masked = img * mask
img_reduced = img[unmasked_hs, ...]  # image with only the unmasked rows

tensor_reduced = torch.from_numpy(img_reduced).float().permute(2, 0, 1).unsqueeze(0)  # [1, C, H, W]
tensor_interp = F.interpolate(tensor_reduced, size=img.shape[:2], mode='bilinear')
img_interp = tensor_interp.squeeze(0).permute(1, 2, 0).numpy()  # This does not in-paint but just expand the image
</code></pre>
<p>Now let’s show <code>[img, mask, img_masked, img_interp]</code>. You can see my way of applying <code>F.interpolate</code> does not <strong>in-paint</strong> the masked pixels in the image but can only expand the unmasked part:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b818e2ec500568cc97427dd0cb38848d6adeed2.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b818e2ec500568cc97427dd0cb38848d6adeed2" title="inpaint"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b818e2ec500568cc97427dd0cb38848d6adeed2_2_690x195.png" alt="inpaint" data-base62-sha1="3VksHgCVsmU0M3Cg5WjdrxM9uLw" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b818e2ec500568cc97427dd0cb38848d6adeed2_2_690x195.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b818e2ec500568cc97427dd0cb38848d6adeed2_2_1035x292.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b818e2ec500568cc97427dd0cb38848d6adeed2.png 2x" data-dominant-color="4B586D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">inpaint</span><span class="informations">1128×319 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.no_grad() Changes Sequence Length During Evaluation Mode

leon2ta: <p>I built a TransformerEncoder model, and it changes the output’s sequence length if I use “with torch.no_grad()” during the evaluation mode.</p>
<p>My model details:</p>
<pre><code class="lang-auto">class TransEnc(nn.Module):
    def __init__(self,ntoken: int,encoder_embedding_dim: int,max_item_count: int,encoder_num_heads: int,encoder_hidden_dim: int,encoder_num_layers: int,padding_idx: int,dropout: float = 0.2):
        super().__init__()
        self.encoder_embedding = nn.Embedding(ntoken, encoder_embedding_dim, padding_idx=padding_idx)
        self.pos_encoder = PositionalEncoding(encoder_embedding_dim, max_item_count, dropout)
        encoder_layers = nn.TransformerEncoderLayer(encoder_embedding_dim, encoder_num_heads, encoder_hidden_dim, dropout, batch_first=True) 
        self.transformer_encoder = nn.TransformerEncoder(encoder_layers, encoder_num_layers)
        self.encoder_embedding_dim = encoder_embedding_dim

def forward(self,src: torch.Tensor,src_key_padding_mask: torch.Tensor = None) -&gt; torch.Tensor:
        src = self.encoder_embedding(src.long()) * math.sqrt(self.encoder_embedding_dim)
        src = self.pos_encoder(src)
        src = self.transformer_encoder(src, src_key_padding_mask=src_key_padding_mask)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">batch_size = 32
ntoken = 4096
encoder_embedding_dim = 256
max_item_count = 64 # max sequence length with padding
encoder_num_heads = 8
encoder_hidden_dim = 256
encoder_num_layers = 4
padding_idx = 0
</code></pre>
<p>I have a tensor (src) containing 32 word-level tokenized sentences (with different paddings) with a shape of (32,64)(batch_size,max_item_count).</p>
<p>When I activate training mode with “model.train()”, set “src_key_padding_mask = src == tokenizer.pad_token_id” and run “logits = model(src = src, src_key_padding_mask = src_key_padding_mask)”, I get logits with an expected shape of (32,64,256)(batch_size,max_item_count,encoder_embedding_dim).</p>
<p>However, when I activate evaluation mode with “model.eval()”, set “src_key_padding_mask = src == tokenizer.pad_token_id” and run with “torch.no_grad(): logits = model(src = src, src_key_padding_mask = src_key_padding_mask)”, I get different logits’ shapes every time like (32,31,256), (32,25,256), etc. I want to get logits with a shape of (32,64,256). How can I solve this problem?</p>
<p>OS: Windows 10 x64<br>
Python: 3.10.12<br>
Torch: Tried on both 1.13.1+cu117 and 2.0.1+cu117, but the problem is still the same.</p>

leon2ta: <p>Downgrading worked! If you suffer from this problem, try</p>
<pre><code class="lang-auto">pip uninstall torch torchvision torchaudio
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
</code></pre>
Overlapping patches using Kornia

Tom_Berry: <p>After using <code>extract_tensor_patches</code> to generate overlapping patches using a non-equal stride and window size,<br>
<code>patches = kornia.contrib.extract_tensor_patches(x, window_size=(3, 3), stride=(1, 1))</code><br>
I’m attempting to reconstruct the tensor using <code>combine_tensor_patches</code>.<br>
<code>output = kornia.contrib.combine_tensor_patches(patches, original_size=x.shape[2], window_size=(3,3), stride=(1,1))</code><br>
Unfortunately, I’m running into the following problem: <code>'NotImplementedError: Only stride == window_size is supported. Got (1, 1) and (3, 3).</code> If anyone has encountered this before, could you please share a solution or suggest an alternative approach? Your insights would be greatly appreciated."</p>
VScode Include paths for .h files?

DerekCresswell: <p>I’m writing some C++ extensions and want my VSCode to be able to autocomplete but I cannot get it to find the needed headers.</p>
<p>In my include paths I have (the code is in a docker container):</p>
<ul>
<li><code>/opt/conda/.../torch/include/torch/csrc/api/include</code></li>
<li><code>/opt/conda/.../torch/include/torch</code></li>
</ul>
<p>This seems to make vscode able to detect the <code>extension.h</code> file but not it’s dependencies <code>python.h</code>. I am wondering if anyone has VSCode setup for this and could share all the include paths that they needed to work on pytorch extensions. Thank you!</p>
<p>EDIT: A bit further, I’ve got an extension running in Python and am also curious how I can set it up so that VSCode recognizes the Python import of my module!</p>
Training graph attention network model for node level predictions

moltisanti: <p>I am trying to train a graph neural network model and I am having problems regarding the models’ architecture. What I have is DataBatch objects that comes from <code>torch_geometric.data</code> library.<br>
In batch object(data), every row in <code>data.y</code> is target variables for nodes in every graph. Batch consists of node embeddings coming from 128 different graphs.What I am interested in is the prediction of first node in every graph and I am not sure how to proceed. I read that graph attention networks are specifically used for node level regression.</p>
<pre><code class="lang-auto">DataBatch(x=[2634, 768], edge_index=[2, 2506], edge_attr=[2506, 1], y=[128, 131], mask=[128, 131], batch=[2634], ptr=[129])
</code></pre>
<p>Here is what I have as architecture</p>
<pre><code class="lang-auto">class GCNAttentionModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, dropout_rate):
        
        #super(GCNModel, self).__init__()
        super(GCNAttentionModel, self).__init__()
        self.conv1 = GATConv(input_dim, hidden_dim)
        self.conv2 = GATConv(hidden_dim, output_dim)

        self.dropout_rate = dropout_rate
        
        
    def forward(self, x, edge_index, edge_attr,batch):
        x = self.conv1(x, edge_index, edge_attr)
        x = torch.nn.LeakyReLU()(x)
        #x = F.dropout(x, p=self.dropout_rate, training=self.training)
        x = self.conv2(x, edge_index, edge_attr)
        x = torch.nn.LeakyReLU()(x)
        #x = F.dropout(x, p=self.dropout_rate, training=self.training)
        return x
</code></pre>
<p>The first level output has the size of <code>torch.Size([2634, 131])</code> and I am not sure if it is what it is supposed to be if I am interested in specific nodes’ prediction. Should output dimension be 1 if I am interested in only one node? If that’s the case, how the target variables neighbor nodes will be aggregated and used, that’s what I couldn’t figure out. Can anyone help or suggest me a way?</p>
Gumbel_softmax function logits?

Lifeng_Jin: <p><a href="https://github.com/pytorch/pytorch/blob/149d4f776b4a896945bdec81b89a150f5c9ee66d/torch/nn/functional.py#L937" rel="nofollow noopener">The link to PyTorch implementation</a></p>
<p>Both in the code and in the docs, the logits argument for the function is annotated as “unnormalized log probabilities”. If this is intended to mean the raw scores before any softmax layer, then I have a hard time understanding why this should work at all. Both in the RelaxedOneHotCategorical distribution implementation and the original Jang’s paper, the logits are clearly normalized. However from Jang’s notebook implementation which is where PyTorch got its gumbel softmax from, the input to that function is straight out from a linear layer, meaning it is not normalized. Does anyone have any insight into this?</p>

Boltzmachine: <p>No you have not fixed it. You are talking about another thing</p>

Boltzmachine: <p>I found the same issue?any new comments?</p>
Jit Model Forward Compatibility with new PyTorch Release

leslie-fang-intel: <p>Dear community, May I ask a question regarding to Jit Model Forward Compatibility?<br>
If I generated a Jit Model with old version PyTorch release (save it as <code>jit.save</code>), Does it guarantee to be runnable with new PyTorch release (load it with <code>jit.load</code>)? If so, where can I find the link of claim?</p>

marksaroufim: <p>There aren’t any forward compatibility tests <a href="https://github.com/pytorch/pytorch/issues/49585" class="inline-onebox">[JIT] Serialization forward compatibility tests · Issue #49585 · pytorch/pytorch · GitHub</a></p>
<p>That said JIT is in maintenance mode so it’s unlikely anyone is going to be making any breaking changes here</p>
Tensor.grad is None if using CUDA device

otavio-silva: <p>I’m trying to compute derivatives of functions in a TensorFlow-like fashion. Consider the following code:</p>
<pre><code class="lang-auto">import torch
x = torch.linspace(-10., 10, 10000, requires_grad=True)
y = x**2
y.backward(torch.ones_like(x))
g = x.grad
</code></pre>
<p>It works just fine, with <code>g</code> being the gradient, such that <code>f(x) = x**2</code> and <code>f'(x) = 2x</code>.<br>
But if I set the device to my GPU by using <code>x = x.cuda()</code>, it does not work, giving the warning:</p>
<pre><code class="lang-bash">&lt;stdin&gt;:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at C:\cb\pytorch_1000000000000\work\build\aten\src\ATen/core/TensorBody.h:491.)
</code></pre>
<p>And then <code>g</code> is <code>None</code>. Why is that? How can I fix it?</p>

ptrblck: <aside class="quote no-group" data-username="otavio-silva" data-post="1" data-topic="186128">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/otavio-silva/48/63088_2.png" class="avatar"> otavio-silva:</div>
<blockquote>
<p>But if I set the device to my GPU by using <code>x = x.cuda()</code>, it does not work, giving the warning</p>
</blockquote>
</aside>
<p>The <code>.to()</code> operation is differentiable and will create a non-leaf tensor as the warning explains.<br>
Either specify the <code>device</code> during the creation:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.linspace(-10., 10, 10000, device="cuda", requires_grad=True)
y = x**2
y.backward(torch.ones_like(x))
g = x.grad
</code></pre>
<p>or create a new leaf tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.linspace(-10., 10, 10000, requires_grad=True)
x = x.to("cuda")
x.detach_().requires_grad_(True)
y = x**2
y.backward(torch.ones_like(x))
g = x.grad
</code></pre>

otavio-silva: <p>It worked now, thanks!<br>
It seems very counter-intuitive to me, why would the <code>Tensor.to</code> function be differentiable? It simply copying a block of memory, no operation is being performed. Or I’m thinking this the wrong way?</p>

ptrblck: <p>Because it allows you to use different devices in the forward and backward pass, which is a fantastic feature, to e.g. CPU-offload, for model sharding, increase the numerical precision for some ops by casting to e.g. <code>float64</code>, etc.</p>
Overlapping control in Kornia extract_tensor_patches

Tom_Berry: <p>Hi,<br>
I’m using Kornia’s <code>extract_tensor_patches</code> to get image patches, and I want to control the overlap between them. I tried adjusting the stride, but it’s supposed to be equal to the window size, which isn’t giving me the desired result.</p>
<p>Any advice on how to manage overlapping without violating the stride-window size equality requirement?</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Tom_Berry" data-post="1" data-topic="186104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom_berry/48/63073_2.png" class="avatar"> Tom_Berry:</div>
<blockquote>
<p>but it’s supposed to be equal to the window size, which isn’t giving me the desired result.</p>
</blockquote>
</aside>
<p>I don’t think this is true as I can specify a smaller stride as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.arange(5*5).view(1, 1, 5, 5)
patches = kornia.contrib.extract_tensor_patches(x, window_size=(3, 3), stride=(1, 1))
print(x)
# tensor([[[[ 0,  1,  2,  3,  4],
#           [ 5,  6,  7,  8,  9],
#           [10, 11, 12, 13, 14],
#           [15, 16, 17, 18, 19],
#           [20, 21, 22, 23, 24]]]])
print(patches)
# tensor([[[[[ 0,  1,  2],
#            [ 5,  6,  7],
#            [10, 11, 12]]],


#          [[[ 1,  2,  3],
#            [ 6,  7,  8],
#            [11, 12, 13]]],


#          [[[ 2,  3,  4],
#            [ 7,  8,  9],
#            [12, 13, 14]]],


#          [[[ 5,  6,  7],
#            [10, 11, 12],
#            [15, 16, 17]]],


#          [[[ 6,  7,  8],
#            [11, 12, 13],
#            [16, 17, 18]]],


#          [[[ 7,  8,  9],
#            [12, 13, 14],
#            [17, 18, 19]]],


#          [[[10, 11, 12],
#            [15, 16, 17],
#            [20, 21, 22]]],


#          [[[11, 12, 13],
#            [16, 17, 18],
#            [21, 22, 23]]],


#          [[[12, 13, 14],
#            [17, 18, 19],
#            [22, 23, 24]]]]])
</code></pre>

Tom_Berry: <p>It seems that the issue arises when I use “combine_tensor_patches.”</p>
<pre><code class="lang-auto">output= kornia.contrib.combine_tensor_patches(patches, original_size=tensor.shape[2], window_size=(3,3), stride=(1,1))
</code></pre>
<pre><code class="lang-auto">NotImplementedError: Only stride == window_size is supported. Got (1, 1) and (3, 3).Please feel free to drop a PR to Kornia Github.
</code></pre>
AOT compilation for GPU inference. Is it supported?

apivovarov: <p>Is it possible to compile Pytorch model for GPU (for inference) and save compiled model in order to load the compiled model in future to avoid jit recompilation?</p>

marksaroufim: <p>Yes! You wanna check out AOT inductor if you’re deploying in c++ <a href="https://github.com/pytorch/pytorch/tree/main/test/cpp/aot_inductor">https://github.com/pytorch/pytorch/tree/main/test/cpp/aot_inductor</a></p>
<p>If you’re still deploying with python something like this should work <a href="https://github.com/pytorch/pytorch/pull/106501" class="inline-onebox">[draft] fsspec code cache by msaroufim · Pull Request #106501 · pytorch/pytorch · GitHub</a></p>

apivovarov: <p>Thank you! I found <code>aot_compile</code>, <code>compile_fx_aot</code> functions in pytorch python code:</p>
<ul>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/cpp/aot_inductor/test.py</a> - AOT inductor test</p>
</li>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/main/torch/_export/__init__.py#L414" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/_export/__init__.py#L414</a> - <code>aot_compile</code>: Traces either an nn.Module’s forward function or just a callable with PyTorch operations inside, generates executable cpp code from the program, and returns the path to the generated shared library</p>
</li>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/main/torch/_export/__init__.py#L168" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/_export/__init__.py#L168</a> - <code>export</code> function: returns ExportedProgram</p>
</li>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/compile_fx.py#L822" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/_inductor/compile_fx.py#L822</a> - <code>compile_fx_aot</code>: returns so path</p>
</li>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/__init__.py#L30" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/_inductor/__init__.py#L30</a> - <code>aot_compile</code>: Ahead-of-time compile a given FX graph with TorchInductor into a shared library</p>
</li>
</ul>

marksaroufim: <p>Yeah this is all very hot of the press, will probably get consolidated by next major release</p>
Installing PyTorch on Macbook Pro 2019

KS84: <p>According to <a href="https://discuss.pytorch.org/t/pytorch-support-for-intel-gpus-on-mac/151996">this</a> thread, PyTorch can be use on Intel Macs, but the functionality is disabled by default. I have changed the code line recommended in <a href="https://discuss.pytorch.org/t/pytorch-support-for-intel-gpus-on-mac/151996/2">this</a> post and compiled pytorch. However, when I run the test code given <a href="https://github.com/chengzeyi/pytorch-intel-mps" rel="noopener nofollow ugc">here</a>, I get the following error:</p>
<pre><code class="lang-auto">-[MTLHeapDescriptorInternal validateWithDevice:]:344: failed assertion `Heap Descriptor Validation
Shared storage mode disallowed'
</code></pre>
<p>Does anybody have any suggestions on how to fix this?</p>
How does one go about overriding the derivative of a primitive matrix operation?

kdu: <p>If I wanted to override the implementation of the backward pass of a primitive matrix operation, say, matmul or convolution, how might I go about it?</p>
<p>Some subquestions:<br>
(a) is there any way to do it (efficiently) in Python (just me hoping, haha)?<br>
(b) Assuming the answer to (a) is no, can I implement each operation once in C++ and then it’ll work for all backends, or would I need to implement each operation for each backend e.g. cuda?</p>
<p>Thanks!</p>

ptrblck: <p>You could write a custom C++ extension as seen <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-c-extension">here</a>. If you are using pure PyTorch operations you won’t need to write backend-specific code. However, you would still have the option to write custom CUDA code if needed.</p>
Trainable parameter 

Tom_Berry: <p>Hi,<br>
How can I integrate a trainable parameter ranging from 2 to 10 using Torch? This parameter doesn’t directly participate in the output operations, but its updates need to be maintained during training.</p>

ptrblck: <aside class="quote no-group" data-username="Tom_Berry" data-post="1" data-topic="186095">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom_berry/48/63073_2.png" class="avatar"> Tom_Berry:</div>
<blockquote>
<p>This parameter doesn’t directly participate in the output operations</p>
</blockquote>
</aside>
<p>Autograd won’t calculate gradients for parameters which weren’t used as they do not have any influence on the output. You can manually assign a <code>.grad</code> attribute but I don’t know how you would calculate it.</p>
Gradient clipping in DNN

Sumesh_Sankar: <p>how to prevent the explosion of gradients of weights via gradient clipping</p>

ptrblck: <p>You can apply <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html#torch.nn.utils.clip_grad_norm_"><code>torch.nn.utils.clip_grad_norm_</code></a> or <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_value_.html#torch.nn.utils.clip_grad_value_"><code>torch.nn.utils.clip_grad_value_</code></a> on the desired parameters.</p>
Parameter group subset handling of optimizers

john-titor: <p>I would like to define parameter groups such that</p>
<pre><code class="lang-python">optimizer([
  {'params' : module.parameters(), 'lr':0.1},
  {'params' : module.submodule.parameters(), 'lr':0.01}
])
</code></pre>
<p>I would expect that this effectively sets the learning rate for parameters but the ones of <code>submodule</code> to <code>lr==0.1</code>, and the learning rate for the parameters of <code>submodule</code> to <code>0.01</code>.</p>
<p>Could someone confirm if this is indeed what is going to happen, or whether any weird behavior is expected because the learning rate of <code>submodule</code> is technically being set twice?</p>
<p>Thanks!</p>

john-titor: <p>Found the answer myself by trying it out:</p>
<pre><code class="lang-python">ValueError: some parameters appear in more than one parameter group
</code></pre>
<p>The question now is: Would it be possible to allow for a definition such as above without the need to explicitly iterate through parameters that were already defined in a previous group?</p>

ptrblck: <p>It depends on your use case and based on the <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">docs</a> you can still pass default arguments to the optimizer, which will be applied to parameters not specified explicitly. Since you are specifying two parameter groups explicitly you would need to filter them properly.</p>

john-titor: <p>Thanks for the feedback. It would be a nice feature to have, rather than raising a <code>ValueError</code></p>

ptrblck: <p>The default arguments are already supported. I don’t think passing overlapping parameter sets into different parameter groups is a nice feature, but a really hard to debug error, which is rightfully disallowed.<br>
Use parameter groups as separate groups with their own optimizer arguments and the default arguments for the rest.</p>
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle) when training "bert-base-uncased"

ACP: <p>I’m running in to this error when training “bert-base-uncased”.</p>
<p>The metric I’m using is</p>
<blockquote>
<p>metric = evaluate.combine([“accuracy”, “f1”, “precision”, “recall”])</p>
</blockquote>
<p>and the training code is</p>
<blockquote>
<p>fold_trainer = Trainer(<br>
model=model,<br>
args=training_args,<br>
train_dataset=tokenized_datasets.select(train_index),<br>
eval_dataset=tokenized_datasets.select(test_index),<br>
compute_metrics=compute_metrics,<br>
)<br>
fold_trainer.train()</p>
</blockquote>
<p>My dataset is 700 texts. The code works fine when running on Google Colab (but time consuming), so I turn to work on a server on terminal using virtual env and downloaded all needed packaged. However, I kept running into this problem. How can I solve this? Why would it run on Colab but not on the server?</p>
<p>I have tried CUDA_LAUNCH_BLOCKING=1, but it did not print out anything (even after 7 hours).</p>

ptrblck: <p>You might be running out of memory so try to reduce the batch size and rerun your code.</p>
cuDNN error: Status Execution failed

Sumesh_Sankar: <p>Anyone tell me why this error is coming, i am new to these terminologies<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a96c650e172de069cf60a7ba1e0fa143ee46a266.png" data-download-href="https://discuss.pytorch.org/uploads/default/a96c650e172de069cf60a7ba1e0fa143ee46a266" title="cuDNNerror"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96c650e172de069cf60a7ba1e0fa143ee46a266_2_690x202.png" alt="cuDNNerror" data-base62-sha1="oaMVu4DN4FrSZbzCWkEdRVf5Wxo" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96c650e172de069cf60a7ba1e0fa143ee46a266_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96c650e172de069cf60a7ba1e0fa143ee46a266_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a96c650e172de069cf60a7ba1e0fa143ee46a266.png 2x" data-dominant-color="F1ECE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuDNNerror</span><span class="informations">1120×329 53.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
Auto-regressive inference with transformer?

anteater: <p>I am trying to use nn.Transformer to train a basic language model. The model trains, however at inference time I am having an issue and noted the following sentence from the docs:</p>
<p>Note: Due to the multi-head attention architecture in the transformer model, the output sequence length of a transformer is same as the input sequence (i.e. target) length of the decoder.</p>
<p>This generally makes sense, but is confusing to me how I am supposed to use inference with this in mind. I know from reading tutorials, the docs, etc that the usual way of performing inference is to begin with a single SOS token, use that as the input to the decoder and append the output to this input and keep going until EOS is generated, one at a time. However, due to that sentence, what is happening to me is my output is doubling in size each time. As in, the sizes of the input to the decoder and therefore the output is going from 1 → 2 → 4 → 8 etc because the inputs and outputs must be the same length.</p>
<p>My expectation is that we auto regressively generate one word at a time, but I’m not sure what I’m misunderstanding or how to do this with this in mind. Any help would be appreciated.</p>
Access gradient of parents of a node during first execution of backprop

kdu: <p>Is there a way to get the gradient of all the “parent” nodes of a node during the execution of backprop (when viewing the graph going backward, e.g. the output node is the root and the input nodes are the leaves)?<br>
For example, in the graph of this expression,</p>
<pre><code class="lang-auto">c = a * b
out = a + c
</code></pre>
<p>the node <code>a</code> has two paths from <code>out -&gt; a</code>, one directly to <code>out</code> and one through <code>c</code> (shown below).</p>
<pre><code class="lang-auto">a ------- out
 \       / 
  \     /
   \   /
    \ /
b----c
</code></pre>
<p>I’d like to manipulate the gradient that gets computed for <code>a</code> by running an arbitrary operation on the gradients accumulated from the parents of <code>a</code>, in this case <code>c</code> and <code>out</code>, probably by using a prehook on the nodes in the computation graph.</p>
<p>While I can get the parent grad_fns of the <code>grad_fn</code> of a node, I’m having trouble figuring out how to get the associated gradient with that node. I was originally hoping I could do <code>parent_grad_fn.variable.grad</code>, but it looks like these parent grad functions don’t have a <code>.variable</code> attribute.</p>
<p>I’ve also attached a <a href="https://colab.research.google.com/drive/14bDpCvMpTWIMCSQcaieQpwuv4DxeOI0x?usp=sharing" rel="noopener nofollow ugc">minimal example in colab here</a> showing how I’ve been able to access the parent grad functions but not the actual grad value associated with them.</p>
<p>Thanks for the help!</p>
How to batch torch.roll with torch.vmap?

LambdaWill: <p>I want to roll the tensor at a list of shift. Here is the toy example.  Can it be vectorized with torch.vmap?</p>
<pre><code class="lang-python">import torch

def custom_roll(tensor, shifts):
    shifted_rows = [torch.cat((row[-shift:], row[:-shift])) for row, shift in zip(tensor, shifts)]
    return torch.stack(shifted_rows)

tensor = torch.tensor([[1, 2, 3, 4, 5],
                       [6, 7, 8, 9, 10],
                       [11, 12, 13, 14, 15]])

shifts = torch.tensor([1, 2, 3])

result = custom_roll(tensor, shifts)
print(result)

</code></pre>
Can I sell style transfer generated images for commercial use

Muhammad_Afnan_Khan: <p>On pytorch site examples there is an tutorial link <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 2.0.1+cu117 documentation</a><br>
That shows the style transfer technique. If I modify the image or training parameter or use the same image with different parameter, Is it possible to commercially use the style transfer newly generated image.</p>
How to train a pretrained CNN model that includes multi-head attention layers

Sashika_Himali: <p>I trained an Efficienetb0 Model by adding two multi-head attention layers. But when I training the model I get  the following warning.How Do I fix this warning and Improve the training accuracy?</p>
<pre><code class="lang-auto">Epoch: 1 | train_loss: 2.0100 | train_acc: 0.2708 | validation_loss: 1.7110 | validation_acc: 0.4147
/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py:1084: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!

Mismatched elements: 3 / 320 (0.9%)
Greatest absolute difference: 0.00018024444580078125 at index (21, 2) (up to 1e-05 allowed)
Greatest relative difference: 7.841616138727894e-05 at index (20, 9) (up to 1e-05 allowed)
  _check_trace(

This is the structure of the model.

![12365|690x388](upload://xLoMu9MfAu1tt4iWwk95mBk9gUa.png)
If you can't see the model structure follow this [https://stackoverflow.com/questions/76865430/pretrained-cnn-model-training-with-multi-head-attention] 


This is my attention layer. 

class MultiHeadAttention(nn.Module):
    def __init__(self, input_dim, num_heads):
        super().__init__()
        self.input_dim=input_dim
        self.num_heads = num_heads
        self.head_dim = input_dim // num_heads


        #print("input dimention", input_dim)
        #print("head dimention", self.head_dim)

        # key, query, value projections for all heads, but in a batch
        self.c_attn = nn.Linear(input_dim, 3 * input_dim, bias=False)
        # output projection
        self.c_proj = nn.Linear(input_dim, input_dim, bias=False)



    def forward(self, x):
        z=x.size()

        x = torch.reshape(x, (x.size(0), -1, x.size(1)))
        batch_size, seq_len, c = x.size()


        B, T, C = x.size()  # batch size, sequence length, embedding dimensionality (n_embd)

        # calculate query, key, values for all heads in batch and move head forward to be the batch dim
        q, k, v = self.c_attn(x).split(self.input_dim, dim=2)
        #head_size = C// self.n_head
        k = k.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)  # (B, nh, T, hs)
        q = q.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)  # (B, nh, T, hs)
        v = v.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)  # (B, nh, T, hs)

        # efficient attention using Flash Attention CUDA kernels
        y = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=True)

        y = y.transpose(1, 2).contiguous().view(B, T, C)  # re-assemble all head outputs side by side

        # output projection
        y = self.c_proj(y)



        attention_output = torch.reshape(y, (z[0], z[1], z[2], z[3]))

        return attention_output</code></pre>
Cannot export MiVOLO model into `onnx` format using `torch.onnx.export`

MasterHM-ml: <p>I am trying to convert a <a href="https://github.com/WildChlamydia/MiVOLO/blob/024d3ae71d42c4147bf21a46bbc2f7521cd94d53/mivolo/model/mi_volo.py#L123" rel="noopener nofollow ugc">pytorch based model</a> into onnx format. The pytorch model is working fine, but I can’t convert it into onnx. I am converting <code>self.model</code> into <code>onnx</code> format like this:</p>
<pre><code class="lang-python">        random_input = torch.randn([8, 6, 224, 224], device=self.device, requires_grad=True)
        onnx_model_name = "mi_volo.onnx"
        # pytorch to onnx
        torch.onnx.export(self.model,
                        random_input,
                        onnx_model_name,
                        verbose=True,
                        opset_version=18,
                        export_params=True,
                        input_names = ['input'],
                        output_names = ['output'],)
</code></pre>
<p>but I am getting the following error:</p>
<pre><code class="lang-auto">============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/home/master/.local/lib/python3.10/site-packages/torch/onnx/symbolic_opset18.py", line 52, in col2im
    num_dimensional_axis = symbolic_helper._get_tensor_sizes(output_size)[0]
TypeError: 'NoneType' object is not subscriptable
</code></pre>
<p>I tried debugging the error, but couldn’t understand it due to less familiarity with the conversion process. The actual line that is causing an error inside <code>torch/onnx/symbolic_opset18.py</code> is</p>
<pre><code class="lang-python">num_dimensional_axis = symbolic_helper._get_tensor_sizes(output_size)[0]
</code></pre>
<p>At the time of the error, I tried inspecting the <code>output_size</code> variable and it is</p>
<pre><code class="lang-auto">1072 defined in (%1072 : int[] = prim::ListConstruct(%958, %963), scope: mivolo.model.mivolo_model.MiVOLOModel::/torch.nn.modules.container.Sequential::network.0/timm.models.volo.Outlooker::network.0.0/timm.models.volo.OutlookAttention::attn)
</code></pre>
<p>Debugging deeper, inside <code>_get_tensor_sizes</code> method, the <a href="https://github.com/pytorch/pytorch/blob/dfd441a12cede6702d1fd160eb808778b62adae0/torch/onnx/symbolic_helper.py#L571C1-L572C20" rel="noopener nofollow ugc">following branch</a> is causing a <code>None</code> return.</p>
<pre><code class="lang-python">    if not _is_tensor(x) or x.type() is None:
        return None
</code></pre>
<p><strong>MiVOLO</strong> - Latest Pull<br>
<strong>Pytorch version</strong> - 2.0.1<br>
<strong>onnx version</strong> - 1.14.1<br>
<strong>OS</strong> - Ubuntu 22.04.3LTS</p>
<p>Any help/direction/discussion will be highly appreciated, thank you.</p>
Dynamically Changing Pluggable CUDA allocator

HanGuo97: <p>Hi, I came across a comment [1] suggesting that it’s possible to change CUDA memory allocators dynamically in different parts of an application. However, the same thread warns that it could be unsafe to do so once an allocator has been utilized. I find these messages a bit contradictory.</p>
<p>I have a specific use case that involves using PyTorch’s default CUDA memory allocator for model parameters and activations, and an external CUDA memory allocator (RMM) for the optimizer state. I am wondering if this is both possible and safe to do?</p>
<p>Thanks in advance for your time!</p>
<p>[1] <a href="https://github.com/pytorch/pytorch/pull/86786#discussion_r1017598777" class="inline-onebox" rel="noopener nofollow ugc">Add Pluggable CUDA allocator backend by emcastillo · Pull Request #86786 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>What’s the reason for mixing both allocators? Do you see a disadvantage in using only one of them?</p>

HanGuo97: <p>My motivation is to use external memory allocators to allocate managed memory (<code>cudaMallocManaged</code>), so they are pageable when oversubscribed.</p>
<p>Is it possible to designate certain parameters as pageable and others as non-pageable without combining allocators?</p>
Speeding up tensor operations by subselecting elements

rada: <p>Hi,</p>
<p>I am performing a series of operations on large 1D torch tensors of the same shape to get several feature vectors. The feature vectors are then multiplied to get the desired result. I am utilizing GPU. Example is:</p>
<pre><code class="lang-auto"># large 1D torch tensors
a = torch.tensor([0.000001, 0.1, 1, 100, 1000]).to("cuda")
b = torch.tensor([0.000002, 0.2, 2, 200, 2000]).to("cuda")

# 1st feature computation
feature_a = a**2

# 2nd feature computation
feature_b = 1/(b**4)

# 3rd feature computation
feature_c = ...

# multiplying feature vectors to get the final result
result = torch.prod([feature_a, feature_b, feature_c], dim=0)
</code></pre>
<p>The thing is, after computation of <code>feature_a</code>, I can already see, that the first element of <code>feature_a</code> will be very close to 0, which in turn will cause the first element of <code>result</code> to be very close to 0.</p>
<p>Is there some way to look for the small values after the calculation of each feature and then to tell PyTorch that it can avoid the computation of these obvious elements in subsequent features and only spend time on the other elements?</p>
<p>The question is not just how to do it but if it is worth doing. The aim is to speed the calculations up, so if some indexing functions would delay the otherwise smooth GPU operations, I will probably stick to the current procedure.</p>
<p>Thanks!</p>
<p>NOTE: I am not interested in the gradients, only the forward pass.</p>

ptrblck: <p>As so often, you would need to profile both approaches for your use case and check if the masking operation would be more expensive then the unnecessary compute.</p>
Confusion about PyTorch Data Augmentation

dbc111: <p>I’m currently using a UNet for image regression and want to explore data augmentation. However, it appears to me that pytorch transformations (as implemented below) replace the original image in the dataset. Considering that the total amount of data in the dataset stays the same, won’t the model’s performance be similar? Don’t know if that assumption is correct, though. Also, are the transforms refreshed per epoch?</p>
<pre><code class="lang-auto">rotation_angle = 90
rotation_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomRotation(degrees=rotation_angle),
    transforms.ToTensor()
])


dataset = FullDataset(dir_input, dir_mask, transform=rotation_transform)
</code></pre>
<p>FullDataset <strong>getitem()</strong> snippit:</p>
<pre><code class="lang-auto">if self.transform:
    aug_input = self.transform(input)
    aug_mask = self.transform(mask)
return aug_input, aug_mask
</code></pre>

ptrblck: <aside class="quote no-group" data-username="dbc111" data-post="1" data-topic="186056">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/f4b2a3/48.png" class="avatar"> dbc111:</div>
<blockquote>
<p>it appears to me that pytorch transformations (as implemented below) replace the original image in the dataset.</p>
</blockquote>
</aside>
<p>The transformation does not replace the data as it’s applied on the fly for each sample.</p>
<aside class="quote no-group" data-username="dbc111" data-post="1" data-topic="186056">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/f4b2a3/48.png" class="avatar"> dbc111:</div>
<blockquote>
<p>Considering that the total amount of data in the dataset stays the same, won’t the model’s performance be similar?</p>
</blockquote>
</aside>
<p>No, since you are randomly transforming each sample in every iteration. Depending on the used transformations the likelihood to repeat exactly the same random config (for all applied transforms) might go towards zero.</p>
Non AutoRegressive Transformer inference script

Rikka370: <p>I’m trying to develop an inference script for a Non autoregressive Transformer, I’ve used nn.transformer() for my model, I’ve got alright results during training, but I had no luck developing an inference script<br>
Since the Decoder layer requires source and target, for my target I’ve used a tensor of size [batch, target_len] filled with start token/start token+pad token but the accuracy is close to 0<br>
Could anyone provide help on how I would develop the inference script or what value I should pass as the target value?</p>
Efficient way to train on GPU and inference on CPU

chess2021: <p>Hi everyone, I am wondering what is the best way to load the model and start making inference on CPU after training the model on GPU:<br>
What I am doing which is working fine but seems inefficient is as follows:</p>
<p>1- Load the data<br>
2- Define, data loader<br>
3- Define network architecture<br>
4- Train the model<br>
5- Save the model using</p>
<blockquote>
<p>torch.save(self.net.state_dict(), save_path)</p>
</blockquote>
<p>Then when I go to inference and prediction on CPU, I do the following:</p>
<p>1- Load data<br>
2- Define dataloader<br>
3- Define network<br>
4 - Load the model and checkpoint<br>
5- Prediction on new data</p>
<blockquote>
<p>model.load_state_dict(torch.load(path,map_location=‘cpu’))</p>
</blockquote>
<p>I tried to do the prediction without steps 1-3 since I thought the saved model and checkpoint has all information but then I got an error.<br>
I am wondering if there is any way to make this procedure more efficient and faster.</p>

ptrblck: <aside class="quote no-group" data-username="chess2021" data-post="1" data-topic="185040">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e47c2d/48.png" class="avatar"> chess2021:</div>
<blockquote>
<p>I tried to do the prediction without steps 1-3 since I thought the saved model and checkpoint has all information but then I got an error.</p>
</blockquote>
</aside>
<p>This is expected since you are only storing the <code>state_dict</code> and thus need to create a new model object. You would script the model via <code>torch.jit.script</code> and load it directly, but note that TorchScript is in maintenance mode. I don’t think the <code>torch.compile</code> export mechanism is done yet but <a class="mention" href="/u/marksaroufim">@marksaroufim</a> could correct me.<br>
Generally, you could also check e.g. <a href="https://pytorch.org/serve/">TorchServe</a> or other serving solutions for inference.</p>

chess2021: <p>Thanks, I thought about this for a while, I am wondering for the prediction, do I have to run the entire train data in my model, what would happen if I only load 10% data and then load the checkpoint and start prediction using new data.</p>

ptrblck: <p>For inference you don’t need to load the training data at all and should just load the samples you want to classify.</p>
Is `heapq` module supported for compilation by Dynamo?

vadimkantorov: <p><code>heapq</code> Python module is implementing heap-based priority queues: <a href="https://github.com/python/cpython/blob/3.11/Lib/heapq.py" rel="noopener nofollow ugc">https://github.com/python/cpython/blob/3.11/Lib/heapq.py</a></p>
<p>Supporting these basic ops is useful for compiling Dijkstra-like loops which treat graph edges or nodes one by one based on some priority recalculation.</p>
<p>Ideally, these Python loops (using <code>heapq.pop</code> and <code>heapq.push</code>) could be compiled to C++ at least…</p>

marksaroufim: <p>This graph broke for me</p>
<pre><code class="lang-auto">import heapq
import torch

@torch.compile(fullgraph=True)
def program():
  h = []
  heapq.heappush(h, 3)
  heapq.heappush(h, 1)
  heapq.heappush(h, 4)
  val = heapq.heappop(h)
  return val * torch.randn(10)

program()
</code></pre>
<p><span class="mention">@voz</span> on github might say this is a good dynamo starter task XD</p>

vadimkantorov: <p>Created an issue <a href="https://github.com/pytorch/pytorch/issues/106885" class="inline-onebox" rel="noopener nofollow ugc">Dynamo graph break when using pyton module `heapq` (manipulates with `list`s) · Issue #106885 · pytorch/pytorch · GitHub</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>