akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there an alternative faster method to deal with for loops

kamat_mayur: <p>Given below is a piece of code that i want to optimize to be faster with backward pass.</p>
<pre><code class="lang-auto">root_transforms[..., :3, 3] = torch.cat((op[..., 0, :], ot[..., 0, :]), dim=0)
integrated_root_trans = root_transforms.clone()

for i in range(1, nframes):
    integrated_root_trans[...,i, :, :] = integrated_root_trans[...,i-1, :, :].clone() @ root_transforms[...,i, :, :]
</code></pre>
<p>Here the shape of root_transforms is (batch_size, nframes, 4, 4) and basically the loop is multiplying the ith frames root_transforms with the i-1th frames root transfoms and the result is the integrated root transform for the ith frame. As it can be observed that the integrated root transform depends on the previous frames integrated root transform. Hence I have to use a for loop. Using this for loop significantly slows down the backward pass. Is there a way I can make this faster??</p>
Hi everyone, I'm new in nlp, I'm trying to build a machine translation model using BERT and I'm having trouble training the model, my predicted tokens all return the id of the token <eos> ( 3) in the first epoch. How do I handle this. Note: I used label s

H_o_Phan: <pre><code>    encoder_output = self.encoder(encoder_input, padding_mask)
    decoder_output = self.decoder(decoder_input, encoder_output, padding_mask, future_mask)
    decoder_output = decoder_output[:, :-1, :]
    decoder_input = decoder_input[:, 1:]
    batch_loss = criterion(
                decoder_output.reshape(-1, decoder_output.size(-1)),
                decoder_input.reshape(-1).long(),
            )
    return batch_loss, torch.argmax(decoder_output, dim=-1)
</code></pre>
<p>criterion = nn.CrossEntropyLoss(ignore_index=0, label_smoothing=0.1)<br>
params = list(transformers.parameters())<br>
optimizer = optim.Adam(params, lr=0.01, weight_decay=0.1, betas=(0.9, 0.98), eps=1e-09)<br>
num_epochs = 1</p>
<p>scheduler = LrSchedule(<br>
transformers.hidden_dim, factor=1, warmup=4000, optimizer=optimizer,<br>
)</p>
<p>for epoch in range(num_epochs):<br>
for batch in dataloader:<br>
ids_src_tensor, ids_trg_tensor, _ = batch<br>
ids_src_tensor = ids_src_tensor.to(device)<br>
ids_trg_tensor = ids_trg_tensor.to(device)<br>
padding_mask = create_padding_mask(ids_src_tensor).to(device)<br>
future_mask = create_future_mask(ids_trg_tensor.shape[1]).to(device)<br>
batch_loss, out = transformers.train(ids_src_tensor, ids_trg_tensor, padding_mask, future_mask, criterion)<br>
batch_loss.backward()<br>
scheduler.step()<br>
scheduler.optimizer.zero_grad()<br>
print(scheduler._rate)</p>
<h1><a name="p-456163-torchnnutilsclip_grad_norm_transformersparameters-max_norm10-1" class="anchor" href="#p-456163-torchnnutilsclip_grad_norm_transformersparameters-max_norm10-1"></a>torch.nn.utils.clip_grad_norm_(transformers.parameters(), max_norm=1.0)</h1>
<pre><code>    print(torch.sum(out, dim = -1))
    print(batch_loss)
</code></pre>
Installing torch using pixi with CUDA version specified

cara: <p>Hello, I am using the pixi package manager to install PyTorch but am not sure how to specify the correct CUDA version. The conda command generated from the PyTorch installer page is:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
</code></pre>
<p>I can specify channels in pixi but am having trouble making sure the correct CUDA dependency is used. Has anyone successfully used pixi for this?</p>

cara: <p>To follow up on this, in case anyone has a similar question: I found a solution by adding the following to my pixi.toml file:</p>
<pre><code class="lang-auto">[system-requirements]
cuda = "12.4"
</code></pre>
<p>Before running:</p>
<pre><code class="lang-auto">pixi add pytorch
pixi add torchvision
</code></pre>
<p>And now torch successfully finds the available GPUs.</p>
Python kernel crashes with GPU

Daniel_Oddo: <p>Hello! i have an alienware 18r2 with a nvidia 4080 i use nvidia 556.12 and i tried the cuda 12.4 with pytorch to train a LTSM model, but the kernel keep crashing after a few minutes, does someone had a similar issue using GPU for machine learning? the only way to train the model is by using ONLY the CPU but thats not the idea.</p>

ptrblck: <p>Check if <code>dmesg</code> is showing and <code>Xid</code>s which could correlate to the crash. If not, run your script in a terminal and check if valid stacktraces could point to a problem.</p>
Update self defined parameters when using distributed data parallel

jetcai1900: <p>I am using distributed data parallel to train the model on multiple gpus. I meet one problem: I have used register_buffer to define one parameter. In addition, I need to manually update it. How could I achieve this? I tried to do the same update as the model is trained on one gpu, but the results are not correct. It seems that the value of this parameter is not synchronized over gpus. Thanks a lot</p>

mrshenli: <p>If this is a model parameter, any reason for using <code>register_buffer</code> instead of <code>register_parameter</code>?</p>
<blockquote>
<p>In addition, I need to manually update it. How could I achieve this?</p>
</blockquote>
<p>If it is a parameter (not a buffer) and if you don’t expect the autograd engine to compute gradients for you, you can set its <code>.requires_grad</code> field to <code>False</code> before passing the model to the DDP ctor. Then, DDP won’t sync it’s grads and optimizer won’t update the parameter value.</p>
<blockquote>
<p>I tried to do the same update as the model is trained on one gpu, but the results are not correct. It seems that the value of this parameter is not synchronized over gpus.</p>
</blockquote>
<p>I might miss something. Looks like you want to <strong>manually</strong> update a parameter, but still want DDP to help synchronize the parameter across the GPUs/processes? I don’t fully understand the use case, could you please elaborate this? If this parameter is manually updated, would it be possible to let all processes to set it to the same value?</p>

jetcai1900: <p><strong>“I might miss something. Looks like you want to  <strong>manually</strong>  update a parameter, but still want DDP to help synchronize the parameter across the GPUs/processes”</strong>’<br>
Yes. This is what I would like to achieve.</p>
<p>For example<br>
class model():<br>
def <strong>init</strong>(self):<br>
a = torch.zeros((3, 1))<br>
self.register_buffer(“a”, a)</p>
<p>def update_a(self, b):<br>
self.a.add_(b)<br>
b is a vector that is dynamic with respect to the input data. So I could not manually set it to the same value.</p>
<aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="88163">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>If this parameter is manually updated, would it be possible to let all processes to set it to the same value?</p>
</blockquote>
</aside>
<p>Let me know whether I describe the problem clearly? Thanks a lot for your help.</p>

xuChenSJTU: <p>hi, <a class="mention" href="/u/jetcai1900">@jetcai1900</a> , have you solved the problem? I met the same problem here, hope you can give me some advice, thanks</p>

guoxiayu0206: <p>Have you solved it? I encounter the same problem</p>
Expected Tensor as element 0 in argument 0, but got JpegImageFile

liampour: <p>Hello guys</p>
<p>I am using this example,</p>
<blockquote>
<pre data-code-wrap="import"><code class="lang-import">import torch
import os
from PIL import Image
import random

list_file='C/.../file_names.txt'    #A txt's files path that contains the names of the images Dataset
raw_dir='C/.../DirImageA'           # the path of the folder where the A type images of the dataset are located
expert_dir='C/.../DirImageB'        # the path of the folder where the B type images of the dataset are located
join = os.path.join


file_list = [  ]
with open(list_file) as f:
    for line in f:                
        name = line.strip()            
        if name:
            p = (join(raw_dir, name), join(expert_dir, name), name)
            file_list.append(p)
            
transformation= [  ]
size=256

transformation.append(transforms.Resize((size,size)))
transformation.append(transforms.RandomHorizontalFlip(0.5))

transformation.append(transforms.ToTensor())
transform=transforms.Compose(transformation)

index=random.randint(1, 1000) #random sample image for training

raw = Image.open(file_list[index][0])
expert =  Image.open(file_list[index][1])

raw_exp = transform(torch.stack([raw, expert])) #The error appears on this line  to torch.stack([raw, expert]) ```
</code></pre>
</blockquote>
<p>and I am getting this error</p>
<blockquote>
<p>expected Tensor as element 0 in argument 0, but got JpegImageFile</p>
</blockquote>
<p>Any suggestions?</p>

ptrblck: <p>Transform the images to tensors first e.g. via:</p>
<pre data-code-wrap="python"><code class="lang-python">image = PIL.Image.open("image.jpeg")

torch.stack([image, image])
# TypeError: expected Tensor as element 0 in argument 0, but got JpegImageFile

x = transforms.ToTensor()(image)
y = torch.stack([x, x])
print(y.shape)
# torch.Size([2, 3, 800, 800])
</code></pre>

liampour: <p>Tank you very much for your answer and help!!</p>
What's the param `conv1.weight_g` used in conv1d

vonct: <p>Recently I want to know the exact param that stored in conv1d layer. But when I print them from a trained-model state_dict, i found that there are 3 tensor.<br>
‘’’<br>
network.0.conv1.bias | torch.Size([10])|<br>
network.0.conv1.weight_g | torch.Size([10, 1, 1])|<br>
network.0.conv1.weight_v | torch.Size([10, 4, 5])|<br>
‘’’</p>
<ul>
<li>My conv1d config is : <code>nn.conv1d(4,20, kernel_size = 5, padding = 4, dilation = 1)</code>.</li>
<li>My pytorch is : <code>1.13.1</code><br>
What is conv1.weight_g means? I couldn’t find any docs about it.</li>
</ul>

ptrblck: <p>These additional parameters are added if you use the deprecated <code>torch.nn.utils.weight_norm</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv1d(3, 3, 3)
print(conv.weight)

conv = torch.nn.utils.weight_norm(conv)
print(conv.weight)
print(conv.weight_g)
print(conv.weight_v)
</code></pre>
<p>The recommendation is to use the supported <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.weight_norm.html#torch.nn.utils.parametrizations.weight_norm"><code>torch.nn.utils.parametrizations.weight_norm</code></a> API.</p>

vonct: <p>Oh! Now I see. Thank you.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Can’t create arm64 version of mv3_coreml_all.pte model for XCode project

swainwri: <p>Following instructions from (<a href="https://github.com/pytorch/executorch/tree/main/examples/demo-apps/apple_ios" class="inline-onebox" rel="noopener nofollow ugc">executorch/examples/demo-apps/apple_ios at main · pytorch/executorch · GitHub</a>)</p>
<p>error on python3 -m examples.apple.coreml.scripts.export --model_name=“mv3"</p>
<p>import executorchcoreml</p>
<p>ImportError: dlopen(/Users/…/executorch/.venv/lib/python3.12/site-packages/executorchcoreml.cpython-312-darwin.so, 0x0002): tried: ‘/Users/…/executorch/.venv/lib/python3.12/site-packages/executorchcoreml.cpython-312-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘x86_64’, need ‘arm64e’ or ‘arm64’)), ‘/System/Volumes/Preboot/Cryptexes/OS/Users/…/executorch/.venv/lib/python3.12/site-packages/executorchcoreml.cpython-312-darwin.so’ (no such file), ‘/Users/…/executorch/.venv/lib/python3.12/site-packages/executorchcoreml.cpython-312-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘x86_64’, need ‘arm64e’ or 'arm64’))</p>
<p>yet the other models built fine: mv3_mps.pte , mv3_xnnpack_fp32.pte, mv3.pte, imagenet_classes</p>

Olivia-liu: <p>Hi, thanks for your question! I’ll forward this to the iOS expert on my team. Please allow us to have some time looking at it.</p>

shoumikhin: <p>Hi <a class="mention" href="/u/swainwri">@swainwri</a>, please try the following in a new Terminal window:</p>
<pre><code class="lang-auto">git clone https://github.com/pytorch/executorch.git --recursive &amp;&amp; cd executorch

python3 -m venv .venv &amp;&amp; source .venv/bin/activate

pip install --upgrade cmake pip setuptools wheel

./install_requirements.sh

./backends/apple/coreml/scripts/install_requirements.sh

python3 -m examples.apple.coreml.scripts.export --model_name="mv3"
</code></pre>

swainwri: <p>Appreciate your time.</p>
<p>I followed your instructions,</p>
<p>(.venv) t@MacBook-Air executorch % python3 -m examples.apple.coreml.scripts.export --model_name=“mv3”</p>
<p>Torch version 2.5.0.dev20240912 has not been tested with coremltools. You may run into unexpected errors. Torch 2.4.0 is the most recent version that has been tested.</p>
<p>Fail to import BlobReader from libmilstoragepython. No module named ‘coremltools.libmilstoragepython’</p>
<p>Failed to load _MLModelProxy: No module named ‘coremltools.libcoremlpython’</p>
<p>Fail to import BlobWriter from libmilstoragepython. No module named ‘coremltools.libmilstoragepython’</p>
<p>Traceback (most recent call last):</p>
<p>File “”, line 198, in _run_module_as_main</p>
<p>File “”, line 88, in _run_code</p>
<p>File “/Users/stevewainwright/executorch/examples/apple/coreml/scripts/export.py”, line 13, in </p>
<p>import executorch.exir as exir</p>
<p>ModuleNotFoundError: No module named ‘executorch’</p>
<p>here’s the output from all your instructions</p>
<p>(Attachment Executorch.txt.zip is missing)</p>
Please someone confirm that my below assumption is correct or not?

Mayuresh: <p>Assumption:<br>
If you set the dataloader class shuffle=True, you can get random samples from ENTIRE dataset into your batches. I mean the batches will be created with random samples of the ENTIRE dataset insead of sequential ones.</p>

Mayuresh: <p><span class="mention">@ptrblck_de</span> can you answer please…</p>

ptrblck: <p>Yes, using <code>shuffle=True</code> will use a <code>RandomSampler</code> internally as seen <a href="https://github.com/pytorch/pytorch/blob/2157e396a3d5de0cc8627432b057d85ea960c778/torch/utils/data/dataloader.py#L376">here</a> and which will shuffle all indices in the <code>dataset</code>.</p>
How to map the elements of a tensor via key-value pairs (sparse version of index_select)

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’m trying to transform an input tensor via a key-value mapping (which is defined via 2 tensor representing the keys and values separately) to a desired output. I did think of using a simple for-loop but I fear that that approach would potentially overwrite previous mapping if I had a larger <code>key_tensor</code> and <code>value_tensor</code>.</p>
<p>Is there an existing function in PyTorch can do the following efficiently?</p>
<pre><code class="lang-auto">input_tensor = torch.tensor([[ 9, 36, 36],
                             [18, 36, 36],
                             [18, 36, 36],
                             [18, 36, 36]])

key_tensor = torch.tensor([36, 18,  9])
value_tensor = torch.tensor([0, 1, 2])

def expected_func(input, keys, values):
  """
  some amazing code here...
  """
  return expected_output

expected_output = tensor([[ 2, 0,  0],
                          [ 1, 0, 0],
                          [ 1, 0, 0],
                          [ 1, 0, 0]])
</code></pre>
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach()

Han_Brian_Lee: <p>I’m trying to freeze first part of my network but I’m getting the following error when I try to freeze them:</p>
<p>RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn’t require differentiation use var_no_grad = var.detach().</p>
<p>I have a big model class A, which consists of models B, C, D.</p>
<p>The flow goes B -&gt; C -&gt; D.</p>
<p>So after training for certain number of iterations, in A.forward(), I’d like to freeze B and train only C and D. I’ve tried doing (inside A.forward()):</p>
<pre><code class="lang-python">for param in self.B.parameters():
  param.requires_grad = False
</code></pre>
<p>However, I’m getting the error message above.</p>
<p>What does the error message exactly mean? My B network contains many convolutions and batchnorms and activation layers in sequential manner. Obviously certain layers come after other layers…</p>
<ol>
<li>what does it mean by only being able to change requires_grad of leaf variables? (I mean… leaf variables don’t really matter right? why would they require gradient flow?)</li>
<li>Also, what does it mean by a computed variable in a subgraph?</li>
<li>what exactly does it mean by var_no_grad = var.detach()?</li>
</ol>

Han_Brian_Lee: <p>Note that I tried building some simple networks to reproduce the problem and I couldn’t… and I can’t share the whole codebase because it is gigantic and my company’s IP.</p>
<p>I’m trying to get some ideas as to what that error points to exactly…</p>

abrahamowos: <p>Hi <a class="mention" href="/u/han_brian_lee">@Han_Brian_Lee</a>  found a solution yet?</p>
Pytorch 2.4.1 can't be installed to the same conda env with PDAL

simonreise: <p>After updating to 2.4.1 <code>import torch</code> is crashing with error <code>OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\ProgramData\miniconda\envs\geo\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies.</code></p>
<p>After trial-and-error I found out that it starts crashing after installing PDAL library to the same conda environment where Pytorch is. That might be due to dependency chain <code>pdal</code> → <code>libpdal-trajectory</code> → <code>ceres-solver</code> → <code>suitesparse</code> → <code>flang 5.0.0</code> (which is outdated) → <code>openmp 5.0</code> (which is also outdated). <code>openmp 5.0</code> replaces <code>libiomp5md.dll</code> file with its older version, and, as Pytorch requires the newer version (from its <code>intel-openmp</code> dependency), it crashes.</p>
<p>Windows 11 23H2 22631.4169<br>
Python 3.12.6 hce54a09_1_cpython conda-forge<br>
Conda 24.7.1 py312h2e8e312_0 conda-forge<br>
Pytorch 2.4.1 py3.12_cuda12.4_cudnn9_0 pytorch</p>
<p>I don’t think that this issue could or need to be solved (cause it is more related to other libs mentioned here rather than to Pytorch), just posting it here for information reasons.</p>
<p>P.S. Force-reinstalling <code>intel-openmp</code> solves the problem (but possibly breaks some of PDAL functions)</p>
How can pytorch call cpp from python?

cyk2018: <p>I want to do something like optimize operator or implement new operators.<br>
But I’m new to pytorch, I watch the code like how pytorch implement relu. and I’m confused how can pytorch call the cpp file from python file?</p>
<p>I 'm working at developing some operators in CUDA I want to publish a paper about my job. But I’m new about this, and my professor is new,too. So If you can give some advise I will be very very thankful!</p>
<p>Thanks for your reply!</p>

ptrblck: <p>You can write custom C++ and CUDA extensions as described <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a>.</p>

cyk2018: <p>Tnanks!!! I will follow this. But I want to know how pytorch work?</p>

ptrblck: <p>PyTorch uses <code>pybind11</code> internally.</p>
Tracing GRU layer in a module with jit.export decorator

Tomiinek: <p>Hello,</p>
<p>I am trying to trace a model that uses a module with a GRU layer. The module inside the model has a <code>forward</code> and <code>online_inference</code> methods. The model uses only the <code>online_inference</code> method, so it is also traced. The module contains a GRU layer. However, I am getting a strange errors when exposing also the <code>online_inference</code> method to trace.</p>
<p>I isolated this reproducible snippet. Here is the code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
from torch import Tensor


class Model(nn.Module):
    
    def __init__(self, hidden_dim = 8, n_layers = 1, batch_first = True):
        super().__init__()
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.gru = nn.GRU(hidden_dim, hidden_dim, num_layers=n_layers, batch_first=batch_first)

    def forward(self, c0: Tensor, state: list[Tensor]) -&gt; Tensor:
        gru_h = state[0]
        c, _ = self.gru(c0, gru_h)
        return c

    @torch.jit.export
    def online_inference(self, c0, state: list[Tensor]) -&gt; Tensor:
        gru_h = state[0]
        c, _ = self.gru(c0, gru_h)
        return c


model = Model().eval()

batch = 2
device = "cpu"
inputs = (
    torch.randn(batch, 64, model.hidden_dim) , 
    [torch.zeros(model.n_layers, batch, model.hidden_dim, device=device)]
)
with torch.no_grad():
    trace = torch.jit.trace(model, inputs, check_trace=False, strict=False)
</code></pre>
<p>which exits with:</p>
<blockquote>
<p>RuntimeError: Couldn’t find method: ‘forward__0’ on class: ‘<strong>torch</strong>.torch.nn.modules.rnn.___torch_mangle_151.GRU (of Python compilation unit at: 0x564ff038b8f0)’</p>
</blockquote>
<p>Removing the <code>@torch.jit.export</code> decorator, the whole <code>online_inference</code>, or gru usage from the online method of course helps, but I dont want to do that. Any ideas what is going on?</p>
How to get the bounding boxes in feature map space?

vivek_maskara: <p>For my video action recognition model, I am using I3D network as a feature extractor.</p>
<p>I am passing a clip of 64 RGB frames to the network and am taking the output of one of the intermediate layers(ie. Mixed_4f.b3b.conv3d) layer as my feature map.</p>
<p>In the original RGB frames, I know the bounding box coordinates of all the objects. Is it possible to get the corresponding bounding box locations in the feature map space?</p>
<p>Is there a 1-1 pixel mapping between the original frame and the feature map.</p>
<p>For eg. my input is B x 64 x 3 x 400 x 400 [B x T x C x W x H] and my output is B x 512 x 16 x 25 x 25 [X x F x T/4 x W/16 x H/16].</p>
<p>Since width and height have been reduced to 1/16 can I interpolate and estimate the coordinates of the objects in the feature map?</p>

Niels_PyTorch: <p>I wanted to do something similar. Yes you can do that. There are 2 main operations for this: ROI-pooling and ROI-aligning. Basically, each bounding box is a certain region of interest (ROI), which is first projected onto the feature map. They differ in the way features are computed for a certain ROI.</p>
<p>I highly recommend <a href="https://www.youtube.com/watch?v=9AyMR4IhSWQ&amp;list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r&amp;index=16" rel="noopener nofollow ugc">this video</a> which explains how both operations work (it starts around the 20th minute). ROI-aligning is actually simpler and performs better than ROI-pooling, since it doesn’t snap the projected region of interest onto the grid cells. Both are implemented in torchvision.</p>
<p>I also recommend <a href="https://stackoverflow.com/a/60062070/7762882" rel="noopener nofollow ugc">this Stackoverflow post</a> that explains how ROI-align works in PyTorch.</p>

vivek_maskara: <p>Thanks for the answer <a class="mention" href="/u/niels_pytorch">@Niels_PyTorch</a>. I just tried out ROI-align and it seems to be serving my purpose well.</p>
<p>I will update this thread with any new learnings/challenges with ROI-align.</p>

Abhishek_R: <p>Hi even im trying to solve the same problem , but my question can we determine the actions of individual object bounding box given its object location and features from the i3d of full frame ?</p>
Expected scalars to be on CPU, got meta instead

tonyr: <p>I’m sorted, I don’t need help, the point of this post is so that the error message gets indexed by the search engines and people don’t waste time like I just have.</p>
<p>I have a version of pytorch-eg-wordLM where I have two optimisers, one for the word embeddings and one for everything else.  I compile my model and train() function separately.  torch.compile works wonderfully except in the case below:</p>
<pre><code class="lang-plaintext">model = torch.compile(RNNModel(...))

sparse = [ 'word', '_orig_mod.word' ]
optDense = torch.optim.AdamW([p for n, p in model.named_parameters() if n not in sparse], fused=True)
optEmbed = torch.optim.AdamW([p for n, p in model.named_parameters() if n in sparse])

def train(batch_size):
    model.train()
    hidden = model.init_hidden(batch_size)
    for batch, i in enumerate(range(0, train_data.size(0) - arg.bptt, arg.bptt)):
	data, targets = get_batch(train_data, i, device=device)
        model.zero_grad()
        hidden = repackage_hidden(hidden)
        output, hidden, _ = model(data, hidden, targets=targets)
        loss = criterALL(output, targets.long())
        item = loss.item()
        loss.backward()
        optDense.step()
        optEmbed.step()
        print('working')
</code></pre>
<p>Which results in the error message:</p>
<pre><code class="lang-plaintext">torch._dynamo.exc.TorchRuntimeError: Failed running call_function &lt;built-in method _foreach_addcdiv_ of type object at 0x7365f3a9f6c0&gt;(*([Parameter(FakeTensor(..., device='cuda:0', size=(32766, 1920), requires_grad=True))], [FakeTensor(..., device='cuda:0', size=(32766, 1920))], (FakeTensor(..., device='cuda:0', size=(32766, 1920)),), FakeTensor(..., size=(1,))), **{}):
Expected scalars to be on CPU, got meta instead.

from user code:
   File "/usr/lib/python3/dist-packages/torch/optim/optimizer.py", line 161, in maybe_fallback
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/torch/optim/adamw.py", line 767, in adamw
    func(
  File "/usr/lib/python3/dist-packages/torch/optim/adamw.py", line 604, in _multi_tensor_adamw
    torch._foreach_addcdiv_(

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
</code></pre>
<p>The answer of course is ‘well don’t do that then’.  I had set fused=True in one optimiser and not in the other.</p>
<p>[ I can of course provide full (cut down) code if that is of interest to anyone ]</p>
Inference breaks in Pytorch lightning model

JAllsopp: <p>Hello,<br>
I’m trying to do inference on a ResNet50 model for an image regression task I’m working on. I’ve changed the last layer of the model to output one value, and the starting layer to accept greyscale images (250x250). I’m using WebDataSet to import the data. However I can’t get inference to produce a good answer when using the shard from the validation set. I know the model is working as I can put a break point in the validation and compare the labels to the output from the model and calculating the MSE manually I get a good match to my validation epoch. I’m using the following code;</p>
<pre><code class="lang-auto">    def inference_data_set(self, batch):
        imgs, _ = batch
        imgs = imgs.to(self.device)
        self.eval()
        with torch.no_grad():
            output = self.forward(imgs)
            
        output = output.squeeze()
        output = output*self.max_ions
        return output
</code></pre>
<p>max_ions converts the output of the network into the numer of ions I’m expecting there to be in the image. However, when I use inference_data_set on one of the validation shards, I get absolute nonsense with values far out of range. Is there anything I’m obviously doing wrong here?</p>
Decreasing Maximum learning rate after every restart

Tejus: <p>Hi guys! I was using a scheduler to decrease the learning rate of my optimizer gradually and the one I was using was <code>CosineAnnealingWarmRestarts()</code>.It has a maximum learning rate which is set by us and anneals the learning rate in a cosine curve manner until it hits a restart where the learning rate is set to maximum again and the cycle restarts. What I wanted to do is decrease the learning rate the optimizer is initialized with every restart. Is that possible in Pytorch?</p>

ptrblck: <p>You could try to manipulate the <code>scheduler.base_lrs</code> using this code:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)

scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
    optimizer, T_0=10)

for epoch in range(40):
    print('scheduler ', scheduler.get_last_lr())
    print('optimizer ', optimizer.param_groups[0]['lr'])
    if (epoch+1) % 10 == 0:
        print('decrease')
        scheduler.base_lrs[0] = scheduler.base_lrs[0] * 0.5
        
    scheduler.step()
</code></pre>
<p>A quick test shows, that the optimizer’s learning rate for the default parameter group is also changed.<br>
However, I haven’t tested the code, so you should double check, if the training is working as expected.</p>

Wenuka: <p>I have tested it and it is working as expected. The only thing I had to add extra was to add an additional condition to make sure it doesn’t go below the min_LR.</p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"></p>

Moses_Lee: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the help!</p>
<p>Another hacky way (inspired from ptrblk) is to check if <code>scheduler.T_cur = 0</code> which will be set to 0 whenever <code>CosineAnnealingWarmRestarts</code> resets its learning rate to the <code>base_lr</code>.</p>
<pre><code class="lang-auto">model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)

scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
    optimizer, T_0=10)

for epoch in range(40):
    print('scheduler ', scheduler.get_last_lr())
    print('optimizer ', optimizer.param_groups[0]['lr'])
    
   if scheduler.T_cur == 0 and epoch !=0:  # to prevent decrease on the first epoch
        print('decrease')
        scheduler.base_lrs[0] = scheduler.base_lrs[0] * 0.5
        
    scheduler.step()
</code></pre>
<p>that way, if you use <code>T_mult</code> to widen the duration of <code>lr</code> decrease, this would still work given that the <code>lr</code> decrease does not necessarily happen every 10 epochs.</p>
Adding a BinomialDistributionLoss to TemporalFusionTransformer

mathDR: <p>Hi all (maybe a basic question) but I would like to construct a BinomialDistributionLoss() for my TemporalFusionTransformer</p>
<p>But such a loss requires <em>both</em> the output of the network <em>and</em> the <code>total_counts</code>.</p>
<p>I can provide the <code>total_counts</code> (and do) within the <code>TimeSeriesDataSet</code> via the <code>time_varying_known_reals</code> variable, but how can I access that value in the loss?</p>
<p>Does the dataloader return each covariate and could I pass it in?</p>
<p>Thanks!</p>

ptrblck: <p>If you can access this attribute in the <code>Dataset</code>, you could return it in the <code>__getitem__</code> which will also make sure the <code>DataLoader</code> returns it. Inside the training loop you could then pass it to your loss function.</p>
Does torch.rot90 support gradient flow?

joohyunglee: <p>Thank you much ahead…</p>

ptrblck: <p>Yes, it does:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 2, requires_grad=True)
out = torch.rot90(x)
out.grad_fn
# &lt;Rot90Backward0 at 0x7f21c7f4e110&gt;

out.mean().backward()
print(x.grad)
# tensor([[0.2500, 0.2500],
#         [0.2500, 0.2500]])
</code></pre>
How model will be effected after change input size for Resnet?

Mert_Oguzhan: <p>Hi! I have been thinking to use resnet architecture for classifying. But my input images are 368x736, If I use resnet for classification, Will model be effected ?  Will be feature vanishing? because the images will be resized? What should I do?</p>

Eduardo_Lawson: <p>Hi! I think you will only know the true answer trying it. But I think it will be effective, because 368x736 isn’t very small. You can check it out this: <a href="https://pytorch.org/hub/pytorch_vision_resnet/" rel="noopener nofollow ugc">https://pytorch.org/hub/pytorch_vision_resnet/</a>. Looking at the description available it will probaly work.</p>