akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (10x4096 and 320x256) in conformer_rnnt_base transcribe method

username: g1777
cooked: <p>i’m using</p>
<pre><code class="lang-auto">spectrogram = log(MelSpectrogram(n_mels=80, n_fft=1024, sample_rate=sr)(waveform)+ 1e-10).squeeze()
</code></pre>
<p>as input for the model.<br>
the error occurs here when calling decoder.forward</p>
<pre><code class="lang-auto">decoder = RNNTBeamSearch(
    model = model,
    blank=10,
    step_max_tokens=24,


)


with torch.no_grad():

    all_transcriptions = []
    all_targets = []
    for i, batch in enumerate(test_dataloader):

        inputs, targets, input_lengths, target_lengths  = batch
        transcriptions,transcriptions_lengths = model.transcribe(inputs,input_lengths)
        print(transcriptions_lengths)
        
        results = []
        for i in range(0,len(transcriptions)):
            result = decoder.forward(transcriptions,transcriptions_lengths,128)
            results.append(result)
</code></pre>
<p>i also tried with conformer_rnnt_model and the only case where it works is when i set both input_dim=80 and<br>
encoding_dim=80.<br>
thanks in advance for any help.</p>
Tensor indexing multiple NNs

username: AlexanderMerkel
cooked: <p>I currently face the problem of wanting to use tensor indexing to select NNs from a list. What I want to do is list_of_NNs [index_tensor] (evaluate_on_this).</p>
<p>This does not work with lists and I could not find a way around this besides using generator expressions which is of course terribly slow. Is there a data structure of function I could find which allows me to do this efficiently?<br>
Specifically, I want to evaluate on batches, of course.</p>
<p>Edit: Bad formatting…</p>

username: ptrblck
cooked: <p>I’m unsure if I understand the use case correctly, but do you want to create multiple modules, store them in a <code>list</code>, and iterate them afterwards?<br>
Would this example work?</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, modules):
        super().__init__()
        self.module_list = nn.ModuleList(modules)
    
    def forward(self, x):
        for module in self.module_list:
            x = module(x)
        return x
    
modules = [
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
]
model = MyModel(modules)
x = torch.randn(1, 10)
out = model(x)
</code></pre>
<p>If not, could you describe your use case a bit more, please?</p>

username: AlexanderMerkel
cooked: <p>Sorry, my initial post had some formatting issues.</p>
<p>My main goal is to define multiple modules, say NN_1,…,NN_m. I then want to be able to store them in some structure, a list in my current code, say list_NN. Then, I want to “tensor index” that list, so given a tensor of indices index_tensor, I want list_NN[tensor_index], where an entry is the corresponding NN to that index.<br>
I then want to evaluate this “tensor of NNs” on some batch of tensors.</p>
<p>I could not get your approach to work because ModuleList also seems to be using “basic” lists.</p>
<p>I also tried using vmap but it also raises, that only single integer tensors can be used for indexing.</p>

username: ptrblck
cooked: <p>Thanks for clarifying. The idea to use <code>vmap</code> sounds reasonable and your example is also quite similar to <a href="https://pytorch.org/functorch/nightly/notebooks/ensembling.html#model-ensembling">this model ensembling</a> example.</p>
Error while using nn.dataparallel

username: s_n
cooked: <p>Hi,</p>
<p>I am getting the below error when I am trying to use dataparallel in my code. Has anyone else seen a similar issue? I am not sure why this error is occurring, when I try to run my code on a single GPU card then I don’t get this error but I only get it when I parallelize it over multiple GPU cards.</p>
<pre><code>outps = model(event_reprs)
</code></pre>
<p>File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 169, in forward<br>
return self.gather(outputs, self.output_device)<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 181, in gather<br>
return gather(outputs, output_device, dim=self.dim)<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py”, line 78, in gather<br>
res = gather_map(outputs)<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py”, line 73, in gather_map<br>
return type(out)(map(gather_map, zip(*outputs)))<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py”, line 73, in gather_map<br>
return type(out)(map(gather_map, zip(*outputs)))<br>
File “/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py”, line 73, in gather_map<br>
return type(out)(map(gather_map, zip(*outputs)))<br>
TypeError: ‘float’ object is not iterable<br>
Uncaught exception. Entering post mortem debugging<br>
Running ‘cont’ or ‘step’ will restart the program</p>
<blockquote>
<p>/home/min/a/abcd/anaconda3/envs/pt17/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py(73)gather_map()<br>
 → return type(out)(map(gather_map, zip(*outputs)))</p>
</blockquote>

username: Knight
cooked: <p>I encountered the same error as you, the reason is that the loss value is of type float when trained by multiple GPUs.</p>
Run prophet example error

username: goodboy-hub
cooked: <p>recently i try to use the facebook prophet model to do some task. but when i ran the example github shown, i met an error. i will appreciate if anyone knew the problem.</p>

username: goodboy-hub
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e" title="屏幕截图 2023-04-08 165527"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e_2_690x371.png" alt="屏幕截图 2023-04-08 165527" data-base62-sha1="tWGLq0m350q0Wi03tWfHJMgCK9o" width="690" height="371" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e_2_690x371.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e_2_1035x556.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1e16d1e7fb227753b43b3a0cb98edbc12252d8e_2_1380x742.png 2x" data-dominant-color="401B1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2023-04-08 165527</span><span class="informations">1470×792 94.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: goodboy-hub
cooked: <p>ok解决了 ,  我换了一台算力高的服务器就好了 <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>
Build pytorch with cutlass

username: ChunLin_Fang
cooked: <p>I just download pytorch 1.11.0 from <a href="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp38-cp38-linux_x86_64.whl" rel="noopener nofollow ugc">here</a>, I find out that <code>lib/libtorch_cuda_cpp.so</code> and<code> lib/libtorch_cuda_cu.so</code> is compiled with cutlass, but AFAIK, cutlass is introduced to pytorch at 1.13.0, Why is this library compiled in advance?</p>

username: ptrblck
cooked: <p>How did you check it and which ops/functions do you see?</p>

username: ChunLin_Fang
cooked: <p>I use unzip to open the whl file, and get into lib directory, This is the command used:<br>
<code>strings libtorch_cuda_cpp.so | grep "cutlass"</code><br>
I can see many cutlass interface and symbols.</p>
Layernorm cann't be quantized in QAT?

username: yyd199948
cooked: <p>The structure of the model after training the converter through QAT is shown below. I noticed that there are no parameters such as scale or zero_point for layernorm. Does this mean that layernorm has not been quantized? Can QAT be used to quantize layernorm? I am using PyTorch 1.13.0 and it seems that layernorm cannot be quantized.<br>
(bn1): QuantizedLayerNorm((8, 32), eps=1e-05, elementwise_affine=True)<br>
(prelu1): ReLU()<br>
(bn2): QuantizedLayerNorm((16, 16), eps=1e-05, elementwise_affine=True)<br>
(prelu2): QuantizedPReLU()<br>
(bn3): QuantizedLayerNorm((32, 8), eps=1e-05, elementwise_affine=True)<br>
(prelu3): QuantizedPReLU()<br>
(dbn3): QuantizedLayerNorm((16, 16), eps=1e-05, elementwise_affine=True)<br>
(dprelu3): QuantizedPReLU()<br>
(dbn2): QuantizedLayerNorm((8, 32), eps=1e-05, elementwise_affine=True)<br>
(dprelu2): QuantizedPReLU()<br>
(dbn1): QuantizedLayerNorm((1, 65), eps=1e-05, elementwise_affine=True)<br>
(dprelu1): QuantizedPReLU()<br>
(quant): Quantize(scale=tensor([0.1354]), zero_point=tensor([0]), dtype=torch.quint8)</p>
How to use average pooling to do average on 4 tensors of size 64*256

username: nikiguo93
cooked: <p>Hi, I have four tensors with size of 64*256. Can I use average pooling to do average on these 4 tensors and the output after the pooling operation should be something like k * 256 ?<br>
Looking forward to hearing from you!</p>

username: ptrblck
cooked: <p>I’m unsure how exactly your input tensor would look like, but assuming it has the shape <code>[batch_size=4, channels, 64, 256]</code>, this might work:</p>
<pre><code class="lang-python">k = 32
pool = nn.AdaptiveAvgPool2d((k, 256))
x = torch.randn(4, 16, 64, 256)
out = pool(x)
print(out.shape)
# torch.Size([4, 16, 32, 256])
</code></pre>

username: nikiguo93
cooked: <p>Hi,thanks. This works for me.</p>
Can you suggest me best name combiner too for excel sheet?

username: Mialee
cooked: <p>First, let’s clarify what a name combiner is. A name combiner is a tool or feature that allows you to combine two or more names into a single name. This can be useful in many situations, such as when you need to create a mailing list or a database of people’s names.</p>
<p>Excel is a powerful spreadsheet software that allows you to organize and manipulate data, including names. One way to combine names in Excel is to use a formula that concatenates (joins together) the first and last names. Here’s an example of such a formula:</p>
<p>=CONCATENATE(A2," ",B2)</p>
<p>In this formula, “A2” and “B2” refer to the first name and last name cells, respectively. The space between the quotation marks creates a space between the first and last names.</p>
<p>However, if you need to combine a large number of names or have more complex requirements, you may want to use a specialized name combiner tool. There are several name combiner tools available online, both free and paid, that can help you combine names in Excel and other spreadsheet software.</p>
<p>One popular tool is the “Combine Names” add-in for Excel. This add-in, which is available for download from the Microsoft Office Store, allows you to easily combine first and last names, as well as middle names and suffixes, into a single cell.</p>
<p>Another popular tool is the “Name Combiner” from 123FormBuilder. This tool allows you to upload an Excel file containing first and last names and automatically combine them into a single column. You can also customize the output format, such as adding a comma between the first and last names.</p>
<p>There are many other name combiner tools available, each with its own features and capabilities. When choosing a name combiner tool for Excel, consider your specific needs and the complexity of your data. Additionally, make sure to choose a tool that is compatible with your version of Excel and is easy to use.</p>

username: ptrblck
cooked: <p>This question seems to be unrelated to PyTorch so I would suggest asking it in an Excel-specific forum.</p>

username: Mialee
cooked: <p>can you share the link for that forum ?</p>

username: ptrblck
cooked: <p>I guess <a href="https://techcommunity.microsoft.com/t5/microsoft-excel/ct-p/Excel_Cat">this Microsoft Excel forum</a> might be a good place.</p>

username: Mialee
cooked: <p>ok thanks, i will check</p>

username: Mialee
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="176863" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I guess <a>this</a><a href="https://techcommunity.microsoft.com/t5/microsoft-excel/ct-p/Excel_Cat" rel="noopener nofollow ugc"> Microsoft Excel forum </a> might be a good place.</p>
</blockquote>
</aside>
<p>I found the right one , finally</p>
How to create a 3d kernel over channels?

username: spike354
cooked: <p>Lets say I give two input epicures of size 234x234. Then I have an input tensor [2,3,234,234]. How do I create a 3d kernel which only take the 3 channels as input and produces 1 output feature map from it?</p>

username: ptrblck
cooked: <p>You could create an <code>nn.Conv2d</code> layer accepting inputs with 3 channels and creating a single output activation map as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 234, 234)
conv = nn.Conv2d(in_channels=3, out_channels=1, kernel_size=3, padding=1)
out = conv(x)
print(out.shape)
# torch.Size([2, 1, 234, 234])
</code></pre>
<p>Note that the filter will be a 4D filter defined as <code>[out_channels, in_channels, height, width]</code>.</p>
Error when calling torch.jit.trace

username: FatemeH
cooked: <p>Hi all,<br>
I am trying to import a pytorch model in tvm. I run the following code to get the torchscripted model from tracing.<br>
import tvm<br>
import torch<br>
import torchvision</p>
<pre><code>model_name = "resnet18"
model = getattr(torchvision.models, model_name)(pretrained=True)
model = model.eval()

input_shape = [1, 3, 224, 224]
input_data = torch.randn(input_shape)
scripted_model = torch.jit.trace(model, input_data).eval()
</code></pre>
<p>But I get this error:</p>
<blockquote>
<p>libc++abi: terminating with uncaught exception of type pybind11::stop_iteration:<br>
Fatal Python error: Aborted</p>
</blockquote>
<p>In torch/jit/_trace.py", line 417 in graph_diagnostic_info.<br>
I found out that the error occurs when the torch tries to iterate through a Dict. Any idea on how to solve this?</p>
<p>I build torch with libstdcpp and tvm with libcpp. I suspected that this might be the cause of this.<br>
Since I cannot build tvm with libstdcpp, I tried building torch with libcpp but I still get the same error. When I get dir on the Dict variable which is casusing the error, I see <strong>pybind11_module_local_v4_clang_libstdcpp_cxxabi1002</strong>. So, it seems that pybind11 is still using libstdcpp. How to build pybind11 submodule with libcpp?</p>
<p>Thank you,<br>
Fateme</p>

username: blrm
cooked: <p><a class="mention" href="/u/fatemeh">@FatemeH</a> Did you try importing torch before importing tvm? That worked for me when I had torch built with libstdcpp and tvm built with libcpp</p>

username: FatemeH
cooked: <p>Thanks for your response. Yes, that’s correct. if I reverse the order of imports it works. But I sometimes I need to call my code through tvm testing module and I get the same failure. So, I need to resolve the issue.</p>
The problem of replacing the resnet layer with nn.Identity in torchvision

username: Ustinian_W
cooked: <p>import torch<br>
import torchvision<br>
from torch import nn<br>
from torchsummary import summary</p>
<p>model = torchvision.models.resnet18(pretrained=False)</p>
<p>model.avgpool = nn.Identity()<br>
model.fc = nn.Identity()<br>
x = torch.randn(size=(2,3,128,128))<br>
y = model(x)</p>
<p>print(x.shape)<br>
print(y.shape)</p>
<p>the output is<br>
torch.Size([2, 3, 128, 128])<br>
torch.Size([2, 8192])</p>
<p>The shape of the last few layers（using torchsummary ）：<br>
BatchNorm2d-64             [2, 512, 4, 4]<br>
ReLU-65             [2, 512, 4, 4]<br>
BasicBlock-66             [2, 512, 4, 4]<br>
Identity-67             [2, 512, 4, 4]<br>
Identity-68                  [2, 8192]</p>
<p>I don’t understand why the last identity layer flattens the image</p>

username: ptrblck
cooked: <p>The <code>nn.Identity</code> layer is not flattening the activation, but <a href="https://github.com/pytorch/vision/blob/1c4f0c49079f6e3a5b3cf2109fd3f83b1041f0d0/torchvision/models/resnet.py#L279">this <code>torch.flatten</code></a> call defined in the <code>forward</code> method via the functional API.</p>

username: Ustinian_W
cooked: <p>Thank you for your answer, can you tell me how to cancel this flatten operation</p>

username: ptrblck
cooked: <p>You could create a custom model and override the <code>forward</code> method with your own.</p>

username: Ustinian_W
cooked: <p>ok, i see, thanks again</p>
Dataset for MICCAI 2008 Coronary Artery Tracking Challenge

username: gregpost
cooked: <p>Hello! Does anybody can give me the link to download the dataset for  MICCAI 2008 Coronary Artery Tracking Challenge (CAT08)? The official website doesn’t work.</p>

username: marioviti
cooked: <p>I have the same issue.</p>
<p>I sent an email to the author that I could find here.</p>
<p><a href="https://github.com/BubblyYi/Coronary-Artery-Tracking-via-3D-CNN-Classification" rel="noopener nofollow ugc">BubblyYi/Coronary-Artery-Tracking-via-3D-CNN-Classification: The PyTorch re-implement of a 3D CNN Tracker to extract coronary artery centerlines with state-of-the-art (SOTA) performance. (paper: ‘Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classiﬁer’) (github.com)</a></p>
<p>You should send one yourself</p>

username: leaxxxxxxx
cooked: <p>Hi,<br>
The website link of this dataset is invalid now and I can’t find this dataset on  google. Could you share this data set? I really need this.</p>

username: tianmaliuxing
cooked: <p>I sent the email, but there was no reply. Sad…Did you got the dataset?</p>
How to convert python deep learning code into c++ dll library

username: tjk
cooked: <p>Hello,</p>
<p>My friend develop a GUI program in C++, and he wants to embed the python deep learning code into the program (a CNN model), so that we can train and test the model by pressing the button of the GUI C++ program. However, the python code with CNN model needs import packages like torch, numpy, scipy (with pytorch), if written using the tensorflow model, it needs import packages like keras. Is it possible to convert the python source code, including the deep learning packages into dll files so that we can call the model training and testing with the C++ GUI program? If it is possible, I need someone offer me some guidance on how to do that.</p>
<p>Thanks!!!</p>

username: ptrblck
cooked: <p>Generally you could script your model and export it via <code>torch.jit.scirpt</code> and <code>torch.jit.save</code>, respectively, which would allow you to load it in a pure C++ application using <code>libtorch</code>.<br>
However, if that’s not an option you could also check the beta <a href="https://github.com/pytorch/multipy"><code>torch::deploy/MultiPy</code> library </a> which allows you to run eager mode PyTorch models in a C++ application.</p>
Way to aggregate loss in DDP training

username: JungHwang
cooked: <p>Hi,<br>
I was implementing logging snippets in Ultralytics YOLOv8, and I realized that they way it calculates loss in training is the following</p>
<pre><code class="lang-python"># Forward
with torch.cuda.amp.autocast(self.amp):
    # a couple of lines of code
    self.loss, self.loss_items = self.criterion(preds, batch)
    if RANK != -1:
        self.loss *= world_size
    self.tloss = (self.tloss * i + self.loss_items) / (i + 1) if self.tloss is not None \
        else self.loss_items
</code></pre>
<p>So it basically get the loss calculated in the main process and multiply the loss by the number of processes.<br>
It seems reasonable, and it shouldn’t be different from the ‘real loss’ by a lot, but it is not the actual aggregated loss from different processes.</p>
<p>Does DDP ever aggregate the all the losses calculated across different processes?<br>
Or above snippet is the only way to get the loss?</p>

username: ptrblck
cooked: <p>No, <code>DDP</code> would not aggregate the losses from different ranks as each rank gets an independent input and calculates “its own” gradients. The gradients are allreduced during the backward pass and eventually all <code>.grad</code> attributes contain the same gradients before the corresponding parameters are updated.<br>
More details about <code>DDP</code> can be found in the <a href="https://pytorch.org/docs/master/notes/ddp.html#internal-design">DDP internal design doc</a>.<br>
I guess your code assumes an approx. same loss on different ranks and just scales it for e.g. logging purpose?</p>

username: JungHwang
cooked: <p>Yeah, the code is from YOLOv8, an open source by Ultralytics for object detection, and it assumed the same losses on different ranks like you said. So I was just wondering if there is a way to get the exact loss calculated across the different ranks, but approximation seems reasonable and good enough for logging in my case.</p>
<p>Thanks for your reply!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="JungHwang" data-post="3" data-topic="176929">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/8edcca/48.png" class="avatar"> JungHwang:</div>
<blockquote>
<p>So I was just wondering if there is a way to get the exact loss calculated across the different ranks</p>
</blockquote>
</aside>
<p>Yes, you could manually allreduce the losses as seen in e.g. <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/777d174008c365a5d62799a86f135a4f171f620e/PyTorch/Classification/ConvNets/image_classification/training.py#L231">this example</a> which sums the losses <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/777d174008c365a5d62799a86f135a4f171f620e/PyTorch/Classification/ConvNets/image_classification/utils.py#L117-L123">here</a>.</p>
Loss isn't changing with LSTM

username: praveenbenedict
cooked: <p>I wrote a simple LSTM classifier to predict the sentiment of movie reviews. Here is the link to the review.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1SEr5YXAT4gXA4ANeu3PENpJeRL-ORd1g?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/1bfe02adf746359a2b88431ee07c9195/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1SEr5YXAT4gXA4ANeu3PENpJeRL-ORd1g?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1SEr5YXAT4gXA4ANeu3PENpJeRL-ORd1g?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Here is the link to the dataset: <a href="https://drive.google.com/file/d/1BpPnO-nuShi1ZaykhS1rBc7qJyC1sQlW/view?usp=share_link" rel="noopener nofollow ugc">https://drive.google.com/file/d/1BpPnO-nuShi1ZaykhS1rBc7qJyC1sQlW/view?usp=share_link</a></p>
<p>While training, the loss isn’t changing at all. Can anyone help me with identifying the issue.</p>
<p>Thanks!</p>

username: vdw
cooked: <p>I only had a quick look on my phone, might have some more time tomorrow.</p>
<p>Anyway, I would at least try with <code>torch.optim.Adam</code>. That basic SGD optimizer is not great.</p>

username: matt.carlson
cooked: <p>I would also try a learning rate scheduler. it’s possible that the model has found a shortcut to return the average of your training data and is stuck in a premature minimum. Does loss decrease at first then settle into a constant number quickly? Also I think 1e-1 is a very high learning rate if no scheduler or manual decay is used.</p>
<p>Here is an example of a learning rate scheduler i’ve had good results with.</p>
<pre><code class="lang-auto">## Calling scheduler.step() every batch
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, len(train_dataloader) * epochs_per_cycle, eta_min=0)

## Calling scheduler.step() every epoch
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, epochs_per_cycle, eta_min=0)
</code></pre>
<p>Another thing Is I have seen advice on this forum to use this order during training -</p>
<pre><code class="lang-auto">loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>As opposed to your implementation</p>
<pre><code class="lang-auto">optim.zero_grad()
loss.backward()
optim.step()
</code></pre>
<p>This is by no means expert advice, but maybe it can point you in the right direction.</p>
Positional Encoding

username: dev1
cooked: <p>Hi everyone. I implemented the positional encoding class just like in the pytorch tutorial:</p>
<p>class PositionalEncoding(nn.Module):</p>
<pre><code>def __init__(self, d_model, max_len):
    super().__init__()

    position = torch.arange(max_len).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
    pe = torch.zeros(max_len, 1, d_model)
    pe[:, 0, 0::2] = torch.sin(position * div_term)
    pe[:, 0, 1::2] = torch.cos(position * div_term)
    self.register_buffer('pe', pe)

def forward(self, x:
    x = x + self.pe[:x.size(0)]
    return self.dropout(x)
</code></pre>
<p>Then I run this:</p>
<pre><code class="lang-auto">rand = torch.randint(10, (BATCH_SIZE, 20))
emb = PositionalEncoding(EMBED_LEN, MAX_SEQ_LEN)
out = emb(rand)
print(out.shape)
</code></pre>
<p>I expect positional encoding output here. What I end up with is:<br>
-&gt;The size of tensor a (20) must match the size of tensor b (768) at non-singleton dimension 2</p>
<p>As with the paper don’t I need to feed input directly to the positional encoder? Is this error expected? What’s the best way to test if this function operates as expected?</p>

username: J_Johnson
cooked: <p>Here is the tutorial I believe you are referring to:</p>
<p><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>Some important context:</p>
<pre><code class="lang-auto">Args:
            x: Tensor, shape [seq_len, batch_size, embedding_dim]
</code></pre>
<p>Note the batch_size is on dim=1. And there is an embedding dim. Yet, the example you provided has batch_size on dim=0 and no embedding dim.</p>

username: matt.carlson
cooked: <p>Hi, I implemented this exact thing with batch_size on dim 0 recently. Here is the code, it may be helpful.</p>
<p>For the <code>forward</code> method</p>
<pre><code class="lang-auto">return self.dropout(token_embedding + self.pos_encoding[:, :token_embedding.shape[1]])
</code></pre>
Self.generate() gives OOM in training

username: sajastu
cooked: <p>Hi, in part of my implemented code in BartForConditionalGeneration class of Huggingface, I need to call self.generate(**kwargs) function for token generation. When I call this function in forward method of BartForConditionalGeneration class, I’m getting OOM error on GPU. However, I can perform validation steps through self.evaluate function of trainer class while training.</p>
<p>Here’s the truncated snippet I have currently in the forward method of BartForConditionalGeneration:</p>
<pre><code class="lang-auto">
...
output = super().forward(
            input_ids=None,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            head_mask=head_mask,
            decoder_head_mask=decoder_head_mask,
            cross_attn_head_mask=cross_attn_head_mask,
            encoder_outputs=encoder_outputs,
            encoder_outputs_weighted=encoder_outputs_weighted,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            decoder_inputs_embeds=decoder_inputs_embeds,
            labels=labels,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=True,
            # output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

# this line gives OOM error on GPU
with torch.no_grad():
        generated_tokens = self.generate(encoder_outputs=BaseModelOutput(last_hidden_state=encoder_outputs[0]), attention_mask=attention_mask)

...
</code></pre>
<p>I have tried a bunch of tricks to resolve this error: like using lower batch_size, and num_beams; however, still getting the same error. What are the possible causes of this behaviour?</p>

username: matt.carlson
cooked: <p>Hi, I’m not familiar with Hugging Face so I’m not sure if this will solve your problem, but maybe I can give you some ideas.</p>
<p>Could you post the error message? It should tell you amount of allocated memory vs how much memory PyTorch is trying to allocate. Also do you have some sort of performance monitor to see if your GPU is OOM?</p>
<p>I believe evaluation passes are performed without gradients where regular forward passes are. If you don’t need to train, this would unnecessarily increase the space used by the GPU. Could you make sure you are using things like</p>
<p><code>model.eval()</code><br>
<code>with torch.no_grad():</code><br>
or<br>
<code>requires_grad=False</code> when instantiating tensors</p>
<p>I’m not familiar with this specific model but in many transformers generating token embeddings is a trainable layer.</p>
How to reuse precalculated attention weights for autoregressive transformers

username: Thibault712
cooked: <p>Hi when I do an inference (not training) of my autoregressive transformer I do it substantially this way (I removed few lines to not affect readibility):</p>
<pre><code class="lang-auto">for i in range(max_batch_sequence_len):
    for layer in self.layers:
        y[:, i] = layer(x, keep_mask, y)[:, i]
</code></pre>
<p>where my layers "forward’ are:</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor, keep_mask: torch.Tensor, y: torch.Tensor) -&gt; torch.Tensor:
        attn_mask = (~keep_mask).unsqueeze(2) &amp; (~keep_mask).unsqueeze(2)
        attn_mask = attn_mask.repeat_interleave(self.num_heads, dim=0)
            
        y_normed = self.layer_norm(y)
        y = y + self.self_attn(y_normed) #A causal mask is applied

        x_normed = self.layer_norm(x)
        y_normed = self.layer_norm(y)
        y = y + self.cross_attn(y_normed, x_normed, attn_mask)
        
        y_normed = self.layer_norm(y)
        y = y + self.ffn(y_normed)
        return y

def self_attn(self, y):
        out, _ = self.attn1(
            query=y, key=y, value=y, need_weights=False, is_causal=True,
        )
        return out

def cross_attn(self, y, x, attn_mask):

    out, _ = self.attn2(
        query=y, key=x, value=x, need_weights=False, attn_mask=attn_mask
    )
    return out
</code></pre>
<p>I can see that using the attention weights and re-inputing them in a certain way I can manage to reduce the computation, especially at step i+1 the attention weights for j&lt;=i have all been already computed.</p>
<p>Has someone here have ever dealt with that and can suggest me a modification of my code?</p>

username: matt.carlson
cooked: <p>Hi I’m not sure if this will help solve your problem but just to give some ideas, typically when I have used transformers i’d do something like this for inference (we wouldn’t want to instantiate every time of course) -</p>
<pre><code class="lang-auto">model = MyTransformer(**args)
model = torch.compile(model)
model.load_state_dict(**args)

predictions = model(src, tgt)

</code></pre>
How can I keep gradient when I change one row

username: JuyiLin
cooked: <p>I have a tensor , input size = (3,4)<br>
I have to change the second row with  new size = (1,4)<br>
How can I change it while keeps the gradient?<br>
When I used these codes, it shows</p>
<pre><code class="lang-auto">        x.masked_fill_(mask, 0)  # set the values of cached nodes in x to 0
        x += emb  # add the embeddings of the cached nodes to x
        return x
</code></pre>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an in-place operation: [torch.cuda.FloatTensor [3, 4]], which is output 0 of ReluBackward0, is at version 2<br>
I change to</p>
<pre><code class="lang-auto">+        out = x.clone()
+        out.masked_fill_(mask, 0)  # set the values of cached nodes in x to 0
+        out += emb  # add the embeddings of the cached nodes to x
+        return out
</code></pre>
<p>But it cannot keep the gradient.</p>
<p>My question is similar with</p><aside class="quote" data-post="6" data-topic="16894">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nima_rafiee/48/16289_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-select-elements-in-tensor-greater-than-and-keep-gradient/16894/6">How to select elements in tensor greater than and keep gradient?</a> 
  </div>
  <blockquote>
    you can use torch.gt() or torch.le() . it results in a tensor with the same dimension as the input tensor then you can do what you want.
  </blockquote>
</aside>
<p>
But it is several years ago so I want to know whether there is new solution.</p>

username: ptrblck
cooked: <p>Cloning the tensor might be the right approach and I don’t understand the last claim:</p>
<aside class="quote no-group" data-username="JuyiLin" data-post="1" data-topic="176883">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juyilin/48/53738_2.png" class="avatar"> JuyiLin:</div>
<blockquote>
<p>But it cannot keep the gradient.</p>
</blockquote>
</aside>
<p>This small code snippet works fine for me:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(3, 4, requires_grad=True)
mask = torch.randint(0, 2, (3, 4)).bool()
emb = torch.randn(1, 4, requires_grad=True)

out = x.clone()
out.masked_fill_(mask, 0)  # set the values of cached nodes in x to 0
out += emb  # add the embeddings of the cached nodes to x

out.mean().backward()

print(x.grad)
# tensor([[0.0833, 0.0833, 0.0833, 0.0000],
#         [0.0833, 0.0000, 0.0833, 0.0000],
#         [0.0000, 0.0833, 0.0833, 0.0000]])

print(emb.grad)
# tensor([[0.2500, 0.2500, 0.2500, 0.2500]])
</code></pre>
<p>Could you describe which gradient exactly won’t be kept?</p>
Return type of transforms.Compose() not recognized by linter

username: gursi26
cooked: <p>The return type of transforms.Compose() is not recognized as a tensor, even when transforms.ToTensor() is one of the transformations in it. Is there something wrong with my code/linter or is this a bug?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76" title="Screenshot 2023-04-06 at 7.37.12 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76_2_690x114.png" alt="Screenshot 2023-04-06 at 7.37.12 PM" data-base62-sha1="oRwiaN4uBuLb5aWjqruMUJXR6No" width="690" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76_2_690x114.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76_2_1035x171.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae40fa31d1c4aca7d00102d5eacbfbcd1f875b76_2_1380x228.png 2x" data-dominant-color="272526"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-04-06 at 7.37.12 PM</span><span class="informations">2018×334 55 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Specifications:</p>
<ul>
<li>MacOS</li>
<li>torch version 2.0.0</li>
<li>torchvision version 0.15.1</li>
</ul>
Thread died at c10::cuda::CUDACachingAllocator::emptyCache();

username: hjhong0213
cooked: <p>In LibTorch,</p>
<p>After the trained model, call c10::cuda::CUDACachingAllocator::emptyCache() to free the GPU memory.<br>
But sometimes the thread died in this <strong>emptyCache()</strong>.</p>
<p>Can anyone help me with this issue?</p>

username: eqy
cooked: <p>This seems unusual, could there potentially be a race between this thread and other parts of your libtorch usage?</p>
How to correctly weight MSE loss for padded sequences

username: Thibault712
cooked: <p>I have embedded sequences of size (batch, sequence, dim), and a padding mask of size ((batch, sequence), False meaning “padding”.<br>
Imagine a NLP problem, where I have different sentences of various number of words while keeping this number below a threshold of “sequence”.</p>
<p>Here an example:</p>
<pre><code class="lang-auto">y_pred = torch.randn(2, 3, 4)
y = torch.randn(2, 3, 4)
keep_mask = torch.tensor([[True, True, False],
                          [True, True, True]])
</code></pre>
<p>I think a fair way would be weighting this loss, such that small sequences contribute to the loss as much as big sequences. Mathematically I don’t want to bias my model to do better on bigger sequences. I have not found a better than:</p>
<pre><code class="lang-auto">loss_fn = nn.MSELoss(reduction="none")
weight_sequence = mask / mask.sum(-1, keepdim=True)
loss = loss_fn(y_pred, y) * weight_sequence.unsqueeze(-1)
loss = loss.mean()
</code></pre>
<p>Maybe you could suggest a better way to perform this or a solution my problem.</p>

username: Michal_Bogacz
cooked: <p>Hi,<br>
can you specify the problem on some example?<br>
Dou you want to create some network for token generation/token classification, where output can be shorter than the length of the target?<br>
MSE is a token based metrics, so the number of tokens shouldn’t matter in my opinion and should be aggregated through the batch. But if you tell more about the problem, I might be able to provide better answer.</p>

username: AbdulsalamBande
cooked: <p>Your current approach to calculating the loss is reasonable, but there might be a more efficient way to achieve the same goal. You can try the following approach:</p>
<ol>
<li>Mask the predicted and target sequences using the keep_mask.</li>
<li>Calculate the loss for each non-padded sequence element.</li>
<li>Divide the sum of losses for each batch by the number of non-padded elements in the batch.</li>
</ol>
<p>Try this</p>
<pre><code class="lang-auto">y_pred = torch.randn(2, 3, 4)
y = torch.randn(2, 3, 4)
keep_mask = torch.tensor([[True, True, False],
                          [True, True, True]])

y_pred_masked = y_pred * keep_mask.unsqueeze(-1)
y_masked = y * keep_mask.unsqueeze(-1)
loss_fn = nn.MSELoss(reduction="none")
loss = loss_fn(y_pred_masked, y_masked)
num_non_padded = keep_mask.sum(-1, keepdim=True).unsqueeze(-1)
mean_loss = loss.sum() / num_non_padded.sum()

</code></pre>
Stacked LSTM vs simple LSTM

username: msabrii
cooked: <p>Good evening,</p>
<p>This is more of a general question. I am new to pytorch and forgot about the existence of a stacked LSTM. I realized changing the num_layers parameter in the LSTM initialization can make it stacked. My question now is in what cases should a stacked LSTM be preferred over a simple one? Is num_layers a hyperparameter to be fine-tuned? Or is it more task-specific?</p>
<p>For context, I am doing a sequence classification task on textual posts to predict the verdict. Is it the case that a stacked LSTM is always preferred? Or is this just another hyperparameter to finetune?</p>

username: AbdulsalamBande
cooked: <p>he choice between a simple LSTM and a stacked LSTM depends on task complexity, dataset size, and computational resources. Stacked LSTMs can capture more complex patterns but may require larger datasets and more resources. The number of layers (num_layers) is a hyperparameter to fine-tune based on validation set performance. Experiment and evaluate different configurations to find the best model for your task. I general</p>
<ol>
<li>
<p>Task complexity: If your task involves capturing more complex and hierarchical patterns, a stacked LSTM might be a better choice, as it can learn different levels of abstraction across layers.</p>
</li>
<li>
<p>Training time and computational resources: Stacked LSTMs typically take longer to train and require more computational resources. If you have limited resources, a single-layer LSTM might be a better option.</p>
</li>
</ol>
OOM when resuming training

username: Ludicio_casanova
cooked: <p>Hello there, i saved a state dict model along the optimizer state, but when i try to resume it , it seems<br>
that the model and its loaded state consume more memory (about 2 GiB more) than the original one when training, which looks stupid, so i just want to ask if it has to be something else than a very bad loading code !<br>
Basically the end of my original model training pipeline looks like that:</p>
<pre data-code-wrap="model"><code class="lang-plaintext">    nn.Conv3d(1, 16, 3, 1, 1), 
    nn.BatchNorm3d(16),
    nn.Conv3d(16, 32, 3, 1, 1),
    nn.Flatten(start_dim=1),
    nn.LazyLinear(128), nn.GELU(),    
    nn.LazyLinear(1), nn.Sigmoid()
).to(device)
optimizer = optim.AdamW(model.parameters(), lr=LEARNING_RATE)
</code></pre>
<p>and the resume pipeline:</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Conv3d(1, 16, 3, 1, 1), 
    nn.BatchNorm3d(16),
    nn.Conv3d(16, 32, 3, 1, 1),
    nn.Flatten(start_dim=1),
    nn.LazyLinear(128), nn.GELU(),    
    nn.LazyLinear(1), nn.Sigmoid()
)
checkpoint = torch.load('blabla.pt')
model.load_state_dict(checkpoint['model'])
model = model.to(device)
optimizer = optim.AdamW(model.parameters(), lr=LEARNING_RATE)
optimizer.load_state_dict(checkpoint['optimizer'])
</code></pre>
<p>Well i should also say that i train and resume on different GPU (Cloud) with the same size but not the same appellation (liek A4000 and P5000),  could it really be the matter (a difference of 2GiB )?<br>
(Tell me if the question isn(t understandable bad english speaker here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> )</p>
<p>And thanks in advance !</p>

username: ptrblck
cooked: <p>Are you using the same PyTorch environment in both setups (the working and failing one)?<br>
Starting with the PyTorch release supporting CUDA 11.7 we enabled CUDA’s lazy module loading, which would reduce the context size and lazily load only needed kernels into the context.<br>
2GB sounds like too much, but might be related.<br>
Also, do you know if 2GB relates to any specific object, e.g. the model’s parameter size, the optimizer state size etc.?</p>

username: Ludicio_casanova
cooked: <p>Sorry i thought i deleted this post since my post got hidden by the bot and by the time I just reduced vram utilization to dodge the error!</p>
<p>It was the same Pytorch environment and same vram capacity but as i mentionned not the same GPU,<br>
if i remember correctly, on the first one the VRAM  utilization was at, at least 98% during training.<br>
Thank you anyway <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !</p>
Random-Transforms: New transform every time __getitem__ is called?

username: mzimmer
cooked: <p>I am wondering whether applying Random Transformations/Augmentations of Images will transform each individual image in the same way in each epoch.</p>
<p>In other words: Is the Image with index <code>idx</code> always the same in every epoch or will the base image <code>idx</code> get a different transform at each call?<br>
Thanks!</p>

username: AbdulsalamBande
cooked: <p>In PyTorch, when you apply random transformations/augmentations to images, they are generally not transformed in the same way in each epoch. The transformations are meant to be random and are applied independently for each image at each call. This means that the image with index idx will likely get a different transformation at each call, making the augmented image different in every epoch.</p>
<p>This randomness in transformations helps the model become more robust and generalize better, as it learns from a wider variety of augmented images.</p>

username: mzimmer
cooked: <p>Thanks for the answer, that clarifies a lot.<br>
What can I do to keep random augmentations fixed for some iterations? I need to compare the same image 2-3 times and I dont want it to be altered every time I get it.</p>
The result of torch.ao.nn.quantized.functional.conv1d is different from manual calculation

username: dopiwoo
cooked: <p><strong>Environment</strong><br>
PyTorch 2.0</p>
<p><strong>Code</strong></p>
<pre><code class="lang-py">import struct
import torch

# define kernel
weight_scale = 0x3bb9211d # IEEE754 representation, 0.005649699363857508
weight_scale = struct.unpack('f', struct.pack('I', weight_scale))[0]
weight_int8 = torch.tensor([[[[  56,    8, -118,  -29,  127,    4,  -52]]]], dtype=torch.int8)
weight = torch.quantize_per_tensor(weight_int8 * weight_scale, scale=weight_scale, zero_point=0, dtype=torch.qint8)

# define input data
x_scale = 0x3d101824 # IEEE754 representation, 0.035179272294044495
x_zero_point = 129
x_scale = struct.unpack('f', struct.pack('I', x_scale))[0]
x_uint8 = torch.tensor([[[190, 213, 231, 247, 255, 251, 234]]], dtype=torch.uint8)
x = torch.quantize_per_tensor((x_uint8.type(torch.int32) - x_zero_point) * x_scale, scale=x_scale, zero_point=
                              x_zero_point, dtype=torch.quint8)

# define bias
bias = 0xbbc834f6 # IEEE754 representation, -0.006109829060733318
bias = struct.unpack('f', struct.pack('I', bias))[0]
bias = torch.tensor([bias], dtype=torch.float32)

# define output scale and output zero point
out_scale = 0x3c64ad2f # IEEE754 representation, 0.013957305811345577
out_scale = struct.unpack('f', struct.pack('I', out_scale))[0]
out_zero_point = 132

# calculation by PyTorch
result_torch = torch.ao.nn.quantized.functional.conv1d(x, weight, bias, stride=1, padding=0, scale=out_scale,
                                                       zero_point=out_zero_point)

# manual calculation
z = weight_int8.squeeze().type(torch.int32) @ (x_uint8.squeeze().type(torch.int32) - x_zero_point)
z = z + torch.round(bias / (x_scale * weight_scale))
unclamped = torch.round(z * (x_scale * weight_scale / out_scale) + out_zero_point)
result_manual = torch.clamp(unclamped, 0, 255).type(torch.uint8)

print(result_torch.int_repr().squeeze(), result_manual)
</code></pre>
<p><strong>What I expect to see</strong><br>
Both PyTorch and manual calculation give 127</p>
<p><strong>Actual result (PyTorch, manual calculation)</strong><br>
tensor(141, dtype=torch.uint8) tensor([127], dtype=torch.uint8)</p>