akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CTC Loss that continues to produce inf values

username: girlsending0
cooked: <p>Hello, I am working on a task that involves speech recognition using CTC Loss.<br>
It’s no different, I use pytorch’s CTC Loss, but the CTC Loss value continues to be derived only as inf.<br>
I tried setting the option to zero_infinity=True, but the value is very small and training does not proceed.<br>
What should I do in this case? I’ve tried many different solutions, but I don’t know why it works this way.</p>
<p>I also found an article that says it’s a length related reason, currently the target length is smaller than the input length, most likely the input length is larger.<br>
I really don’t know why.<br>
I would really appreciate your advice.</p>
<p>Below is my code.</p>
<p><code>criterion_CTC = nn.CTCLoss(blank=0, reduction='mean', zero_infinity=False).to(device)</code></p>
<p>and in train phase,</p>
<p><code>CTC_output, LM_output = model(input_embeddings_batch, input_lengths, input_attn_mask_invert_batch, input_attn_mask_batch, target_ids_batch)             CTC_output = F.log_softmax(CTC_output, dim=2)             CTC_output = CTC_output.transpose(0, 1) # [113, 80, 27] # [sequence, batch, cls probability] =&gt; 입력 시퀀스가 목표 시퀀스에 정렬되기에 짧으면, infinite 문제 발생             print('CTC_output', CTC_output.size())# 1351, 80, 29             print('labels', labels.size()) # 80, 394             print('input_lengths', input_lengths) #80              print('label_lengths', label_lengths) #80</code></p>
<p>each component shape and results is shown in below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3de05e9b14723da789bf5fa6af0e76a147e7337.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3de05e9b14723da789bf5fa6af0e76a147e7337" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3de05e9b14723da789bf5fa6af0e76a147e7337.png" alt="image" data-base62-sha1="uegqzn0L78Ht5qN16biOv44KILl" width="690" height="158" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1270×292 31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please comment to me… I will very thank you…</p>
Issues with running model on GPU

username: External_happy
cooked: <p>I am training a ViT-B/16 on a dataset with 4K samples using 24 NVIDIA Quadro RTX 6000 GPUs.<br>
The training was going well until I add a line to make an inference on the same model twice using different inputs at the same iteration. This resulted in the error message “RuntimeError: CUDA out of memory. Tried to allocate 148.00 MiB” appearing every time. I tried to release the GPU cache memory, but that did not help. Can someone help me with this issue?</p>

username: ptrblck
cooked: <p>If you are not disabling the gradient calculation each forward call will store intermediate results which are needed for the gradient calculation during the forward pass. Either call <code>backward()</code> between these forward passes or disable gradient calculation by wrapping the forward calls into a <code>torch.no_grad()</code> context.</p>
Conditional Tensor Shape for PyTorch Linear Layers

username: vivekjoshy
cooked: <p>I have a dataset loaded into a dataloader with the following features.</p>
<pre data-code-wrap="txt"><code class="lang-plaintext">x: (node_features * batch_size, 4)
edge_index: (2, edge_size)
y: (node_features * batch_size, )
z: (node_features * batch_size, )
boolean: (batch_size, )
</code></pre>
<p>I am using a Graph Neural Network to learn an output tensor <code>p</code> of shape <code>(node_features * batch_size)</code>. I have access to all data during training, but not during evaluation, validation or inference time. In that time I only have access to <code>x</code> and <code>edge_index</code>.</p>
<p>How do I learn from all the data in the dataloader during training, but only use <code>x</code> and <code>edge_index</code> for testing?</p>
<p>One might think, simply concatenating the other values to the output before passing it to the linear layer would work, but because the dimensions change during inference time, the linear layer will complain about the un-concatenated data. How do I solve this issue?</p>
Using torch.compile there's a long call without any CUDA activity at the beginning of each iteration

username: polarbart
cooked: <p>I compiled my model with <code>torch.compile</code>. I see significant speedups, though when profiling the network I also see a long call to <code>_compile.&lt;locals&gt;.compile_inner</code> at the beginning of each iteration/call to the model without any activity on the GPU. What is happening there? This profile was taken after 60 iterations, so it’s definitely no initial preparation.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4d05b4c4e1728ab80dbf7885b01288aae38be07.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4d05b4c4e1728ab80dbf7885b01288aae38be07" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4d05b4c4e1728ab80dbf7885b01288aae38be07_2_690x177.png" alt="image" data-base62-sha1="s55Z95yqqd87HB9WOViCLp00gOb" width="690" height="177" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4d05b4c4e1728ab80dbf7885b01288aae38be07_2_690x177.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4d05b4c4e1728ab80dbf7885b01288aae38be07_2_1035x265.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4d05b4c4e1728ab80dbf7885b01288aae38be07_2_1380x354.png 2x" data-dominant-color="E5E3EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1776×456 39.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: bdhirsh
cooked: <p>That’s a good indication that <code>torch.compile</code> is probably recompiling, which is what’s showing up in the trace.</p>
<p>One way to diagnose why you’re still seeing compiles happen after 60 iterations would be to run your code with:</p>
<pre><code class="lang-auto">TORCH_LOGS="recompiles,graph_breaks"
</code></pre>
<p>Which should give you some insight into what is changing run-to-run in each iteration that prevents torch.compile from re-using previous compiled artifacts.</p>

username: polarbart
cooked: <p>Thanks a lot for your help!<br>
Indeed these were recompiles. Removing all graph breaks with the help of <code>explain</code> resolved the issue.</p>
Error importing Camembert model

username: matt58
cooked: <p>Hello ! I try to import the Camembert model available on Pytorch hub but I have an error. I do not know how to solve it.</p>
<p>The code :<br>
import torch<br>
camembert=torch.hub.load(“pytorch/fairseq”,“camembert.v0”)</p>
<p>The error :<br>
Traceback (most recent call last):</p>
<p>Cell In[7], line 1<br>
camembert=torch.hub.load(“pytorch/fairseq”,“camembert.v0”)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\hub.py:566 in load<br>
model = _load_local(repo_or_dir, model, *args, **kwargs)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\hub.py:592 in _load_local<br>
hub_module = _import_module(MODULE_HUBCONF, hubconf_path)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\hub.py:106 in _import_module<br>
spec.loader.exec_module(module)</p>
<p>File :940 in exec_module</p>
<p>File :241 in _call_with_frames_removed</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\hubconf.py:39<br>
from fairseq.hub_utils import (  # noqa; noqa</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\fairseq_<em>init</em>_.py:20<br>
from fairseq.distributed import utils as distributed_utils</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\fairseq\distributed_<em>init</em>_.py:7<br>
from .fully_sharded_data_parallel import (</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\fairseq\distributed\fully_sharded_data_parallel.py:10<br>
from fairseq.dataclass.configs import DistributedTrainingConfig</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\fairseq\dataclass_<em>init</em>_.py:6<br>
from .configs import FairseqDataclass</p>
<p>File ~/.cache\torch\hub\pytorch_fairseq_main\fairseq\dataclass\configs.py:1127<br>
<span class="mention">@dataclass</span></p>
<p>File ~\AppData\Local\anaconda3\Lib\dataclasses.py:1230 in dataclass<br>
return wrap(cls)</p>
<p>File ~\AppData\Local\anaconda3\Lib\dataclasses.py:1220 in wrap<br>
return _process_class(cls, init, repr, eq, order, unsafe_hash,</p>
<p>File ~\AppData\Local\anaconda3\Lib\dataclasses.py:958 in _process_class<br>
cls_fields.append(_get_field(cls, name, type, kw_only))</p>
<p>File ~\AppData\Local\anaconda3\Lib\dataclasses.py:815 in _get_field<br>
raise ValueError(f’mutable default {type(f.default)} for field ’</p>
<p>ValueError: mutable default &lt;class ‘fairseq.dataclass.configs.CommonConfig’&gt; for field common is not allowed: use default_factory</p>
<p>Thanks a lot for your help !<br>
Mattéo</p>
Error in multiple inputs multimodel autoencoder

username: Noorulain_Islam
cooked: <p>I have followed the code for multiple inputs multimodal autoencoder which is written by <a class="mention" href="/u/ptrblck">@ptrblck</a> in this post “<a href="https://discuss.pytorch.org/t/an-autoencoder-with-multiple-inputs/139138" class="inline-onebox">An autoencoder with multiple inputs</a>”. I am getting an error when I write training loop to reconstruct my multiple inputs. <a class="mention" href="/u/ptrblck">@ptrblck</a> , can you please help, how should I write the training loop to reconstruct the multiple inputs which is present in this code:</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.encoders = nn.ModuleList()<br>
for _ in range(9):<br>
self.encoders.append(nn.Linear(4, 3))</p>
</blockquote>
<pre><code>    self.encoder = nn.Sequential(
        nn.Linear(9*3, 4),
        nn.ReLU(),
        nn.Linear(4, 3)
    )
    self.decoder = nn.Sequential(
        nn.Linear(3, 4),
        nn.ReLU(),
        nn.Linear(4, 9*3)
    )
    self.decoders = nn.ModuleList()
    for _ in range(9):
        self.decoders.append(nn.Linear(3, 4))
    
def forward(self, inputs):
    out = []
    for idx, enc in enumerate(self.encoders):
        out.append(enc(inputs[idx]))
    out = torch.cat(out, dim=1)
    print("out", out)
    z = self.encoder(out)
    out = self.decoder(z)
    out = torch.split(out, 3, dim=1)
    outs = []
    for idx, dec in enumerate(self.decoders):
        outs.append(dec(out[idx]))
    return outs
</code></pre>
<blockquote>
<p>model = MyModel()<br>
inputs = [torch.randn(1, 4) for _ in range(9)]<br>
outs = model(inputs)<br>
loss_fn = nn.MSELoss()</p>
</blockquote>
<h1><a name="create-the-optimizer-1" class="anchor" href="#create-the-optimizer-1"></a>Create the optimizer</h1>
<p>optimizer = torch.optim.Adam(params=model.parameters(), lr=0.01)</p>
<p>torch.manual_seed(42)</p>
<p>epochs = 2</p>
<p>train_loss_values = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
test_loss_values = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
epoch_count = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for epoch in range(epochs):<br>
### Training<br>
model.train()<br>
inp = inputs<br>
y_pred = model(inp)<br>
for idx, y_pr in enumerate(y_pred):<br>
loss = loss_fn(y_pr, inp[idx])<br>
<span class="hashtag-raw">#print</span>(loss)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
train_loss.append(loss)</p>
<p>Error<br>
line 173, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
Updating learning rate during training

username: sroyyors
cooked: <p>hi,<br>
I was wondering how I can update the learning rate in the Adam::optim optimizer after a few interactions of training in libtorch? In the python version there is lr_scheduler, but it seems like this is not implemented yet in the libtorch?</p>
<p>Is there a way to set the learning rate manually after each epoch?<br>
Here is how I initialize my optimizer:</p>
<p><code>torch::optim::Adam optimizer(model-&gt;parameters(), torch::optim::AdamOptions(learning_rate).weight_decay(weight_decay))</code></p>
torchrec - [WinError 193] %1 is not a valid Win32 application

username: jamesmachine
cooked: <p>Hi, I used pip install torchrec, which was successful. Yet, when I tried to import torchrec, this error showed up.</p>
<p>“[WinError 193] %1 is not a valid Win32 application”</p>
<p>This error only happened in my vscode. (My laptop is system type is: “64-bit operating system, x64-based processor” with Nvidia GeForce MX450) Running the tutorial code in colab was not a problem that everything ran smoothly.</p>
<p>Is there any problem? Any solutions? Any Suggestions?</p>

username: rodrigobraz
cooked: <p>Have you ever solved that problem? I am facing the same.</p>
<p>Someone <a href="https://github.com/pytorch/pytorch/issues/41585" rel="noopener nofollow ugc">here</a> mentioned setting up Python 64-bit solved the problem, but that did not work for me.</p>
Torchvision.io.read_image() distorting image data

username: Alex_Alessi
cooked: <p>I am building a custom Dataset object to format and iterate though my data set for a img-seq project I am working on. I am using the <code>torchvision.io.read_image()</code> function to load in a given image as a tensor.</p>
<p>I am using the following code to validate the functionality of <code>read_image()</code> and I am seeing some odd behavior.</p>
<pre><code class="lang-auto">img2 = read_image(tmp_img_dir)
img2 = T.ToPILImage()(img2)
img2.show()
</code></pre>
<p>where <code>tmp_img_dir</code> is simply an absolute path to a .png image.</p>
<p>I will include the original image and the one displayed by .show()</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a812cbfd2a479476603f6bea89fda2b311cd4fd4.png" alt="Screenshot from 2023-11-12 14-46-31" data-base62-sha1="nYQucgrC7OWMnamCbMdatcGCygY" width="334" height="152"></p>
<p>As you can see the image appears to be compressed leftwards. This does not occur when reading in this image using any other python library such as PIL or matplotlib.image.</p>
<p>Is this error a result of the way in which i am displaying the image or my use of the read_image() function?</p>
<p>If anyone has seen this before please do tell!</p>

username: Alex_Alessi
cooked: <p>(Apparently since I am a new user I cannot include more than 1 embedded image) soooo here is the original undistorted image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f2600cf91a7bc9325032389e19c837a07578dfa.png" alt="110002955-1_57_1" data-base62-sha1="2a0AaQxjLskq37TJ9mp9i6Deev0" width="322" height="120"></p>
<p>The original and distorted images have the same dimension</p>
gradient of p2 norm does not agree with analytic solution

username: zheng_xiong
cooked: <p>I want to scale a vector by its p2 norm, i.e., <code>y = x / norm(x, p=2)</code>. The analytic gradient of this operation w.r.t. each element in the vector is <code>1 / ||x||_2 - x_i^2 / ||x||_2^3</code>. However, I found that the gradient computed by pytorch is different from this analytic solution. Not sure if I made some mistakes here or pytorch does not compute it correctly?<br>
Here is a minimal example to reproduce this inconsistent issue:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

# Function to compute the gradient of vector normalization
def gradient_vector_norm(x):
    # Convert input to PyTorch tensor if not already
    x = torch.tensor(x, requires_grad=True, dtype=torch.float)

    # Compute the normalized vector
    norm_x = x.norm(p=2)
    y = torch.div(x, norm_x).sum()
    y.backward()

    return x.grad, norm_x

# Test the function
vector = [3.0, 4.0]
gradient, norm_x = gradient_vector_norm(vector)

print("Gradient w.r.t. each element:", gradient)
analytic_gradient = 1. / norm_x - torch.tensor(vector) ** 2 / (norm_x ** 3)
print (analytic_gradient)
</code></pre>
<p>Pytorch give a gradient of [ 0.0320, -0.0240], while the analytic solution is [0.1280, 0.0720], which are inconsistent.</p>
Can I launch training from another node?

username: coincheung
cooked: <p>Hi,</p>
<p>Due to some reasons, I need to launch training from another node. I hope it can work like this:<br>
In node1:</p>
<pre><code class="lang-auto">torchrun --master_addr ip_of_node3 --master_port port_of_node3 train.py
</code></pre>
<p>In node2:</p>
<pre><code class="lang-auto">torchrun --master_addr ip_of_node3 --master_port port_of_node3 train.py
</code></pre>
<p>In node3:</p>
<pre><code class="lang-auto">torchrun_or_other_script start.py
</code></pre>
<p>It means that, only node1 and node2 has gpus and training only happens on node1 and node2, and node3 is only used to start training on node1 and node2.</p>
<p>I do this because I do not know the actual ip address of node1 and node2, but I can access node3 from both node1 and node2. Can I do this with pytorch ?</p>
How does backward propagation work in a nested neural network

username: Jazzy66
cooked: <p>Hi, everyone. I am working on a dynamic programing problem in which a nested neural network is implemented. For the sake of better understanding my question, simply assume the loss is</p>
<p>L = G(k’) - H(k’')</p>
<p>where G and H are two functions we do not need to know but variable k’ and k’’ stem from neural network NNs such that</p>
<p>k’ = NNs(k), k’’ = NNs(k’)</p>
<p>so k is the original input. Apparently, to obtain output k’’ the same neural network gets nested once:</p>
<p>k → NNs → k′ → NNs → k′′</p>
<p>My question is simple:</p>
<p>(1) how does backward propagation work in this nested neural network setting (one backpropagation for each transition I guess)? does it work the same as the regular one?<br>
(2) could this nested NNs setup slow down the computation?<br>
(3) what could go wrong under this nested NNs setup?</p>

username: ptrblck
cooked: <ol>
<li>Autograd will create a computation graph during the forward pass and will use it to backpropagate through it during the <code>backward</code> call.</li>
<li>Yes, increasing the computation graph adds more operations to the forward and backward passes and increases the memory usage.</li>
<li>Depending on how the entire recursion is used you might run into issues where stale forward activations are used during a backward pass. However, if PyTorch doesn’t raise any errors your code should be fine.</li>
</ol>

username: Jazzy66
cooked: <p>Thanks for the heroic answer. My code produce NaN loss when I increase the dimensionality of NNs output, lol. I have one more question, if I may ask, what is the potential challenge except for the issues concerning stale forward activations.</p>
Torch_geometric , HeteroData , GNN, multi-dimensional features with varying dimensions

username: SKTheLearner
cooked: <p>Hi<br>
I am experimenting with torch_geometric and HeteroData et al…</p>
<p><em>following specifically MovieLens link regression example…</em></p>
<p>In the example ‘user_features’ are one-dimensional…ie, each user is represented by an one-dimensional vector…and HeteroData is created and all fine…</p>
<p>My question is WHAT IF my user_features are multi and different dimensional…ie, each user is represented by a matrix and these matrices have different dimensions…</p>
<p>eg: I have a list of ‘Data’ objects as following…</p>
<p>tmp_users = [Data(x=[28, 36]),<br>
Data(x=[22, 36]),<br>
Data(x=[32, 36]),<br>
Data(x=[30, 36]),<br>
Data(x=[35, 36]<br>
]</p>
<p>How do I use this ‘tmp_users’ to create the  ‘user’ Nodes/features as in the MovieLens example…<br>
ie,</p>
<p>data = HeteroData()</p>
<p><strong>data[‘user’].x = ???</strong></p>
<p>data[‘movies’].x=…</p>
<p>Help much appreciated…</p>
<p>Regards</p>
SwinTransformer issue by pytorch pretrain models

username: februarysea
cooked: <p>I am currently experimenting with pretrained SwinTransformer models from torchvision and check my model structure by torchinfo. The code is as follows:</p>
<pre><code class="lang-auto">from torchvision.models import swin_t, Swin_T_Weights
import torch.nn as nn
from torchinfo import summary

model = swin_t(weights=Swin_T_Weights.DEFAULT)
features = nn.Sequential(*list(model.children())[:-1])
print(summary(model, input_size=(4, 3, 224, 224)))
print(summary(features, input_size=(4, 3, 224, 224))).  

</code></pre>
<p>The output from torchinfo is a bit strange. These are the results for the last few layers. I believe the output shape of AdaptiveAvgPool2d in the feature module should also be [4, 768, 1, 1], but instead, it is [4, 7, 1, 1]. Can you tell me what’s wrong with my code?</p>
<pre><code class="lang-auto"># model
├─LayerNorm: 1-2                                   [4, 7, 7, 768]            1,536
├─AdaptiveAvgPool2d: 1-3                           [4, 768, 1, 1]            --
├─Linear: 1-4                                      [4, 1000]                 769,000

# features
├─LayerNorm: 1-2                                   [4, 7, 7, 768]            1,536
├─AdaptiveAvgPool2d: 1-3                           [4, 7, 1, 1]              --

</code></pre>
Zero CTC Loss Unchanged by Parameter Tweaks

username: pidgeotto
cooked: <p>I’ve hit a wall with an issue that’s as intriguing as it is frustrating, and I’m hoping to tap into the collective wisdom of this community to find a resolution.</p>
<p>I’m working on a CRNN model for an OCR task, and no matter what parameters I adjust, my CTC loss remains stubbornly at zero. This has turned into a real head-scratcher, and I’m looking for fresh perspectives on what might be going awry.</p>
<ul>
<li>My model’s architecture seems solid at a glance, but during training, the loss doesn’t reflect any learning or improvement.</li>
<li>I’ve tried adjusting learning rates, experimenting with different optimizers, and even tweaking the loss function’s parameters – all to no avail.</li>
</ul>
<p><strong>Here’s a glimpse of my setup:</strong></p>
<ul>
<li><strong>Model</strong>: Custom CRNN designed for OCR.</li>
<li><strong>Data</strong>: Image tensors with corresponding text labels, verified to be correctly formatted.</li>
<li><strong>Loss Function</strong>: <code>nn.CTCLoss</code>, with <code>blank=0</code> and <code>zero_infinity=True</code> to avoid NaN issues.</li>
<li><strong>Optimizer</strong>: <code>optim.Adam</code> with a learning rate that I’ve varied from 0.001 down to 0.0001.</li>
</ul>
<p><strong>What I’ve attempted so far:</strong></p>
<ul>
<li>Ensuring the data loader is correctly shuffling and batching images along with their respective labels.</li>
<li>Double-checking the image preprocessing steps and label encoding.</li>
<li>Monitoring gradients and weights for any signs of life.</li>
</ul>
<p>Has anyone encountered a similar issue? Could there be an overlooked aspect of the CRNN architecture or the CTC loss function that’s causing this? Any insights, suggestions, or debugging tips would be immensely appreciated.</p>
<p>I’ve attached snippets of my code below for reference and am open to any questions that might help clarify the issue.</p>
<p>Thank you for your time, and I look forward to any advice you can share!</p>
<pre><code class="lang-auto">
import torch.nn as nn

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

images = torch.randn(4,1,300,192).to(device)

output = crnn_model(images)

labels = torch.randn(4,17932)

# print(output)

input_lengths = torch.full(size=(images.size(0),), fill_value=1, dtype=torch.long).to(device)

target_lengths = torch.tensor([17932,17932,17932,17932], dtype=torch.long).to(device)

criterion = nn.CTCLoss(blank=0,zero_infinity=True).to(device)

loss = criterion(output, labels, input_lengths, target_lengths)

print(loss)
</code></pre>
<p>Even this dummy data gives 0 loss.</p>
<p>For reference, 17932 is the buffered size of the bounding boxes of the labels in my data set.</p>
<p>I have transcription data (The contents of what is labeled) that I noticed isn’t getting used, and have tried using it instead, which also results in 0 loss.</p>
Torch.tensor precision not same as numpy.array

username: bradbell
cooked: <p>In the torch tensor case below, why do s * pi and t * pi have different precision ?</p>
<pre><code class="lang-auto">#! /usr/bin/env python3
r'''
Expected Results:
   s * pi - t * pi as numpy arrays  =  [0.]
   s * pi - t * pi as torch tesors =  tensor([0.], dtype=torch.float64)

Actual Results:
   s * pi - t * pi as numpy arrays  =  [0.]
   s * pi - t * pi as torch tesors =  tensor([-8.7423e-08], dtype=torch.float64)
'''
import torch
import numpy
#
# pi
pi = numpy.pi
#
# s, t
s  = numpy.ones(1)
t  = numpy.array( [1.0] )
print( 's * pi - t * pi as numpy arrays = ', s * pi - t * pi )
#
# s, t
s  = torch.tensor( numpy.ones(1) )
t  = torch.tensor( [1.0] )
print( 's * pi - t * pi as torch tesors = ', s * pi - t * pi )
</code></pre>

username: ptrblck
cooked: <p>By default PyTorch uses <code>float32</code> for performance reasons so the small difference is expected.<br>
Use <code>t  = torch.tensor( [1.0], dtype=torch.float64 )</code> to initialize <code>t</code> as a <code>DoubleTensor</code> and the result should show <code>0.</code>.</p>

username: bradbell
cooked: <p>Thanks for the clarification. I did not expect that a torch.tensor would act differently than numpy in this regard.</p>
Model Runs Slowly Unless I Restart Kernel

username: tempaccountforissue
cooked: <p>I’m pretty new to PyTorch, so please excuse my lack of knowledge on this subject. I’m running a PyTorch model inside of a Jupyter kernel. For some reason, if I try to run the model a second time, it is extremely slow. However, if I restart the kernel, it is able to run quickly again. I was wondering why this is? Also, is there a way to get the model to run quickly without having to restart the kernel after every run?</p>

username: ptrblck
cooked: <p>I’ve never seen such a behavior and would recommend profiling the workload e.g. with the native PyTorch profiler or Nsight Systems to check where the slowdown is coming from.</p>

username: tempaccountforissue
cooked: <p>Thanks for the suggestion. I took a look at the PyTorch profiler, but I have no idea how to get it working with the model I’m using. I’m working with <a href="https://github.com/openai/shap-e" rel="noopener nofollow ugc">OpenAI’s Shap-E</a>. Here is the code I’m using to run the model:</p>
<pre><code class="lang-auto">import torch

from shap_e.diffusion.sample import sample_latents
from shap_e.diffusion.gaussian_diffusion import diffusion_from_config
from shap_e.models.download import load_model, load_config
from shap_e.util.notebooks import create_pan_cameras, decode_latent_images, gif_widget

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

xm = load_model('transmitter', device=device)
model = load_model('text300M', device=device)
diffusion = diffusion_from_config(load_config('diffusion'))

batch_size = 4
guidance_scale = 15.0
prompt = "a shark"

latents = sample_latents(
    batch_size=batch_size,
    model=model,
    diffusion=diffusion,
    guidance_scale=guidance_scale,
    model_kwargs=dict(texts=[prompt] * batch_size),
    progress=True,
    clip_denoised=True,
    use_fp16=True,
    use_karras=True,
    karras_steps=64,
    sigma_min=1e-3,
    sigma_max=160,
    s_churn=0,
)
</code></pre>
<p><code>latents = sample_latents(</code> is the line that is executing extremely slowly.</p>
<p>Would you mind helping me profile this?</p>

username: corunethron
cooked: <p>Try to separate all the lines before batch_size and do not run them more than once. I assume that heavy models are loaded into memory second time and sampling starts before garbage collector (if there is one in python) deletes previously loaded models.</p>
Inference time issue

username: cbd
cooked: <p>By running the same code on cloud giving difference in inference time of 10 seconds for 1000 images. What may be the reason?</p>

username: ptrblck
cooked: <p>Everything could be related, starting from the hardware, driver, math libs, PyTorch release, to your actual code.<br>
Profile your workload with the native PyTorch profiler or e.g. Nsight Systems and compare both profiles from theses systems.</p>
Faster RCNN focuses on wrong areas to make predictions

username: Manu_Johny
cooked: <p>Hello everyone,</p>
<p>I trained a FasterRCNN model on 400 annotated hip xrays. I want that the model detects the areas around the hip joints. After training more than 15 epochs, I am getting very good losses and IoU for both training and validation datasets. But when I use GradCam to look how the model predicts the bounding boxes, I can see that the model considers Background areas of the xrays to make predictions. Examples are the black background on the side of the hip etc. Why does this happen and what can I do such that the model makes predictions based on the hip joint areas?</p>
<p>I would appreciate any help!</p>

username: ptrblck
cooked: <p>The model will use any information to lower the loss. Did you check if these background areas are somehow specific to one class? E.g. in the past publications reported initially very good results also using medical data until it was realized d.g. the “unhealthy” class images all contained a medical device introduced after the diagnosis and treatment, which the model obviously learned.</p>
How to Monitor and Optimize GPU and CPU Resource Usage in PyTorch?

username: Mengbo_Zhou
cooked: <p>Hello everyone,</p>
<p>I have been training and fine-tuning large language models using PyTorch recently. During this process, I am looking to better understand and monitor the inter-GPU communication, the transfer of parameters and operators, as well as the usage of GPU memory and CPU memory.</p>
<p>Specifically, I am facing the following challenges:</p>
<ol>
<li>How can I monitor the communication and data transfer efficiency in a multi-GPU environment?</li>
<li>What are the best practices to accurately measure and optimize the memory usage of the GPU in my model?</li>
<li>Are there tools or techniques that can help me better monitor the usage of CPU memory during training?</li>
</ol>
<p>I have tried using nvidia-smi to observe GPU usage, but I find the information provided somewhat limited. I am looking for more detailed analysis, especially in a distributed training context.</p>

username: ptrblck
cooked: <ol>
<li><a href="https://developer.nvidia.com/dcgm">NVIDIA DCGM</a> can be used to monitor clusters.</li>
<li>You could use a profiler, such as Nsight Systems, to profile your code and to check for unexpected memory increases. PyTorch also provides native profiling tools or you could even log the memory usage manually in your code.</li>
</ol>
How to forward pass?

username: Manaswi_Mancha
cooked: <p>I am trying to train the LEDForConditionalGeneration. The example in the documentation uses model.generate. How do I use forward pass so I can get loss values and do backprop? The code is</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, LEDForConditionalGeneration

model = LEDForConditionalGeneration.from_pretrained("allenai/led-large-16384-arxiv")
tokenizer = AutoTokenizer.from_pretrained("allenai/led-large-16384-arxiv")

ARTICLE_TO_SUMMARIZE = '''Transformers (Vaswani et al., 2017) have achieved state-of-the-art
    results in a wide range of natural language tasks including generative language modeling
    (Dai et al., 2019; Radford et al., 2019) and discriminative ... language understanding (Devlin et al., 2019).
    This success is partly due to the self-attention component which enables the network to capture contextual
    information from the entire sequence. While powerful, the memory and computational requirements of
    self-attention grow quadratically with sequence length, making it infeasible (or very expensive) to
    process long sequences. To address this limitation, we present Longformer, a modified Transformer
    architecture with a self-attention operation that scales linearly with the sequence length, making it
    versatile for processing long documents (Fig 1). This is an advantage for natural language tasks such as
    long document classification, question answering (QA), and coreference resolution, where existing approaches
    partition or shorten the long context into smaller sequences that fall within the typical 512 token limit
    of BERT-style pretrained models. Such partitioning could potentially result in loss of important
    cross-partition information, and to mitigate this problem, existing methods often rely on complex
    architectures to address such interactions. On the other hand, our proposed Longformer is able to build
    contextual representations of the entire context using multiple layers of attention, reducing the need for
    task-specific architectures.'''
inputs = tokenizer.encode(ARTICLE_TO_SUMMARIZE, return_tensors="pt")

# Global attention on the first token (cf. Beltagy et al. 2020)
global_attention_mask = torch.zeros_like(inputs)
global_attention_mask[:, 0] = 1

# Generate Summary
summary_ids = model.generate(inputs, global_attention_mask=global_attention_mask, num_beams=3, max_length=32)
print(tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
</code></pre>

username: ptrblck
cooked: <p>To perform the forward pass on your model you can directly call the model with inputs as: <code>out = model(input)</code> if it’s a standard PyTorch model. Higher level APIs might be using their own abstractions and you would need to check their docs or implementation in code.</p>
Custom data and dataloader object

username: Noorulain_Islam
cooked: <p>my dataset is of this form, I want to create data and dataloader in pytorch on my final_train_data, final_test_data, final_train_label, final_test_label. I want to divide my data into 2 batches. Can you help me?<br>
train_data = t.randn(12, 12)<br>
test_data = t.randn(6,6)<br>
train_label = t.randn(12, 12)<br>
test_label = t.randn(6, 6)<br>
train_data1 = t.randn(12, 12)<br>
test_data1 = t.randn(6,6)<br>
train_label1 = t.randn(12, 12)<br>
test_label1 = t.randn(6, 6)</p>
<p>final_train_data = [train_data, train_data1]<br>
final_test_data = [test_data, test_data1]<br>
final_train_label = [train_label, train_label1]<br>
final_test_label = [test_label, test_label1]</p>
E/log: error java.lang.ClassNotFoundException: Didn't find class "org.pytorch.PyTorchAndroid"

username: Taras_Khalep
cooked: <p>Hello Pytorch’s community. I looked at the existing topics on this issue, but did not find a suitable solution for myself.</p>
<p>The problem is: I getting</p>
<blockquote>
<p>E/log: error java.lang.ClassNotFoundException: Didn’t find class “org.pytorch.PyTorchAndroid” on path: DexPathList[[zip file “/data/app/~~5hf8jgRJFfzqOKLB8ckRxQ==/com.example.ndk_camera_jni-gl6E_Jg9_yyZyXm5mIbJ8A==/base.apk”],nativeLibraryDirectories=[/data/app/~~5hf8jgRJFfzqOKLB8ckRxQ==/com.example.ndk_camera_jni-gl6E_Jg9_yyZyXm5mIbJ8A==/lib/arm64, /data/app/~~5hf8jgRJFfzqOKLB8ckRxQ==/com.example.ndk_camera_jni-gl6E_Jg9_yyZyXm5mIbJ8A==/base.apk!/lib/arm64-v8a, /system/lib64, /system_ext/lib64]]<br>
A/.ndk_camera_jn: java_vm_ext.cc:577] JNI DETECTED ERROR IN APPLICATION: JNI NewGlobalRef called with pending exception java.lang.ClassNotFoundException: Didn’t find class “org.pytorch.PyTorchAndroid”</p>
</blockquote>
<p>I have tried different version: 1.12, 1.9, 1.10. Also, I have tried ‘org.pytorch:pytorch_android_lite:1.9.0’ (lite version), but all time getting the same error.</p>
<p>Just one point need to admit, that I have entry point on NDK layer. It means, that my application started execution from c++ jni. After that it use some method from java layer.</p>
<p>Does somebody face this problem? Maybe someone has a suggestion where should I to find solution</p>
<p>Best regards, Taras</p>

username: jakjak
cooked: <p>Hi Taras,<br>
I got the same issue at runtime when using native C++ Pytorch API on android and was able to fix it.<br>
I was missing the implementation of pytorch in the app/build.gradle in dependencies,</p>
<p>You need<br>
dependencies {<br>
implementation ‘org.pytorch:pytorch_android_lite:1.10.0’<br>
extractForNativeBuild ‘org.pytorch:pytorch_android_lite:1.10.0’<br>
…<br>
}</p>
<p>Hope it helps,</p>
<p>Best.</p>

username: chan_z
cooked: <p>try<br>
target_link_libraries(test fbjni pytorch_jni)<br>
first fbjni and than pytorch_jni<br>
not<br>
target_link_libraries(test pytorch_jni fbjni)</p>
Number of parameters and FLOPs in network

username: cbd
cooked: <p>There are two model. One model has number of parameters 1 M and second has number of parameters 3 M but both have the same FLOPs value. What will be the advantage of model having less number of parameters?</p>

username: ptrblck
cooked: <p>Size would be one advantage.</p>

username: cbd
cooked: <p>Can you bit explain how size would be the advantage? Is it require less hardware to load model or require less run time memory or faster than model?</p>

username: J_Johnson
cooked: <p>For instance, say you want to run a model inference on a Raspberry Pi. Suppose you’ve got 4MB of RAM. Part of that will be for overhead. At half precision, you could run the 1MB model, which will use &gt;2MB just for the weights.</p>
Default argument not captured in FX graph

username: Abel_Nieto
cooked: <p>When I <code>torch.export</code> the sample module in the official tutorial:</p>
<pre><code class="lang-auto">import torch
from torch.export import export

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = torch.nn.Linear(100, 10)

    def forward(self, x, y):
        return torch.nn.functional.relu(self.lin(x + y), inplace=True)

mod = MyModule()
exported_mod = export(mod, (torch.randn(8, 100), torch.randn(8, 100)))
print(type(exported_mod))
print(exported_mod(torch.randn(8, 100), torch.randn(8, 100)))
</code></pre>
<p>The resulting graph doesn’t capture the default value of the <code>alpha</code> parameter for the sum <code>x + y</code>:</p>
<pre><code class="lang-auto">    %add : [num_users=1] = call_function[target=torch.ops.aten.add.Tensor](args = (%arg2_1, %arg3_1), kwargs = {})
</code></pre>
<p>By contrast if I explicitly pass in alpha</p>
<pre><code class="lang-auto">        return torch.nn.functional.relu(self.lin(torch.add(x, y, alpha = 0.5)), inplace=True)

</code></pre>
<p>Then the graph correctly contains the third argument</p>
<pre><code class="lang-auto">    %add : [num_users=1] = call_function[target=torch.ops.aten.add.Tensor](args = (%arg2_1, %arg3_1), kwargs = {alpha: 0.5})
</code></pre>
<p>Not including the default value of the argument (when the argument is left unspecified by the caller) is a problem for backends that want to work with the FX graph, because it means that the backend has to reach out of the FX graph to find out the default value.</p>
<p>This is what e.g. the shark-turbine importer does in <a href="https://github.com/nod-ai/SHARK-Turbine/blob/main/python/shark_turbine/dynamo/importer.py#L609" rel="noopener nofollow ugc">https://github.com/nod-ai/SHARK-Turbine/blob/main/python/shark_turbine/dynamo/importer.py#L609</a></p>
<p>Instead of iterating over e.g. the arguments of a <code>call_function</code> node, they get a <code>FunctionSchema</code> from the node (a protected field) and iterate over that.</p>
<p>This seems like a bug in <code>torch.export</code> since it breaks encapsulation. Thoughts?</p>
QARAC: Question Answeting, Reasoning and Consistency

username: PeterBleackley
cooked: <p>Hello, I’d like to introduce myself and a project I’ve been working on.</p>
<p>I’m a self-employed data scientist, with a particular interest in NLP. I’m interested in trying to make NLP models more factually reliable, so I’ve started a project to investigate some ideas I’ve had on the subject. It’s called <a href="https://playfultechnology.co.uk/qarac-question-answering-reasoning-and-consistency.html" rel="noopener nofollow ugc">QARAC</a>, which stands for <em>Question Answering, Reasoning and Consistency</em>, which are the training objectives of my models.</p>
<p>I’m using RoBERTa as my base model, and was originally using Keras to implement it. However, my training system requires training 3 models together on 4 objectives, so I’ve <a href="https://playfultechnology.co.uk/qarac-porting-to-pytorch.html" rel="noopener nofollow ugc">ported it to PyTorch</a>, which seems more suitable for that.</p>
Adding layers to text generation model (Transfer Learning)

username: Ryan_Maccarty
cooked: <p>Hello everyone,</p>
<p>I am currently working on implementing transfer learning to address catastrophic forgetting in my text generation model. The pre-trained model is based on GPT2 and initially trained on financial data (referred to as dataset A). Now, I aim to fine-tune it on a different dataset, dataset B. However, I have encountered an issue: despite training for 2 hours, there is no improvement, and the loss remains constant.</p>
<p>I suspect that either my custom model is not constructed correctly, or there might be an issue with my training loop. I would appreciate any insights or suggestions you may have</p>
<pre data-code-wrap="Python"><code class="lang-plaintext">class CustomModel(torch.nn.Module):
  def __init__(self, pretrained_model, config):
    super(CustomModel, self).__init__()
    self.transformer = pretrained_model
    self.config = config

    self.ffn1 = torch.nn.Sequential(
        torch.nn.Linear(self.config.vocab_size, self.config.n_embd),
        torch.nn.GELU(),
        torch.nn.Linear(self.config.n_embd, self.config.n_embd)
    )
    self.layer_norm1 = torch.nn.LayerNorm(self.config.n_embd)

    self.ffn2 = torch.nn.Sequential(
        torch.nn.Linear(self.config.n_embd, 2*self.config.n_embd),
        torch.nn.GELU(),
        torch.nn.Linear(2*self.config.n_embd, self.config.n_embd)
    )
    self.layer_norm2 = torch.nn.LayerNorm(self.config.n_embd)

    self.Linear = torch.nn.Linear(self.config.n_embd, self.config.vocab_size)

  def forward(self, input_ids, attention_mask=None, token_type_ids=None):
    outputs = self.transformer(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)

    hidden_states = self.ffn1(outputs.logits)
    hidden_states = self.layer_norm1(hidden_states)

    hidden_states = self.ffn2(hidden_states)
    hidden_states = self.layer_norm2(hidden_states)

    logits = self.Linear(hidden_states)

    return logits
</code></pre>
<p>The training loop:</p>
<pre data-code-wrap="Python"><code class="lang-plaintext">epochs=3
step = 0
model.train()
total_loss = 0.

for epoch in range(epochs):
  train_loader = tqdm(train_loader, total=len(train_loader))

  # I also applied gradual unfreezing
  if epoch &lt; 2:
    for param in model.transformer.parameters():
      param.requires_grad = False
  else:
    for param in model.transformer.parameters():
      param.requires_grad = True

  for batch in train_loader:
    input_ids, attention_mask, token_type_ids, targets = batch

    input_ids = input_ids.to(device)
    targets = targets.to(device)

    optimizer.zero_grad()
    outputs = model(input_ids)
    loss = criterion(outputs.view(-1, config.vocab_size), targets.view(-1))
    loss.backward()
    optimizer.step()
    scheduler.step()
    total_loss += loss.item()
    train_loader.set_description(f"Epoch {epoch+1}")
    train_loader.set_postfix(loss=loss.item())
</code></pre>
Different results with same input but with different evaluation order

username: YuryS
cooked: <p>I have trained a simple network for sentence classification on 4 classes using<br>
from pytorch_pretrained_bert.modeling import BertForSequenceClassification.</p>
<p>Then I evaluate it on 2 sequences (sentences) placing to DataLoader</p>
<ol>
<li>only second sentence, and</li>
<li>both sentences.</li>
</ol>
<p>In the first case the result for the second sentence is<br>
tensor([[-0.3797,  4.1902, -3.0362, -0.9368]])<br>
for the second case the result for the same second sentence is<br>
tensor([[-0.0066, -2.3150,  3.2263, -0.3096]])</p>
<p>The snippet of my code is the following:</p>
<p>def evaluate(logger, model, device, eval_dataloader, eval_label_ids, num_labels, verbose=True):<br>
model.eval()<br>
for input_ids, input_mask, segment_ids, label_ids in eval_dataloader:<br>
input_ids = input_ids.to(device)<br>
input_mask = input_mask.to(device)<br>
segment_ids = segment_ids.to(device)<br>
label_ids = label_ids.to(device)</p>
<pre><code>    with torch.no_grad():
        logits = model(input_ids, segment_ids, input_mask, labels=None)
        print ( str(logits) )
</code></pre>
<p>The results are independent from the used device - cpu or gpu.<br>
The eval_batch_size = 1 in both cases.<br>
If I place more sentences to DataLoader then then the results again are varied.<br>
What could be the cause of such behaviour of the model<br>
and how to fix this problem ?</p>
ValueError: Height and Width of image, mask or masks should be equal.disable shapes check by setting is_check_shapes=False of Compose class

username: sccomp
cooked: <p>I would like to mention that the shape of image before transformation is → <strong>Original image shape: (360, 640, 3) Original mask shape: (360, 640) transformed image shape: (320, 320, 3) transformed mask shape: (320, 320)</strong></p>
<p>and <strong>img_trans is defined as → img_trans = A.Compose([A.Resize(a_config.INPUT_IMAGE_HEIGHT, a_config.INPUT_IMAGE_WIDTH)],is_check_shapes=False)</strong></p>
<p>The code snippet is-&gt;</p>
<pre><code class="lang-auto">def image_mask_transformation(image, mask, img_trans, aug_trans=None, normalize=True,
                              var_min=10, var_max=400):

    
    
    print("Original image shape:", image.shape)
    transformed = img_trans(image=image, mask=mask)
    image = transformed["image"]
    mask = transformed["mask"]
    print("transformed image shape:", image.shape)
    # image, mask still uint 8 in 0, 255 range
</code></pre>
<p>But I am getting an error → </p>
<p><strong>ValueError: Caught ValueError in DataLoader worker process 0.</strong></p>
<p><strong>ValueError: Height and Width of image, mask or masks should be equal. You can disable shapes check by setting a parameter is_check_shapes=False of Compose class (do it only if you are sure about your data consistency).</strong></p>
<p>I am not sure whether I am getting the error because initially a 3rd dimension is there for image and not for the mask. Hence I have tried to add the <strong>is_check_shapes=False</strong> while defining img_trans but I am still getting the error.</p>
<p>Can you suggest whether and where I should add <strong>is_check_shapes=False</strong> or the error can be solved in any other way?</p>