akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convergence issue with PyTorch distributed

davidm: <p>Hi,<br>
I’m trying to do distributed training using torch.distributed package while hosting different parts of the model on different nodes. These nodes are initialized independently, and aggregated outputs of the first part of the model are used as inputs for the second model. The backpropagation is done in the same way. So firstly, the second model calculates gradients w.r.t. inputs and these are sent back to the first part of the model on multiple clients that do the backward pass.</p>
<p>The code looks approximately like this:</p>
<ul>
<li>Init top:</li>
</ul>
<pre><code class="lang-auto">model_top = nn.Sequential([...])
opt_top = Adam()
loss_top = nn.BCELoss()
</code></pre>
<ul>
<li>Init bottom:</li>
</ul>
<pre><code class="lang-auto">model_bottom = nn.Sequential([...])
opt_bottom = Adam()
</code></pre>
<p>Training loop:</p>
<ul>
<li>Bottom:</li>
</ul>
<pre><code class="lang-auto">output = model_bottom(sample)
dist.send(output, 0, global_group)
gradient = torch.zeros_like(output).to(device)
dist.recv(gradient, 0, global_group)
output.backward(gradient)
self.optimizer_.step()
</code></pre>
<ul>
<li>Top:</li>
</ul>
<pre><code class="lang-auto">outputs = []
for j in range(1, global_size):
    output = torch.zeros((len(batch), 1000))
    dist.recv(output, j, global_group)
    outputs.append(output)

outputs = torch.autograd.Variable(
    torch.stack(outputs), requires_grad=True
)
outputs = outputs.to(torch.float32).mean(dim=0)

predictions = top_model(outputs)
loss = loss_top(predictions, data)
loss.backward(retain_graph=True)
if outputs.grad is not None:
    gradients = outputs.grad
else:
    logger.warning("Gradient not computed in backward pass!")
    gradients = torch.autograd.grad(loss, outputs)[0]
opt_top.step()

for j in range(1, global_size):
    dist.send(gradients[j - 1], j, global_group)
</code></pre>
<p>The questions here are:</p>
<ul>
<li>Is this correct approach towards custom distributed training?</li>
<li>Is there a reason why the model should not converge? (It converges when running on single machine)</li>
<li>Is there a better way how to write this in PyTorch?</li>
</ul>

fduwjj: <aside class="quote no-group" data-username="davidm" data-post="1" data-topic="170403">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/71e660/48.png" class="avatar"> davidm:</div>
<blockquote>
<ul>
<li>rrect approach towards custom distributed training?</li>
<li>Is there a reason why the model should not converge? (It converges when running on single machine)</li>
<li>Is there a</li>
</ul>
</blockquote>
</aside>
<p>Thanks for your question. What you have described sounds like pipeline parallelism. Have you tried our solution: PiPPy? (Ref: <a href="https://github.com/pytorch/tau/blob/main/README.md" rel="noopener nofollow ugc">https://github.com/pytorch/tau/blob/main/README.md</a>) We do the split automatically for you. Also what’s your model in your use case?</p>
<p>cc: <a class="mention" href="/u/kwen2501">@kwen2501</a></p>
Training on GPU is very slow

aquorio15: <p>Hi i am traning a massive computer ision model with almost 130M parameteres. But the issue is my training is very slow even on multi-gpu setup. Although this may not be the right comparison, but while using fairseq  for Machine Translation, i found the training to be very fast. Is there any reson to that, am i missing any additional packages that i need to install other than Cudann.</p>
<p>Thank you</p>

ptrblck: <p>You would have to profile the workload to understand where the bottleneck in your overall training pipeline is in order to optimizer it further. You could use the native PyTorch profiler or e.g. Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a>.</p>
Distributed processing

Andrea85: <p>Hello, I am trying to train a machine learning model in python by using the package PyTorch.<br>
By now, I have used a single machine with one GPU, but it is not enough to train the model in reasonable time.<br>
I have got acces to myriad, so that I can try to parallelize the process and having the power of a supercomputer.<br>
To parallelize the training I have been using the module “torch.multiprocessing”, where I load the dataset on the Object “DataLoader”<br>
and I use the method Process().<br>
Here I am a bit confused, because it looks like there are two possible parallelizations to perform:</p>
<ol>
<li>I can pass the number of threads in the Dataloader class passing the “num_workers” as input</li>
<li>I can append multiple processes with the method Process() and join them after the job.<br>
Given that my goal is to use multiple GPUs with multiple cores, are 1) and 2) the right ways? Or these parallelization are still related to the CPUs?<br>
Should I see distributed processing isntead?<br>
I have this doubt because I have not seen any speed improvement by now.<br>
It would be really appreciated any kind of help, thanks!!</li>
</ol>

ptrblck: <p>I don’t know the specs of your new system, but one possible and widely used approach would be to apply a data parallel training strategy via <code>DistributedDataParallel</code> with uses a single process per GPU.<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">This tutorial</a> might be a good starter.</p>
About multiprocessing in dataloader when num_workers > 0

SminYu: <p>Hello, I use Pytorch for my research to train the machine-learning model.</p>
<p>As I know, “Dataloader” in PyTorch makes subprocesses and parallelizes the job that loads the data from drives to memory when the attribute “num_worker” is larger than zero.<br>
However, I am a little bit confused that I understand exactly the process.<br>
Are all subprocesses involved in loading data every time, or does each subprocess load and prepare data sequentially?</p>
<p>Best.</p>

nivek: <p>The main process generates the indices using <code>Sampler</code> and sends them to the workers, which then prepare the specific sample and send the result back to the main process.</p>
Is there a way of calculating the dice score for each class separately in multi class segmentation using Pytorch?

Abdelrahman_Youssef: <p>I am training a UNet for multi-class segmentations. The classes I have are 0,1,2 which correspond to the background, object, and tip classes respectively. In binary segmentation, I could simply use the <code>torchmetrics</code> function <code>dice</code> inside the training loop to calculate the dice score between each prediction and the target, then average that per epoch to get the output. How can such a definition be generalized to a multi-class problem in Pytorch?</p>
Torch cuda extension with cuda libraries

Bokyeong1001: <p>Hello</p>
<p>I am trying to use the cublas and cusparse in my cuda code and wrap with torch CUDAExtension.<br>
I am wondering if I put creating cublas or cusparse handle code in cuda code, does it mean it creates handle every time I use the extension I made?<br>
Is there any way to create handle only one time and reuse it?</p>

eqy: <p>You might want to check if you can directly use or copy the approach in e.g., <a href="https://github.com/pytorch/pytorch/blob/9ae63bd87c21a59094b3f8344bac8910990d395b/aten/src/ATen/cuda/CuSparseHandlePool.cpp" rel="noopener nofollow ugc">pytorch/CuSparseHandlePool.cpp at 9ae63bd87c21a59094b3f8344bac8910990d395b · pytorch/pytorch (github.com)</a></p>
Training eval() vs Testing eval() results differ

Bled_Clement: <p>I am trying to replicate the results I get on my evaluation dataset in training with the same dataset in testing. For context, I am trying to denoise images and they are being passed a simple NN consisting of a few conv, relu and BN layers.</p>
<p>My problem is that my results are quite close but there is always some disparity. For instance in training the model gets 37.49 dB and in testing it gets 37.40.</p>
<p>I have done a bit of reading and it seems that this has to do with batchnorm in the model but setting<code> track_running_stats</code> to false hasnt helped as some people suggest.</p>
<p>I also make sure to call <code>model.eval()</code> on my model and I use <code>with torch.no_grad():</code></p>
<p>The only method I have found to bring the PSNR close is to test using a small minibatch (3 images) but this is not practical as the end goal is to process images seperately. I think this increase might also be due to the PSNR function itself processing larger batches at a time.</p>
Getting 768 feature embedding from ViT

Star_Cloud: <p>I have been trying to extract the 768 feature embedding from ViT model. I tried getting the <code>outcome</code> as output but it is of size 32.</p>
<pre><code class="lang-auto"># References:
# timm: https://github.com/rwightman/pytorch-image-models/tree/master/timm
# DeiT: https://github.com/facebookresearch/deit
# --------------------------------------------------------

from functools import partial

import torch
import torch.nn as nn

import timm.models.vision_transformer


class VisionTransformer(timm.models.vision_transformer.VisionTransformer):
    """ Vision Transformer with support for global average pooling
    """
    def __init__(self, global_pool=False, **kwargs):
        super(VisionTransformer, self).__init__(**kwargs)

        self.global_pool = global_pool
        if self.global_pool:
            norm_layer = kwargs['norm_layer']
            embed_dim = kwargs['embed_dim']
            self.fc_norm = norm_layer(embed_dim)

            del self.norm  # remove the original norm

    def forward_features(self, x):
        B = x.shape[0]
        x = self.patch_embed(x)

        cls_tokens = self.cls_token.expand(B, -1, -1)  # stole cls_tokens impl from Phil Wang, thanks
        x = torch.cat((cls_tokens, x), dim=1)
        x = x + self.pos_embed
        x = self.pos_drop(x)

        for blk in self.blocks:
            x = blk(x)

        if self.global_pool:
            x = x[:, 1:, :].mean(dim=1)  # global pool without cls token
            outcome = self.fc_norm(x)
        else:
            x = self.norm(x)
            outcome = x[:, 0]

        return outcome


def vit_base_patch16(**kwargs):
    model = VisionTransformer(
        patch_size=16, embed_dim=768, depth=12, num_heads=12, mlp_ratio=4, qkv_bias=True,
        norm_layer=partial(nn.LayerNorm, eps=1e-6), **kwargs)
    return model
</code></pre>
The gradient norm and the noise

ost: <p>I am training ResNet101 on tiny-imagenet. How to compute gradient l2 norm and the noise of the gradient (see the last two strings) per epoch? L is a classic loss function.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10f9baf042807e4a97647b9794df62822abcd354.jpeg" alt="norm and noise" data-base62-sha1="2qaGpA3LnzBxJGRbjk4mXOWwS1u" width="517" height="135"></p>
<p>Firstly I need to compute them per iteration and then average.  Is it OK for gradient norm?</p>
<pre><code class="lang-auto">for batch_num, (X, y) in enumerate(train_loader):
        X = X.to(device)
        y = y.to(device)
        optimizer.zero_grad()
        y_pred = model(X)
        loss = loss_fn(y_pred, y)
        loss_value += loss.item()
        loss.backward()
        optimizer.step()
        
        # Compute gradients for batch
        all_batch_gradients = []
        for param in model.parameters():
            all_batch_gradients.append(param.grad.view(-1,1).to("cpu"))
       all_batch_gradients = torch.concat(all_batch_gradients)
</code></pre>
Guidance to understand the dataloader architecture in a minute level

Vinayaka_Hegde: <p>Hi,</p>
<p>I’m currently trying to understand the dataloader architecture of pytorch as a part of my research. It’s very important for me to gain minute level understanding on few things such as:</p>
<ol>
<li>how the sampler yields indices to the dataloader</li>
<li>how the _MultiProcessingDataLoaderIter class works internally</li>
<li>How the _index_queues for every worker are populated</li>
<li>Why exactly the _task_info queue is used when we already have index_queues and _data queue to store indices to be processed and final data respectively. Is it because, to store out of order indices from the sampler?</li>
<li>Why do we need _rcvd_idx and _send-idx parameters? Are these just counts? Does rcvd_idx - send_idx provide the number of elements that are being processed ? What’s the use of these exactly ?</li>
<li>Why are we calling _try_put_index again in _process_data? Since we already initialized the prefetch_factor * num_workers number of indices to the workers?</li>
<li>After prefetching a batch, how do we load the next pre-fetch batch ?</li>
<li>Is there any reason for the data-queue to store (send_idx, data) ?</li>
</ol>
<p>And multiple such minute questions about the architecture of dataloader and what’s happening internally…</p>
<p>I’ve searched a lot about dataloader architecture implementation, but couldnt find any resources. And have gone through the source code for dataloader, fetch, sampler, dataset, worker and other files.</p>
<p>I sincerely request you to please point me any resources / contacts which would help me in understanding the architecture of data loader.</p>
<p>Do you suggest me to do like a dry run by using a dummy input and try to make sense of the values at every step in the code ?</p>
<p>Thanks !</p>
How does pytorch handle backward propagation with torch.round()?

duartejfs: <p>The torch.round() function is not continuous. How is the gradient calculation handled with it? Are there any references I can look into?</p>

RaulPPelaez: <p>This might help:<br>
<a href="https://pytorch.org/docs/stable/notes/autograd.html#gradients-for-non-differentiable-functions" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html#gradients-for-non-differentiable-functions</a></p>
Set and unset trainability

funnym0nk3y: <p>I don’t know if I’m doing everything how it’s supposed to but I have a very weird issue in the following code example:</p>
<pre><code class="lang-python">import torch
from torch import nn
from torch.nn import functional
from torch.utils.data import DataLoader, random_split
from torchvision import datasets, models, transforms

from torchvision.models import efficientnet_v2_m, EfficientNet_V2_M_Weights
from torchinfo import summary

weights = EfficientNet_V2_M_Weights.DEFAULT
backbone = efficientnet_v2_m(weights=weights)

for params in backbone.features[-1].parameters():
    print(params)
    params.requires_grad = False
    print(params)
    params.requires_grad = True
    print(params)

</code></pre>
<p>Output for one tensor:</p>
<pre><code class="lang-plaintext">...
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477],
       requires_grad=True)
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477])
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477])
</code></pre>
<p>It seems that <code>requires_grad = False</code> removes the field entirely. I’m not able to let the layer train again by setting <code>params.requires_grad = True</code>.</p>
<p>I could set this flag again by directly calling <code>params.requires_grad_()</code> however.</p>
<p>Is this the way it should work? Or whats going on here?</p>

ptrblck: <p>I’m not able to reproduce the issue in a current nightly release and see:</p>
<pre><code class="lang-python">...
Parameter containing:
tensor([3.4256, 3.3162, 3.1029,  ..., 3.3091, 2.9864, 3.3672],
       requires_grad=True)
Parameter containing:
tensor([3.4256, 3.3162, 3.1029,  ..., 3.3091, 2.9864, 3.3672])
Parameter containing:
tensor([3.4256, 3.3162, 3.1029,  ..., 3.3091, 2.9864, 3.3672],
       requires_grad=True)
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477],
       requires_grad=True)
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477])
Parameter containing:
tensor([-1.6943, -1.6580, -1.4941,  ..., -1.5908, -1.5684, -1.7477],
       requires_grad=True)
</code></pre>

funnym0nk3y: <p>Thanks for looking into it! Unfortunately I too can’t reproduce it today. Somehow the issue resolved itself…</p>
How to read data just once when using IterableDataset with num_workers > 0?

Yang_Zhou: <p>I’m trying to load a large npy file for training and I’m using IterableDataset. However, the memory cost increase when I set num_worker &gt; 0 due to dataset replicated on each worker process.<br>
So is there any solution for me to just read the dataset with one worker then deal with it seperately with each worker?</p>

Wenlu_Sun: <p><a href="https://pytorch.org/docs/stable/data.html?highlight=concat#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html?highlight=concat#torch.utils.data.IterableDataset</a><br>
this doc can solve your question, define a <code>worker_init_fn</code> that configures each dataset copy differently</p>
Architecture to learn rotati two 3d objects

Gabriele_Orlando: <p>hi,<br>
I am struggling with the definition of an architecture.</p>
<p>so, this is my problem:</p>
<ol>
<li>I have 2 3D objects made by points in the space</li>
<li>There is a fixed and a mobile object</li>
<li>I want to predict a rotation matrix and a translation that moves the mobile object to the right position</li>
<li>both objects have a variable number of points</li>
</ol>
<p>does anybody have a suggestion?</p>
Where exactly does the data loader access the disk?

Vinayaka_Hegde: <p>Hi, I needed to know where the data loader opens the file to read/ access the disk?<br>
In the fetch function below, I can see that we get the data. But, I’m not exactly able to find out where <strong>getitem</strong> is defined exactly?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d7f9265e129bd638e07c3e1e4f039dfd7562034.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d7f9265e129bd638e07c3e1e4f039dfd7562034" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d7f9265e129bd638e07c3e1e4f039dfd7562034_2_690x202.png" alt="image" data-base62-sha1="dl7J1zHnE3zHVZ23OEPT6y5N4Uc" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d7f9265e129bd638e07c3e1e4f039dfd7562034_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d7f9265e129bd638e07c3e1e4f039dfd7562034_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d7f9265e129bd638e07c3e1e4f039dfd7562034_2_1380x404.png 2x" data-dominant-color="F7F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1524×447 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One possible explanation could be that <strong>getitem</strong> could be used from the custom datasets. Example: datasets/mnist.py. Where <strong>load_data</strong> calls read_sn3_pascalvincent_tensor and this function internally access the file by opening like so :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04c8c84e7f5d962bed9563315c9b33577c85aac9.png" data-download-href="https://discuss.pytorch.org/uploads/default/04c8c84e7f5d962bed9563315c9b33577c85aac9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04c8c84e7f5d962bed9563315c9b33577c85aac9_2_690x175.png" alt="image" data-base62-sha1="Gk5e1c6cRjAPIu6PoE1FApbJeV" width="690" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04c8c84e7f5d962bed9563315c9b33577c85aac9_2_690x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04c8c84e7f5d962bed9563315c9b33577c85aac9_2_1035x262.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/04c8c84e7f5d962bed9563315c9b33577c85aac9.png 2x" data-dominant-color="F7F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1141×290 24.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The <code>__getitem__</code> method is defined in your <code>Dataset</code>.</p>
Problem with shapes in autograd.grad

Miguel_Andres: <p>Hi. I’m trying to compute a gradient of a non-scalar tensor with torch.autograd.grad. The input shape is (B, Ns, 3) and the output shape is (B, Ns, Nf).  I use the following code to compute it:</p>
<pre><code class="lang-auto">gradients = torch.autograd.grad(
        outputs=sigma,
        inputs= xyz,
        grad_outputs=torch.ones_like(sigma),
    )[0]
</code></pre>
<p>The problem is that the gradients shape is (B, Ns, 3) when I was expecting (B, Ns, Nf, 3). So far I tried to compute de gradients for each element Nf of the output ‘sigma’ separately and it seems to work, but is there a way to compute it directly and more efficiently by using just autograd.grad?</p>
<p>Thanks</p>
Forward() takes 2 positional arguments but 3 were given in case of predefined Transformer Decoder layer

Amartya_Chowdhury: <p>I am training a model based on Transformer Decoder. The decoder layer i have taken from the predined pytorch website, But when i am giving the encoder output and the input to the decoder i am gettin a ’ forward() takes 2 positional arguments but 3 were given’.</p>
<p>class Decoder(nn.Module):<br>
def <strong>init</strong>(<br>
self,<br>
in_channels:int=1,<br>
patch_size:int=16,<br>
num_transformer_layers:int=6,<br>
embedding_dim:int=768,<br>
mlp_size:int=3072,<br>
num_heads:int=12,<br>
attn_dropout:int=0.1,<br>
mlp_dropout:int=0.1,<br>
embedding_dropout:int=0.1,<br>
out_features:int=512<br>
):<br>
super().<strong>init</strong>()<br>
self.DecoderPrenet = Prenet(input_size=embedding_dim, output_size=out_features<em>2,<br>
hidden_size=embedding_dim)<br>
self.embedding = PatchEmbedding(in_channels=in_channels,<br>
patch_size=patch_size,<br>
embedding_dim=embedding_dim)<br>
self.pe = PositionalEncoding(d_model=768)<br>
self.embedding_dropout = nn.Dropout(p=embedding_dropout)<br>
self.mel_linear = nn.Linear(embedding_dim, hp.num_mels * hp.outputs_per_step)<br>
self.transformer_decoder = nn.Sequential(</em> [nn.TransformerDecoderLayer(d_model=embedding_dim,<br>
nhead=num_heads,<br>
dim_feedforward=mlp_size,<br>
dropout=mlp_dropout, activation=‘gelu’,<br>
batch_first=True, norm_first=True) for _ in range(num_transformer_layers)])<br>
self.PostConvNet = PostConvNet(num_hidden=embedding_dim)<br>
self.encoder = SpeechtoText()</p>
<p>def forward(self, x, src):<br>
batch_size = 16<br>
memory = self.encoder(src)<br>
x = self.embedding(x)<br>
x = self.DecoderPrenet(x)<br>
x = self.pe(x)<br>
expand = x.expand(batch_size, -1, -1)<br>
x = self.embedding_dropout(expand)<br>
x = self.transformer_decoder(x, memory)<br>
mel_out = self.mel_linear(x)<br>
y = torch.flatten(mel_out, start_dim=0, end_dim=1)<br>
postnet_input = y.transpose(0, 1)<br>
postconvnet = self.PostConvNet(postnet_input)<br>
out = postnet_input + out<br>
out = out.transpose(1, 0)<br>
k = out.shape[1]<br>
out = torch.reshape(out, (batch_size, m, k))<br>
return out</p>
<p>i tried using torchsummary. I am still receiving the same issue. Can anyone help me with this</p>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/forward-takes-2-positional-arguments-but-3-were-given-for-predefined-transformer-decoder-layer/170375">here</a> from another account?</p>

aquorio15: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> yes sorry my primary work account was having some problems so i had to repost the question from another account. Sorry for the inconvenience. Can you kindly delete the question as i am unable to do it from my end</p>
Outputting Positive Semidefinite Covariance Matrices from Neural Network

jxmmy7777: <p>Hello all,</p>
<p>I am working on a project that involves training a neural network to output covariance matrices. However, I am having trouble ensuring that the matrices are always positive semidefinite. I’ve read about techniques like Cholesky decomposition and matrix exponential, but I’m not sure how to implement them in PyTorch.</p>
<p>Has anyone here had experience with this issue and could provide some guidance on how to ensure that the network outputs positive semidefinite matrices? Any help or advice would be greatly appreciated.</p>
<p>Thank you.</p>
<p>Best regards,</p>
How to interpret (understand) this OOM information

111466: <p>I am confused with the error message. It seems that the machine owns 23.70GiB memory, and it is enough for 30.00 MiB. Why thrown error information below? Could you help me clarify the reson? Thnank you !</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 30.00 MiB (GPU 0; 23.70 GiB total capacity; 1.60 GiB already allocated; 13.03 GiB free; 1.66 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>

ptrblck: <p>This error seems to be indeed at least misleading since your device has &gt;13GiB of free memory and I doubt it’s all fragmented. In any case, could you post a minimal and executable code snippet so that I could try to reproduce and debug the issue, please?</p>

111466: <p>Sorry, but I can not provide you with a desirable code. Because the original code is very large and complicated . And I guess that  the multiprocessing library of pytorch utilzed by the code may be related to this error.</p>
How to compile weights into model?

Flam3y: <p>I have weights from …\yolov3\runs\train\exp6\weights, but when i used best.pt in imageAI code, i get this error: “ModuleNotFoundError: No module named ‘models’” How cain build a model from weights?</p>

ptrblck: <aside class="quote no-group" data-username="Flam3y" data-post="1" data-topic="170482">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flam3y/48/56508_2.png" class="avatar"> Flam3y:</div>
<blockquote>
<p>How cain build a model from weights?</p>
</blockquote>
</aside>
<p>You cannot build a model architecture just from its parameters alone and would need the model definition.<br>
Could you also explain your use case in more detail as it seems that the <code>models</code> class is currently undefined so maybe just an <code>import</code> is missing.</p>
Segmentation fault while importing torch

MrPerfkt: <p>hi,<br>
Im trying to import torch and torchvision on my jetson nano<br>
i have followed this tutorial <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048" class="inline-onebox" rel="noopener nofollow ugc">PyTorch for Jetson - Announcements - NVIDIA Developer Forums</a><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3fb145e177056d8da95917bbb7892da0230c13c.png" alt="Screenshot from 2023-01-17 19-13-22" data-base62-sha1="wwOjUXBi17eDvk0X2vbStPLMres" width="640" height="108"><br>
I am new to this please help</p>

ptrblck: <p>Could you check if any of <a href="https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html">these wheels</a> work work for you?</p>
Why is my loss function going down and jumping

redback.bjfu.ws: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a4afd2b58e7d068dc383b491c7a5d7c98bbd1ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/4a4afd2b58e7d068dc383b491c7a5d7c98bbd1ed" title="1674009887723"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a4afd2b58e7d068dc383b491c7a5d7c98bbd1ed.png" alt="1674009887723" data-base62-sha1="aBdZKayf4pv3Y2hdhHlzFFmPI7z" width="248" height="499" data-dominant-color="EBEDEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1674009887723</span><span class="informations">613×1231 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

J_Johnson: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4ff639c7ad464c5d9da607061fdf85d5056064b6.jpeg" alt="MuYS8" data-base62-sha1="bpnfTdMOX6UmnaTU84blMrBMePQ" width="600" height="400"></p>
<p>It’s not always a straight path to the best fit. In fact, usually it’s pretty bumpy.</p>

redback.bjfu.ws: <p>*Thank you for your answer and letting me know that this is normal.</p>
Training and validation loss characteristics

Hafsa_Moontari_Ali: <p>I am working on an image reconstrction problem. I am observing the performance by training different models. For every model, in the first epoch, the difference between training loss and validation loss is almost same. Afterwards, from the second epoch the difference between training and validation loss significantly decreases from the first epoch’s loss. The differences between them becomes very low, and this low loss value tends to continues in all the epochs.<br>
Can anyone help me understand why this is happening? Is it one form of overfitting?</p>

J_Johnson: <p>When you start training, your model does very poorly on the train data since it’s never seen the data. So if you average out that first epoch and print it, you’re getting an average of “never seen this before” and “this is starting to make sense”.</p>
<p>By the time it gets to the test data, it’s starting to understand the problem and usually performs better. Additionally, the test data doesn’t use any dropout layers. Hence lower loss is initially expected on the test data. But eventually, these two converge, which suggests you are starting to overfit on the train data.</p>
Torch_tensorrt vs AMP

sardanian: <p>What are the differences of converting a model to tensorrt via torch_tensorrt vs using PyTorch AMP for inference?</p>
<p>I’m using precisions of float and half (not int8) on a convolution and skip connections.</p>
<p>Does the model come out with that same graph in both models? Or does one reduce elements</p>
<p>Is one faster?<br>
Is one more accurate?<br>
Other advantages or disadvantages?</p>

sardanian: <p>Does anybody have experience with the differences of these?</p>

eqy: <p>The difference would ultimately depend on what your model is doing, but in general AMP would be much more limited in what optimizations it can apply to your model for inference compared to TensorRT which in theory can apply arbitrary fusions and use specialized kernels for inference. For example, AMP in eager mode won’t opportunistically fuse successive pointwise operations, and would insert potentially unnecessary up/down-casts for successive pointwise operations. Additionally, you may see greater CPU overhead from AMP/eager-mode dispatching compared to TensorRT but the severity again depends on what your model is doing (e.g., ratio of compute/arithmetic-bound ops vs. bandwidth-bound ops) and the coverage of TensorRT optimizations for your use-case.</p>
Is a different learning rate per neuron or parameter possible?

adamgoodtime: <p>I have seen that it is possible to have a different learning rates for a layer or group of parameters using the code copied below.</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': mylayer.weight},
                {'params': mylayer.bias, 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>Is it possible to have a different learning rate per parameter e.g.</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': mylayer.weight, 'lr': [np.random.random() for i in range(len(mylayer.weight)]},
                {'params': mylayer.bias, 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>?</p>

eqy: <p>You may want to take a look at the Adam optimizer, <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam" class="inline-onebox" rel="noopener nofollow ugc">Adam — PyTorch 1.13 documentation</a>, which in effect computes a per-weight learning rate, but I’m not sure it is common practice to specify/initialize this explicitly.</p>
Scale to 0 and 1

hustine316: <h1>
<a name="i-have-a-ct-image-whose-values-of-pixels-are-between-0-4000-how-can-i-convert-it-to-an-image-with-0-to-1-1" class="anchor" href="#i-have-a-ct-image-whose-values-of-pixels-are-between-0-4000-how-can-i-convert-it-to-an-image-with-0-to-1-1"></a>I have a ct image whose values of pixels are between 0-4000. How can I convert it to an image with 0 to 1?</h1>
<p>I have tried using the transform.ToTensor() but does not work,  notice the image is still between 0 and 4000.</p>

eqy: <p><code>ToTensor()</code> will simply call <code>.div(255)</code> if it gets an image of type <code>ByteTensor</code> but it will not do this for other data types (which I suspect you have if the range is 0-4000). If you know the range of the image, you can simply call div <code>.div(MAX)</code> yourself.</p>
CUBLAS_STATUS_NOT_INITIALIZED when running CudaGRAPH

jakkes12: <p>Hi,</p>
<p>I am attempting to make use of the CUDAGraph API in libtorch. However, I run into the following error</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
</code></pre>
<p>Sample code</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;ATen/cuda/CUDAGraph.h&gt;
#include &lt;c10/cuda/CUDAStream.h&gt;


torch::Tensor linear_sigmoid(const torch::Tensor &amp;A, const torch::Tensor &amp;x, const torch::Tensor &amp;b)
{
    return torch::sigmoid(A.matmul(x) + b);
}


int main()
{
    torch::StreamGuard stream_guard{at::cuda::getStreamFromPool(true)};
    auto x = torch::randn({5, 5, 1}).cuda();
    auto A = torch::randn({5, 2, 5}).cuda();
    auto b = torch::randn({5, 2, 1}).cuda();

    at::cuda::CUDAGraph graph{};
    graph.capture_begin();
    auto output = linear_sigmoid(A, x, b);
    graph.capture_end();
    std::cout &lt;&lt; output &lt;&lt; "\n";

    x.copy_(torch::randn({5, 5, 1}).cuda());
    A.copy_(torch::randn({5, 2, 5}).cuda());
    b.copy_(torch::randn({5, 2, 1}).cuda());

    graph.replay();
    std::cout &lt;&lt; output &lt;&lt; "\n";
}
</code></pre>

eqy: <p>I don’t see warmup iterations being done here, which are required (e.g., see <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" class="inline-onebox" rel="noopener nofollow ugc">Accelerating PyTorch with CUDA Graphs | PyTorch</a>).</p>

jakkes12: <p>Amazing, thank you for the quick reply.</p>
<p>Updated code snippet for anybody coming here in there future:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;ATen/cuda/CUDAGraph.h&gt;
#include &lt;c10/cuda/CUDAStream.h&gt;


torch::Tensor linear_sigmoid(const torch::Tensor &amp;A, const torch::Tensor &amp;x, const torch::Tensor &amp;b)
{
    return torch::sigmoid(A.matmul(x) + b);
}


int main()
{
    torch::StreamGuard stream_guard{at::cuda::getStreamFromPool()};
    auto x = torch::randn({5, 5, 1}).cuda();
    auto A = torch::randn({5, 2, 5}).cuda();
    auto b = torch::randn({5, 2, 1}).cuda();
    auto output = linear_sigmoid(A, x, b);

    at::cuda::CUDAGraph graph{};
    graph.capture_begin();
    output = linear_sigmoid(A, x, b);
    graph.capture_end();
    std::cout &lt;&lt; output.mean() &lt;&lt; "\n";

    for (int i = 0; i &lt; 10; i++) {
        x.copy_(torch::randn({5, 5, 1}).cuda());
        A.copy_(torch::randn({5, 2, 5}).cuda());
        b.copy_(torch::randn({5, 2, 1}).cuda());
        graph.replay();
        std::cout &lt;&lt; output.mean() &lt;&lt; "\n";
    }
}

</code></pre>
Towards Reproducible Training Results

untitled47: <p>In <a href="https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281" class="inline-onebox">Manual seed cannot make dropout deterministic on CUDA for Pytorch 1.0 preview version</a>, it is mentioned that seeding does not help in reproducibility when a model contains modules like <code>nn.Dropout</code> .  In that forum, a potential solution would be using <code>torch.set_rng_state()</code>. Here are my findings:</p>
<ol>
<li>LeNet5 does not contain dropout layer while seeding failed to give reproducible results. Seeds are configured as below.</li>
</ol>
<pre><code class="lang-auto">    np.random.seed(args.seed)
    os.environ['PYTHONHASHSEED'] = str(args.seed)

    # Set seed for pytorch
    torch.manual_seed(args.seed)  # Set seed for CPU
    torch.cuda.manual_seed(args.seed)  # Set seed for the current GPU
    torch.cuda.manual_seed_all(args.seed)  # Set seed for all the GPUs
    cudnn.benchmark = False
    cudnn.deterministic = True
</code></pre>
<ol start="2">
<li>When <code>torch.nn.DataParallel</code> is used, the results among different runs varies slightly more obvious.</li>
</ol>
<p>I would be grateful if you could explain why these could happen.</p>

ptrblck: <p>Which PyTorch version are you using and did you follow the steps described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>? Since the issue you’ve linked is quite old I would assume things have changed already and the described CUDA seeding issue was fixed.</p>

untitled47: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you so much for your reply.</p>
<p>I am using pytorch 1.13.0 and I did follow the steps shown in  <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.13 documentation</a>.</p>
<p>Here are some clues:</p>
<ol>
<li>
<p>I noticed that seeding as shown above kept data sequence fixed (with dataset shuffle and random transforms on samples).</p>
</li>
<li>
<p>In addition, seeding itself guaranteed the model (LeNet5 in my case) to be initialized in the same way during different runs.</p>
</li>
</ol>
How to interpret the global max pooling operation in graph neural networks?

zaratruta: <p>0</p>
<p>I’m trying to use pytorch geometric for building graph convolutional networks. And I’m trying to interpret the result of the max pooling operation, which is described in this link:</p>
<p><a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.pool.global_max_pool" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.pool.global_max_pool</a></p>
<p>If I understood it correctly, the result is a feature vector r_i (which the dimensionality of the features of the nodes in the graph), which represents the maximum, considering the feature vectors of all nodes in the graph.</p>
<p>But for me, it is obscure how the feature vectors are compared in order to find the vector that represents the maximum.</p>
<p>Can you help me?</p>
<p>Best regards.</p>

tiramisuNcustard: <p>Sounds to me you are getting a vector of size N (where N = number of features) and this vector contains the maximum value of the features among all the samples (i.e images).</p>
Building Static Libraries Fail

will-leeson: <p>The project I’m building requires that libtorch be built statically. I am following the instructions in the <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" rel="noopener nofollow ugc">docs directory</a>, but it fails at the end of the build. Specifically, when linking libshm. I get the following error</p>
<blockquote>
<p>Linking CXX shared library …/…/…/…/lib/libshm.so<br>
clang: error: no such file or directory: ‘$&lt;TARGET_FILE:dnnl_graph&gt;’</p>
</blockquote>
<p>Whats curious to me is that it is build libshm.so instead of libshm.a. I have the environment variable BUILD_SHARED_LIBS set to OFF, so I’m not sure why this is the case. Any suggestions? I’m not sure what dnnl_graph is and why it is not there</p>

phoenix: <p>Were you able to solve this problem?</p>
<p>If yes, then can you share a fix?</p>

phoenix: <p>I didn’t face the libshm.so linking error with Pytorch <code>v1.9.1</code>.</p>
<p>However, I faced a different error which I was able to solve with <code>-DBUILD_PYTHON:BOOL=OFF</code> as explained here: <a href="https://discuss.pytorch.org/t/errors-when-building-static-linked-libtorch/117186" class="inline-onebox">Errors when building static linked LibTorch</a></p>
LSTM: using different sequence lengths for input and target

Planet_Boost: <p>I want to use a denser time series to predict a less dense time series.</p>
<p>I first had input (X) with shape [33405, 4, 25] and target (Y) with shape [33405, 4, 7], in which 33405 is the amount of samples, 4 is the sequence length and 25 &amp; 7 are the number of features. They thus had a similar sequence length.</p>
<p>I used the following model:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=1, dropout=0, activation='tanh'):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers=num_layers, batch_first=True)
        self.dropout = nn.Dropout(dropout)
        self.linear = nn.Linear(hidden_size, output_size)
        init.xavier_uniform_(self.linear.weight)
        
    def forward(self, x):
        x, _ = self.lstm(x)
        x = self.dropout(x)
        x = self.linear(x)
        return x
</code></pre>
<p>I got correctly an output of shape [batch_size, 4, 7].</p>
<p>However, I want to do something similar, but now use a more dense time series (sequence length 92) to predict the same sequence of 4. This means that I have an input X that has a sequence length of 92 and a target Y that has a sequence length of 4. My input (X) now has shape [33540, 92, 7] and target (Y) shape [33540, 4, 7].</p>
<p>I use the same model, but now my output has shape [4, 92, 7]. However, I want it, again, to be [batch_size, 4, 7].</p>
<p>I’m a newbie with LSTM and RNNs. Is it possible to work with an X and Y that have different sequence lengths? If so, how should I alter my model to get the desired output?</p>
<p>Thank you!</p>
Calculate f1-score a for Transformer model

C0NGTRI123: <p>def flat_accuracy(preds, labels):<br>
pred_flat = np.argmax(preds, axis=1).flatten()<br>
labels_flat = labels.flatten()<br>
return np.sum(pred_flat == labels_flat) / len(labels_flat)</p>
<p>I have problem how change find the accuracy to f1-score</p>

flutter85: <p>Do you have a solution?</p>
Save torch model in Databricks Cluster

sergimassabates1: <p>Hi. I’m using a Databricks CPU Cluster to train my NN on Pytorch. The problem is that when I try to save my model, I can only save it on the Cluster CPU (which is unnaccessible) and when I restart the cluster I lose my saved model.<br>
How could I make it to save it into local path(windows)? If it is not possible, could it be possible to save the file on a table-like format and then reformat it into PyTorchModelType?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <aside class="quote no-group" data-username="sergimassabates1" data-post="1" data-topic="81777">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/87869e/48.png" class="avatar"> sergimassabates1:</div>
<blockquote>
<p>I can only save it on the Cluster CPU (which is unnaccessible)</p>
</blockquote>
</aside>
<p>I’m not sure I understand this sentence correctly.<br>
Are you able to access some folders on this machine?<br>
If so, you should be able to store the <code>state_dict</code> there (similar to your script files).</p>

sergimassabates1: <p>I can acces into some folders of the Cluster but they are non-savable paths for the model. It seems like I’m saving the .pt into a non accessible path.</p>

ptrblck: <p>Were are you storing the script files? Could you save the <code>state_dict</code> to the same location?</p>

sergimassabates1: <p>Any path that I try that is not the direct path -&gt; “filename.pt” returns me this error:</p>
<p>FileNotFoundError: [Errno 2] No such file or directory:</p>

ptrblck: <p>Are you getting this error from <code>torch.save(model.state_dict(), 'filename.pt')</code>?</p>

sergimassabates1: <p>This gets correctly saved and can be loaded but when cluster is restarted it disappears:</p>
<blockquote>
<p>torch.save(model.state_dict(), ‘filename.pt’)</p>
</blockquote>
<p>Andthis returns de error mentioned before:</p>
<blockquote>
<p>torch.save(model.state_dict(), ‘dbfs:/existingfilepath/filename.pt’)</p>
</blockquote>

ptrblck: <p>So the files disappear after a restart?<br>
This seems to be a Databricks issue and I’m unfortunately not familiar with their cluster.<br>
Could you add some persistent user folders, where you could store data or maybe mount another drive to the machine?</p>

nima_rafiee: <p>The path should be like:<br>
<code>‘/dbfs:/existingfilepath/filename.pt’</code></p>