akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Libtorch script can't access cuda

SeHSie: <p>I’ve been trying to get the C++ frontend set up in visual studio on my machine. I installed the cuda 11.1 compatible version of libtorch, but in my project the is_available command returns 0. below my code I have included the output of the collect-env.py script. Running other commands like torch::eye() works, so I’m not sure why it says PyTorch version N/A or that cuda isn’t available when I’ve installed cuda 11.1.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

#include &lt;opencv2/core.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;

using namespace torch;

struct Net : torch::nn::Module {
    Net( int kNoiseSize)
        : conv1(torch::nn::ConvTranspose2dOptions(kNoiseSize, 256, 4)
            .bias(false)), batch_norm1(1) {
        register_module("conv1", conv1);
        register_module("batch_norm1", batch_norm1);
    }
    torch::Tensor forward(torch::Tensor input) {
        input = conv1(input);
        input = batch_norm1(input);
        return torch::relu(input).clone();
    }
    torch::nn::ConvTranspose2d conv1;
    torch::nn::BatchNorm2d batch_norm1;

};


int main() {
    std::cout &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;

}
</code></pre>
<p>Collecting environment information…<br>
PyTorch version: N/A<br>
Is debug build: N/A<br>
CUDA used to build PyTorch: N/A<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Enterprise<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: version 3.21.0-rc1<br>
Libc version: N/A</p>
<p>Python version: 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.18363-SP0<br>
Is CUDA available: N/A<br>
CUDA runtime version: 11.1.74<br>
GPU models and configuration: GPU 0: Quadro M2000M<br>
Nvidia driver version: 471.68<br>
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\cudnn_ops_train64_8.dll<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] Could not collect<br>
[conda] Could not collect</p>
Fully connected layers with no activation functions

modeler: <p>If you don’t specify any activation functions on torch.nn.Linear, is the default the identity activation?</p>
<p>If so, in a network where you have only have fully connected layers (torch.nn.Linear) with identity activation, you can just replace all fully connected layers with a single fully connected layer?</p>
<p>I.e., the following 2 are equivalent:</p>
<blockquote>
<p>class LinearRegressionModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>  # self.fc1 = nn.Linear(num_TCs * num_time_steps, 500)
  self.fc1 = nn.Linear(100, 1)
  self.fc2 = nn.Linear(100, 500)
  self.fc3 = nn.Linear(500, 250)
  self.fc4 = nn.Linear(250, 1)
</code></pre>
<p>def forward(self, x):</p>
<pre><code>  out = self.fc1(out)

  return out
</code></pre>
</blockquote>
<blockquote>
<p>class LinearRegressionModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>  self.fc1 = nn.Linear(100, 100)
  self.fc2 = nn.Linear(100, 500)
  self.fc3 = nn.Linear(500, 250)
  self.fc4 = nn.Linear(250, 1)
</code></pre>
<p>def forward(self, x):<br>
out = self.fc1(out)<br>
out = self.fc2(out)<br>
out = self.fc3(out)<br>
out = self.fc4(out)</p>
<pre><code>  return out
</code></pre>
</blockquote>
Always the same gradient at a warmstart model

smartmart: <p>I followed the pytorch tutorial for warmstarting a model (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 2.2.0+cu121 documentation</a>). But when I want to train the model and check the gradients, I always get the same gradient every iteration and the model gets worse every iteration. This is the code where I first save the model and the reload the model. Then I set up the optimizer for training</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89203315dd207813f1c2982d356dbaa72264a6ac.png" alt="image" data-base62-sha1="jz4oQbU1zyW3mMP2bkk323wogHy" width="678" height="269"></p>
Zero gradient in firt layer

Stefano_Francesco_Pi: <p>Hey guys,</p>
<p>I have a very simple model but I’m struggling in understanding why I obtain zero gradients for the first layer and very lowee gradients for the last one. My code is:</p>
<p>import os</p>
<p>import torch</p>
<p>from torch import nn</p>
<p>from torch import optim</p>
<p>from torch.utils.data import DataLoader</p>
<p>from torchvision import datasets, transforms</p>
<p>device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’</p>
<p>training_set = datasets.MNIST(</p>
<pre><code>root="MNIST",

train=True,

download=True,

transform=transforms.ToTensor()
</code></pre>
<p>)</p>
<p>test_set = datasets.MNIST(</p>
<pre><code>root="MNIST",

train=False,

download=True,

transform=transforms.ToTensor()
</code></pre>
<p>)</p>
<p>train_dataloader = DataLoader(training_set, batch_size=64, shuffle=True)</p>
<p>test_dataloader = DataLoader(test_set, batch_size=64, shuffle=True)</p>
<p>model = nn.Sequential(</p>
<pre><code>nn.Linear(28*28, 50),

nn.ReLU(),

nn.Linear(50, 10)
</code></pre>
<p>)</p>
<p>optimizer = optim.SGD(model.parameters(), lr=0.1)</p>
<p>criterion = nn.CrossEntropyLoss()</p>
<p>input, output = next(iter(train_dataloader))</p>
<p>input = input.view(input.shape[0], input.shape[2]*input.shape[3])</p>
<p>optimizer.zero_grad()</p>
<p>logits = model(input)</p>
<p>loss = criterion(logits, output)</p>
<p>loss.backward()</p>
<p>optimizer.step()</p>
<p>Someone can spot the mistake? Thank you</p>

ptrblck: <p>I’m not sure how you’ve checked the gradient in the first layer, but running your code for a single step I get:</p>
<pre data-code-wrap="python"><code class="lang-python">print((model[0].weight.grad))
&gt; tensor([[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]])
</code></pre>
<p>While this looks like an all-zero gradient, note that the output is truncated.<br>
Visualizing an example gradient of the first layer via:</p>
<pre data-code-wrap="python"><code class="lang-python">f, axarr = plt.subplots(1, 2)
axarr[0].imshow(model[0].weight.grad[0].view(28, 28))
axarr[1].imshow(model[0].weight.grad[0].view(28, 28) == 0)
</code></pre>
<p>gives:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f66680b23e1fea9d52244dffdaaa64ba1a080a8a.png" alt="image" data-base62-sha1="z9L4grI5MX8OWi3UKmCeDXI9VMK" width="368" height="184"></p>
<p>So I would assume that the raw inputs containing zeros (the borders of the MNIST images) are creating the zero gradient.</p>

Stefano_Francesco_Pi: <p>Hey, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your answer. It is very helpful, never thought of plotting the gradients <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=10" title=":smile:" class="emoji" alt=":smile:"></p>
Making a tensor sparse

MRRP: <p>Hi Pytorch Community<br>
I have a tensor of shape (6, 3, 5, 5) and for my project i want to keep a fraction 0 &lt; q &lt; 1 of indices that are related to the q largest elements in that tensor and set the others to zero. Let me show you an example of what i am exactly talking about in a smaller measure. Assume that i have a tensor of shape (2, 2) like below:</p>
<blockquote>
<p>a = [[1 ,3], [4, 2]]</p>
</blockquote>
<p>Now imagine that i want to keep 50 percent of all the elements in this tensor. But those 50 percent have to be the largest ones among them so the final result will be like:</p>
<blockquote>
<p>b = [[0, 3], [4, 0]]</p>
</blockquote>
<p>I will appreciate if you can give me a hint or a solution on how to do this.<br>
Thank you</p>
How to implement this with fast pytorch function?

ljmzlh: <p>Say i have a float tensor A with dim[n][n], and 2 int tensor X,Y with dim[N].<br>
each value in X,Y is a int between [0,n-1]<br>
now i want to create a tensor B with dim[N,N], satisfying B[i][j]=A[X[i]][Y[j]]</p>
<p>help me, plz</p>

ptrblck: <p>If I understand your use case correctly, this should work:</p>
<pre><code class="lang-python">N = 5
A = torch.randn(N, N)
x, y = torch.randint(0, N, (N,)), torch.randint(0, N, (N,))

B = torch.zeros(N, N)

# slow approach for verification
for i in range(N):
    for j in range(N):
        B[i, j] = A[x[i], y[j]]

# avoiding loops
C = A[x.unsqueeze(1), y]

# check for equal values
print((B == C).all())
&gt; tensor(True)
</code></pre>

ljmzlh: <p>thx, really helpful!</p>
<p>can i further avoid the outer loop for batch ?<br>
Say</p>
<pre><code class="lang-python">A.size=(Batch,N,N)
x.size=y.size=(Batch,N)

for b in range(Batch):
    B[b]=A[b,x[b].unsqueeze(1),y[b]]
</code></pre>

ptrblck: <p>Yes, this should work:</p>
<pre><code class="lang-python">batch_size, N = 4, 5
A = torch.randn(batch_size, N, N)
x, y = torch.randint(0, N, (batch_size, N)), torch.randint(0, N, (batch_size, N))

B = torch.zeros(batch_size, N, N)

# loop approach
for b in range(batch_size):
    B[b]=A[b,x[b].unsqueeze(1),y[b]]


# avoiding loops
C = A[torch.arange(A.size(0)).unsqueeze(1).unsqueeze(2), x.unsqueeze(2), y.unsqueeze(1)]

# check for equal values
print((B == C).all())
&gt; tensor(True)
</code></pre>

ljmzlh: <p>Thanks very much! it works</p>
Download docs in pdf and epub format

Giorgio: <p>I’ve noticed that some websites allow to download the documentation (tutorials and complete language documentation) in epub and pdf format. This makes the consultation and study so much easier.</p>
<p>For example this happens in the <a href="http://python.org" rel="noopener nofollow ugc">python.org</a> website.</p>
<p>Could you consider doing this also for pytorch?</p>
<p>Thanks<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/936092bae9e137c72fc18e64362ac40d68fa3f60.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/936092bae9e137c72fc18e64362ac40d68fa3f60" title="Screenshot_20210819-103757"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/936092bae9e137c72fc18e64362ac40d68fa3f60_2_690x485.jpeg" alt="Screenshot_20210819-103757" data-base62-sha1="l1L5TYsFr0vWT5vvV9EV2wwwIvK" width="690" height="485" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/936092bae9e137c72fc18e64362ac40d68fa3f60_2_690x485.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/936092bae9e137c72fc18e64362ac40d68fa3f60_2_1035x727.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/936092bae9e137c72fc18e64362ac40d68fa3f60_2_1380x970.jpeg 2x" data-dominant-color="EAEAEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20210819-103757</span><span class="informations">1920×1350 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Giorgio: <p>I’ve seen that the pytorch documentation is generated  with Sphinx, which supports the automatic conversion to epub etc, so this should be quite simple to automate.</p>
Optuna fforward function?

PlzHelp: <p>Hi, I am trying to optimize the hyperparameters of a Transformer model with optuna. Now what I am struggling with, is to figure out how i can create a forward esque object, that can returned from the trial function. As i understand it, you need to return a sequential, but I need to have skip connections and other transformations in the “sequential” object, do any of you know if this is possible, and if so how you do it?</p>
Infer value of retain_graph and create_graph

schaefertim: <p>During my backward hook, I’d like to know the values of <code>retain_graph</code> and <code>create_graph</code>.<br>
For <code>create_graph</code>, the value can be determined with <code>torch.is_grad_enabled()</code>.<br>
Is there a similar function for <code>retain_graph</code>?<br>
If no, is there a workaround?</p>
<p>Here is a code illustration at which point in the code I’d like to determine the value.</p>
<pre><code class="lang-auto">from torch import nn, is_grad_enabled, rand, randint


def _hook(module, grad_input, grad_output):
    print(f"{module.__class__.__name__}, retain_grad=???, create_graph={is_grad_enabled()}")


def forward_backward(retain_graph=False, create_graph=False):
    print(f"\ntest: retain_grad={retain_graph}, create_graph={create_graph}")
    loss = loss_fn(model(rand(8, 5)), randint(4, (8,)))
    loss.backward(retain_graph=retain_graph, create_graph=create_graph)


model = nn.Linear(5, 4)
loss_fn = nn.CrossEntropyLoss()
model.register_full_backward_hook(_hook)

forward_backward()
forward_backward(retain_graph=True)
forward_backward(retain_graph=True, create_graph=True)
forward_backward(retain_graph=False, create_graph=True)
</code></pre>
<p>with output</p>
<pre><code class="lang-nohighlight">test: retain_grad=False, create_graph=False
Linear, retain_grad=???, create_graph=False

test: retain_grad=True, create_graph=False
Linear, retain_grad=???, create_graph=False

test: retain_grad=True, create_graph=True
Linear, retain_grad=???, create_graph=True

test: retain_grad=False, create_graph=True
Linear, retain_grad=???, create_graph=True
</code></pre>

tom: <p>The easiest probably is to just ask grad / backward for the parameter it got (after fixing the retain_grad vs. retain_graph typo):</p>
<pre><code class="lang-python">from torch import nn, is_grad_enabled, rand, randint


def _hook(module, grad_input, grad_output):
    fr = ([f[0] for f in inspect.stack() 
           if '/torch/autograd/'in f.filename][-1])  # could be made more precise to check for backward/grad
    retain_graph = (inspect.getargvalues(fr).locals['retain_graph'])
    print(f"{module.__class__.__name__}, {retain_graph=}, create_graph={is_grad_enabled()}")


def forward_backward(retain_graph=False, create_graph=False):
    print(f"\ntest: retain_graph={retain_graph}, create_graph={create_graph}")
    loss = loss_fn(model(rand(8, 5)), randint(4, (8,)))
    loss.backward(retain_graph=retain_graph, create_graph=create_graph)


model = nn.Linear(5, 4)
loss_fn = nn.CrossEntropyLoss()
model.register_full_backward_hook(_hook)

forward_backward()
forward_backward(retain_graph=True)
forward_backward(retain_graph=True, create_graph=True)
forward_backward(retain_graph=False, create_graph=True)
</code></pre>
<p>You did not ask whether what you want to do is a good idea. This is good because in all likelihood is is not a good idea. You are creating quite brittle things here from casual observations with a given PyTorch version in very simple use-cases.</p>
<p>Best regards</p>
<p>Thomas</p>
Advice about Transformers for Time series

radi-cho: <p>Hello there! I’ve been experimenting with using attention mechanisms and Transformers for time series forecasting. I am familiar with the already published discussions, e.g. <a href="https://discuss.pytorch.org/t/using-transformer-on-timeseries/104759">this</a>. My idea is to pass some time series to a transformer - [1,2,3,4,5] and get the forecast for [6, 7, 8, 9, 10]. Note that each item (1…n) should be a vector, not a scalar because I want to use multiple indicators for finance data.</p>
<ol>
<li>I see that in many examples TransformerDecoder is not used at all and the models consist of encoder and linear final layer only. Why is that and are my thoughts that implementing a decoder structure should increase the performance right? In which cases a decoder is necessary and why the majority of samples and tutorials don’t use it?</li>
<li>When implementing a decoder one should give it targets and memory. Let’s suppose that for my use case the source to the encoder will be [1,2,3,4,5], the targets for the decoder will be [6,7,8,9,10] and the memory will be the output of the encoder. So while training it should be alright because we have both the sources and the targets in the set. But then how should one handle the predictions on new data? We do not have the targets there. But from my experience passing just zeros or ones (with positional encoding) to the decoder just confuses it more. Am I wrong to think of the targets as [6,7,8,9,10]? How should I properly implement it for training and then predicting on new data?</li>
</ol>
Loss Function for smaller values

duddal: <p>Hello,</p>
<p>I would like to discuss a topic related to Loss Functions.</p>
<p>I have an object detection model that predicts 3 things (class, bbox, and distance). I am using 2 input images for the model.</p>
<p>I would like to discss about the last predicted value, i.e. distance.  The input is the depth image which is normalized between 0 and 1. This value ranges between 0 and 1 as an output and then is multiplied by 200 in the eval stage to get the correct prediction. Currently I am using the following loss function:</p>
<pre><code class="lang-auto">dist_loss_funciton = nn.SmoothL1Loss(reduction='none')
dist_loss = torch.sum(dist_loss_funciton (pred_dist, gt_dist)) / batch_size
</code></pre>
<p>Note that here the value is predicted for each bounding box detected. I get the depth loss as 0 after a few epochs, however the values aren’t correct.</p>
<p>I think this is because the model thinks that because the output value is between 0 and 1, anything between that range is okay and is not bad.</p>
<p>How can I change this loss function so as to accommodate the aforementioned? i.e. penalize the loss function if the value is between 0.3 and 1.0, or something?</p>
Torch ImageFolder but for regression Task

JanoschMenke: <p>Hi Guys,</p>
<p>I am trying to find a solution, I am trying to perform a regression on some images.<br>
How would I go about adapting the <code>ImageFolder</code> or <code>DatasetFolder</code> to read in the numeric values rather than the classes.</p>
<p>I was thinking to just include the target value in the name of the image</p>

ptrblck: <p>The best approach would be to create a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and implement the logic to create the target (e.g. by reading them from the image names etc.) manually.<br>
<code>ImageFolder</code> uses the folder structure to create the class indices, so it wouldn’t be a good fit for your use case (changing the internals would most likely be equal to creating a custom dataset).</p>

JanoschMenke: <p>Thank you very. I was not aware of the example you linked to. I think this will be enough for me to get it to work.</p>
Size missmatch when loading lightning model

MoRoBe: <p>I use PyTorch Lightning for saving checkpoints. Unfortunately when I try to load a model from a checkpoint I get a size missmatch of the weight tensors. This is because the model.load_from_checkpoint() method instantiates a new model object but fails to infer the correct hyperparameters. Thus the layers have the wrong size when trying to load the weights.</p>
<p>I was wondering whether (and how) to load the weights saved in the checkpoint into a manually pre-instantiated model?</p>
<p>Because I use different models within the same script based on a CLI Argument passing the hyperparameters to the load_from_checkpoint() method would lead to either a bunch of duplicated code or alternatively a substantial amount of refactoring I’d like to avoid.</p>
<p>I’m grateful for any suggestions, if you need further information pls ask.</p>

MoRoBe: <p>For future reference: I ended up doing the refactoring so I could pass the params needed to correctly instantiate the model as a keyword dictionary, regardless whether I was creating a new model or loading a saved one.</p>
Does sync batch normalization in pytorch not work?

two_Two: <p>Hi,sync batch normalization has been really bothering me for a lone time  ,last time I posted a question here does not get much response .<br>
The situation is when I train with sync  batch normalization ,the training process stops…<a href="https://github.com/ultralytics/yolov5/issues/3754" rel="noopener nofollow ugc">The author of yolov5 can reproduce this issue </a><br>
And When I use pytorch version 1.6.0 ,same phenomenon occurs…<br>
I am not sure how many people meet this issue and does pytorch dev team is aware of this issue ?</p>
<p>Below I post my code of how I do DDP training :</p>
<pre><code class="lang-auto">
    args = build_args()
    local_rank = args.local_rank
    args.world_size = int(os.environ["WORLD_SIZE"])  if "WORLD_SIZE" in os.environ else 1

    #only work for ddp mode
    if local_rank != -1:
        args.total_batch_size = args.batch_size
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda",local_rank )
        dist.init_process_group(backend='nccl', init_method='env://')

        assert args.batch_size % args.world_size==0 ,"batch size shuld be multiple of number of cuda device."
        args.batch_size = args.total_batch_size // args.world_size
    
    model = MODEL()

    if local_rank != -1:
        #sync bn
        model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model).to(device)
        model.to(device)
        model = DDP(model , device_ids = [local_rank] , output_device = local_rank)
    else :
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        model.to(device)

#And  forward looks like this :  
with torch.set_grad_enabled(True):
    with torch.cuda.amp.autocast():
        outputs = model(inputs)[0]
        _, preds, = torch.max(outputs, 1)
        loss = criterion(outputs, labels)

       scaler.scale(loss).backward()
       scaler.step(optimizer)
       scaler.update()

</code></pre>
<p>Hope someone can help me out ,Thanks in advance !</p>

ptrblck: <p>I’m not aware of any issues, so could you please add the missing pieces to your code to make it executable, so that we could try to reproduce it?</p>

two_Two: <p>my code is based on yolov5 on github , and author can actually reproduce this issue , for more detail , please visit : <a href="https://github.com/ultralytics/yolov5/issues/3754" rel="noopener nofollow ugc">Training process stop when using --sync_bn</a><br>
Thanks!!!</p>

two_Two: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi,I would like to know can you reproduce this issue? Thanks</p>

ptrblck: <p>Sorry, I haven’t had a chance to look into the linked repository and check how to execute which script to try to reproduce it yet. Generally, providing a standalone minimal code snippet would be easier to use, but I also understand that it’s not possible for all user to provide these.</p>
Pytorch GPU inference Out Of Memory(OOM)

yolo_549: <p>Hello<br>
I’m stucking with this problem for about a week.<br>
Currently, i’m working on the code of Hifi-GAN official code with my own model.</p>
<p>But the main problem is that<br>
my GPU0 suddenly increases and goes out of memory when the validation process goes on.</p>
<p>I tried to set batch size as 8 and 16 but both results came out same as out of memory…</p>
<p>I would appreciate anyone who can make me free from this problem…Thank you.</p>

<h1>
<a name="validation-1" class="anchor" href="#validation-1"></a>Validation</h1>
<pre><code>            if steps % a.validation_interval == 0:  # and steps != 0:
                generator.eval()
                torch.cuda.empty_cache()
                val_err_tot = 0
                with torch.no_grad():
                    for j, batch in enumerate(validation_loader):
                        x, y, _, y_mel = batch
                        y_g_hat = generator(x.to(device))  
                        y_mel = torch.autograd.Variable(y_mel.to(device, non_blocking=True))
                        y_g_hat_mel = mel_spectrogram(y_g_hat.squeeze(1), h.n_fft, h.num_mels, h.sampling_rate,
                                                      h.hop_size, h.win_size,
                                                      h.fmin, h.fmax_for_loss)
                        val_err_tot += F.l1_loss(y_mel, y_g_hat_mel).item()

                        if j &lt;= 4:
                            if steps == 0:
                                sw.add_audio('gt/y_{}'.format(j), y[0], steps, h.sampling_rate)
                                sw.add_figure('gt/y_spec_{}'.format(j), plot_spectrogram(x[0]), steps)

                            sw.add_audio('generated/y_hat_{}'.format(j), y_g_hat[0], steps, h.sampling_rate)
                            y_hat_spec = mel_spectrogram(y_g_hat.squeeze(1), h.n_fft, h.num_mels,
                                                         h.sampling_rate, h.hop_size, h.win_size,
                                                         h.fmin, h.fmax)
                            sw.add_figure('generated/y_hat_spec_{}'.format(j),
                                          plot_spectrogram(y_hat_spec.squeeze(0).cpu().numpy()), steps)

                    val_err = val_err_tot / (j+1)
                    sw.add_scalar("validation/mel_spec_error", val_err, steps)

                generator.train()

        steps += 1

    scheduler_g.step()
    scheduler_d.step()</code></pre>
Can you recommend a library for measuring image similarity?

hix: <p>I’m using PSNR recently.</p>
<p>However, the PSNR shows a low PSNR value even though the photos are more similar.</p>
<p>I think these problems are the downsides of PSNR.</p>
<p>Can you recommend an image similarity measurement method that improves the PSNR problem?</p>

Mazhar_Shaikh: <p>In general, PSNR may not be a good metric for most natural image domains. To measure perceptual similarity between images, may be an alternative method could be the structural similarity index <a href="https://ece.uwaterloo.ca/~z70wang/publications/ssim.pdf" rel="noopener nofollow ugc">https://ece.uwaterloo.ca/~z70wang/publications/ssim.pdf</a><br>
To go even farther, you could use inception distance as a sort of semantic similarity metric, similar to how it is used in FID scoring for Image GANs, <a href="https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance#:~:text=Unlike%20the%20earlier%20inception%20score,used%20to%20train%20the%20generator" class="inline-onebox" rel="noopener nofollow ugc">Fréchet inception distance - Wikipedia</a>.</p>

hix: <p>Thank for your recommand I’m try it</p>
MSELoss() with ReluBackward0

anoud_abdalziz: <p>could you help me?<br>
when I use criterion mse loss as mse = nn.MSELoss() ,it release this error:<br>
i tried different solutions in discussions but i cannot solve it.</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [20, 1, 6, 32, 32]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>

KFrank: <p>Hi Anoud!</p>
<aside class="quote no-group" data-username="anoud_abdalziz" data-post="1" data-topic="130583" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anoud_abdalziz/48/39701_2.png" class="avatar"> anoud_abdalziz:</div>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [20, 1, 6, 32, 32]], which is output 0 of ReluBackward0</p>
</blockquote>
</aside>
<p>This is a shot in the dark, but is there any chance that you have a<br>
<code>torch.nn.ReLU (inplace = True)</code> as a “layer” in your model?<br>
If so, try changing it to <code>inplace = False</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

anoud_abdalziz: <p>thanks for responding ,i solved it by adding ‘detach’  for  mse_loss = criterion_mse(fake_img.detach(), real_img)<br>
.</p>

ptrblck: <p>For the sake of completeness: detaching a tensor to avoid an inplace operation issue is unfortunately not a solution as described <a href="https://discuss.pytorch.org/t/element-0-of-tensors-does-not-require-grad-and-does-not-have-a-grad-fn/32908/104">here</a>.</p>

anoud_abdalziz: <p>sorry , I could not understand what do you mean, Just I need MseLoss values to update the loss of the generator not to update MseLoss .<br>
g_loss = self.lambda_mse * mse_loss+dversarial_loss<br>
g_loss.backward()</p>
AMP and CUDNN_STATUS_NOT_INITIALIZED

victorc25: <p>Hello. I’m hitting a strange error and can’t really find a cause.</p>
<p>When training a pix2pix model, if AMP is disabled, ie using “with autocast(enabled=False):”, everything works fine and the model trains, no issues at all, but when enabled, the backward step: amp_scaler.scale(loss).backward() produces a “RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED” error.</p>
<p>I could identify that the issue is produced exactly where the Discriminator loss is calculated if done with autocast enabled and later the backward step has the error. Disabling autocast in just this step (while keeping it enabled everywhere else) also works fine, meaning the Generator works fine (including the GAN component) with autocast enabled.</p>
<p>Any clues what could be the cause of this?</p>

ptrblck: <p>This error message could point to an out of memory issue, since cuDNN seems to have trouble with its initialization. Could you lower the batch size and rerun the code, please?</p>
Confusion Matrix for multi-class classification problem

aleemsidra: <p>I am working on a 4-class classification problem. I have got some results using ResNet. I want to evaluate the results further by a confusion matrix. I have seen sklearn has the method to calculate it. Since I am using CNN, which gives results for multiple epochs, how can I plot the confusion matrix for the best epoch?</p>

ptrblck: <p>Usually you would calculate the validation loss and accuracy after each epoch and could also store the predictions for the entire validation dataset and calculate the confusion matrix afterwards.<br>
If you want to only compute the confusion matrix for the “best” model model, store the <code>state_dict</code> of the model if the validation loss decreases (or the validation accuracy increases), reload the <code>state_dict</code> after the training is finished, and calculate the confusion matrix afterwards.</p>

aleemsidra: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> so just like accuracy and validation loss is calculated for every epoch, other evaluation measures like <strong>confusion matrix</strong>, F1-score, recall, precision are also calculated for every epoch? Or do we calculate these measures only for the best model?</p>

ptrblck: <p>I don’t think there is any rule and it depends on your use case, i.e. the calculation of each of these metric stats will of course slow down your code a bit and you would have to decide, if you want and need to track these additional stats for each epoch.<br>
The “slowdown” of course also depends on the general use case. E.g. if an epoch already takes ~1h to finish, you wouldn’t notice the additional calculations even if they would take ~1min (I would assume they are faster as well).</p>
Does this LSTM loop code break the computational graph in PyTorch?

Henry_Chinner: <p>The code below is from <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Sequence Models and Long Short-Term Memory Networks — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<pre><code class="lang-auto">for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)
</code></pre>
<p>For me, it seems like handling the LSTM in this way breaks the computational graph as hidden keeps on getting overridden. Should all the hidden states not be stored in an array so the computational graph can be maintained, so backprop can flow through the hidden states?</p>

googlebot: <p>it is not overwritten with python’s semantics; the function returns a tuple, it is deconstructed, and the <em>name</em> ‘hidden’ is reassigned, hiding the old object referenced by that name in current scope</p>
Tensor Creation uses 0.5 MB of GPU memory?

mr-mojo: <p>I recently stumbled upon something I don’t understand: When creating a float-tensor on GPU with only 1 element, I would assume it to take up 4 bytes of memory. However, torch.cuda.max_memory_allocated() returns 512 bytes? Why is that?</p>
<p>MWE to replicate:</p>
<pre><code class="lang-auto">import torch 
print(torch.cuda.max_memory_allocated())
a = torch.tensor(1.0, device='cuda')
print(torch.cuda.max_memory_allocated())   # 512 bytes 
print(torch.cuda.max_memory_reserved())    # 2097152 bytes = 2 MB 
</code></pre>
<p>I’m aware that the CUDA context must be created on the GPU as well, which is why nvidia-smi shows values much higher than 0.5MB, around 1GB, but that’s not what I’m asking here. Why does PyTorch reserve half a MB for a 4byte tensor, and why does it cache 2MB when creating said tensor?</p>
<p>Following <a href="https://stackoverflow.com/questions/54361763/pytorch-why-is-the-memory-occupied-by-the-tensor-variable-so-small" rel="noopener nofollow ugc">this answer</a> on Stackoverflow, I did</p>
<pre><code class="lang-auto">import sys 
print(sys.getsizeof(a))            # 64
print(sys.getsizeof(a.storage())   # 60
</code></pre>
<p>which unfortunately is even more confusing. Anybody knows what’s going on here?</p>

googlebot: <p>that’s because pytorch manages its own buffers, you don’t want to do 4 byte allocations from “system” memory manager</p>

mr-mojo: <p>Thanks, any resources on where I can read up on this?</p>

googlebot: <p>not sure, that’s rather a common practice for high performance c++ programming - using specialized allocators tuned for program’s allocation patterns. with cuda’s limited memory it is even more essential, mostly to decrease fragmentation.</p>
<p>but, yeah, this happens to be documented - <a href="https://pytorch.org/docs/master/notes/cuda.html#memory-management" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch master documentation</a></p>
Measure the inference performance for multi-tasking

zzsunshine: <p>Hi all,</p>
<p>I want to measure the GPU inference time for multi-tasking on a single GPU. I used the code here:</p>
<pre><code class="lang-auto">with torch.no_grad():
    	starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
        starter.record()      
        output = model(input_batch)
        ender.record()
                
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)
</code></pre>
<p>However, cuda.synchronize seems to synchronize across GPU. But I want to measure the GPU inference time for each of them separately. How to do synchronization for each one and measure the actual GPU inference time?</p>
<p>Appreciate for any help.</p>

ptrblck: <p><code>torch.cuda.synchronize</code> accepts a <code>device</code> argument as seen in the <a href="https://pytorch.org/docs/stable/generated/torch.cuda.synchronize.html">docs</a>. Also the <code>Event</code> object provides a <code>synchronize()</code> method in case you want to use it.<br>
PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

zzsunshine: <p>Thanks for reply.  torch.cuda.synchronize accepts a  device  argument. However, I co-run multi-tasking on the same single GPU, the device will be only one. I am not sure if I can use torch.cuda.synchronize under co-running condition.</p>
<p>For event object,  it seems that torch.cuda.Event.synchronize() does not works here.</p>
<pre><code class="lang-auto">with torch.no_grad():
    	starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
        starter.record()      
        output = model(input_batch)
        ender.record()    
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)</code></pre>

ptrblck: <p>I’m not sure how you are “co-running”, but in case you are using streams you might want to sync them.</p>

zzsunshine: <p>Thanks for your help!</p>
Question of assigning optimizer param

jhp: <p>Hi, I have a quick question about parameters passed into the optimizer.<br>
I knew that only the parameters assigned in the optimizer were updated by gradients, but after the optimizer was defined and start training, I make the parameters that were not assigned to the optimizer to compute gradient(requires_grad=True), which caused a huge change in performance.<br>
In this situation, does the parameter not assigned to the optimizer updated? All does that parameter affects other assigned parameter when updating?</p>

tom: <p>The training step parts of “run backward” and “optimizer step” are independent in PyTorch, so the first computes gradients for everything that has <code>requires_grad=True</code> and is ignorant of whether you use them in the optimizer step. So you would keep allocating gradients for the unused parameters but never use (nor zero) them.</p>
<p>Best regards</p>
<p>Thomas</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (28x28 and 784x128)

vimal4353: <p>Hello all,<br>
I am newbie to torch …I am getting this error while passing the new data to the model</p>
<p>"Model "<br>
Sequential(<br>
Linear(in_features=784, out_features=128, bias=True)<br>
ReLU()<br>
Linear(in_features=128, out_features=64, bias=True)<br>
Tanh()<br>
Linear(in_features=64, out_features=10, bias=True)<br>
LogSoftmax(dim=1)<br>
)</p>

KFrank: <p>Hi Vimal!</p>
<aside class="quote no-group" data-username="vimal4353" data-post="1" data-topic="130565" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vimal4353/48/41740_2.png" class="avatar"> vimal4353:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">   Linear(in_features=784, out_features=128, bias=True)
</code></pre>
</blockquote>
</aside>
<p>The first layer of your model is a <code>Linear</code> with <code>in_features = 784</code>.  It.<br>
therefore expects to be passed a batch of 1-d vectors of length <code>784</code>.<br>
(More precisely, a <code>Linear</code> expects a tensor whose last dimension is<br>
<code>in_features</code> and that has any number of leading “batch” dimensions,<br>
including zero.  So you can think of this as being a potentially<br>
multidimensional “batch” of 1-d vectors, if you will.)</p>
<p>It looks like you are passing in a image with <code>width = height = 28</code>.<br>
You need to <a href="https://pytorch.org/docs/1.9.0/generated/torch.flatten.html" rel="noopener nofollow ugc">flatten()</a> your 28x28 image into a length-784 vector (or,<br>
more generally, <code>flatten()</code> the last two dimensions of your batch of<br>
images) before passing it to your <code>Linear</code>.</p>
<p>Note, you can include the flattening operation as a “layer” in your<br>
model by using the <code>Module</code> (class or function-object) version, <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.Flatten.html" rel="noopener nofollow ugc">Flatten</a>.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; batch_of_3_images = torch.randn (3, 28, 28)
&gt;&gt;&gt; batch_of_3_images.shape
torch.Size([3, 28, 28])
&gt;&gt;&gt; lin = torch.nn.Linear (in_features = 784, out_features = 128, bias = True)
&gt;&gt;&gt; flat = torch.nn.Flatten (start_dim = -2)
&gt;&gt;&gt; flat_batch = flat (batch_of_3_images)
&gt;&gt;&gt; flat_batch.shape
torch.Size([3, 784])
&gt;&gt;&gt; lin (flat_batch).shape
torch.Size([3, 128])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

vimal4353: <p>Thanks much Frank !..I will try</p>
Expected torch.Size([4, 1, 40, 99]) but getting torch.Size([4, 1, 257, 197])

asha97: <p>Here 4 is the batch size and I am trying to extract audio , hence channel is 1.</p>
<p>audio extractor</p>
<pre><code class="lang-auto">audio, audioR, audio2, audio2R = torch.split(spec, [1, 1, 1, 1], dim=1)
</code></pre>
<p>required value is audio</p>
<p>decoder</p>
<pre><code class="lang-auto">vid_path, 
    container, 
    sampling_rate, 
    num_frames, 
    clip_idx, 
    num_clips=10, 
    target_fps=30, 
    aug_audio=[1, 1, 3, 6], 
    decode_audio=True,
    num_sec=1,
    aud_sample_rate=48000,
    aud_spec_type=1,
    use_volume_jittering=False,
    use_temporal_jittering=False,
    z_normalize=False,


fps = float(container.streams.video[0].average_rate)
    frames_length = container.streams.video[0].frames
    duration = container.streams.video[0].duration

    if duration is None:
        # If failed to fetch the decoding information, decode the entire video.
        decode_all_video = True
        video_start_pts, video_end_pts = 0, math.inf
    else:
        # Perform selective decoding.
        decode_all_video = False
        start_idx, end_idx = get_start_end_idx(
            frames_length,
            sampling_rate * num_frames / target_fps * fps,
            clip_idx,
            num_clips,
        )
        timebase = duration / frames_length
        video_start_pts = int(start_idx * timebase)
        video_end_pts = int(end_idx * timebase)

    frames = None
    # If video stream was found, fetch video frames from the video.
    if container.streams.video:
        video_frames, max_pts = pyav_decode_stream(
            container,
            video_start_pts,
            video_end_pts,
            container.streams.video[0],
            {"video": 0},
        )
        container.close()

        frames = [frame.to_rgb().to_ndarray() for frame in video_frames]
        frames = torch.as_tensor(np.stack(frames))

    # Get wav
    if decode_audio:
        try: 
            # Get spectogram
            fr_sec = start_idx / fps
            spec = load_audio(
                vid_path, 
                fr_sec, 
                num_sec=num_sec, 
                sample_rate=aud_sample_rate, 
                aug_audio=aug_audio, 
                aud_spec_type=aud_spec_type, 
                use_volume_jittering=use_volume_jittering,
                use_temporal_jittering=use_temporal_jittering,
                z_normalize=z_normalize,
            )
        except:
            print(f"Bad audio of video: {vid_path}", flush=True)
            if spec is not None:
                print(f"Bad spec shape of {vid_path}: {spec.shape}", flush=True)
            if wav is not None:
                print(f"Bad wav shape of {vid_path}: {wav.shape}", flush=True)
            return None, None, None, None
        return frames, spec, fps, decode_all_video
    else:
        return frames, None, fps, decode_all_video



</code></pre>
KL Divergence calculation

Nil_MSh: <p>I want to calculate the kl divergence for two probability distributions. but one is a tensor of size (64, 936, 32, 32) and the other is (64, 939, 32, 32). as you can see the difference is small. how can I make them the same size without ruining the data and kl divergence value.<br>
I tried adding 0s but it doesn’t work.</p>

AlphaBetaGamma96: <p>Could you explain what each of the dimensions represent?</p>
<p>Also, if I’m not mistaken, the KL divergence term contains the log of the probability distribution so if you were to add in zeros you’re effectively adding in negative infinity to your data which is going to cause a problem when backpropagating your loss.</p>

Nil_MSh: <p>well originally (1,64,32,32) was the activation of one layer of my resnet model. 1= batchsize, 64 = neurons in the layer. and I saved it for each 936 pics that I gave to my model. I reshaped and permuted that for another purpose and so now I have a tensor of size (64,939,32,32). it’s the same with the other tensor.<br>
How can I reshape one of them and which one should I reshape?<br>
What if I was to remove the excess data and make them both of size (64, 936, 32, 32)? how should I do it? I think in my case it is wouldn’t hurt the purpose.</p>

AlphaBetaGamma96: <p>So, from my understanding you have 2 different Tensors with different batchsizes? Is that correct?</p>

Nil_MSh: <p>yes. you could say that</p>

AlphaBetaGamma96: <p>Hmmm, that’s odd that your batch dimension gets changed via the function. Are you sure it’s not meant to change a different dimension of your Tensor?</p>
<p>Would you mind explain what that function does?</p>
<p>Ideally, your loss function should take 2 Tensors with the same batch dimension as you’re effectively measuring the difference between 2 probability distributions with your data.</p>

Nil_MSh: <p>it’s a bit complicated. I’m experimenting something!<br>
what can be done about the tensor reshaping? how do I delete data? I can permute it so it’ll be (64, 32, 32, 939) and then delete data to make it 936. but how do I delete index of a tensor in pytorch?</p>

AlphaBetaGamma96: <p>With regards to deleting particular indices of an array, you could just redefine the array to be the old over the given indices, i.e.,</p>
<pre><code class="lang-auto">data = torch.randn(64,939,32,32)
data.shape #returns torch.Size([64, 939, 32, 32])
data = data[:,0:936,:,:]
data.shape #retuns torch.Size([64, 936, 32, 32])
</code></pre>
<p>One thing that came to mind was, could you not apply this custom function (which changes the batchsize) after you calculate the KL-divergence? So, I assume you have 2 Tensors which represent <code>prediction</code> and <code>targets</code>? Could you calculate the KL-divergence between the two, then apply your custom function afterward? and include the extra 3 batch value in some other way?</p>
Max_pool2d(): argument 'input' (position 1) must be Tensor, not ReLU

Aditya_Raj_P15VSS201: <p>I am running this neural Network</p>
<p>class MultiLabelNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MultiLabelNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3,64, 5)<br>
self.pool = nn.MaxPool2d(2,2)<br>
self.conv2 = nn.Conv2d(64, 128, 5)<br>
self.conv3 = nn.Conv2d(128, 256, 5)<br>
self.conv4 = nn.Conv2d(256,320,5)<br>
self.fc1 = nn.Linear(250880,2048)<br>
self.fc2 = nn.Linear(2048, 1024)<br>
self.fc3 = nn.Linear(1024, 512)<br>
self.fc4 = nn.Linear(512, 6)<br>
def forward(self, x):<br>
x = self.conv1(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv2(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv3(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv4(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = x.view(-1, 250880)<br>
x = self.fc1(x)<br>
x = self.fc2(x)<br>
x = self.fc3(x)<br>
x = self.fc4(x)<br>
return x</p>
<p>but getting error</p>
<p>TypeError: max_pool2d(): argument ‘input’ (position 1) must be Tensor, not ReLU</p>
<p>Please help me. I am not able to understand what is wrong here</p>

JuanFMontesinos: <p>nn.ReLu is a class, not a function.<br>
When you do <code>x=nn.ReLu(x)</code> you are instantiating the class <code>nn.ReLu</code>, not computing a relu. You can either replace nn.ReLu by it’s corresponding <code>nn.functional.relu</code> or to instantiate the activation in the init:<br>
<code>self.relu=nn.ReLu()</code> and replace <code>x = nn.ReLU(x)</code> by ``x=self.relu(x)</p>

Aditya_Raj_P15VSS201: <p>Thank you so much for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>