akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Semantic Segmentation - Error: 0D or 1D target tensor expected, multi-target not supported

SimCan: <p>I would like to know why I get this error using <code>CrossEntropyLoss()</code> for the semantic segmentation task.<br>
Inputs have shape <code>[B, C, W, H]</code>, and targets have shape <code>[B, W, H]</code>.<br>
Target is <em>not</em> one-hot encoded.</p>
<p><strong>Error</strong>:</p>
<pre><code class="lang-auto">   3012     if size_average is not None or reduce is not None:
   3013         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 3014     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   3015 
   3016 

RuntimeError: 0D or 1D target tensor expected, multi-target not supported
</code></pre>

ptrblck: <p>I guess your model output has another shape than reported here, as your shapes would work:</p>
<pre><code class="lang-python">B, C, H, W = 2, 3, 4, 4

output = torch.randn(B, C, H, W, requires_grad=True)
target = torch.randint(0, C-1, (B, H, W))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target) # works

output = torch.randn(B, C)
loss = criterion(output, target)
# RuntimeError: 0D or 1D target tensor expected, multi-target not supported
</code></pre>

SimCan: <p>I double-checked the model output and it has shape [B, C, H, W], in my case [2, 3, 256, 256].<br>
The “target” instead previously had shape [B, C, H, W], after I applied <code>torch.squeeze(dim=1)</code>, it has shape [B, H, W] → [2, 256, 256]<br>
I still do not understand why I get that kind of error</p>
When do we call the prune.remove() function when using the pruning toolbox to prune a cnn?

uzair789: <p>Hi,</p>
<p>I am using the pruning tool box to prune a resnet18. I see that there is a function called prune.remove() which aims to make the pruning permanent. When should this function be called ideally?</p>
<p>If i train a model, and then perform l1 structured pruning, make the pruning permanent by calling the prune.remove() and then finetune, the performance of the model is differnet from if i train the model, perform l1 structured pruning, fine tune and then make the pruning permanent right before deployment. I experimented with pruning_amount of 0.7, and the former method’s value is almost equal to the baseline unpruned model while the latter’s performance drops significantly which I think should be the case since we lost 70% of the filters in each layer.</p>
<p>Can somebody please shed some light on this. I have my pruning code below</p>
<pre><code class="lang-auto">    prune_strategy = prune.ln_structured
    addn_params = {'n':1, 'dim':1}
    path_to_checkpoint = 'results/{}/checkpoint_200.pth'.format(checkpoint_folder)
    checkpoint = torch.load(path_to_checkpoint)
    #model.load_state_dict(checkpoint['model_state_dict'])
    model.load_state_dict(checkpoint)
    print("Model loaded successfully : ", path_to_checkpoint)
    num_params_before, num_params_after = 0, 0
    for name, module in model.named_modules():
        # print(name, module)
        # not pruning fc layers
        if isinstance(module, torch.nn.Conv2d):
            prune_strategy(module, name='weight', amount=args.prune_amount, **addn_params)
            prune.remove(module, name='weight')
            # if I run the prune.remove here while the layers are being pruned, the performace of the model 
            # is as good as the baseline. If i comment it out, the performance drops.
</code></pre>
<p>Thank you,<br>
Uzair</p>

ube: <p>Hi there,<br>
I am working on pruning too. And I find this walkthrough helpful.<br>
check it out <a href="https://www.youtube.com/watch?v=bQt0CLXXAqg&amp;t=353s" class="inline-onebox" rel="noopener nofollow ugc">The Lottery Ticket Hypothesis and pruning in PyTorch - YouTube</a></p>
Replacing all ReLU(inplace=True) with ReLU(inplace=False) for all pretrained models

razla: <p>Hey,</p>
<p>I want to replace all ReLU(inplace=True) with ReLU(inplace=False) - but I want it to be generic for all pretrained models - vgg16, resnet, etc.</p>
<p>Is it possible? since each model has different architecture and thus different hierarchy.</p>
<p>Thank you!</p>

ptrblck: <p>I think you should be able to use <code>torch.fx</code> with its ability to manipulate the graph as described <a href="https://pytorch.org/docs/stable/fx.html#direct-graph-manipulation">here</a>. In the example they are replacing <code>add()</code> with <code>mul()</code> calls and I assume you can use the same or similar approach to replace the <code>ReLU</code> modules.</p>

razla: <p>Thank you, but this solution doesn’t work - maybe I need to make it recursive…<br>
It tries to find the function calls, but there might be these function calls within a sequential layer or any other embedded layer…</p>
How do I stop updates to a part of BN's running_mean and running_var buffers?

Qwwq: <p>I am training a model in which only a part of the BN’s weights and biases are updated. I can do this by setting the gradients of the ‘frozen’ part to zero, during the backward pass. However, the <code>running_mean</code> and <code>running_var</code> buffers still get ‘updated’ with every iteration. How do I stop this from happening?</p>
<p>Thanks.</p>

ptrblck: <p>Call <code>.eval()</code> on the batchnorm layer(s) to disable the updates of the running stats in each forward pass.</p>

Qwwq: <p>Pardon my lack of clarity in the original post. The situation I’m in is something like this. I have a network that contains a BN layer say:</p>
<p><code>bn = nn.BatchNorm2d(64)</code></p>
<p>So the layer above has 64 features.<br>
<code>bn.weight.shape # 64</code><br>
Also, <code>bn.running_mean.shape # 64</code></p>
<p>Now, while training I want the first 8 elements of the <code>bn</code> layer to not get updated and update the remaining 56 of them. I can easily achieve that on the <code>bn.weight</code> tensor by setting the gradients of the first 8 elements in the backward pass to 0 using <code>torch.register_hook</code> on the tensor generated by <code>bn.weights</code>.</p>
<p>But I can’t stop <code>bn.running_mean</code>’s first 8 elements from not being updated. Every time there is a <code>forward</code> call on the <code>bn</code> layer, all of <code>bn.running_mean</code>’s elements get updated.</p>
<p>What I want is to stop <code>bn.running_mean</code> from updating the first 8 elements and have the rest of the 56 elements update as usual. How do I achieve this?</p>
<p>Thanks.</p>

Qwwq: <p>I am currently the following layer implementation (that supports partial updates to <code>running_mean</code> and <code>running_var</code> through the <code>partial_features</code> variable.</p>
<pre><code class="lang-auto">class PartialBatchNorm2d(nn.BatchNorm2d):
    def __init__(
        self,
        num_features,
        eps=0.00001,
        momentum=0.1,
        affine=True,
        track_running_stats=True,
        partial_features=0,
    ):
        super().__init__(
            num_features=num_features,
            eps=eps,
            momentum=momentum,
            affine=affine,
            track_running_stats=track_running_stats,
        )
        self.register_buffer('partial_features', torch.zeros((1), dtype=torch.int32))
        self.partial_features[0] = partial_features

    def forward(self, x):
        running_mean_copy = self.running_mean.data.clone()
        running_var_copy = self.running_var.data.clone()

        x = super().forward(x)

        if self.training and self.track_running_stats:
            self.running_mean.data[: self.partial_features] = running_mean_copy[: self.partial_features]
            self.running_var.data[: self.partial_features] = running_var_copy[: self.partial_features]

        return x

</code></pre>
Calculating the gradient of a sparse Tensor argument to mm is not supported

Mandeep-Rathee: <p>In my forward pass, I am doing element-wise multiplication of sparse and dense matrix. I am not using “*” for element-wise multiplication. I have defined another function for sparse dense matrices element-wise multiplication. I am getting this error while doing backpropagation. Can someone please help me.</p>
<p>thanks in advance</p>
<p>raceback (most recent call last):<br>
File “train.py”, line 107, in <br>
train(epoch)<br>
File “train.py”, line 75, in train<br>
loss_train.backward()<br>
File “/home/rathee/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/rathee/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 97, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: calculating the gradient of a sparse Tensor argument to mm is not supported.</p>

lyf: <p>who can solve this problem？ this is very difficult for me.thanks a lot.</p>
Update a CNN model's weights manually

Z_Huang: <p>If I want to update a CNN model’s weights manually, could it be done by simply changing and modifying the model.state_dict()'s weight values?</p>

Karthik_Ganesan: <p>Yes, you can just edit the weights directly. Load the model state dict into a new dict, change the values there and then load that state dict. But you have to be careful doing this during training. But if you do this for a trained model and you want to modify the weights just for inference its safer to do that.</p>

Z_Huang: <p>thanks. what might be the recommended functions to be used for changing the state_dict() weight values? do I have to set the param.requires_grad = False in order to update the weight values in state_dict()?</p>

Karthik_Ganesan: <p>You can just create a new ordered dict and copy the <code>state_dict()</code> to that. You can then edit the new dict and then load it to the model using the usual <code>load_state_dict()</code> function.</p>
<p>No, you don’t have to set param.requires_grad = False explicitly. But if you are only changing the weights for inference and not training, then you can just do a <code>model.eval()</code> first.</p>
Change Midas depth image by editing input

kokawa2003: <p>Now ,I am checking  <a href="https://pytorch.org/hub/intelisl_midas_v2/" rel="noopener nofollow ugc">Midas</a><br>
and watching its output depth image.</p>
<p>I want to change its some unclear output depth image more clear  by some function(OpenCV …)<br>
But I cannot succeed it .<br>
I want to know how to emphasize its output by editing its input image<br>
Do you know how to do it?</p>
Creating April Tag

Joeojazz: <p>Hello I am trying to track April Tags with OAK-D-Lite and RPi4 using a PyTorch Model. The reason I would like to use PyTorch instead of OpenCV is OAK-D-Lite requires it own unique file type which I can convert to from PyTorch model (<a href="https://docs.luxonis.com/en/latest/pages/tutorials/creating-custom-nn-models/#run-your-own-cv-functions-on-device" class="inline-onebox" rel="noopener nofollow ugc">Run your own CV functions on-device — DepthAI documentation | Luxonis</a>). The RPi4 is to slow only able to track at around 20FPS the goal is to be closer to 60FPS. Any suggestions on how I should go about this? My best guess right now is to dig through April tags C code (located <a href="https://github.com/AprilRobotics/apriltag" class="inline-onebox" rel="noopener nofollow ugc">GitHub - AprilRobotics/apriltag: AprilTag is a visual fiducial system popular for robotics research.</a>) and change it to PyTorch.</p>
Sanity check for my first LSTM model

Nyakov: <p>Hi. I am trying to create my first LSTM model for timeseries forecasting.<br>
There is things that I don’t really understand.<br>
Is it true that one LSTM cell created and input applied sequential to it sample by sample?<br>
And num_layers controls how many LSTM cells created?<br>
Why we using hidden state as output instead of actual output?<br>
Am i correctly extracting hidden state of last layer? (hn[self.num_layers-1])</p>
<p>Any advises for my code? Am I doing something wrong?</p>
<pre><code class="lang-auto">class ShallowRegressionLSTM(nn.Module):
    def __init__(self, num_features, hidden_units, num_layers, device):
        super().__init__()
        self.num_features = num_features  # this is the number of features
        self.hidden_units = hidden_units
        self.num_layers = num_layers
        self.device = device

        self.lstm = nn.LSTM(
            input_size=num_features,
            hidden_size=hidden_units,
            batch_first=True,
            num_layers=self.num_layers
        )

        self.linear = nn.Linear(in_features=self.hidden_units, out_features=1)

    def forward(self, x):
        # Reset hidden state for current batch
        batch_size = x.shape[0]
        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_units, device=self.device).requires_grad_()
        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_units, device=self.device).requires_grad_()

        # Do forward pass
        _, (hn, _) = self.lstm(x, (h0, c0))
        out = self.linear(hn[self.num_layers-1]).flatten()  # First dim of Hn is num_layers

        return out
</code></pre>
Are the hidden_state,cell_state reset for each element/time-step of LSTM?

Gabriele_Gennaro: <p>I have this configuration of my encoder :</p>
<ol>
<li>
<p>class Encoder_LSTM(nn.Module):</p>
</li>
<li>
<pre><code>def __init__(self,dict_config = dict_config):
</code></pre>
</li>
<li>
<pre><code>    super(Encoder_LSTM, self).__init__()
</code></pre>
</li>
<li>
<pre><code>    self.input_size = dict_config["input_dim"]
</code></pre>
</li>
<li>
<pre><code>    self.hidden_size = dict_config["hid_dim"]
</code></pre>
</li>
<li>
<pre><code>    self.output_size = dict_config["output_dim"]
</code></pre>
</li>
<li>
<pre><code>    self.num_layer = dict_config["num_layer"]
</code></pre>
</li>
<li>
<pre><code>    self.Encoder_LSTM = nn.LSTM(self.input_size, self.hidden_size , batch_first=True , num_layers=self.num_layer , dropout=0.2 )
</code></pre>
</li>
<li>
<pre><code>def forward(self, input):
</code></pre>
</li>
<li>
<pre><code>    self.batch_size = input.shape[0]
</code></pre>
</li>
<li>
<pre><code>    self.hidden_init = torch.zeros(self.num_layer, self.batch_size, self.hidden_size, device=device)
</code></pre>
</li>
<li>
<pre><code>    self.h_0 = self.hidden_init
</code></pre>
</li>
<li>
<pre><code>    self.c_0 = self.hidden_init
</code></pre>
</li>
<li>
<pre><code>    self.output,  (h,c) = self.Encoder_LSTM(input, (self.h_0, self.c_0))
</code></pre>
</li>
<li>
<pre><code>    return self.output
</code></pre>
</li>
</ol>
<p>I think I am a bit confused about the “forward” method. It seems like every time you call it, it initialize the hid_state/cell_state. Now,  I was wandering , when you have a sequence , is the forward method call on each timestep? If yes, wouldn’t it mean that for each time step the hid_state/cell_state are reset ?</p>
LSTM produces zero outputs

Laya1: <p>I have a very simple model that has a <code>torch.nn.LSTM</code> that takes in a sequence of length <code>30</code> and produces a sequence of length <code>30</code>. The <code>LSTM</code> is followed by two <code>Linear</code> layers with <code>ReLU</code> activation to produce the final predictions of length <code>30</code>. The number of features in both input and output of <code>LSTM</code> is <code>1</code>. The <strong>predictions have intermittent zero values, while the target at the time of training is never zero</strong> (I use <code>[0, 1]</code> normalization, and then replace the zeroes by <code>1e-4</code>).</p>
<p>I have tried changing:</p>
<ol>
<li>optimizer</li>
<li>depth</li>
<li>number of features in LSTM cells</li>
</ol>
<p>and regardless, the output has zeroes at some places in the predictions. Why is this happening and <strong>how do I ensure that the output of the model is never zero</strong> (because the physical quantity I am predicting cannot be zero)?</p>
What does this parameter mean?(torch.ones_like(inp)

Chris36839: <p>inp = torch.eye(5,requires_grad=True)<br>
out = (inp+1).pow(2)<br>
out.backward(<strong>torch.ones_like(inp)</strong>, retain_graph=True)<br>
print(f"First call\n{inp.grad}")</p>

ptrblck: <p><code>torch.ones_like(inp)</code> will create a tensor in the same shape and <code>dtype</code>, <code>device</code>, and <code>layout</code>  as <code>inp</code> filled with <code>1</code>s and is used as the <code>gradient</code> argument to the <code>backward</code> call.</p>
Will different weight initialization matter that much?

jzy95310: <p>The problem is quite simple. I’m trying to fit an ensemble of 2-layer MLPs with all fully connected layers to the function y = sin(x) with 5 training data points (marked as red dots in the figures below). When I use the default initialization strategy (which, I believe, is a type of uniform initialization) for weights and biases, the model can almost perfectly fit to the training data points. However, when I change the initialization strategy to Gaussian initialization, the model can only learn a constant mean. Does the weight initialization matter that much in this case? I personally don’t think uniform or Gaussian initialization has such a large difference.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/621447ac8e9828dae6e95abbf5db6cc1672a9b90.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/621447ac8e9828dae6e95abbf5db6cc1672a9b90" title="Screen Shot 2022-09-22 at 10.24.32 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/621447ac8e9828dae6e95abbf5db6cc1672a9b90_2_690x269.jpeg" alt="Screen Shot 2022-09-22 at 10.24.32 PM" data-base62-sha1="dZEeZiHOVmLY3BYEtxUNOjwrrP2" width="690" height="269" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/621447ac8e9828dae6e95abbf5db6cc1672a9b90_2_690x269.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/621447ac8e9828dae6e95abbf5db6cc1672a9b90_2_1035x403.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/621447ac8e9828dae6e95abbf5db6cc1672a9b90_2_1380x538.jpeg 2x" data-dominant-color="F2F0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-09-22 at 10.24.32 PM</span><span class="informations">2268×886 155 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Transformer give zero train and val loss in just 3-4 epochs!

Gaurav_Sharma: <p>I am trying to build an architecture (CNN+transformer), but it looks like the transformer is doing way too good, as it takes only 3 epochs to zero the losses. My assumption is that in some way, it is able to see the entire sequence at once, but I am not able to figure it out as I am providing “tgt_mask”.</p>
<pre><code class="lang-auto">import math
import torch
import torch.nn as nn
from models.encoding.positional_encoding_for_xfmer import PositionalEncoding


class Transformer(nn.Module):

    def __init__(self, output_dim, dec_hid_dim, max_len, nheads, n_encoder_layers,
                    n_decoder_layers, dropout, device):

        super(Transformer, self).__init__()
        self.pos = PositionalEncoding(dec_hid_dim, dropout, max_len)
        self.device=device
        self.output_dim = output_dim
        self.dec_hid_dim = dec_hid_dim
        self.embed = nn.Embedding(output_dim, dec_hid_dim)
        self.xfmer = nn.Transformer(
                                d_model=dec_hid_dim,
                                nhead=nheads,
                                num_encoder_layers=n_encoder_layers,
                                num_decoder_layers=n_decoder_layers,
                                dropout=dropout,
                                )
        self.modify_len = nn.Linear(330, max_len)
        self.out = nn.Linear(dec_hid_dim, output_dim)

    def _generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def forward(self, src, trg, sos_idx, pad_idx, is_test):
        # src : (B, L, dec_hid_dim)
        src = self.modify_len(src.permute(0,2,1)).permute(2,0,1)
        src = src * math.sqrt(self.dec_hid_dim)
        trg = trg.permute(1,0)  # (max_len, B)
        trg = self.embed(trg) * math.sqrt(self.dec_hid_dim)
        src = self.pos(src) # (max_len, B, dec_hid_dim)
        trg = self.pos(trg) # (max_len, B, dec_hid_dim)
        trg_mask = self._generate_square_subsequent_mask(trg.shape[0]).to(self.device)

        xfmer_dec_outputs = self.xfmer(src, trg, tgt_mask=trg_mask)
        xfmer_dec_outputs = self.out(xfmer_dec_outputs)

        return xfmer_dec_outputs.permute(1,0,2)

</code></pre>
<p>Loss plot:</p>
<p><strong><img src="https://lh4.googleusercontent.com/RChH1R9_1mDgzlRHJmzFWqTbk_0MdRlJ0qv1ZNA5FJiXGXz6LZJnDlCmm1SgJkyzFILqu8_3UUxDyYd0KCmG3tjVdKKKuun9UTwN5WZt72_HVOiESUP831n3DrhMdk5OPlw8yzwl2KIpW0O1iAfyQ_GqIF2rPZFKFEVx9rQ-XVb8L6Bch_8qQJ_bag" alt="" width="251" height="212"></strong></p>
<p>Any suggestion will be helpful. Thank you in advance!</p>
Inconsistent relative matmul times across machines

lshamis_fb: <p>Hey Folks, I was trying to time-test the benefit of matrix factorization and have gotten wildly different results across machines.</p>
<pre><code class="lang-auto">import timeit
import torch

x = torch.rand(1000)

w = torch.rand([5000, 1000])

a = torch.rand([5000, 100])
b = torch.rand([100, 1000])

print("wx   ", timeit.timeit(lambda: w@x, number=10000))
print("a(bx)", timeit.timeit(lambda: a@(b@x), number=10000))
</code></pre>
<p>On machine A:</p>
<pre><code class="lang-auto">wx    1.0115966480225325
a(bx) 0.8654864309355617

</code></pre>
<p>On machine B:</p>
<pre><code class="lang-auto">wx    2.858995377959218
a(bx) 0.16533887601690367
</code></pre>
<p>On machine C:</p>
<pre><code class="lang-auto">wx    2.5966870239935815
a(bx) 2.271036416757852
</code></pre>
<p>On machine D:</p>
<pre><code class="lang-auto">wx    1.2024586703628302
a(bx) 4.790976291522384
</code></pre>
<p>These results are consistent and reproducible per machine.</p>
<p>I care about the relative timing per machine, not the absolute. One machine might be faster, but should still scale relative to the size of the matrix.</p>
<p>The measurements are not affected by pytorch version, having tried 1.5.0 and 1.12.1<br>
And the dtype is float32 and device is cpu, across the machines.</p>
<p>Any idea why this is so machine dependent?</p>
<p>Thanks in advance!</p>
CPU inference causes OOM with repeated calls to forward

cjjgrj: <p>I have some libtorch code that is doing inference on the cpu using a model trained in pytorch that is then exported to torchscript. The code below is a simplified version of a method that is being repeatedly called.</p>
<p><code>m_model</code> is the .ts file loaded via:</p>
<pre><code class="lang-auto">    m_model = torch::jit::load(path);
    m_model.eval();
</code></pre>
<p>Every call it seems that more of the torch graph is being allocated, and it isn’t being freed causing the program to eventually OOM and crash. Commenting out the forward call causes the the memory usage to stabilize.</p>
<p>My understanding is that InferenceMode guard should turn off autograd memory buildup which seems to be the normal cause of these issues.</p>
<pre><code class="lang-auto">void Backend::perform(std::vector&lt;float *&gt; in_buffer,
                      std::vector&lt;float *&gt; out_buffer) {
  c10::InferenceMode guard;

  at::Tensor tensor_out;
  at::Tensor tensor_in = torch::zeros({ 1, 16, 2 }); 
  std::vector&lt;torch::jit::IValue&gt; inputs = { tensor_in };

  // calling forward on the model "decode," this is where 
  // the memory leak happens
  tensor_out = m_model.get_method("decode")(inputs).toTensor(); 
  
  auto out_ptr = tensor_out.contiguous().data_ptr&lt;float&gt;();

  for (int i(0); i &lt; out_buffer.size(); i++) {
    memcpy(out_buffer[i], out_ptr + i * n_vec, n_vec * sizeof(float));
  }
}
</code></pre>
<p>I tried mimicking this in pytorch (by repeatedly calling forward from a loop), and there’s no memory issues</p>
<p>My system:<br>
OS: Windows 10/11<br>
pytorch version: 1.11.0<br>
libtorch version: 1.11.0</p>
C++ Memory Management API

collinmccarthy: <p>I believe the documentation is missing regarding the C++ memory management API. I opened an issue but was told to post here instead.</p>
<h2>
<a name="from-issue-85436httpsgithubcompytorchpytorchissues85436-1" class="anchor" href="#from-issue-85436httpsgithubcompytorchpytorchissues85436-1"></a>From <a href="https://github.com/pytorch/pytorch/issues/85436" rel="noopener nofollow ugc">issue 85436</a>:</h2>
<p>I assume there should be a C++ memory management API like Python, since <a href="https://github.com/pytorch/pytorch/issues/14618" rel="noopener nofollow ugc">Issue 14618</a> to do this was marked as completed. Does this API exist or no?</p>
<p>If not do I need to get the CUDA caching allocator and read from the <code>DeviceStats</code> struct, e.g. <a href="https://github.com/pytorch/pytorch/blob/d23ce29761dbc0e817fa80dcd35d9b8d30f16bbb/c10/cuda/CUDACachingAllocator.h#L60" rel="noopener nofollow ugc">here</a>? I’m assuming this isn’t exactly future proof so I would rather not do this unless I have to.</p>

ptrblck: <p>Yes, using the linked <code>struct</code> could work, but I also think your issue is a valid feature request, so feel free to reopen your GitHub issue and explain your feature request of having a matching C++ API for the memory stats.</p>
How to initialise hooks for a model loaded using an engine file

Sai_Kishore1: <p>I have an engine file created for a resnet model, i am not interested in the final outputs but the intermediate layers, how to initilase hooks for a model initialised using engine file.</p>

ptrblck: <p>Could you explain in more detail what an engine file is and how you’ve generated it, please?</p>

Sai_Kishore1: <p>Hey, so the engine file is the tensorrt version of the model. So a model path is a .pth file then that is fed to  ONNX which returns a .onnx file. that is parsed to generate a tensorrt engine.<br>
I have attached the link to the method. <a href="https://learnopencv.com/how-to-convert-a-model-from-pytorch-to-tensorrt-and-speed-up-inference/" rel="noopener nofollow ugc">Link</a>.</p>

ptrblck: <p>I’m unsure how this is related to PyTorch as it seems you are using TensorRT?</p>

Sai_Kishore1: <p>Yes, i was wondering if there was any way of doing hooks in pytorch which would be suitable to tensorrt</p>

ptrblck: <p>I’m not aware of any way to export hooks. However, if you want to always return the intermediate activations, you might want to return them in the <code>forward</code> in a custom module, which should then be exportable. Would this work?</p>

Sai_Kishore1: <p>Yes this would work. Thanks a lot for your help.</p>
How to disable torchscript warnings

zac: <p>Hi all,</p>
<p>I am converting our model to torchscript using <code>torch.jit.script</code>, but when I execute it, it prints out several lines of warning statement as below.</p>
<pre><code class="lang-auto">[W register_special_ops.cpp:229] Warning: Creating a tensor from an empty intlist will create a tensor of default floating point type  (currently Float) in python but a tensor of type int in torchscript.
</code></pre>
<p>Model’s result is the same with the eager model. I searched on pytorch git repo but couldn’t locate where this warning message is trigged and how to solve or locate the place that the tensor was created from an empty list from our code either. I am using pytorch=12.1 from conda.</p>
<p>Please help, thanks in advance!</p>
(Python edit-able) implementation of the GRU, not in lower-level languages?

John_Schwarcz: <p>Hello, thanks in advance.</p>
<p>I have been doing research on the LSTM using the edit-able implementation located at:<br>
<em>/python3.10/site-packages/torch/nn/quantizable/rnn.py</em></p>
<p>I am now trying to do the same for the GRU, however I cannot find a similar edit-able implementation.<br>
Is there an edit-able version of the GRU in Pytorch?</p>
<p>By <em>“edit-able” I mean that I can edit it in Python code, as I don’t know the lower level languages</em></p>
[RNNCell vs RNN]

j-min: <p>[RNNCell vs RNN]<br>
What is the better way when implementing RNN decoder?<br>
I used to work with tensorflow, so I am familiar with implementing RNN decoder by calling <code>RNNCell</code>s for each unrolling step.<br>
However, it seems many implementation calls <code>RNN</code> with input whose <code>seq_len</code> size is 1 for each time step, including <a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">official seq2seq tutorial</a>.<br>
As I see, <code>RNN</code> corresponds to <code>tf.nn.dynamic_rnn</code> or <code>tf.nn.static_rnn</code> of tensorflow, which internally call <code>RNNCell</code> for number of given unrolling steps.<br>
I am curious if calling <code>RNN</code> for multiple times causes unrolling slower than calling <code>RNNCell</code> for multiple times.<br>
I found <a href="https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/Models.py#L53" rel="nofollow noopener">OpenNMT-py</a> used RNNCell for decoding.<br>
Do people even use RNNCells in PyTorch?</p>

smth: <p>people do use RNNCell, but largely if your computation fits in the regime of RNN, then people prefer that (because RNN uses CuDNN and is ~3 to 4 times faster).</p>

beneyal: <p>Could you please clarify what do you mean by “the regime of RNN”? I still don’t really understand the difference between the RNN/LSTM/GRUCell family and the “simple” RNN/LSTM/GRU layers…</p>
<p>Thanks!</p>

Mostafa_Elhoushi: <p>Not sure about the definition of RNNCell in Pytorch, but quoting from <a href="https://www.tensorflow.org/guide/keras/rnn" rel="noopener nofollow ugc">TF Tutorial on RNN</a>:</p>
<blockquote>
<p>In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.</p>
</blockquote>

RoeAnt: <p>I understand the tensorflow approach. But how can we wrap some custom RNNCell easily in pytorch to create an RNN?</p>
Isn't NT-Xent loss commutative?

joohyunglee: <p>I think InfoNCE type losses, e.g. NT-Xent from <a href="https://arxiv.org/abs/2002.05709" rel="noopener nofollow ugc">SimCLR v1</a>, are commutative. However, SimCLR v1 mentions in <code>Algorithm 1</code> that they added the loss after commutating the input of NT-Xent loss: <code>loss = l(a, b) + l(b, a)</code>.</p>
<p>Isn’t adding commutated value unnecessary? Thank you ahead.</p>
How to compute the element-wise gradients efficiently?

kfxw: <p>Hello guys!</p>
<p>I recently have an efficiency problem when I’m going to compute the element-wise gradients.</p>
<p>To make my problem clear, assume there is an input tensor A in shape [M]. The computational graph is like</p>
<pre><code class="lang-python">B=model(A)  # A, B are both in the shape of [M]
</code></pre>
<p>Now I’d like the gradient as <code>sum(dA[i]/dB[i])</code> and <code>sum(d^2A[i]/dB[i]^2)</code>, and I implemented it as</p>
<pre><code class="lang-python">import torch
sum_1st_grad = 0
sum_2nd_grad = 0
for i in range(A.shape[0]):   # A.shape[0]=M
    grad1 = torch.autograd.grad(B[i],  A[i], retain_graph=True)[0]
    grad2 = torch.autograd.grad(grad1, A[i], retain_graph=True)[0]
    sum_1st_grad += grad1
    sum_2nd_grad += grad2
</code></pre>
<p>but it is SUPER slow as you may expect.</p>
<p>Have you ever had similar problems? Do you have any idea for a more efficient implementation?</p>
<p>Thanks!</p>

Skinish: <p>Did you find a solution?</p>

kfxw: <p>you can check out torch.vmap, which is a new feature that came out few months ago<br>
<a href="https://pytorch.org/tutorials/prototype/vmap_recipe.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/vmap_recipe.html</a></p>

Skinish: <p>I guess that is similar to functorch.vmap, right? Should even be the same underlying code.<br>
Check <a href="https://colab.research.google.com/drive/14HL6FhLrum7F2MygajQ6U3l0_izZWmdF?usp=sharing" rel="noopener nofollow ugc">this notebook</a>, I’m not able to calculate gradients with it, it’s too memory intensive</p>
Is it possible to do the integration of neural network

jaswant_singh: <p>I want to do the integration over one input of the neural network. Is it possible?</p>

ParGG: <p>Could you elaborate more? What do you want to integrate exactly?</p>

jaswant_singh: <p>Thank you for your reply.</p>
<p>def fun(x):<br>
<span class="hashtag">#print</span>("calling this function\t ",x_val)<br>
d_x=x[:, 0]<br>
in_Q2=[d_x[i].item() for i in range(len(d_x))]<br>
in_xy=[[in_Q2[i],x_val] for i in range(len(in_Q2))]<br>
in_xy=torch.tensor(in_xy,dtype=torch.float32).clone().detach()<br>
output_net=model(in_xy)<br>
output_net.requires_grad_(True)<br>
<span class="hashtag">#print</span>(f"Out put of the model:\t {list(model.parameters())} ",)</p>
<pre><code>            return output_net
</code></pre>
<p>pred = torch.tensor([mc.integrate(fun,dim=1,N=100,integration_domain=[[0, 1]],backend=“torch”,).item()],requires_grad=True).float()</p>
<p>loss = loss_fn(pred, val_yb)<br>
opt.zero_grad()<br>
loss.backward()<br>
opt.step()</p>
<p>sorry for the late reply. I have done the integration of the neural network but now the problem is that not updating the model parameters(weights and biases)</p>
Exception in getCustomClassType() with TORCH_LIBRARY_IMPL(quantized, QuantizedCPU, m)

Kareem_Shaik: <p>m.impl(TORCH_SELECTIVE_NAME(“quantized::conv1d_relu”),     QConv1dInt8::run);</p>
<p>TORCH_CHECK(false, "Can’t find class id in custom class type map for ", tindex.name());</p>
Is there a simple way to make the updated paramaters as interim tensors?

WuYunfan: <p>This is a problem related to higher gradients.<br>
In this simple case:</p>
<pre><code class="lang-auto">a = torch.tensor([1., 2., 3.], requires_grad=True)
b = torch.tensor([1., 2., 3.], requires_grad=True)
c = a * b
loss = c.sum()
loss.backward(create_graph=True)**
c = a - a.grad
loss2 = c.sum()
</code></pre>
<p>Here, <strong>b</strong> is the tensor I need to optimize, and <strong>a</strong> is model parameter. In the above codes, <strong>c</strong> is the updated version of tensor <strong>a</strong>, <strong>loss2</strong> is a loss which could backward to tensor <strong>b</strong>.  Howerver, the computation graph from <strong>a</strong> to <strong>loss2</strong> may be complex as <strong>a</strong> can be a parameter of a huge model.</p>
<pre><code class="lang-auto">a = torch.tensor([1., 2., 3.], requires_grad=True)
b = torch.tensor([1., 2., 3.], requires_grad=True)
opt = SGD([a], lr=1.)
c = a*b
loss = c.sum()
loss.backward(create_graph=True)
opt.step()
loss2 = a.sum()
</code></pre>
<p>In this version, <strong>loss2</strong> could not backward to tensor <strong>b</strong>. Is there a simple way to implement my needs, i.e., making the updated paramater <strong>a</strong> as an interim tensor induced by <strong>b</strong>, and <strong>a</strong> can be used further to compute <strong>loss2</strong>.</p>
Multiple inference models on jetson GPU

Martin_Pedersen: <p>Hello</p>
<p>I am trying to run a Yolov5 model and a resnet152 model on my Jetson Xavier NX16 GPU<br>
At first, the Yolov5 model was struggling to utilize the GPU and my inference time was about 4-5 seconds, but my Resnet152 inference time was around 0.2 seconds. After some digging, I tried switching to a TensorRT model for Yolov5 to get better results, this resulted in Yolov5 inference in 0.3 seconds, but Resnet152 inference scaled to 3-4 seconds.<br>
I have tried using</p>
<pre><code class="lang-auto">Yolov5model.cuda()
Yolov5results = Yolov5model(input)
Yolov5model.cpu()
Resnet152model.cuda()
Resnet152results = Resnet152model(input)
Resnet152model.cpu()
</code></pre>
<p>But it didn’t seem to change anything.<br>
The two models don’t need to run at the same time, so I don’t see why the GPU can’t manage two models loaded in one script<br>
Is there a way to make the two models share resources? Do I need to convert the Resnet152 model to TensorRT as well because of some prioritizing in the Jetson GPU?</p>
Calculating second-order partial derivatives of loss function

arm-on: <p>Hi! I’m trying to calculate: 1) the hessian of the loss function w.r.t. the model parameters 2) first calculate the gradient of loss function w.r.t. the inputs, then differentiate w.r.t the model parameters.<br>
This is my code (first install torchmetrics using <code>pip install torchmetrics</code>):</p>
<pre><code class="lang-auto">import torch
from torchmetrics import HingeLoss
import numpy as np

class simple(torch.nn.Module):
    def __init__(self):
        super(simple, self).__init__() 
        self.linear = torch.nn.Linear(in_features=784, out_features=1, bias=True)
    def forward(self, x):
        output = self.linear(x)
        return output

model = simple()

x = torch.Tensor(np.random.rand(5,784)) # 5 dummy data points
y = torch.Tensor([0,1,0,1,0]) # 5 dummy labels

outputs = model(x)
loss_fn = HingeLoss()
loss = loss_fn(outputs, y)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75.png" data-download-href="https://discuss.pytorch.org/uploads/default/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75_2_690x100.png" alt="pytorch" data-base62-sha1="c2Lk4V8JPtF1T3n8WowAJZ4JlYx" width="690" height="100" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75_2_690x100.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75_2_1035x150.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546a0d6de27f32763e5c9a031e7e1f6cc5a2bd75_2_1380x200.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">1988×290 26.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The above-mentioned matrices should be of shape <code>785x785</code> and <code>785x784</code>, respectively.<br>
I’ve searched a lot but none of the previously mentioned questions solved my problem. Is there a neat way to calculate these values?</p>
DataPipeline and DDP training

kimihailv: <p>I do not understand how to use datapipelines in multi-gpu training. Specifically, I have following questions:</p>
<ol>
<li>Where should shuffling happen? In datapipline or in dataloader?</li>
<li>Is it correct to use DistributedSampler with dataloader + datapipline?</li>
<li>Is there any examples of DDP + datapipline usage?</li>
</ol>
Sudden explosion of the training loss

RobbenRibery: <p>Hi I am using the output of my network to parameterise a Dirichlet distribution, however the loss function explodes sky high, when their is a point like [1,0,0] coming in as the observation.</p>
<p>I wondered if the statistical modelling community knows any methods to the resolution of this problem?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2507642095670ef43356275a6c6b7eccf09ed28.png" alt="image" data-base62-sha1="n9TJPeBmCUYlpGX3QPnHflF2XWg" width="377" height="248"></p>
Changing kernel size in wav2vec pretrained model

Ahmed_Khaled: <p>I got an error that is almost the one solved here <a href="https://discuss.pytorch.org/t/kernel-size-cant-greater-than-actual-input-size/21579" class="inline-onebox">Kernel size can't greater than actual input size</a><br>
the solution, as provided, is to lower the kernel size to 1 (to make it lower than the input size).</p>
<p>How can I do this with the pretrained wav2vec (WAV2VEC2_ASR_BASE_960H) model?</p>

ptrblck: <p>Changing the kernel in a pretrained model might not be the best idea as you would need to retrain the new kernel.<br>
Instead of changing the model you could increase the spatial size of your inputs.</p>

Ahmed_Khaled: <p>Thanks.<br>
So, can retraining happen in certain layer(s) (at which we need to adjust the kernel size) without retraining the other layers? can you point out some references to look up this using PyTorch, if it’s possible to do so?</p>

ptrblck: <p>Yes, you can fine-tune specific layers only as described in e.g. <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">this tutorial</a>.</p>
What is the benefit of torch.multiprocessing for shared memory?

phuongndvn: <p>I try to test the benefit of using torch.multiprocessing for shared memory. But when I use Queue from multiprocessing and torch.multiprocessing, there is nothing changes (processing time). Here is my test code:</p>
<pre><code class="lang-auto">import sys
import time

import torch
from torch.multiprocessing import Process
from torch.multiprocessing import Queue

# from multiprocessing import Process
# from multiprocessing import Queue


def torch_shared_mem_process(q):
    while True:
        data = q.get()
        if data is None:
            return
        print('Received data:', len(data), data)


def test_mem_share():
    q = Queue()
    p = Process(target=torch_shared_mem_process, args=(q,))
    p.start()

    def sample_data():
        return torch.zeros([100, 3, 1080, 1920], dtype=torch.float)

    data = sample_data()
    start = time.time()
    q.put(data)
    q.put(None)
    p.join()

    print(f'Finished sending tensor!')

    took_seconds = time.time() - start
    return took_seconds

def main():
    shared_memory = test_mem_share()
    print(f'Took {shared_memory:.1f} s with shared memory.')


if __name__ == '__main__':
    sys.exit(main())
</code></pre>
Pytorch Geometric - Loss always 'NaN'

Akash_142: <p>Hello,</p>
<p>I am training a simple GCN model. For some reason, my loss is always NaN from the beginning.<br>
Please let me know if you need the whole code and data.</p>
<p>Thank you!</p>
<p>Here is my data:<br>
Num_graphs = 800<br>
X (num_nodes, num_fea) = [64,1]<br>
Y (num_nodes, num_fea) = [64,1]</p>
<p>Here is my model:</p>
<p>class Graph_GCN(nn.Module):</p>
<pre><code>def __init__(self, nfeat, nhid, nclass):

    super(Graph_GCN, self).__init__()
    self.conv1 = GCNConv(nfeat, nhid)
    self.conv2 = GCNConv(nhid, nclass)

def forward(self, data):

    input = data.x.float()
    edge_index = data.edge_index
    
    X = self.conv1(input,edge_index)
    X = self.conv2(X, edge_index)

    return X
</code></pre>
Device error while working on colab

zoey_yj: <p>I’ve encountered the following error while transfer computation to cpu and send back cuda again while working on colab.</p>
<pre><code class="lang-auto">RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at "../aten/src/ATen/native/cuda/Loops.cuh":89, please report a bug to PyTorch. argument 1: expected a CUDA device but found cpu
</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing the error, please?</p>
Training on sparse tensors

nirzaa: <p>I want to train my neural network on sparse tensors that I’ve made by using:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.to_sparse.html#torch-tensor-to-sparse" rel="noopener nofollow ugc">pytorch sparse tensors</a><br>
But while trying to propagate my sparse tensors into the CNN I get this error:<br>
<code>RuntimeError: Input type (torch.cuda.sparse.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</code></p>
<p>What should I change in my network in order to be able to train on sparse tensors?</p>
<p>Thank you</p>

nirzaa: <p>I read and learn a lot from your comments here <a class="mention" href="/u/ptrblck">@ptrblck</a>, maybe you could help me here?</p>
<p>Thank you!</p>

cpuhrsch: <p>Hello <a class="mention" href="/u/nirzaa">@nirzaa</a>,</p>
<p>We currently don’t have support for convolutions for sparse inputs. Could I kindly ask you to open a feature request on <a href="https://pytorch/pytorch" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch</a> with your intended use case (model, application area, dataset)? Also, please include details on the intended semantics if you want the input to be treated differently if it has a sparse layout vs. a regular strided layout.</p>
<p>Thanks,<br>
Christian</p>

nirzaa: <p>the link to the issue I opened on github is here:<br>
<a href="https://github.com/pytorch/pytorch/issues/78262" rel="noopener nofollow ugc">link to the issue on github</a></p>
<p>Thank you very much</p>

DerekGloudemans: <p>This library from facebookresearch at least at first glance seems to implement the desired features for basic CNNs. <a href="https://github.com/facebookresearch/SparseConvNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/SparseConvNet: Submanifold sparse convolutional networks</a></p>
Analyzing the inference of a of a model

Nubman: <p>So im new to pytorch<br>
But im wondering is there is a way to gain information on a model during an inference like what neurons were a activated or a history of what each output was at each layer as the data moved through the model?<br>
Is that beyond the scope of what of what pytorch can do? if not is it beyond the scope of someone with an amateurish knowledge of machine learning can do?</p>
<p>Im far from experienced in these kinds of things but im really curious to see what is happening under the hood.</p>

ptrblck: <p>Yes, it’s possible to check the intermediate activations e.g. via forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
Knowing if training on multi-GPU

aRI0U: <p>Hi. I have the model that I sometimes train on a single GPU and sometimes on multiples GPUs. In its architecture there are batchnorm’s so I decided to use <code>model = SyncBatchNorm.convert_sync_batchnorm(model)</code> to avoid performance loss when training on Multi-GPU.</p>
<p>However, the thing is that now when I want to train on one GPU I get the following error:</p>
<pre><code class="lang-auto">...
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py", line 731, in forward
    world_size = torch.distributed.get_world_size(process_group)
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 867, in get_world_size
    return _get_group_size(group)
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 325, in _get_group_size
    default_pg = _get_default_group()
  File "/home/alain/miniconda3/envs/ts/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 429, in _get_default_group
    raise RuntimeError(
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
</code></pre>
<p>I tried to do something conditional like:</p>
<pre><code class="lang-auto">import torch.distributed as dist
model = SyncBatchNorm.convert_sync_batchnorm(model) if dist.get_world_size() &gt; 1 else model
</code></pre>
<p>but I got a similar error when trying to evaluate <code>dist.get_world_size()</code>.</p>
<p>Also, I use Pytorch Lightning, I don’t know whether it can help/be an issue.</p>
<p>Does anyone has an idea?<br>
Thanks a lot!</p>

ptrblck: <p>Try to use <code>torch.distributed.is_initialized()</code> in a condition.</p>