akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Problem loading parallel datasets even after using SubsetRandomSampler

Krishna_Garg: <p>I have two parallel datasets <code>dataset1</code> and <code>dataset2</code> and following is my code to load them in parallel using <code>SubsetRandomSampler</code> where I provide <code>train_indices</code> for dataloading.</p>
<p>P.S. Even after setting <code>num_workers=0</code> and seeing <code>np.random</code>, the samples do not get loaded in parallel. Any suggestions are heartily welcome including methods other than <code>SubsetRandomSampler</code>.</p>
<pre><code class="lang-auto">import torch, numpy as np
from torch.utils.data import Dataset, DataLoader, SubsetRandomSampler

dataset1 = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
dataset2 = torch.tensor([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

train_indices = list(range(len(dataset1)))
np.random.seed(12)
np.random.shuffle(train_indices)
sampler = SubsetRandomSampler(train_indices)

dataloader1 = DataLoader(dataset1, batch_size=2, num_workers=0, sampler=sampler)
dataloader2 = DataLoader(dataset2, batch_size=2, num_workers=0, sampler=sampler)

for i, (data1, data2) in enumerate(zip(dataloader1, dataloader2)):
  x = data1
  y = data2
  print(x, y)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([5, 1]) tensor([15, 18])
tensor([0, 2]) tensor([14, 12])
tensor([4, 6]) tensor([16, 10])
tensor([8, 9]) tensor([11, 19])
tensor([7, 3]) tensor([17, 13])
</code></pre>
<p>Expected Output:</p>
<pre><code class="lang-auto">tensor([5, 1]) tensor([15, 11])
tensor([0, 2]) tensor([10, 12])
tensor([4, 6]) tensor([14, 16])
tensor([8, 9]) tensor([18, 19])
tensor([7, 3]) tensor([17, 13])
</code></pre>

ptrblck: <p>Since you are using a random sampler, the random indices are expected.<br>
If you want to yield the same (shuffled) indices from both <code>DataLoader</code>s, create the indices first, and use a custom sampler:</p>
<pre><code class="lang-python">class MySampler(torch.utils.data.sampler.Sampler):
    def __init__(self, indices):
        self.indices = indices
        
    def __iter__(self):
        return iter(self.indices)
    
    def __len__(self):
        return len(self.indices)


dataset1 = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
dataset2 = torch.tensor([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

train_indices = list(range(len(dataset1)))
np.random.seed(12)
np.random.shuffle(train_indices)

sampler = MySampler(train_indices)

dataloader1 = DataLoader(dataset1, batch_size=2, num_workers=0, sampler=sampler)
dataloader2 = DataLoader(dataset2, batch_size=2, num_workers=0, sampler=sampler)

for i, (data1, data2) in enumerate(zip(dataloader1, dataloader2)):
  x = data1
  y = data2
  print(x, y)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">tensor([5, 8]) tensor([15, 18])
tensor([7, 0]) tensor([17, 10])
tensor([4, 9]) tensor([14, 19])
tensor([3, 2]) tensor([13, 12])
tensor([1, 6]) tensor([11, 16])
</code></pre>
<p>You could also try to pass a <code>generator</code> to the <code>SubsetRandomSampler</code>, but I would recommend to create the indices explicitly since trying to manipulate the PRNG often yields misunderstandings and errors.</p>
Is there a faster way to assign values in a matrix in LibTorch?

Alio: <p>I’m assigning new values to a matrix and I’m doing it one by one for each individual value. Is there a faster way to do this in LibTorch? If I was using PyTorch (in Python instead of C++) I would be able to do this with one line,<br>
<code>matrix_two[:, tril[0], tril[1]] = matrix_one</code></p>
<p>But since I can’t use indexing like this to store values in C++ I did it this way, and it is very slow when in the <code>forward</code> function of a model.</p>
<p>The first ‘for’ loop “o” of ten iterations in the code below is just to help show the time it takes to run, though when it is in a model it runs even slower.</p>
<blockquote>
<pre><code>for (int o = 0; o&lt;10; o++) {
    auto matrix_one = torch::rand({42, 903});
    auto matrix_two = torch::zeros({42, 42, 42});
    auto trl = torch::tril_indices(42, 42, 0);

    for (int p = 0; p&lt;matrix_one.sizes()[0]; p++) {
        for (int z = 0; z&lt;matrix_one[p].sizes()[0]; z++) {
            matrix_two[p][trl[0][z]][trl[1][z]] = matrix_one[p][z];
        }
    }
    std::cout &lt;&lt; o &lt;&lt; "\n";
}
</code></pre>
</blockquote>
Problem with Uninstalling PyTorch and CUDA

Ali_Aghababaei: <p>Hi everyone,<br>
I have been working with Pytorch and recently wanted to use CUDA of my PC to benefit from GPU capabilities. I figured out that a few versions of CUDA had been installed on the Windows, which made me decide to uninstall all the versions and re-install a new one. Also, previously I had installed PyTorch via the official sitr via this command <code>conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia</code> .<br>
Based on what the NVIDIA documentation says , I executed the following command in the command prompt <code>conda uninstall cuda</code> . Also, I unistalled the NVIDIA driver from the control panel. However, in the anaconda pkgs directory, the previous subpackages still exist, e.g, cuda-toolkit, cuda-nvrtc, cuda-libraries, cuda-nvcc etc. So my question is How can I “completely” get rid of CUDA?<br>
I appreciate your consideration.</p>

ptrblck: <aside class="quote no-group" data-username="Ali_Aghababaei" data-post="1" data-topic="166609">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>Based on what the NVIDIA documentation says , I executed the following command in the command prompt <code>conda uninstall cuda</code> .</p>
</blockquote>
</aside>
<p>I don’t know which documentation you are following, but you should just uninstall the previously installed packages.</p>
<p>E.g. in:</p>
<pre data-code-wrap="python"><code class="lang-python">conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
</code></pre>
<p>you’ve installed <code>pytorch</code>, <code>torchvision</code>, and <code>torchaudio</code>, with the CUDA 11.7 dependency.<br>
Run <code>conda uninstall pytorch torchvision torchaudio</code> and it will uninstall these tree libs.</p>
<p>Also note that your local CUDA toolkit won’t be used, since the conda binaries and pip wheels ship with their own CUDA runtime (unless you build custom CUDA extensions).</p>
How do num_workers>1 and batch_size=None interact?

dasturge: <p>I’m curious how the dataloader behaves differently if it’s instructed to use multiple workers but also told that the dataset will take care of batching.  Does this mean each worker will load separate batch?</p>

ptrblck: <p>I would expect to see the same behavior, i.e. each worker is responsible to load, process, and create a single batch. If your <code>Dataset</code> is now responsible to create the full batch already, I guess the number of calls into <code>Dataset.__getitem__</code> would differ:</p>
<ul>
<li>In the default use case each worker would call <code>batch_size</code> times into <code>__getitem__</code> to load and process each sample. Afterwards, it would create the batch in its <code>collate_fn</code>.</li>
<li>In your use case I assume each workers calls once into <code>__getitem__</code> and uses the <code>collate_fn</code> to create/process the batch samples afterwards.</li>
</ul>
Libtorch module as C++ class member

coolvision: <p>I want to load libtorch module once and then use it on several times when a class method is called.<br>
I tried to do it this way:</p>
<pre><code class="lang-auto">class myClass {
      void process();
      bool module_loaded = false;
      torch::jit::script::Module module;
};

void myClass::process() {
        if (!module_loaded) {
            module = torch::jit::load("model.pt");
            module_loaded = true;
        }
        // .....
        auto outputs = module.forward(inputs).toTensor();
        // .....
}
</code></pre>
<p>This cause a crash on class destruction, and in any case copying of torch::jit::script::Module does not seem right, as it has <strong>TH_DISALLOW_COPY_AND_ASSIGN</strong> in the code.<br>
Is there a better way to use torch::jit::script::Module so that it’s not loaded every time, for several consecutive calls?</p>

chavezrodz: <p>Try loading the model in the class constructor instead of using a boolean flag like that, Its working for me. I am also making the module a private class member.</p>
Visualize param.grad before loss.backward()

papoo13: <p>When I try to visualize the grad values of my classifier head, through pdb.set_trace() and in shell, it returns nothing. Does this mean that param.grad does not exist at this stage?</p>
<pre><code class="lang-auto">pdb.set_trace()
for i, param in enumerate(self._model_and_loss._model.resnet.fc.parameters()):
          if i==1:
                     pass

        loss.backward()
</code></pre>

ptrblck: <p>The <code>.grad</code> attribute will be lazily create in the first <code>backward</code> call. Afterwards it would be set to zero via <code>optimizer.zero_grad()</code> or <code>model.zero_grad()</code> and would still be valid.<br>
If <code>set_to_none=True</code> is used in <code>.zero_grad()</code> the attribute will be deleted again, which allows you to save memory and to avoid an unneeded accumulation kernel in the next <code>backward</code> call.</p>

papoo13: <p>thank you for your reply.<br>
I must have added that I am in the very first iteration. So, in the first iteration and before the loss.backward(), weight.grad should return nothing? am I correct?</p>

ptrblck: <p>Yes, the <code>.grad</code> attribute will be initialized to <code>None</code> before the first <code>backward</code> call.</p>
Getting parameters of specific layers in cvae

Mohammed_Zidane: <p>Hi,<br>
I need to get the parameters of all layers of a conditional variational autoencoder EXCEPT the parameters of the first layer.<br>
Any suggestions?<br>
Thanks</p>

ptrblck: <p>Most likely you haven’t properly registered the first layer, so make sure it’s a subclass of <code>nn.Module</code> and either directly assigned to an attribute or to e.g. <code>nn.ModuleList</code> or <code>nn.ModuleDict</code>.</p>

Mohammed_Zidane: <p>Thanks for your reply but I think there is a misunderstanding. I did not mean that I do not get the parameters of the first layer while calling model.parameters() for example, I meant what I want to do is to get all the parameters without the parameters of the first layer, in other words, I need to do some processes on the parameters of the model starting from the second layer.<br>
Thanks</p>

ptrblck: <p>You are right, I thought you see all parameters “except” the ones from the first layer.<br>
In this case, you might want to filter the parameters of the first layer out e.g. via using their <code>name</code> attribute via <code>model.named_parameters()</code>.</p>
How to indexing a slice of data in a N-D Tensor?

zgjja: <p>Suppose I have a batch of data with shape (2, 7, 7, 30), now if i want to rewrite:<br>
[0, 2, 3, 0:2]    [0, 2, 3, 5:7]  and<br>
[0, 3, 3, 0:2]    [0, 3, 3, 5:7]<br>
How to do it without using for loop but Pytorch API?</p>
<p>To illustrate my problem more clear, i want to indexing something like<br>
[<br>
:(whole batch),<br>
{A 2-D tensor indicating the 1,2 dim of batch data},<br>
{the real index i want to change}<br>
],<br>
How can i do it ?</p>

KFrank: <p>Hi zgjja!</p>
<aside class="quote no-group" data-username="zgjja" data-post="1" data-topic="166498" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zgjja/48/55036_2.png" class="avatar"> zgjja:</div>
<blockquote>
<p>How to do it without using for loop but Pytorch API?</p>
</blockquote>
</aside>
<p>If I understand correctly what you are trying to do, you can assign to the<br>
tensor in question after indexing into it more-or-less as you posted.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; t = torch.arange (2 * 3 * 3 * 5).reshape (2, 3, 3, 5)
&gt;&gt;&gt; t[0, 2]
tensor([[30, 31, 32, 33, 34],
        [35, 36, 37, 38, 39],
        [40, 41, 42, 43, 44]])
&gt;&gt;&gt; t[0, 2, 2, 0:2] = torch.tensor ([66, 99])
&gt;&gt;&gt; t[0, 2]
tensor([[30, 31, 32, 33, 34],
        [35, 36, 37, 38, 39],
        [66, 99, 42, 43, 44]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Training with variable length sequences with LSTM

Yuerno: <p>Hi all! I’ve gone through a bunch of similar posts about this topic, and while I’ve figured out the idea of needing to use padding and packing, I still haven’t been able to find how to properly pass this data into a loss function. Could anyone check if my logic of code makes sense? Thanks in advance!</p>
<ol>
<li>I create a padded set of data as follows:</li>
</ol>
<pre><code class="lang-auto">seq_lengths = torch.LongTensor(list(map(len, observations_history)))
observations_history = pad_sequence(observations_history).to(self.device)
</code></pre>
<ol start="2">
<li>Then I pass that padded data and sequence length data into the forward pass of my neural network:</li>
</ol>
<pre><code class="lang-auto"># Embedding layer
x = F.relu(self.fc1(x))

# Padded LSTM layer
x = pack_padded_sequence(x, seq_lengths, enforce_sorted=False)
self.lstm1.flatten_parameters()
x, _ = self.lstm1(x)
x, x_unpacked_len = pad_packed_sequence(x)
time_dimension = torch.tensor(0).to(x.device)
last_timestep_idx = (
    (seq_lengths - 1).view(-1, 1).expand(len(seq_lengths), x.size(2))
).to(x.device)
last_timestep_idx = last_timestep_idx.unsqueeze(time_dimension)
x = x.gather(time_dimension, last_timestep_idx).squeeze(time_dimension)

# Remaining layers
x = F.relu(self.fc2(x))
q_values = self.fc3(x)
return q_values
</code></pre>
<ol start="3">
<li>Those <code>q_values</code> then get passed into an MSE loss function: <code>qf1_loss = 0.5 * F.mse_loss(qf1_a_values, next_q_value)</code>
</li>
</ol>
<p>A couple of questions here:</p>
<ol>
<li>
<p>My first embedding FC layer directly uses the padded data. Is this ok? It obviously seems a bit of a waste computationally, but I don’t think FC layers can also used packed data, can they?</p>
</li>
<li>
<p>With all the padding and packing going on in the intermediate layers, is this backprop happening correctly using only the non-padded data? Or is it also passing gradients through the padded data and thus making incorrect gradient updates? What is the correct way to set this up?</p>
</li>
</ol>
The heght and width of the input video/frame for the PyTorchVideo library's pretrained models

Tm4: <p>I need to use a pretrained model using <a href="https://github.com/facebookresearch/pytorchvideo" rel="noopener nofollow ugc">PyTorchVideo</a> library that provides pretrained model for several models. The main problem of mine is that I could not find any information that for using the pretrained models for nference, what should be the spatial size of the input video/frame( I mean height and width)? To be clearer, I need to use the pretrained models of X3D and r2p1D but I cannot find how should I select the spatial size of input video/frames?</p>

ptrblck: <p>The <a href="https://github.com/facebookresearch/pytorchvideo/blob/main/tutorials/torchhub_inference_tutorial.ipynb"><code>torchhub_inference_tutorial</code></a> from the same repository might be a good starter and shows <code>crop_size = 256</code>.</p>

Tm4: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>  thanks for the answer, one thing is ambiguous for me. Since it is mentioned that <code>crop_size = 256</code>, it means the input size should be (256,256)?</p>

JuanFMontesinos: <p>A typical approach is resizing the videos to a large size (as a pre-processing step). Then cropping parts of each video (as a data augmentation step). That’s why  (I think) it’s called crop_size.<br>
You have training recipes here <a href="https://github.com/facebookresearch/pytorchvideo/tree/main/pytorchvideo_trainer" class="inline-onebox" rel="noopener nofollow ugc">pytorchvideo/pytorchvideo_trainer at main · facebookresearch/pytorchvideo · GitHub</a></p>
Sampling grid - use vector field as input

MartinPerry1: <p>I have a vector field given by dx, dy changes.<br>
My field has shape [batch, H, W, 2], where 2 corresponds to dx and dy. How can I use this together with <code>grid_sample</code> to warp the image? I can use <code>affine_grid</code>, but not with the field directly. I need to convert the field to mappings, but I don’t know exactly how (without iterating the field value by value and computing position manually, which is slow).</p>
<pre><code class="lang-auto">field = some field given by dx, dy

# ?? transform field to be usable with grid_sample ??

out = F.grid_sample(input, field, mode="bilinear", padding_mode="zeros", align_corners=False)
</code></pre>
<p>I have also found this extension - <a href="https://github.com/seung-lab/torchfields" class="inline-onebox" rel="noopener nofollow ugc">GitHub - seung-lab/torchfields: A PyTorch add-on for working with image mappings and displacement fields, including Spatial Transformers</a>, but I am not sure how to use it correctly.</p>
Loss during learning

ArkadyBogdanov: <p>Hello, what to do when the loss starts to jump all over during learning?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2124ecb5a602e94a22441a52de7750c9dab3814d.png" alt="Screenshot from 2022-11-22 15-20-37" data-base62-sha1="4JcRLzL1GVSyp0zz09rJqWDb5CJ" width="472" height="327"></p>
<p>Thank you</p>

dimou_gk: <p><a class="mention" href="/u/arkadybogdanov">@ArkadyBogdanov</a><br>
Greetings,<br>
By learning I assume you mean…training. Most of the times when this happens is because you have a very small learning rate so your model tries to “approximate”  thus slowly reaching to a local minimum to the point the loss is starting to increase again.<br>
Unfortunately you aren’t providing much information so I can’t conclude to something. In general try to change the hyperparameters of your model.</p>
Faster R-CNN box coder

sigma_x: <p>At evaluation time FRCNN decodes boxes in the following way:</p>
<ol>
<li>Get N candidates from RPN, dim Nx4</li>
<li>Filter them through RoIAlign to get N RoIs, dim NxKxHxW</li>
<li>Predict C+1 classes and box deltas from each RoI, dim Nx(C+1) and Nx(C+1)x4</li>
<li>Compare each (C+1) box predictions from each RoI to its source candidate</li>
<li>Thus obtain decoded boxes dim Nx(C+1)x4</li>
</ol>
<p>What I don’t wholly understand about BoxCoder class (see _utils.py in detection module) is if it works for cases when I want to decode one set of box deltas wrt multiple references. In case there’s M deltas and R references, and I want to obtain R decoded boxes for each delta, i.e. the output will be dim MxRx4.</p>
<p>Is this possible?</p>
Time inference before and after using pruning Pytorch is same

aouatif_ziane: <p>Hi all,</p>
<p>I used Pruning Pytorch in my model (detection YOLOX)</p>
<p>I prune my model , then i tested i got the same inference time</p>
<p>is what I used in a false way ?</p>
<p>Time before prune : 0.3s<br>
Time after prune : 0.3s</p>
<p>Thanks in advance</p>

aouatif_ziane: <p>any help please to improve my result !</p>
What is the best way to visualize My Network?

spnova12: <p>What is the best way to visualize My Network(model)?<br>
Is it tensorboard?<br>
Or is there a better alternative?<br>
If tensorboard is the best way, is there anything I lose when tensorflow is not installed?</p>
<p>I get this message “TensorFlow installation not found - running with reduced feature set.”<br>
When I use tensorboard, Would it be adversely affected by visualizing the Network(model) without tensorflow?</p>

ConvolutionalAtom: <p>Hi Dong,</p>
<p>I created a package that you can use to visualize model (or any graph execution), called torchview<br>
You can see it here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/mert-kurttutan/torchview">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/mert-kurttutan/torchview" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/7d4336304b875723ebb5a3fef799a31e4a85dbcbc0ed2fa6e92f69e71145d561/mert-kurttutan/torchview" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/mert-kurttutan/torchview" target="_blank" rel="noopener nofollow ugc">GitHub - mert-kurttutan/torchview: torchview: visualize pytorch models</a></h3>

  <p>torchview: visualize pytorch models. Contribute to mert-kurttutan/torchview development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
It can give visualization without using any external platform (e.g. tensorboard). So you dont need to use tensorboard or anything.</p>
Error using torch.load() (.pt file is corrupted)

olivier_vl: <p>Hi everyone,</p>
<p>I’m running a YoloV5 model on Azure ML using a compute cluster. In the train file I changed opt.save_dir from <code>str(increment_path(Path(opt.project) / opt.name, exist_ok=opt.exist_ok))</code> to <code>'logs/'</code>, since that is required when running on AzureML (to save the weights somewhere). The folder is indeed created, but when downloading last.pt/best.pt and running <code>!python detect.py --img 512 --data data/images --weights last.pt</code> the following error will occur: <code>RuntimeError: PytorchStreamReader failed reading file data/0: invalid header or archive is corrupted</code>. The same error will occur when using <code>torch.load(best.pt)</code></p>
<p>EDIT: The validation part (using best.pt) when the training process is finished does work, but after downloading the “best.pt” file locally and loading it, it gives errors.</p>
<p>Anyone who might know what the cause of the error is?</p>
<p>Thanks in advance,<br>
Kind regards,<br>
Olivier</p>
Controlling host to device memory transfer in a custom layer

rinkujadhav2013: <p>Is it possible (as in not too complicated) to not only run my custom C++ layer but also control when and how tensors are transferred to and from GPU for that layer?<br>
The only related thing I could find is <a href="https://ieeexplore.ieee.org/document/9599280" rel="noopener nofollow ugc">this paper</a> where the authors were somehow able to make use of unified memory in PyTorch but their code isn’t available and it seems like that had to put in quite some effort to achieve that.</p>

ptrblck: <p>I’m not sure what exactly your use case is, but wouldn’t it work to move the tensors around in the <code>libtorch</code> C++ API or what exactly is missing?</p>

rinkujadhav2013: <p>Is it possible to move a tensor partially to the GPU both for the forward and the backward pass?<br>
My use case is that a layer can be too big to fit into the GPU memory so I’m processing it in slices…it might be straightforward to do it manually for the forward pass but I don’t know how to do it for the backward pass.</p>

rinkujadhav2013: <p>I’ll assume that it’s a bit too complicated to do or at least too complicated to talk about here and mark this as resolved.</p>
Crnn how to obtain a gradcam for characters

Zrufy: <p>Hi all, I’ve been looking for a solution to this problem for some time.<br>
I have a crnn from which I would like to get a heatmap to see and see where there is more concentration and see which characters it focuses on and more.<br>
But after so many attempts I still haven’t been able to get the result I was looking for.<br>
Could anyone help me?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py">
  <header class="source">

      <a href="https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py" target="_blank" rel="noopener nofollow ugc">meijieru/crnn.pytorch/blob/master/models/crnn.py</a></h4>


      <pre><code class="lang-py">import torch.nn as nn


class BidirectionalLSTM(nn.Module):

    def __init__(self, nIn, nHidden, nOut):
        super(BidirectionalLSTM, self).__init__()

        self.rnn = nn.LSTM(nIn, nHidden, bidirectional=True)
        self.embedding = nn.Linear(nHidden * 2, nOut)

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        T, b, h = recurrent.size()
        t_rec = recurrent.view(T * b, h)

        output = self.embedding(t_rec)  # [T * b, nOut]
        output = output.view(T, b, -1)

        return output
</code></pre>



  This file has been truncated. <a href="https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
This is the project I’m referring to</p>
Problem with import of class

Saguaro: <p>I would like to import a class from torch_geometric, but I keep receiving the error. I use the following line of code: <code>from torch_geometric.transforms import AddTrainValTestMask</code><br>
The error is: <code>ImportError: cannot import name 'AddTrainValTestMask' from 'torch_geometric.transforms' </code></p>
<p>Documentation you may find here: <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/transforms.html" class="inline-onebox" rel="noopener nofollow ugc">torch_geometric.transforms — pytorch_geometric 1.6.3 documentation</a></p>
<p>Thank you!</p>

zhihuat: <p>I met the same problem, have you fixed it?</p>

Saguaro: <p>Update torch geometric in your environment</p>

javadr: <aside class="quote no-group" data-username="Saguaro" data-post="1" data-topic="116836">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saguaro/48/45129_2.png" class="avatar"> Saguaro:</div>
<blockquote>
<p>AddTrainValTestMask</p>
</blockquote>
</aside>
<p>It seems this class has been removed from version 2.<br>
Is there any class that has the same function?</p>
Question about behavior of Register_forward_hook in loops

tanabe.d.zc32s: <p>Hi there.</p>
<p>I tried to implement Grad-CAM with Register_forward_hook, but ran into a problem when I let the loop process estimate test data from Dataloader.</p>
<p>If I apply the following hook with Register_forward_hook</p>
<pre><code class="lang-auto">def forward_hook(module, inputs, outputs):
    global feature
    feature = outputs[0].clone()

forward_handle = target_layer.register_forward_hook(forward_hook)
</code></pre>
<p>I thought that the “feature” would be updated by the forward propagation on each loop, but it was not updated and the “feature” based on the first estimated data was fixed and output.</p>
<p>Therefore, I incorporated the forward_handle definition expression into the loop to achieve the behavior I wanted, but I don’t think this is a smart way to go about it.</p>
<p>Is this behavior a specification of register_forward_hook? If I am doing it wrong I would like your advice.</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="tanabe.d.zc32s" data-post="1" data-topic="166493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tanabe.d.zc32s/48/46394_2.png" class="avatar"> tanabe.d.zc32s:</div>
<blockquote>
<p>I thought that the “feature” would be updated by the forward propagation on each loop,</p>
</blockquote>
</aside>
<p>Yes, that’s the case. The registered forward hook will be triggered in each forward pass as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layer = nn.Identity()
        
    def forward(self, x):
        x = self.layer(x)
        return x


activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook


model = MyModel()
model.layer.register_forward_hook(get_activation('identity'))

x = torch.zeros(1, 10)
output = model(x)
print(activation['identity'])
# tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

x = torch.ones(1, 10)
output = model(x)
print(activation['identity'])
# tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
</code></pre>
<p>I’m unsure what might be causing your issue, so please feel free to post a minimal, executable code snippet reproducing your issue.</p>

tanabe.d.zc32s: <p>Thank you for reply.</p>
<p>Thank you for the clarification.<br>
I understand more about Pytorch now.</p>
<p>I have solved my issue by myself.<br>
The cause was a very trivial mistake in fixing the index number when retrieving the output of the intermediate layer stored in a list object.</p>
Requires_ grad and false invalid during repeated freezing and unfreezing

bqdeng: <p>I’m using requires_ grad=true and false During repeated freezing and thawing, it is observed that a parameter v requires_ It is no problem for grad to change from true to false during training. Parameter v stops updating, but once it changes from false to true again, parameter v will always be updated, even if it requires_ The grad attribute has changed to false several times.</p>
<p>What is the problem?</p>

ptrblck: <p>Parameters can be still updated even if they are frozen if an optimizer with internal running states is used. E.g. <code>Adam</code> lazily initialized running estimates for each updated parameter and will use these to update the parameter even if their gradients are set to zero.<br>
You could delete their <code>.grad</code> attribute, which would let the optimizer skip these parameters in its <code>step()</code> method, or you could also use <code>optimizer.zero_grad(set_to_none=True)</code>, which will also delete the <code>.grad</code> attributes.</p>

bqdeng: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="166568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>optimizer.zero_grad(set_to_none=True)</p>
</blockquote>
</aside>
<p>thank you!<br>
Excuse me,<br>
<code>optimizer.zero_grad(set_to_none=True)</code><br>
is valid.</p>
<p>Why is it? Will it affect the accuracy of training?</p>
Error with Older GT710 with cuda capability 3.5 and pytorch

kmn_works: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d2b4b6f7690e5286daf318ab9995f95844c32e0.png" alt="image" data-base62-sha1="mqnIIwYBgKSbg7YvsSbmkEUvCQE" width="654" height="336"><br>
C:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda_<em>init</em>_.py:132: UserWarning:<br>
Found GPU0 NVIDIA GeForce GT 710 which is of cuda capability 3.5.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 3.7.</p>
<p>warnings.warn(old_gpu_warn % (d, name, major, minor, min_arch // 10, min_arch % 10))<br>
[2022-11-21 16:15:41,672] [INFO] Set up nlp object from config<br>
[2022-11-21 16:15:41,687] [INFO] Pipeline: [‘transformer’, ‘ner’]<br>
[2022-11-21 16:15:41,694] [INFO] Created vocabulary<br>
[2022-11-21 16:15:41,696] [INFO] Finished initializing nlp object<br>
Traceback (most recent call last):<br>
File “C:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\site-packages\cupy\cuda\compiler.py”, line 66, in _run_cc<br>
universal_newlines=True)<br>
File “C:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\subprocess.py”, line 411, in check_output<br>
**kwargs).stdout<br>
File “C:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\subprocess.py”, line 512, in run<br>
output=stdout, stderr=stderr)<br>
subprocess.CalledProcessError: Command ‘[‘C:\Program’, ‘Files\NVIDIA’, ‘GPU’, ‘Computing’, ‘Toolkit\CUDA\v11.8\bin\nvcc.EXE’, ‘-gencode=arch=compute_35,code=sm_35’, ‘–ptx’, ‘-DFIRST_PASS=1’, ‘–std=c++11’, ‘-IC:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\site-packages\cupy\_core\include’, ‘-IC:\Users\Kharanshu.Naghera\AppData\Local\Programs\Python\Python37\lib\site-packages\cupy\_core\include\cupy\_cuda\cuda-11’, ‘-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include’, ‘-ftz=true’, ‘C:\Users\KHARAN~1.NAG\AppData\Local\Temp\tmpujlij1rh\preprocess.cu’]’ returned non-zero exit status 1.</p>

ptrblck: <p>The error message:</p>
<pre><code class="lang-python">Found GPU0 NVIDIA GeForce GT 710 which is of cuda capability 3.5.
PyTorch no longer supports this GPU because it is too old.
The minimum cuda capability supported by this library is 3.7.
</code></pre>
<p>explains that your GPU is too old and not supported anymore.<br>
You might try to build PyTorch from source for your compute capability 3.5.</p>

kmn_works: <p>thx for reply, i tried buildng it from scratch didn’t work i think i’ll execute everything on cpu</p>
PyTorch Mask 50% image during training process with medium boxes

Khawar_Islam: <p>I would like to mask 50% image randomly with medium boxes during training a network. Not more than two boxes are at same position. I have an idea but it is little bit hard how to implement during training loop.</p>
<p><strong>Code Idea</strong></p>
<ol>
<li>you have image_width, image_height, and image_channels and image size is 224</li>
<li>So we define a range: <code>boxes=np.arange(49)</code>
</li>
<li>batch.size() <code>(100, 224, 224, 3)</code>
</li>
<li>So we resize it to batchview=batch.view(-1, 224*7, 7, 3)</li>
<li>batch.size() <code>(100, 224, 224, 3)</code> change to  <code>batchview=batch.view(batch_size, 7*7, -1, 3)</code>
</li>
</ol>
<p><strong>Input</strong></p>
<p><a href="https://i.stack.imgur.com/utfEh.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/utfEh.png" alt="enter image description here" width="286" height="284"></a></p>
<p><strong>Output</strong></p>
<p><a href="https://i.stack.imgur.com/GcO3n.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/GcO3n.png" alt="enter image description here" width="284" height="281"></a></p>
<p><strong>Training loop</strong></p>
<pre><code>def train(train_loader, model, criterion, optimizer, epoch, args):
batch_time = AverageMeter('Time', ':6.3f')
data_time = AverageMeter('Data', ':6.3f')
losses = AverageMeter('Loss', ':.4e')
top1 = AverageMeter('Acc@1', ':6.2f')
top5 = AverageMeter('Acc@5', ':6.2f')
progress = ProgressMeter(
    len(train_loader),
    [batch_time, data_time, losses, top1, top5],
    prefix="Epoch: [{}]".format(epoch))

# switch to train mode
model.train()

end = time.time()
for i, data in enumerate(train_loader):
    # measure data loading time
    data_time.update(time.time() - end)
    images = data['image']
    target = data['label']

    if torch.cuda.is_available():
        images = images.cuda(args.gpu, non_blocking=True)
        target = target.cuda(args.gpu, non_blocking=True)
</code></pre>
Remove column from tensor?

R0b: <p>What is the proper way to remove a column from a tensor?</p>
<p>What I’m searching is the proper way to do the following (removing the 3):</p>
<pre><code class="lang-auto">t = torch.tensor([0,1,2,3,4,5])
t = torch.cat( ( t[:3], t[4:] ) )
</code></pre>
<p>Thanks in advance.</p>

Max1: <p>Does <a href="https://stackoverflow.com/a/19286855/3356441" rel="nofollow noopener">boolean indexing</a> work with torch tensors? Haven’t tried it myself yet.</p>
<pre><code>a = np.arange(9, -1, -1)     # a = array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])
b = a[np.arange(len(a))!=3]  # b = array([9, 8, 7, 5, 4, 3, 2, 1, 0])</code></pre>

Suraj: <p>Hi,</p>
<p>Did anyone figure an efficient way to pop up (delete) an entire column from a torch tensor? Do we have a function in PyTorch to do that? just like we have numpy.delete. Thanks!</p>

123nishant: <p>HI <a class="mention" href="/u/dev-team">@dev-team</a>  ,</p>
<p>Can somebody help out here ?</p>

erzaliator: <p>Okay, so here is a less memory efficient way to delete last n columns. I have only tested it on a 2-D tensor.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; def delete_last_n_columns(a, n):
...     n_rows = a.size()[0]
...     n_cols = a.size()[1]
...     assert(n&lt;n_cols)
...     first_cols = n_cols - n
...     mask = torch.arange(0,first_cols)
...     b = torch.index_select(a,1,mask) # Retain first few columns; delete last_n columns
...     return b
... 
&gt;&gt;&gt; a.size()
torch.Size([10, 536])
&gt;&gt;&gt; b = delete_last_n_columns(a,512)
&gt;&gt;&gt; b.size()
torch.Size([10, 24])
</code></pre>

J_Johnson: <p>While the below solution may be a bit heavy-handed for removing only one column, it works great when you need to remove many columns:</p>
<pre><code class="lang-auto">import torch

t=torch.arange(6)
print(t)

tensor([0, 1, 2, 3, 4, 5])

c=[0,1,2,4,5]
print(t[c])

tensor([0, 1, 2, 4, 5])
</code></pre>
<p>You can also use an array of booleans when deciding what to keep. This is probably the most efficient method when you want to create arguments to determine which indices to act on.</p>
<pre><code class="lang-auto">a = torch.arange(10)
row_cond=a!=5
print(a[row_cond])

tensor([0, 1, 2, 3, 4, 6, 7, 8, 9])
</code></pre>
<p>The nice thing about this method is you can also quickly call the negation via ~.</p>
<pre><code class="lang-auto">print(a[~row_cond])

tensor([5])
</code></pre>
<p>With this method, you can also strictly perform an operation only on certain values:</p>
<pre><code class="lang-auto">a[row_cond]=a[row_cond]*2
print(a)

tensor([ 0,  2,  4,  6,  8,  5, 12, 14, 16, 18])
</code></pre>

Astariul: <p>Using slices and concatenations :</p>
<pre><code class="lang-auto">import torch

x = torch.arange(4 * 8).reshape((4, 8))

print(x)

x = torch.cat([x[:, :1], x[:, 4:]], dim=1)

print(x)
</code></pre>
<p>prints :</p>
<blockquote>
<p>tensor([[ 0,  1,  2,  3,  4,  5,  6,  7],<br>
[ 8,  9, 10, 11, 12, 13, 14, 15],<br>
[16, 17, 18, 19, 20, 21, 22, 23],<br>
[24, 25, 26, 27, 28, 29, 30, 31]])<br>
tensor([[ 0,  4,  5,  6,  7],<br>
[ 8, 12, 13, 14, 15],<br>
[16, 20, 21, 22, 23],<br>
[24, 28, 29, 30, 31]])</p>
</blockquote>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 3]], which is output 0 of TBackward, is at version 4; expected version 3 instead

TMRajeh: <p>I’m facing this problem. some help, please?<br>
I tried many solutions before, but they didn’t work. I have a number “N” of agents, and each agent owns an independent actor and critic. Each agent has different states according to the label given to each agent.</p>
<pre><code class="lang-auto">all_agents = []

all_agents.append (Agent (actor_dims, critic_dims))

for agent_idx, agent in enumerate (all_agents):
    i = agent.agent_label
    critic_value_ = agent.target_critic.forward (states_[i], new_actions_cluster[i]).flatten ()

    critic_value = agent.critic.forward (states[i], old_actions_cluster[i]).flatten ()

    target = rewards[:, agent_idx] + agent.gamma * critic_value_

    critic_loss= F.mse_loss (critic_value.float (), target.float ())

    agent.critic.optimizer.zero_grad ()
    critic_loss.backward (retain_graph=True)

    actor_loss = agent.critic.forward (states[i], mu_cluster[i]).flatten ()
    actor_loss = -(T.mean (actor_loss))

    agent.actor.optimizer.zero_grad ()
    actor_loss.backward ()

    agent.critic.optimizer.step ()
    agent.actor.optimizer.step ()```
</code></pre>
<pre><code class="lang-auto">##################################
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 3]], which is output 0 of TBackward, is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!```</code></pre>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256/45">here</a>.</p>
Model runs faster on CPU than on GPU

michalt38: <p>I’m loading a yolov5 model on GPU in the following way:</p>
<p><code>model = torch.hub.load('ultralytics/yolov5', 'yolov5s')</code></p>
<p>I have measured it’s performance:</p>
<pre><code class="lang-auto">start = timer()
results = model(test_image.jpg')
end = timer()
print(end - start)
</code></pre>
<p>It gives me around 1 second which is very slow. However, when I run it on CPU it gives me about 0.2 seconds:</p>
<p><code>model = torch.hub.load('ultralytics/yolov5', 'yolov5s', device='cpu')</code></p>
<p>Why it is so slow on GPU and how can I boost it’s perfomance?</p>

ptrblck: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timer. Besides that, you should add warmup iterations and profile a few steps to get valid results.</p>
Cuda programming using Pytorch

Sumesh_Sankar: <p>I am using a loop, which uses three functions, these three function call another three function. So i run this logic in GPU, i got an error, “out of memory”. please tell me how do i manage the memory, by clearing the unwanted parameters after use and present in the GPU.</p>

ptrblck: <p>You can delete Python objects via <code>del object</code> and allow PyTorch to reuse the memory.<br>
Note however, that the underlying will only be freed if no references point to the deleted object anymore.</p>
Train loss and validation loss is almost same

Hafsa_Moontari_Ali: <p>The training loss and validation loss of my model is almost same from the second epoch. Then the training and validation loss keep reducing, but both the losses remain almost same as well.<br>
Does this mean the model is overfitting?</p>

ptrblck: <p>No, overfitting would be visible as a gap between the training and validation losses, where the training loss would be lower.</p>
Backpropagating the Loss from one model to another model

Summer_Breeze: <p>Hello! I need help with using one model to calculate the loss for another model! Essentially I use <code>model1</code> to generate a face image based on a real face image. <code>model2</code> which is the media pipe face mesh calculates the face mesh for both the real and fake faces and does MSEloss on it. I want to then backpropagate that loss to <code>model1</code>. Because I run the MSE on the output of the face mesh, it won’t backpropagate the gradients of <code>model1</code>. How can I do this?</p>

ptrblck: <aside class="quote no-group" data-username="Summer_Breeze" data-post="1" data-topic="166421">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/summer_breeze/48/55015_2.png" class="avatar"> Summer_Breeze:</div>
<blockquote>
<p>Because I run the MSE on the output of the face mesh, it won’t backpropagate the gradients of <code>model1</code>.</p>
</blockquote>
</aside>
<p>I don’t fully understand this claim so could you explain the issue in more detail, please?<br>
<code>MSELoss</code> won’t detach the computation graph and is differentiable, so I don’t know what the root cause of the failure is.</p>
How do I contribute to pytorch wiki?

keineahnung2345: <p>Hello, I’ve found a minor error in pytorch wiki(<a href="https://github.com/pytorch/pytorch/wiki/Codegen-and-Structured-Kernels" rel="noopener nofollow ugc"> Codegen and Structured Kernels</a> to be specific), I can clone it to my computer but how do I fork it and make a pull request?<br>
Thanks in advance.</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contribution Guide</a> as well as <a class="mention" href="/u/tom">@tom</a>’s <a href="https://lernapparat.de/fixing-your-first-pytorch-bug">blog post</a> including a video might be a good starter.</p>

keineahnung2345: <p>I’ve scanned the guide and blog post but they don’t mention how to create a pull request for the <strong>wiki page of pytorch</strong>. Because pytorch wiki page seem to belong to a different repository(<a href="https://github.com/pytorch/pytorch.wiki.git" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch.wiki.git</a>) and I can’t find any ways to fork the wiki page. So I’m afraid that the question is not answered.</p>

ptrblck: <p>I would stick to the same process or creating a GitHub issue first, discussing your changes, and maybe it’ll be as easy as directly edit the file for a maintainer.</p>
I`ve changed the voxel size from the config file and I keep getting the error below. Could you please give me some hints and tips?

Kubra_Budancamanak: <p>Traceback (most recent call last):                                                                                                      | 0/928 [00:00&lt;?, ?it/s]<br>
File “train.py”, line 221, in <br>
main()<br>
File “train.py”, line 168, in main<br>
train_model(<br>
File “/home2/kubra/OpenPCDet/tools/train_utils/train_utils.py”, line 150, in train_model<br>
accumulated_iter = train_one_epoch(<br>
File “/home2/kubra/OpenPCDet/tools/train_utils/train_utils.py”, line 52, in train_one_epoch<br>
loss, tb_dict, disp_dict = model_func(model, batch)<br>
File “/home2/kubra/OpenPCDet/tools/…/pcdet/models/<strong>init</strong>.py”, line 42, in model_func<br>
ret_dict, tb_dict, disp_dict = model(batch_dict)<br>
File “/home/kubra/anaconda3/envs/OpenPCDet/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home2/kubra/OpenPCDet/tools/…/pcdet/models/detectors/pointpillar.py”, line 11, in forward<br>
batch_dict = cur_module(batch_dict)<br>
File “/home/kubra/anaconda3/envs/OpenPCDet/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home2/kubra/OpenPCDet/tools/…/pcdet/models/backbones_2d/base_bev_backbone.py”, line 103, in forward<br>
x = torch.cat(ups, dim=1)<br>
RuntimeError: Sizes of tensors must match except in dimension 3. Got 36 and 35 (The offending index is 0)</p>
Zeroing certain weights in the classifier before weights update

papoo13: <p>Before updating the classifier’s head, I want to zero the grad of certain weights in the classifier. I am trying to implement a cross_entropy by first selecting a subset of nodes in the classifier (indexing them using the batch labels). I want to update those nodes’ connections and avoid changing the gradient of the rest of the weights in the classifier. My task is continual learning. In each step, I have a dataset with a subset of classes of the main dataset.<br>
After implementing the code for freezing the weights, I am getting meaningless results (worse and different than the time I was not freezing the weight). My hunch is that maybe I am indexing the weight matrix wrong.<br>
This is the code.</p>
<pre><code>    if self._args.freeze_classifier_weights is True:
            unq_task_targets = torch.unique(example_dict['task_labels'][0], sorted=True)
            
            
            row_indexes = list(set(self.classes) - set(unq_task_targets.tolist()))
            row_indexes_tensor = torch.tensor(row_indexes)
            for i, param in enumerate(self._model_and_loss._model.resnet.fc.parameters()):
                        #  we have 3 values in the parameters list: [torch.size([512]), torch.size([170,512]), torch.size([10])]
                        if i==1: #(we need the weights and not the biases therefore i=0)
                            param.grad[row_indexes_tensor,:] = torch.zeros_like(param.grad[row_indexes_tensor,:])
</code></pre>
<p>I appreciate any help and guidance on this issue.</p>