akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Commandline hangs when trying to use GPU (CTRL+C doesn't kill process)

marek-mottmac: <p>Hi,</p>
<p>I’m attempting to use Pytorch on a new Windows machine. I can’t seem to get a working pytorch configuration.</p>
<p>I am using Windows 10 with Nvidia RTX A6000. I have tried several driver versions, cuda versions and pytorch versions in various configurations but I can’t seem to use pytorch.</p>
<p>I don’t get an error message. Instead, any time I try to send anything to the GPU, the commandline freezes completely and CTRL+C doesn’t stop execution, I have to CTRL+Pause/Break to stop execution. For example the line torch.rand(3,3,device=‘cuda:0’) causes the cmd to hang. (but torch.rand(3,3) works fine)</p>
<p>My current software versions are:<br>
graphics driver version 460.89<br>
Cuda 11.1<br>
pytorch 1.8.2</p>
<p>torch.cuda.is_available() returns true<br>
torch.version.cuda returns 11.1</p>
<p>Any ideas of what I should try are appreciated.</p>

marek-mottmac: <p>Hello, I’ve solved my issue and it was caused by the Windows power performance being set to ‘power saving’. Switching it to ‘high performance’ has allowed me to use pytorch normally.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=10" title=":upside_down_face:" class="emoji only-emoji" alt=":upside_down_face:"></p>
No grad fn attribute for custom loss term

Vishal_Jain1: <p>Ive implemented a new loss function that tries to measure the circularity of a segmentation based on image moments. Ive combined this with the dice loss so my new loss is:<br>
L = dice_loss + log10(circ_loss)/3 + 1</p>
<p>When printing the variables L, dice_loss and circ_loss in the above I noticed that the circ_loss term had no grad_fn attribute.</p>
<p>I’ve got very little knowledge as to what this means, but I am worried this means backward prop is not taking into account the circ loss term. There is no error during training and the overall loss seems to be decreasing, but I’m just wondering what the lack of a grad fn attribute means.</p>
<p>Can provide any code that could be helpful.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d.png" data-download-href="https://discuss.pytorch.org/uploads/default/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d_2_690x49.png" alt="image" data-base62-sha1="5F46pDoYs3oXgpSNUTmojFWpxpz" width="690" height="49" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d_2_690x49.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d_2_1035x73.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27af41475ea87de1f2d3bd02a0fab0b4b340ab2d_2_1380x98.png 2x" data-dominant-color="060707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2772×200 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>EDIT:<br>
Adding requires_grad = true during initalisation of leaf tensors worked to bring the grad_fn attribute. So code does not have the issue detailed in the question. Not sure how to close question.</p>

AlphaBetaGamma96: <p>Could you print the entire loss function code?</p>
<p>Initially the error seems to be coming from how you define <code>circ_loss</code> as it has no <code>grad_fn</code> attribute (which can be seen for <code>dice_loss</code> and <code>L</code> via the <code>grad_fn=&lt;RsubBackward</code>&gt; and <code>grad_fn=&lt;AddBackward0&gt;</code> terms respectively). If you’re creating some data (or importing it from a numpy array) make sure to tell PyTorch it requires a gradient via the <code>requries_grad_()</code> attribute.</p>

gphilip: <p>Could you provide the code that computes <code>circ_loss</code>, starting from whatever inputs it is based on?</p>
Which backend is used by default on GPU/CPU?

jqliu: <p>which backend is used by default on GPU/CPU ?</p>

ptrblck: <p>The CPU is used by default and you can check it via creating a tensor without specifying the <code>device</code>: <code>print(torch.randn(1).device)</code>.</p>

jqliu: <p>Thanks for your reply. What I want to know is, which backend is used by default on GPU (nvidia 2080Ti). Is it cudnn or cuda?</p>

ptrblck: <p>cuDNN is a library to accelerate workloads, such as convolutions etc., and will be used if it’s available, while CUDA is used to execute code on the GPU.<br>
The binaries ship with both while you would need to install cuDNN separately into your local CUDA toolkit if you want to build PyTorch from source to be able to use it.</p>
RuntimeError: Module 'NnapiModule' has no attribute 'weights' (This attribute exists on the Python module, but we failed to convert Python type: 'list' to a TorchScript type.):

Realdr4g0n: <p>hello.</p>
<p>I am trying to quantize my custom model and apply nnapi.<br>
nnapi prototype “<a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.7.1 documentation</a>”</p>
<p>I followed the example provided here.<br>
First, as in mobilenet v2, a quant pair was created, inherited as quantizable_my_model, and the pretrained weight was loaded, and the example was followed.</p>
<p>But nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(MY_MODEL,MY_TENSOR)<br>
The following error occurred in.</p>
<pre><code class="lang-auto">RuntimeError:
Module'NnapiModule' has no attribute'weights' (This attribute exists on the Python module, but we failed to convert Python type:'list' to a TorchScript type.):
  File "/home/ubuntu/anaconda3/envs/nnapi_2/lib/python3.6/site-packages/torch/backends/_nnapi/prepare.py", line 35
    def init(self):
        assert self.comp is None
        self.weights = [w.contiguous() for w in self.weights]
                                                ~~~~~~~~~~~~ &lt;--- HERE
        comp = torch.classes._nnapi.Compilation()
        comp.init(self.ser_model, self.weights)
</code></pre>
<p>Please help me what to do.</p>

Vasiliy_Kuznetsov: <p>cc <a class="mention" href="/u/david_reiss">@David_Reiss</a> , do you know who can take a look at this?</p>

honglin_gao: <p>I meet the same problem. Did you address it?</p>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking arugment for argument weight in method wrapper_cudnn_batch_norm)

ekuama: <pre><code class="lang-auto">class DenseLayer(nn.Module):
    def __init__(self, in_channels, growth_rate):
        super(DenseLayer, self).__init__()
        self.norm = nn.BatchNorm2d(in_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv = nn.Conv2d(in_channels, growth_rate, kernel_size=3, stride=1, padding=1, bias=True)
        self.drop = nn.Dropout2d(p=0.2)

    def forward(self, x):
        x = self.norm(x)
        x = self.relu(x)
        x = self.conv(x

        x = self.drop(x)
        return x  
class DenseBlock(nn.Module):
    def __init__(self, in_channels, growth_rate, n_layers, upsample=False):
        super(DenseBlock, self).__init__()
        self.upsample = upsample
        self.layers = nn.ModuleList([DenseLayer(in_channels + i * growth_rate, growth_rate) for i in range(n_layers)])

    def forward(self, x):
        if self.upsample:
            new_features = []
            for layer in self.layers:
                out = layer(x)
                x = torch.cat([x, out], 1)
                new_features.append(out)
            return torch.cat(new_features, 1)
        else:
            for layer in self.layers:
                out = layer(x)
                x = torch.cat([x, out], 1)  # 1 = channel axis
            return x

class Bottleneck(nn.Module):
    def __init__(self, in_channels, growth_rate, n_layers):
        super(Bottleneck, self).__init__()
        self.bottleneck = DenseBlock(in_channels, growth_rate, n_layers, upsample=True)

    def forward(self, x):
        return self.bottleneck(x)
</code></pre>
<p>Above is the code I am currently using. When I make a call to the bottleneck module, I get the error RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking arugment for argument weight in method wrapper_cudnn_batch_norm)</p>

ptrblck: <p>The error is raised if the input and parameters are on different devices, which could happen if you forget to push either one of them to the GPU.<br>
Make sure to use <code>model.to('cuda')</code> as well as <code>input = input.to('cuda')</code> before executing the code.</p>
<p>PS: your current code is a bit hard to read so please format it by wrapping it into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

ekuama: <p>Thanks for the response. I just realized that I forgot to push my decoder to GPU as well.</p>
Different cpu inference results on different machines using libtorch

hanabi: <p>Hi,<br>
I trained a pytorch model using CUDA using pytorch python, and try to inference using C++.<br>
First, I exported the model using torch script (via tracing), with eval mode (tracing is done below the line model.eval()). Also I changed the device to cpu before tracing. The machine for training is Ubuntu 18.04, with GTX3070.</p>
<p>My inference tests are done in my laptop (Windows10) and linux clusters (RedHat linux), but the inference results are different from each other. Both machines don’t have GPUs, so the only way for inference is with CPUs.</p>
<p>Even if the differences are very small (i.e. 0.9824 vs 0.9827 and so on…) I don’t want any discrepancy between them. How can I avoid this?</p>
<p>The libtorch version in those machines are the same (1.9.0)</p>
The shortest distance from the same element in a matrix

QianMo-wangxiaoyu: <p>[[1,0,1,0],     =&gt;[[1.414,0,1.414,0],    caculate the ‘1’ element shortest distance<br>
[0,1,0,0]]     =&gt;[0,1.414,    0  ,0]]</p>
<p>[[1,0,1,1],    =&gt;[[1.414,0, 1,1],     caculate the ‘1’ element shortest distance<br>
[0,1,0,0]]     =&gt;[0,1.414,0,0]]</p>
Using libtorch get different results

zsir: <p>Hello！ <img src="https://discuss.pytorch.org/images/emoji/apple/cold_face.png?v=10" title=":cold_face:" class="emoji" alt=":cold_face:"><br>
I trained a model (using GPU) under ubuntu, then I put this model on the cpu (.cpu()) and turned it into a .pt file through torch.jit.trace, and another computer (windows) uses the cpu version libtorch load model, under the same input, returned different results.<br>
Do you have any idea where this difference could be coming from?</p>

ptrblck: <p>How large are these differences? Since you are tracing the model, I assume that no data-dependent control flow is expected? Are you using static inputs (e.g. <code>torch.ones</code>) or “real” data?</p>

zsir: <p>There is no control flow in the model and inputs is duplicate data，output is a matrix,some elements in the matrix have an error of about e-2. <img src="https://discuss.pytorch.org/images/emoji/apple/persevere.png?v=10" title=":persevere:" class="emoji" alt=":persevere:"></p>

ptrblck: <p>Thanks for the update! Are you seeing the same difference with the plain PyTorch model without tracing it and using <code>liborch</code>? Also, are you seeing a difference on the same OS between the Python and <code>libtorch</code> model?</p>
List hooks applied to layer

ZimoNitrome: <p>When adding a hook to a layer, let’s say by <code>nn.utils.spectral_norm(my_layer)</code>, how can I confirm that the hook in fact registered?</p>
<p>If I added more hooks, could I list them all somehow?</p>

nitaifingerhut: <p>I think you access this protected attribute of the layer<br>
something like <code>layer._forward_hooks</code> which holds an OrderedDict of all registered forward hooks.</p>
<p>an example:</p>
<ol>
<li>get the model’s children: x = list(model.children())</li>
<li>get forward hooks of the 1st layer: x[0]._forward_hooks</li>
</ol>
<p>similarly, you can see:<br>
<code>layer._backward_hooks</code><br>
<code>layer._forward_pre_hooks</code></p>

ZimoNitrome: <p>Thanks, I will try this out!</p>
<p>This was surprisingly difficult to find in thr documentation/tutorials.</p>
Specify device_ids in barrier()

aguirguis: <p>Hello!<br>
For some reason, I’m getting this warning:</p>
<pre><code class="lang-auto">[W ProcessGroupNCCL.cpp:1569] Rank 6 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
</code></pre>
<p>This is triggered on <code>model.cuda(device_id)</code></p>
<p>The problem is that the code sometimes really hang!<br>
I was wondering if there is an explanation to what’s going on and optimally a solution to this issue.</p>
<p>Thank you very much.</p>

amirhf: <p>I am assuming you are using a distributed launch. The warning message is self-explanatory. It seems like in each particular process other GPUs are still visible to the process. Ideally, on local_rank X, you want to only GPU X to be visible. There are some workarounds:</p>
<ol>
<li>Set CUDA_VISIBLE_DEVICES = int(os.environ[“LOCAL_RANK”]) in your main worker function</li>
<li>use <code>torch.distributed.barrier(device_ids=int(os.environ["LOCAL_RANK"])</code>
</li>
</ol>
<p>In your case, RANK 6 is using GPU 0 for barrier but it should use GPU 6 as barrier.<br>
You have to also be careful about how you set the device you want to use in your script. the set device should also be equal to the LOCAL_RANK.</p>

aguirguis: <p>Thanks <a class="mention" href="/u/amirhf">@amirhf</a> for your answer.<br>
This indeed solved the issue.</p>

amsword: <p>recently, i also saw such issues with pytorch 1.9, but if it is pytorch 1.7.1, there is no such issue.</p>
Unrolled UNet, memory not increasing - should we expect a linear scaling with #unrolled steps?

qcyza: <p>Hi, I am applying U-Net iteratively, and try to get the gradient update of the very first input, however, the network memory is not increasing significantly (not scale linearly with the number of unrolled steps, every. extra unrolled step only add ~10mb gpu memory). Does this make sense or are there may be some bugs…?</p>
<p>Flow:<br>
input x → Unet → x_1 → Unet → x_2 → Unet → … → x_n-1 → Unet → x_n → loss<br>
and the gradient calculated is respect to x</p>
How to get "finfo" in C++ torchlib like that in pytorch

dbsx: <p>I am using C++ torchlib, but I don’t know what to do it in c++ like that in pytorch:</p>
<pre><code class="lang-python">        min_real = torch.finfo(self.logits.dtype).min
# or
        min_real = torch.finfo(self.logits.dtype).tiny
</code></pre>

ptrblck: <p>I don’t think <code>finfo</code> is exposed in <code>libtorch</code>, but you should be able to use e.g.:</p>
<pre><code class="lang-auto">const auto int_min = std::numeric_limits&lt;int&gt;::min();
const auto int_max = std::numeric_limits&lt;int&gt;::max();
</code></pre>
"RuntimeError: CUDA error: out of memory" for no reason

1309123499: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b088dff2355abc53cf534f8479928874ec4b8f17.png" alt="image" data-base62-sha1="pbHhPQXv5G0EQDiuLFerOJH9c0v" width="638" height="193"><br>
even with the simplest calculation i got Error OOM and i have already reinstall conda and cuda.<br>
as to the env info<br>
PyTorch version: 1.4.0+cu100<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0</p>
<p>OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
CMake version: Could not collect</p>
<p>Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.0.130<br>
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti<br>
Nvidia driver version: 460.91.03<br>
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.5<br>
[pip3] torch==1.4.0+cu100<br>
[pip3] torchvision==0.5.0+cu100<br>
[conda] torch 1.4.0+cu100 pypi_0 pypi<br>
[conda] torchvision 0.5.0+cu100 pypi_0 pypi</p>
<p>thanks for your attention!</p>

Rahul_Chand: <p><a class="mention" href="/u/1309123499">@1309123499</a><br>
You can check the GPU usage via <code>nvidia-smi</code> command. There you can see if there is an already existing process that is taking up the GPU memory &amp; kill it if needed.</p>

1309123499: <p>That is the problem, about 5-20m/11G. I just cannot use the gpu.</p>

ptrblck: <p>Make sure to restart the machine after installing CUDA or the driver, as this is often the root cause of these issues.</p>

1309123499: <p>I connect the remote server to train my model.  After encountering this problem, I have reinstalled the cuda and the rebuilt the conda environment, as well as reconnected to the server. and this problem happened between two debugings, which means i did nothing about the  machine or the environment. It  really makes me confused.</p>
non_blocking=True reserves extra memory in GPU 0

ManiadisG: <p>I am running experiments using DDP and I observed that, when setting non_blocking=True to move the batch to the GPUs, memory is reserved at GPU 0, even if it is not being used by the experiment (in which case utilization remains at 0%). When non_blocking=False, this does not happen.</p>
<p>Is this normal? What might be causing it?</p>

ptrblck: <p>I’m unable to reproduce the additional memory usage of GPU0 using the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DDP example</a> and adding <code>non_blocking=True</code> to the <code>to()</code> operation, so could you post a minimal, executable code snippet, please?</p>
High score for blurred image ssim metric

hix: <p>I think SSIM sometimes scores higher in Blurred images than in some clear images.</p>
<p>I want to mention the problem with this SSIM, but I don’t know why.</p>
<p>Can you tell me why SSIM gets a higher score on the blurred image?</p>
<p>This is Paper Example<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c06c9f2af5357d656d9b18c8aee66483ae395ba.png" alt="image" data-base62-sha1="aQyPDlSTdnXXPLY4NWwS8qLQWUO" width="408" height="260"></p>
<p>(<a href="https://www.researchgate.net/figure/Comparison-of-SSIM-and-MSE-performances-for-Einstein-image-altered-with-different_fig5_283461887" rel="noopener nofollow ugc">https://www.researchgate.net/figure/Comparison-of-SSIM-and-MSE-performances-for-Einstein-image-altered-with-different_fig5_283461887</a>)</p>
<p>I think h image so claer than g Image but ssim metric high score of g image</p>
Share the output of one DNN to another DNN

Mohamed_Nabih: <p>I have two DNNs the first one returns two outputs. I want to use one of these outputs in a second class that represents another DNN as in the following example:</p>
<pre><code class="lang-auto">import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim

    class Net(nn.Module):
        def __init__(self):
            ..
        def forward(self, x):
            ..
            return x, z


    class Net2(nn.Module):
        def __init__(self):
            ..
        def forward(self, v):
            y = torch.cat(v, x)
            return y
</code></pre>
<p>I want to pass the output (x) to the second class to be concatenated to another variable (v). I found a solution to make the variable (x) as a global variable, but I need another efficient solution</p>

ptrblck: <p>You can pass <code>x</code> as another argument to the <code>forward</code> of <code>Net2</code> and wouldn’t need to use global variables.</p>

Mohamed_Nabih: <p>do you mean in <code>Net2</code> the forward function will be like as <code>def forward(self, v, x):</code></p>

ptrblck: <p>Yes, exactly. This allows you to pass both inputs to it which avoids the usage of some global variables.</p>
Mat1 and mat2 shapes cannot be multiplied (16x13 and 16x1)

vijayshankarrealdeal: <pre><code class="lang-auto">import torch 
import torch.nn as nn
import torchvision
from torch.utils.data import DataLoader ,Dataset 
import numpy as np
import math
from tqdm.auto import tqdm

class WineDataset(Dataset):
  def __init__(self):
    xy = np.loadtxt('/content/wine.csv',delimiter=',',dtype = np.float32,skiprows=1)
    self.x = torch.from_numpy(xy[:,1:]) 
    self.y = torch.from_numpy(xy[:,[0]])
    self.shape = xy.shape[0]

  def __getitem__(self,index):
    return self.x[index],self.y[index]

  def __len__(self):
    return self.shape

dataset = WineDataset()


class Model(nn.Module):
  def __init__(self):
    super(Model,self).__init__()
    self.fc1 = nn.Linear(13,30)
    self.fc2 = nn.Linear(30,1)

  def forward(self,x):
    out = self.fc1(x)
    out = torch.nn.functional.relu(out)
    out = self.fc2(x)
    return out


model = Model()
loss = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())

train_losses = []
for i in range(5):
  train_loss = []
  for x,y in tqdm(loader):
    optimizer.zero_grad()
    output = model(x)
    loss_ = loss(output,y)
    loss_.backward()
    train_loss.append(loss_.item())
    optimizer.step()
  train_loss = np.mean(train_loss)
  train_losses[i] = train_loss
</code></pre>

nitaifingerhut: <p>you did not specify sizes of input…<br>
this error means your weights sizes mismatch with input size.<br>
your model expects input of size (N,*,13) where N is batch size.<br>
I guess your input need some transpose or something alike to match the initialized weights size.</p>
How can I calculate this steadily decreasing WGAN metric?

ZimoNitrome: <p>The loss of Wasserstein GANs is typically unconstrainted:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/693d03bbd9cc64df1e49450375624d38ed926051.png" alt="image" data-base62-sha1="f0YRaotV4hQYPLN4PSbwWwUq3S1" width="640" height="480"><br>
(<a href="https://machinelearningmastery.com/how-to-code-a-wasserstein-generative-adversarial-network-wgan-from-scratch/" rel="noopener nofollow ugc">image source</a>)</p>
<p>But I have seen examples of some WGAN metric that is steadily decreasing, for example this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b16159e52cc237ba5bdfd0508afceb8ffb19a36d.jpeg" alt="image" data-base62-sha1="pjb5laJd0lTCQHjs20Lwnzzjrt3" width="577" height="371"><br>
from the original <a href="https://arxiv.org/pdf/1701.07875.pdf" rel="noopener nofollow ugc">Wasserstein GAN paper</a>. But how can this “Wasserstein metric” easily be calculated?</p>
<p>Generally, the discriminator loss is just D(x) - D(G(Z)) and the generator loss is D(G(Z)) but these produce losses akin to the top graph above. Admittedly I lack the math skills of understanding the original WGAN paper.</p>
Installation problem (OSError: [Errno 2])

frenchbones: <p>Hello!</p>
<p>I am having trouble installing torch. Here is the error:</p>
<pre><code class="lang-auto">ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\h_bas\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\caffe2\\python\\serialized_test\\data\\operator_test\\collect_and_distribute_fpn_rpn_proposals_op_test.test_collect_and_dist.zip'
</code></pre>
<p>OS: Windows <code>10 21H1 (build 19043.1202)</code><br>
pip version: <code>21.2.3</code><br>
Python version: <code>3.9.7</code></p>
<p>I’ve been looking for fixes, but I couldn’t find anything.<br>
Would anyone have a solution? (Sorry if this as already been posted somewhere here)<br>
If anyone has more questions on my setup I’ll be happy to provide more info.</p>
<p>Thank you.</p>
<p>PS: This is what I am running to install</p>
<pre><code class="lang-auto">pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>NVIDIA-SMI: <code>471.96</code><br>
CUDA Version: <code>11.4</code></p>

Rahul_Chand: <p><a class="mention" href="/u/frenchbones">@frenchbones</a></p>
<p>Hi, this usually happens because the file path is too big. You can try the following <a href="https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/" rel="noopener nofollow ugc">link</a> to increase the acceptable length of file path.</p>
<pre><code class="lang-auto">C:\\Users\\h_bas\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\caffe2\\python\\serialized_test\\data\\operator_test\\collect_and_distribute_fpn_rpn_proposals_op_test.test_collect_and_dist.zip'
</code></pre>

frenchbones: <p>Thanks a lot that fixed it!</p>
Error - "one of the variables needed for gradient computation has been modified by an inplace operation" (not resolved using previous such posts)

Vaibhav_Raj: <p>I am trying to use the code from <a href="https://github.com/escuccim/vaegan-pytorch/blob/master/VAEGAN_Implementation_128.ipynb" class="inline-onebox" rel="noopener nofollow ugc">vaegan-pytorch/VAEGAN_Implementation_128.ipynb at master · escuccim/vaegan-pytorch · GitHub</a> to train a VAEGAN.<br>
This code is from 2 years ago and raises the error “one of the variables needed for gradient computation has been modified by an inplace operation”. This error is not encountered in PyTorch 1.4 but I would like to know how to deal with it in the latest versions.</p>
<p>I have looked for suggestions in many topics available on Github and Pytorch Discussions and got some like</p>
<ol>
<li>Detaching/Cloning variables - This process doesn’t give an efficient solution because there are terms like the mse_loss (which uses output from the discriminator) that are added to the encoder’s loss and thus don’t allow easy detaching (would require multiple runs of the model per forward pass to use detach)</li>
<li>Calculate losses first and then do backward().</li>
<li>Use the inputs argument in backward to update only the required model component (encoder, decoder or discriminator).<br>
This is what I tried to replace the corresponding part of the source code by:</li>
</ol>
<pre><code class="lang-auto">loss_encoder.backward(inputs=list(net.encoder.parameters()))
optimizer_encoder.step()
net.zero_grad()

if train_dec:
    loss_decoder.backward(inputs=list(net.decoder.parameters()))
    optimizer_decoder.step()
    net.discriminator.zero_grad()
if train_dis:
    loss_discriminator.backward(inputs=list(net.discriminator.parameters()))
    optimizer_discriminator.step()
</code></pre>
<p>However, this ended up giving the same error. Please guide on how this can be dealt with</p>

Vaibhav_Raj: <p>Turns out that with a combination of the “inputs” and “retain_graph”, it works.<br>
I also noted that the problem was because the updates that I had made to the code were not working live on Colab.</p>
Factorials that will work with cuda?

ccook: <p>Is there a way to do factorials that will work with cuda? I can use <code>scipy.special.factorial</code> if using the CPU, but it gives an error when using cuda (because it’s using numpy under the hood).</p>

KFrank: <p>Hi ccook!</p>
<aside class="quote no-group" data-username="ccook" data-post="1" data-topic="131716" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ccook/48/30334_2.png" class="avatar"> ccook:</div>
<blockquote>
<p>Is there a way to do factorials that will work with cuda?</p>
</blockquote>
</aside>
<p>Yes.  The Euler gamma function is essentially the factorial function.<br>
Because it grows so fast we often use the logarithm of the gamma<br>
function for numerical work.  Pytorch provides <a href="https://pytorch.org/docs/1.9.0/generated/torch.lgamma.html" rel="noopener nofollow ugc">torch.lgamma()</a>, a<br>
full-featured, cuda-enabled, autograd-enabled pytorch tensor function.</p>
<p>Best.</p>
<p>K. Frank</p>

ccook: <p>Wow, excellent – this worked perfectly. Thank you.</p>
<p>(One note for future readers is that it’s <code>torch.lgamma(k+1).exp() ~= factorial(k)</code> – the <code>+1</code> is needed).</p>
Question about ".data"

S200331082: <p>I have some doubts about pieces of codes:</p>
<pre><code class="lang-auto">    def init_conv(self, inp, out, weight, bias, convert=True):
        conv = nn.Conv2d(inp, out, 3, 2, 1, bias=True)
        if convert:
            weight = self.convert_to_torch_weight(weight)
        conv.weight.data = torch.Tensor(weight).float()  
        conv.bias.data = torch.Tensor(bias).float()  
        return conv
</code></pre>
<p>I just can’t understand  the “.data” in them, can anyone explain what it is used for?<br>
I did’t find the parameter “data” in “nn.Conv2d” from “conv.py”<br>
many thanks</p>

googlebot: <p>it is an internal attribute, that is mostly used instead of torch.no_grad() context, as autograd misses tensor mutations via .data. and you have to initialize parameter values with autograd disabled, that’s what your code does.</p>

gphilip: <aside class="quote no-group" data-username="S200331082" data-post="1" data-topic="131724">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/s200331082/48/39704_2.png" class="avatar"> S200331082:</div>
<blockquote>
<p>I did’t find the parameter “data” in “nn.Conv2d” from “conv.py”</p>
</blockquote>
</aside>
<p><code>Conv2d.weight</code> is inherited from the superclass <code>_ConvNd</code> of <code>Conv2d</code>, as you can see in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html" rel="noopener nofollow ugc">the source code</a>.</p>
<p>As you can see from the above link, <code>ConvNd.weight</code> is an object of type <code>Parameter</code>. And <code>Parameter</code>  has an attribute <code>data</code>, as you can see <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" rel="noopener nofollow ugc">here</a>.</p>
Batch size 44, bad idea?

Samuel_Bachorik: <p>Hi, as I know everyone use batch sizes multiplied by two. 8,16,32,64,128 etc…<br>
Is it bad to use batch size for example 44? My GPU can not take 64 but I want to save some time so I don’t want to use 32. Batch 44 I can fit to GPU, can I use it or it is bad for training ?</p>
<p>Thank you!</p>

googlebot: <p>no, it is fine, with multi-dimensional arrays the negative effect is greatly dimished, it is also mostly about features/channels dimension. still, it may be marginally better to choose batch sizes that are divisible by 2,4,8,16,32.</p>
Output Mismatch between pytorch model and converted onnx model for blazeface

Baishali_Chaudhury: <p>Hi</p>
<p>pytorch version = 1.6.0+cpu<br>
onnxruntime version =1.7.0<br>
environment =ubuntu</p>
<p>I am trying to export a pretrained pytorch model for “blazeface” face detector in onnx.<br>
Pytorch model definition and weights file taken from : <a href="https://github.com/hollance/BlazeFace-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hollance/BlazeFace-PyTorch: The BlazeFace face detector model implemented in PyTorch</a><br>
I exported the pytorch model to onnx using torch.onnx.export. And then used onnxruntime  to compare the outputs from pytorch model and onnx model. But the outputs dont match</p>
<p>Code used:</p>
<p>import io<br>
import sys</p>
<p>import numpy as np<br>
import onnxruntime<br>
import torch.onnx<br>
import onnx</p>
<p>from blazeface import BlazeFace</p>
<p>def to_numpy(tensor):<br>
return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()</p>
<p>def main():<br>
print(torch.<strong>version</strong>)<br>
<span class="hashtag">#print</span>(onnx.<strong>version</strong>)<br>
print(onnxruntime.<strong>version</strong>)<br>
net = BlazeFace()<br>
net.load_weights(“blazeface.pth”)<br>
torch.onnx.export(net, torch.randn(1, 3, 128, 128, device=‘cpu’), “blazeface.onnx”,<br>
input_names=(“image”,), output_names=(“preds”, “confs”), opset_version=9, verbose=True<br>
)</p>
<pre><code>onnx_model = onnx.load("blazeface.onnx")
onnx.checker.check_model(onnx_model)

#comparing onnx output to pytorch output
torch_model = BlazeFace()
torch_model.eval()
x = torch.ones(1, 3, 128, 128, requires_grad=True)
torch_out = torch_model(x)
ort_session = onnxruntime.InferenceSession("blazeface.onnx")
# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(x)}
ort_outs = ort_session.run(None, ort_inputs)
# compare ONNX Runtime and PyTorch results
np.testing.assert_allclose(to_numpy(torch_out[0]), ort_outs[0], rtol=1e-03, atol=1e-05)

print("Exported model has been tested with ONNXRuntime, and the result looks good!")
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
sys.exit(main() or 0)</p>
<p>Output:<br>
AssertionError:<br>
Not equal to tolerance rtol=0.001, atol=1e-05</p>
<p>Mismatched elements: 14335 / 14336 (100%)<br>
Max absolute difference: 99.00309<br>
Max relative difference: 799.63086<br>
x: array([[[ 0.335632,  0.003543,  0.032026, …,  0.004705, -0.224979,<br>
-0.469438],<br>
[ 0.180302,  0.292883, -0.117604, …, -0.218451,  0.165066,…<br>
y: array([[[-9.223551e-01,  3.763577e-01,  2.190542e+01, …,<br>
1.208612e+00,  5.133753e+00, -6.580073e+00],<br>
[-7.786160e-01,  1.016221e+00,  2.847925e+01, …,…</p>
<p>Process finished with exit code 1</p>

Tomas_Mendoza: <p>Hello,</p>
<p>Can you please share with us how you solved the problem? I am having a similar issue but don’t know how to fix it. Thanks!</p>
Causal 2D convolution

Superklez: <p>I’m trying to implement a causal 2D convolution, wherein the “width” of my “image” is temporal in domain. What I’ve implemented so far is as follows (it’s rather simple and only works with kernel sizes that are odd):</p>
<pre><code class="lang-auto">class CausalConv2d(nn.Module):
    def __init__(
        in_channels: int,
        out_channels: int,
        kernel_size: tuple = (3, 3)
    ):
        super(CausalConv2d, self).__init__()
        self.pad = nn.ZeroPad2d((kernel_size[1] - 1, 0, kernel_size[0] // 2,
            kernel_size[0] // 2))
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size)

    def forward(self, x: Tensor):
        x = self.pad(x)
        x = self.conv(x)
        return x
</code></pre>
<p>Is this implementation correct? Any help would be great.</p>
Same input and ouput 3rd dimmension in ConvTranspose3d

Rituraj_19_Dutta_100: <p>I’m using Transposed Convolutions in my network and I want to keep the shape of the output 3rd(indexing from 1) dimension same as in input. How can I do that with ConvTranspose3d() ?</p>
<pre><code class="lang-auto">import torch

import torch.nn as nn

x = torch.randn(1,3,1,128,128).cuda()

layer = nn.ConvTranspose3d(in_channels=3,out_channels=3,kernel_size=4,stride=2,padding=1).cuda()

output = layer(x)

print(output.shape)
</code></pre>
<pre><code class="lang-auto">Actual Output: (1,3,2,256,256)
Expected Output : (1,3,1,256,256)
</code></pre>

ptrblck: <p>You can specify the <code>kernel_size</code>, <code>stride</code>, <code>dilation</code>, and <code>padding</code> for each dimension separately and using a single scalar value is a convenient way to set all dimensions to the same value.<br>
This should thus work:</p>
<pre><code class="lang-python">x = torch.randn(1,3,1,128,128).cuda()

layer = nn.ConvTranspose3d(
    in_channels=3,
    out_channels=3,
    kernel_size=(1, 4, 4),
    stride=(1, 2, 2),
    padding=(0, 1, 1)).cuda()

output = layer(x)

print(output.shape)
&gt; torch.Size([1, 3, 1, 256, 256])
</code></pre>
How to set correct_bias in pytorch optim.Adamw ike huggingface transformers Adamw

cool_boys: <p>here is <a href="https://huggingface.co/transformers/_modules/transformers/optimization.html#AdamW" rel="noopener nofollow ugc">huggingface AdamW</a> and there is correct_bias  parameter. And I want to know which parameter it corresponds to in <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adamw.html#AdamW" rel="noopener nofollow ugc">pytorch optim.Adamw</a>. Is it amsgrad ? Thx!</p>

gphilip: <aside class="quote no-group" data-username="cool_boys" data-post="1" data-topic="131715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cool_boys/48/42117_2.png" class="avatar"> cool_boys:</div>
<blockquote>
<p>Is it amsgrad</p>
</blockquote>
</aside>
<p>From a quick look at the paper from which AMSgrad is taken, the answer is NO. I am not an expert, though, so this may be wrong.</p>
Loss and Gradients Fluctuating very rapidly

Harsh_Choudhary: <p>I am using a 3 layered neural net with a single input and single output I am using  loss.backward() and then updating the weights but my loss fluctuates very rapidly and it never converges, However I am pretty sure that my loss function has a global minima.</p>

gphilip: <p>It is very difficult to help you based on the abstract description that you provided in your post. This is just because there are literally millions of code scenarios that could show the behaviour that you describe, and since your description is so vague, we have no clue which of these scenarios apply to you.</p>
<p>You will have a <em>far better</em> chance of getting help with your problem if you provide the following (more concrete) information:</p>
<ul>
<li>
<strong>Complete,  executable code</strong> which exhibits the behaviour that you want to address
<ul>
<li>Ideally, this should include just enough code to demonstrate the points that you want us to address, while being executable. The fewer the lines of code involved, the easier (usually …) it is for others to read, understand, and debug it.</li>
<li>If possible, try to come up with code that takes in random tensors of the proper shapes as input, instead of actual data (such as images or text from the real world).</li>
</ul>
</li>
<li>A description of what you expected from this code, and what you actually saw.</li>
</ul>
Attention weights in TransformerEncoderLayer

rasoolfa: <p>Hi,</p>
<p>How to access attention weights/outputs in TransformerEncoderLayer/TransformerEncoder?<br>
[ Referring to <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a> ]<br>
Thanks</p>

sachinruk: <p>Bump.</p>
<p>I think this is pretty important, and huggingface transformers do this well. Is there a way to access the last Attention matrix in <code>TransformerEncoder</code>?</p>

cifkao: <p>I was dealing with a similar problem and couldn’t find a solution, so I created a <a href="https://github.com/cifkao/nopdb" rel="noopener nofollow ugc">library called <strong>NoPdb</strong></a>, which allows to retrieve the value of an arbitrary tensor (in fact, any variable inside any Python function).</p>
<p>In the Transformer tutorial, it should be enough to do something like:</p>
<pre><code class="lang-python">import nopdb

# Evaluate the model while capturing local variables of the 1st attention layer
with nopdb.capture_calls(model.transformer_encoder.layers[0].self_attn.forward) as calls:
    evaluate(model, test_data)

# Now we have access to the attention weights and outputs
print(calls[0].locals['attn_output_weights'])
print(calls[0].locals['attn_output'])
</code></pre>
<p>(Note that this way, we are accumulating some tensors over the whole dataset. To avoid running out of GPU memory, it would be better to capture the variables for each batch separately, only select the tensors we need and move them to the CPU. This is left as an exercise for the reader. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:">)</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/cifkao/nopdb">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/cifkao/nopdb" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://repository-images.githubusercontent.com/339834164/d91b4e00-9201-11eb-84bd-f563feba4211" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/cifkao/nopdb" target="_blank" rel="noopener nofollow ugc">GitHub - cifkao/nopdb: NoPdb: Non-interactive Python Debugger</a></h3>

  <p>NoPdb: Non-interactive Python Debugger. Contribute to cifkao/nopdb development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Working with 3D images and handle them

vinc2905: <p>Hey everyone,</p>
<p>I am currently working on a CNN that will use PET images to classify Alzheimer’s. I have preprocessed the images with an external program and already converted them into tensors. These tensors have the size [1, 169, 208, 179]. Is it advisable to scale down this size due to the probable high computing power and if so, which values would be suitable?</p>
<p>Or would it even be better if I convert the images to tensors myself and then resize them in the same step? If so, how do I handle this with the 3D images, since I’ve mostly only seen tutorials with 2D images?</p>
<p>If someone could explain this to me briefly, it would be a great help.</p>
Using dual model

han-yeol: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36468feed81e964919e238ee2cd558cd7ea1b75d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/36468feed81e964919e238ee2cd558cd7ea1b75d" title="IMG_0477"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36468feed81e964919e238ee2cd558cd7ea1b75d_2_690x390.jpeg" alt="IMG_0477" data-base62-sha1="7K8XF5BN0cdZiz6GTZZ7Truu1LD" width="690" height="390" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36468feed81e964919e238ee2cd558cd7ea1b75d_2_690x390.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36468feed81e964919e238ee2cd558cd7ea1b75d_2_1035x585.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36468feed81e964919e238ee2cd558cd7ea1b75d_2_1380x780.jpeg 2x" data-dominant-color="F3F1E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_0477</span><span class="informations">1767×1000 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I faced a big problem.</p>
<p>Model 1 was trained using a single model. Only inferences will be made without any backpropagation,<br>
Since model 2 is a heavy model, it must be trained through two GPUs in parallel.</p>
<p>Can I get some code or advice?</p>
Unable to load Adam optimizer

44_Pashikanti_Pranit: <p>I am try to classify dogs from dogs-vs-cats dataset and the article which i am referencing is <a href="https://towardsdatascience.com/pytorch-vision-binary-image-classification-d9a227705cf9" rel="noopener nofollow ugc">here</a><br>
The Error i am getting is attached in below image and code is <a href="https://colab.research.google.com/drive/1dsj34GTnLzIAe48e4Iv3wvTIGqQka4L6#scrollTo=q7f9zkL345oF" rel="noopener nofollow ugc">here</a>)]<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec23149bf090c03371ce8fa57d993608c060b627.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec23149bf090c03371ce8fa57d993608c060b627" title="Screenshot (374)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec23149bf090c03371ce8fa57d993608c060b627_2_690x388.png" alt="Screenshot (374)" data-base62-sha1="xGXQfbvdL7zmVNhPFBGBAvTnzWT" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec23149bf090c03371ce8fa57d993608c060b627_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec23149bf090c03371ce8fa57d993608c060b627_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec23149bf090c03371ce8fa57d993608c060b627_2_1380x776.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (374)</span><span class="informations">1920×1080 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

gphilip: <p>Typo: “paramters” should be “parameters”.</p>

44_Pashikanti_Pranit: <p>Yeah, i get i was just going to  update the post thanks</p>
Change values in specific tensor positions

ptrch_c_m: <p>Hello,</p>
<p>I have 2D tensors, where each row starts with <code>1</code> and continues with a variable length of <code>1</code>s followed by <code>0</code>s.<br>
I want to turn the first and the last <code>1</code> of each row to <code>0</code>.<br>
For instance, for a tensor with <code>torch.Size([3, 7])</code>, that initially is</p>
<pre><code class="lang-auto">tensor([[1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 0, 0]])
</code></pre>
<p>and I want to transform it to the tensor</p>
<pre><code class="lang-auto">tensor([[0, 1, 1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0, 0, 0],
        [0, 1, 1, 1, 0, 0, 0]])
</code></pre>
<p>What is the most straight forward way to make this transformation?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = torch.tensor([[1, 1, 1, 1, 0, 0, 0],
                  [1, 1, 1, 0, 0, 0, 0],
                  [1, 1, 1, 1, 1, 0, 0]])

tmp = (x == 1).cumsum(dim=1) - 1
idx_start = tmp[:, 0] 
idx_stop = tmp[:, -1]
idx = torch.stack((idx_start, idx_stop), dim=1)
x[torch.arange(x.size(0)).unsqueeze(1), idx] = 0

print(x)
&gt; tensor([[0, 1, 1, 0, 0, 0, 0],
          [0, 1, 0, 0, 0, 0, 0],
          [0, 1, 1, 1, 0, 0, 0]])
</code></pre>

ptrch_c_m: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, this is a straight forward solution, since you build the tensor positions that are about to change.</p>
<p>That was my first attempt too, and while trying, I thought that maybe there is a build-in function doing that. Anyway, it is a nice solution.</p>
How to access to a specific layer in a model

Mathieu_Grosso: <p>hello,<br>
i would like to access many layers of the model and i am struggling to do it.<br>
For now this is what I had done but it does not seem to work and I don’t know how I can do it otherwise (my goal is to access to all layers except for 2 of them).</p>
<pre><code class="lang-auto"> for name,param in self.model.named_parameters():
                # ic(name[2])
                if name=='conv_demosaic.1.weight' :
                    # or name['conv_demosaic.1.bias'] or name['conv1_a.0.weight'] or name['conv1_a.0.bias']
                    print('hello')
                    ic(name)
                else:
                    self.model.name.copy_(loaded_state_dict[name])
                    print('yes')
                    ic(name)
</code></pre>
<p>Thanks  !</p>

ptrblck: <p>I’m not sure what exactly is not working in your code snippet, so could you explain the issue a bit more?</p>