akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BatchNorm2d occurs RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

username: BINGO
cooked: <p>I don’t know why when I set the affine=True of BatchNorm2d, it will occur error : RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
But when I set the affine=False of BatchNorm2d , the error will not appear.<br>
My code is here, anybody can help me ?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b400175d67efa745af097c4d0573ebea94380fc2.png" alt="image" data-base62-sha1="pGm8H1fMZ19dqUvKg3YhjKwvB6O" width="342" height="124"></p>

username: ptrblck
cooked: <p>Which PyTorch, CUDA and cudnn version and which GPU are you using?<br>
I cannot reproduce this error in the latest stable release.</p>

username: BINGO
cooked: <p>Pytorch==1.1.0 Cuda=10.1.0, cudnn=7.6.4,  Gpu = RTX2080Ti. I could use BatchNorm function in other model, but in this code coccur error.</p>

username: ptrblck
cooked: <p>Are you seeing this error by running the posted code screenshot?</p>

username: BINGO
cooked: <p>Yes , I did …</p>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest stable version (<code>1.4.0</code>) and retry the code again, please?</p>

username: BINGO
cooked: <p>Thank you. I worked !<br>
But I successfully used BatchNorm in other model using PyTorch=1.1.0. I don’t know what it happend? The codes is shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c08556eca04d9a59aa79fa08aff0d67d86533e40.png" data-download-href="https://discuss.pytorch.org/uploads/default/c08556eca04d9a59aa79fa08aff0d67d86533e40" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c08556eca04d9a59aa79fa08aff0d67d86533e40_2_690x333.png" alt="image" data-base62-sha1="rt7lSTwX58hwpJcgoxvlmqzu5d6" width="690" height="333" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c08556eca04d9a59aa79fa08aff0d67d86533e40_2_690x333.png, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c08556eca04d9a59aa79fa08aff0d67d86533e40.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c08556eca04d9a59aa79fa08aff0d67d86533e40.png 2x" data-dominant-color="333030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">774×374 39.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Not sure what happened and I cannot remember this particular issue, but since it’s fixed in the current version, you should be good to go. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>What could have been the issue, is that for your particular GPU a specific path was chosen by cudnn, which was faulty and thus raised this issue.</p>
Unstable behavior while iterating over Dataloader (collate_fn TypeError issue)

username: suraj.pt
cooked: <p>I have a custom Dataset object, which I am iterating over using a Dataloader.</p>
<p>Dataset code:</p>
<pre><code class="lang-auto">class PatientDataset(data.Dataset):  
    def __init__(self, datadir, list_IDs): 
        self.datadir = os.path.join('data',datadir)
        self.list_IDs = list_IDs
        self.all_labels = {}
        avbl_dicts = {x.replace('_label_dict.pkl', '') : utils.pkl_load(os.path.join(self.datadir,x)) \
                        for x in os.listdir(self.datadir) if '_label_dict.pkl' in x} # {tgt_type : tgt_dict}
                        
        for ID in self.list_IDs:
            self.all_labels[ID] ={tgt_type:tgt_dict[ID] for tgt_type, tgt_dict in avbl_dicts.items()}

    def __len__(self):
        return len(self.list_IDs)

    def __getitem__(self, index):
        ID = self.list_IDs[index]
        X = utils.pkl_load(os.path.join(self.datadir, 'sequences', ID+'.npy')) # Numpy seq
        y = self.all_labels[ID] # Dict of available target values
        return X, y`
</code></pre>
<p>Iteration code:</p>
<pre><code class="lang-auto">def fn_that_iterates(dl):
    for batch, (X,y) in enumerate(dl):
          # do something

test_ds = pickle_load('test.dataset')) # Loads the pickled dataset object
dl = DataLoader(test_ds, batch_size=len(test_ds))
#iterate
fn_that_iterates(dl)
</code></pre>
<p>Sometimes when I run this as a script, I get a collate_fn TypeError because it got <code>&lt;PatientDataset&gt;</code> instead of the <code>(numpy array, dict)</code> that my Dataset <code>__getitem__</code> returns.</p>
<p>Running the same code in IPython never gives me the error.</p>
<p>Has anyone else faced this? Do we know why this happens?</p>

username: ptrblck
cooked: <p>How did you save the <code>test.dataset</code> and what’s your use case to store it?<br>
Based on the posted code it seems you are lazily loading the data anyway, so storing the <code>Dataset</code> object might be unnecessary.<br>
Are you seeing the same error, if you recreate the <code>PatientDataset</code> and pass it to the <code>DataLoader</code>?</p>

username: suraj.pt
cooked: <p>Thank you for your response <a class="mention" href="/u/ptrblck">@ptrblck</a>. It is now running as expected; this happened without a single code change, so I am very confused.<br>
I <code>pickle.dump()</code>ed it to a binary file. I’m storing it because I need to use the same datasets for a series of experiments.<br>
I didn’t get the error when I run the above in IPython, or if I recreate the Dataset and pass it to the DataLoader. It was only happening when I ran it in a script from the command line.</p>
How to get the class names to class label mapping

username: avijit_dasgupta
cooked: <p>I am using ResNet-18 for classification purpose. I have used dataloader to load the data. But how get the label to class name mapping? Does it load in alphabetical order?</p>

username: avijit_dasgupta
cooked: <p>For future reference:</p>
<p>I have used this from <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py</a> to find the mapping-</p>
<pre><code>def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    return classes, class_to_idx</code></pre>

username: zym1010
cooked: <p><a class="mention" href="/u/avijit_dasgupta">@avijit_dasgupta</a> hi, I have the same problem. Can you provide a class_idx to class name mapping? That would be super convenient.</p>

username: Ray_Luo
cooked: <p>Something like this:</p>
<pre><code class="lang-auto">dataset = datasets.ImageFolder(data_folder, transform=data_transforms)
print(dataset.class_to_idx)
</code></pre>

username: vipin14119
cooked: <p>For idx to classes mappping, just get list of keys from dict returned from <code>dataset.class_to_idx</code></p>
How to run two models in both CPU and GPU concurrently?

username: Darshik_A_S
cooked: <p>I have two pre-trained ResNet18 models and I’m trying to run these models concurrently in both CPU and GPU to achieve maximum throughput.</p>

username: TinfoilHat0
cooked: <p>Well, you should only pass one of them to the GPU in this case, don’t think you have to do anything else.</p>
<p>Though, I think it’d still b faster if you use GPU for both of them.</p>

username: Darshik_A_S
cooked: <p>Thank you, but I’m trying to run these models heterogeneously. Does PyTorch support this type of compatibility?</p>

username: ptrblck
cooked: <p>As <a class="mention" href="/u/tinfoilhat0">@TinfoilHat0</a> suggested, you could just call both models as:</p>
<pre><code class="lang-python">model0 = MyModel().to('cuda')
model1 = MyModel()

x0 = torch.randn(1, 1).to('cuda') 
x1 = torch.randn(1, 1)

out0 = model0(x0) # calls asynchronously to CUDA
out1 = model1(x1)
</code></pre>
In multiple loss function situation, order of loss is matter in gradient calculation?

username: Yupjun
cooked: <p>Let’s say I want to train Autoencoder with two loss function(L1, L2).<br>
Does the call order at the code line is the matter at gradient calculation?<br>
For example, the code line will be looks like this.</p>
<pre><code class="lang-auto">loss_1 = nn.L1Loss()
loss_2 = nn.MSELoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)
output_1 = loss1(input, target)
output_2 = loss2(input, target)
loss = loss_1 + loss_2
loss .backward()
</code></pre>
<p>If I change the loss calculation sequence into this,</p>
<pre><code class="lang-auto">loss_1 = nn.L1Loss()
loss_2 = nn.MSELoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)
output_2 = loss2(input, target)
output_1 = loss1(input, target)
loss = loss_1 + loss_2
loss .backward()
</code></pre>
<p>Does this kind of change will affect the model output? I think it will because L2 norm is more restricted version of norms so the model need more epochs to merge.</p>

username: ptrblck
cooked: <p>Both codes should yield the same result, as you are summing the losses anyway before calling <code>backward()</code>. Are you seeing any issues with one or the other approach?</p>
Import torch, module not found

username: ashu
cooked: <p>import torch<br>
File “C:\Users\Ashutosh\AppData\Local\Programs\Python\Python36\lib\site-packages\torch_<em>init</em>_.py”, line 79, in <br>
from torch._C import *<br>
ModuleNotFoundError: No module named ‘numpy.core._multiarray_umath’</p>

username: Sainadh086
cooked: <p><a class="mention" href="/u/ashu">@ashu</a>  If your are facing this problem every time you import torch the it is better to reinstall pytorch package from below link<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/?utm_source=Google&amp;utm_medium=PaidSearch&amp;utm_campaign=%2A%2ALP+-+TM+-+General+-+HV+-+IN&amp;utm_adgroup=Installing+PyTorch&amp;utm_keyword=%2Binstalling%20%2Bpytorch&amp;utm_offering=AI&amp;utm_Product=PyTorch&amp;gclid=Cj0KCQjw6cHoBRDdARIsADiTTzbuFmRhTpzTPy2Joty4WjdyYcyqz1oT3hcCwgvxduXWDL9zOScqpd0aAlBxEALw_wcB" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/?utm_source=Google&amp;utm_medium=PaidSearch&amp;utm_campaign=%2A%2ALP+-+TM+-+General+-+HV+-+IN&amp;utm_adgroup=Installing+PyTorch&amp;utm_keyword=%2Binstalling%20%2Bpytorch&amp;utm_offering=AI&amp;utm_Product=PyTorch&amp;gclid=Cj0KCQjw6cHoBRDdARIsADiTTzbuFmRhTpzTPy2Joty4WjdyYcyqz1oT3hcCwgvxduXWDL9zOScqpd0aAlBxEALw_wcB" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Choose your specification like OS, Package(eg: anaconda, pip), cuda version for gpu use.</p>

username: ssalome
cooked: <p>I am facing the same issue. I have tried what you’ve suggested but it didnt work for me…What can be done?</p>

username: Sainadh086
cooked: <p>May I know which Python Ide and OS you’re using?</p>
Convolution 2D with feature selection as input

username: Akankshya_Patra
cooked: <p>I have an input of the form 1000* 5* 1* 50 where 1000 is the batch size with 5 features, having 45* 1 input.<br>
I have an output label of 1000* 1* 2. This is a regression task.<br>
Now, I would like to take individual features into account for example using only feature 1 or only feature 2 or combination of feature 1 and 2 and so on.<br>
How may I set the convolutions to be?<br>
Until now, I am using Conv1d with 1000* 1* 50 input feature wise and then concatenating wherever required with another Conv1d.</p>
<p>What should I choose for the kernal size  and how to do this in one model? is there any option to choose which features I can select while using Conv2d?</p>

username: ptrblck
cooked: <p>I’m not sure I understand the question correctly.<br>
If you want to use a specific input channel only, you could slice the input at the corresponding channel.<br>
On the other hand you could use grouped convolutions to process each input channel individually.</p>
<p>Could you explain a bit more, what you mean by selecting the feature?</p>
Torchvision.transforms.functional.rotate( fill with nearest pixels

username: TomAvrech
cooked: <p>Rotate, but instead of filling the corners with a single color, fill with the nearest pixels’ colors, how ?</p>

username: TomAvrech
cooked: <p>Ok, since I didn’t find an official solution, mine:</p>
<p>frame_removal_length = 3<br>
padding = max(IMAGE_SIZE[0], IMAGE_SIZE[1])<br>
rgb = T.functional.resized_crop(rgb, frame_removal_length, frame_removal_length, IMAGE_SIZE[0] - 2 * frame_removal_length, IMAGE_SIZE[1] - 2 * frame_removal_length, IMAGE_SIZE) # Only if the image has some kind of a frame needed to be remvoed.<br>
rgb = T.Pad(padding=padding, fill=0, padding_mode=‘edge’)(rgb)<br>
rgb = T.functional.rotate(img=rgb, angle=degree_to_rotate, fill=(0,) *  len(rgb.getbands()))<br>
rgb = T.functional.resized_crop(rgb, padding, padding, IMAGE_SIZE[0], IMAGE_SIZE[1], IMAGE_SIZE)</p>
<p>Enjoy !</p>
L2 regulizer for keeping parameters in a range

username: yegane
cooked: <p>Hi all,<br>
I want to implement a regulizer that keeps the weights in a certain range, and does not push the weight itself to become zero, but push the <code>|w(t)-w(t-1)|&gt;a</code> to zero. It means that in that range, I don’t want to have any effect on the training.</p>
More efficient way to write in Pytorch

username: g33sean
cooked: <p>Hello I am new in PyTorch, I was wondering what be a more efficient way to write this in Pytorch, possibly getting rid of the for loops or maybe the conditional statements</p>
<p>def fit_dz(z,v_pieces,w_pieces):<br>
z,_=torch.sort(z)<br>
g_dz=torch.zeros(z.shape,)<br>
for i in range(len(z)):<br>
for j in range(len(w_pieces)-1):<br>
if 0 &lt; z[i] &lt; v_pieces[0]:<br>
j=0<br>
g_dz[i]= ((v_pieces[j]-0)/(w_pieces[j]-0))<br>
elif v_pieces[j] &lt; z[i] &lt; v_pieces[j+1]:<br>
g_dz[i]= ((v_pieces[j+1]-v_pieces[j])/(w_pieces[j+1]-w_pieces[j]))<br>
return g_dz</p>
<p>z, v_pieces,w_pieces are just 1-d tensors.</p>
Basic doubt dim in squeeze and unsqueeze

username: sandeep1
cooked: <p>How does dim dim in squeeze and unsqueze work?<br>
x = torch.zeros(2, 1, 2, 1, 2)<br>
x.squeeze()<br>
torch.Size([2, 2, 2])</p>
<p>y=torch.squeeze(x,1)<br>
torch.Size([1, 2, 1, 2, 1, 2])   .//no change</p>
<p>y=torch.squeeze(x,2)<br>
results :-<br>
torch.Size([1, 2, 2, 1, 2])</p>

username: KFrank
cooked: <p>Hi Sandeep!</p>
<aside class="quote no-group" data-username="sandeep1" data-post="1" data-topic="73285" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sandeep1/48/12454_2.png" class="avatar"> sandeep1:</div>
<blockquote>
<p>How does dim dim in squeeze and unsqueze work?</p>
</blockquote>
</aside>
<p><code>torch.squeeze()</code> removes those dimensions of a tensor that have<br>
<code>size = 1</code>.  This is explained reasonably clearly in the documentation<br>
for <a href="https://pytorch.org/docs/stable/torch.html#torch.squeeze" rel="noopener nofollow ugc">torch.squeeze</a>.</p>
<p>Specifically, if you give <code>squeeze()</code> no <code>dim</code> argument, it removes all<br>
<code>size = 1</code> dimensions.  If you give <code>squeeze()</code> a value for <code>dim</code>, it<br>
removes just that dimension, but only if it has <code>size = 1</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>x = torch.zeros(2, 1, 2, 1, 2)</p>
</blockquote>
</aside>
<p>This looks like a typo.  Based on the results you give below, I think<br>
this should be <code>x = torch.zeros(1, 2, 1, 2, 1, 2)</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
How can I get a list of all tensors that requires_grad?

username: kaiseryet
cooked: <p>So let’s say I have a list of tensors, some have <code>requires_grad=False</code>, some have <code>requires_grad=True</code>. How can I get a list of all the tensors that come with <code>requires_grad=True</code>?</p>

username: Marco_Piccoli
cooked: <pre><code class="lang-auto">def get_graded_tensors(my_list):
    new_list = []
    for i, item in enumerate(my_list):
        if item.requires_grad:
            new_list.append(item)
    return new_list
</code></pre>
Exception in device=TPU:0: module 'torch_xla.core.xla_model' has no attribute 'rendezvous'

username: sudddddd
cooked: <p>I wanted to try the PyTorch lightning tutorial on MNIST on colab TPU. But, after running it with some modifications, I am getting</p>
<blockquote>
<p>Exception in device=TPU:0: module ‘torch_xla.core.xla_model’ has no attribute ‘rendezvous’</p>
</blockquote>
<p><a href="https://colab.research.google.com/drive/1HyzIKjwE6KOL-8vQRHoIGQJfswHGpbeE" rel="noopener nofollow ugc">Here</a> is my colab notebook. Please help.</p>
Bizarre OOM error

username: divinho
cooked: <pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 104.00 MiB (GPU 0; 31.72 GiB total capacity; 48.83 MiB already allocated; 30.61 GiB free; 68.00 MiB reserved in total by PyTorch)
</code></pre>
<p>This does not make any sense? I’m using pytorch 1.4.0.</p>

username: ptrblck
cooked: <p>Could you check the memory usage via <code>nvidia-smi</code>?<br>
Other processes might use the device (or some zombie processes still occupy the GPU).</p>

username: divinho
cooked: <p>I’ve checked that using watch and that’s not it.</p>
<p>I think it’s related to some sort of cuda version mismatch. Do you know if it’s possible to use pytorch with cuda9.2 while <code>nvidia-smi</code> shows cuda10.0 ?</p>
<p>edit: It runs on a 1080ti and v100 but hangs on p40, version 1.3.1 cuda9.2</p>
<p>edit2: Nevermind, seems like the grid where I am is not setup correctly.</p>
"Mirror" autoencoder - how-to share parameters between layers?

username: lesser_evil
cooked: <p>Hello,</p>
<p>I am trying to build a “mirror” autoencoder where the encoder parameters would be the same as decoder parameters.</p>
<p>In the simplest case, autoencoder:</p>
<pre><code class="lang-auto">• conv
• fc_1
• fc_2
• convT
</code></pre>
<p>How can I denote parameters of fc_1 to be the same (mirrored) as parameters of fc_2?<br>
Similarly, how can I share parameters between the 1st convolution and last deconvolution layers?</p>
<p>Thank you for the consideration!</p>

username: ptrblck
cooked: <p>I think the easiest way would be to define the parameters manually and use the functional API for all operations, e.g.:</p>
<pre><code class="lang-python">out = F.conv2d(intput, weight)
</code></pre>
<p>where you’ve defined the <code>weight</code> as a parameter before.</p>

username: lesser_evil
cooked: <p>Thank you, sounds like what is needed!</p>
At dispatch long types

username: sh0416
cooked: <p>Hi, I am working on extending pytorch using c++.</p>
<p>Is there a macro that realize function into long type tensor??</p>
<p>Also, if there is any documentation for this kind of API?</p>
<p>If it is, please provide link for me.</p>
<p>Thanks,</p>

username: Edwinzero
cooked: <p>You might want to dig into the source code of Pytorch.<br>
That is how I start working on C++ extension for Pytorch…</p>

username: sh0416
cooked: <p>Yeah, I think so.<br>
For now, I just fix the type of tensor.<br>
Thanks,</p>
Different behavior of auto grad on PyTorch1.0 and >=1.1

username: Layne
cooked: <p>Hi, I’m trying to port my code from version 1.0 to newer version of PyTorch.<br>
The following code snippet works fine on PyTorch1.0 but encounters RuntimeError on PyTorch1.3.<br>
Do you have any idea about this?</p>
<p>code:</p>
<pre><code class="lang-auto">criterion_kl = nn.KLDivLoss(reduction='sum')
model.eval()

# generate adversarial example
x_adv = x_natural.detach() + 0.001 * torch.randn_like(x_natural)
for _ in range(self.perturb_steps):
    x_adv.requires_grad_()
    with torch.enable_grad():
        loss_kl = criterion_kl(F.log_softmax(model(x_adv), dim=1),
                            F.softmax(model(x_natural), dim=1))
    grad = torch.autograd.grad(loss_kl, [x_adv])[0].detach() 
    x_adv = x_adv.detach() + self.step_size * torch.sign(grad)
    x_adv = torch.min(torch.max(x_adv, x_natural - self.epsilon), x_natural + self.epsilon)
    x_adv = torch.clamp(x_adv, 0.0, 1.0)
</code></pre>
<p>error message:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [640]] is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Traceback (most recent call last):
  File "main_adv.py", line 326, in &lt;module&gt;
    main()
  File "main_adv.py", line 187, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "main_adv.py", line 240, in train
    output, loss = criterion(input, target, index, epoch, model, optimizer)
  File "/home/code/losses/trades.py", line 51, in __call__
    grad = torch.autograd.grad(loss_kl, [delta])[0].detach()
  File "/home/code/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 157, in grad
    inputs, allow_unused)
</code></pre>

username: ptrblck
cooked: <p>Do you see the offending operation in the stack trace?<br>
Based on the code snippet I cannot find the inplace operation, which might cause this error. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Layne
cooked: <p>No. That’s all I see in the stack trace.<br>
BTW I was using distributed dataparallel and then encountered the error. However,  it seems everything well when I run the code on single GPU device even using PyTorch1.1<br>
Maybe this is related to the distributed scheme?</p>
Why am I not getting a gradient with respect to my input data?

username: mitchello
cooked: <p>I’m running a model where I need to get the gradient of the loss function w.r.t my input data so I can use it to update previous networks that are in series. I thought I was calling everything right, but I have encounted a problem where the gradients of my loss function w.r.t input data is zero or very close to it. What am I doing wrong? The gradients exist throughout the network, loss is nonzero, etc.<br>
Here’s the code for the backpropagation. after_graph is the output of a graph neural network, and net2 is my graph.</p>
<pre><code class="lang-auto">after_graph2 = Variable(after_graph.clone(), requires_grad=True)
        
        after_net2 = self.net2(after_graph2)
        # back prop
        loss = self.loss_func(after_net2, y)
        # find gradients of net 2
        loss.backward(retain_graph = True)
        
        loss_graph_output = after_graph2.grad
</code></pre>
<p>Here’s the net:</p>
<pre><code class="lang-auto">net2 = nn.Sequential(
    nn.Linear(input_size2, hidden_sizes2[0]),
    nn.Tanh(),
    nn.Linear(hidden_sizes2[0], output_size2),
    nn.Tanh()
)
</code></pre>
<p>What could be going wrong?</p>

username: ptrblck
cooked: <p>If the gradients are not <code>None</code> but close to zero, the gradient calculation was performed, but your gradient might vanish.<br>
Using your model I get valid gradients for the input:</p>
<pre><code class="lang-python">net2 = nn.Sequential(
    nn.Linear(10, 10),
    nn.Tanh(),
    nn.Linear(10, 10),
    nn.Tanh()
)

x = torch.randn(1, 10, requires_grad=True)
out = net2(x)
out.mean().backward()
print(x.grad)
&gt; tensor([[-0.0204,  0.0356, -0.0084, -0.0122,  0.0018, -0.0206,  0.0270,  0.0311,
          0.0075,  0.0081]])
</code></pre>
<p>PS: <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code>, so you can use tensors now.</p>
AttributeError: 'NoneType' object has no attribute 'register_forward_hook'

username: CCL
cooked: <p>I’m trying to register a forward hook function to the last conv layer of my network. I first printed out the names of the modules via:</p>
<pre><code class="lang-auto">for name, _ in model.named_modules():
    print(name)
</code></pre>
<p>Which gave me <code>"0.conv"</code> as the module name. However, when I tried to do the following, the above error was triggered by line 2:</p>
<pre><code class="lang-auto">def hook_feature(module, in_, out_):
    features.append(out_.cpu().data.numpy())

model._modules.get("0.conv").register_forward_hook(hook_feature)
</code></pre>
<p>What am I doing wrong and how do I fix it? Thanks!</p>

username: mailcorahul
cooked: <p>This seems like there is no module named <code>0.conv</code> as the exception says <code>'NoneType' object has no attribute ‘register_forward_hook’</code>. Please check if the module name passed to get method is correct.</p>
<p>Can you post the output of print statement showing the network’s module list?</p>

username: CCL
cooked: <p>Apologies for the late reply! Here’s the output:</p>
<pre><code class="lang-auto">...
0.decoder1.dec1norm1
0.decoder1.dec1relu1
0.decoder1.dec1conv2
0.decoder1.dec1norm2
0.decoder1.dec1relu2
0.conv
1
1.outc1
1.outc2
</code></pre>

username: CCL
cooked: <p>Solved with help from Shai (Stack Overflow): <code>model._modules["0"]._modules.get("conv").register_forward_hook(hook_feature)</code></p>
Python covert to c++ torch copy operation

username: 21fa417e3fb06e56040c
cooked: <p>Hi all</p>
<p>How does the code below convert to  c++?<br>
I need your help.</p>
<p>thanks all</p>
<pre><code class="lang-python"> // batched_imgs ,pad_img tensor list 
batch_shape = [len(images)] + max_size
batched_imgs = images[0].new_full(batch_shape, 0)
    for img, pad_img in zip(images, batched_imgs):
         pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
</code></pre>

username: 21fa417e3fb06e56040c
cooked: <p>I solved it.<br>
do it like this</p>
<pre><code class="lang-auto">    torch::Tensor batched_imgs = tensors[0].new_full({ (int64_t)tensors.size() ,tensors[0].size(0),max_size_first, max_size_second },0);
   
    std::vector&lt;torch::Tensor&gt; image_sizes;
    for (int i = 0; i &lt; tensors.size(); ++i) 
    {
        batched_imgs[i].narrow(/*dim=*/1, /*start=*/0, /*length=*/tensors[i].size(1))
            .narrow(/*dim=*/2, /*start=*/0, /*length=*/tensors[i].size(2)).copy_(tensors[i]);
    }
</code></pre>

username: yf225
cooked: <p>Starting from the current nightly build (and PyTorch 1.5 soon), for</p>
<pre><code class="lang-auto">pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
</code></pre>
<p>we can write</p>
<pre><code class="lang-auto">using namespace torch::indexing;
pad_img.index({Slice(None, img.size(0)), Slice(None, img.size(1)), Slice(None, img.size(2))}).copy_(img)
</code></pre>
<p>Here is the general translation for <code>Tensor::index</code> and <code>Tensor::index_put_</code> functions:</p>
<pre><code class="lang-auto">Python             C++ (assuming `using namespace at::indexing`)
-------------------------------------------------------------------
0                  0
None               None
...                "..." or Ellipsis
:                  Slice()
start:stop:step    Slice(start, stop, step)
True / False       true / false
[[1, 2]]           torch::tensor({{1, 2}})
</code></pre>

username: 21fa417e3fb06e56040c
cooked: <p>Oh Good, I Look forward<br>
Thank You</p>
How to compare 1 image to 26 image in siamese neural network

username: Itha_LItha
cooked: <p>i use siamese network, right now i inser 2 image then will processing an show the answer<br>
if answer is 1 its mean the picture is same, but if answer is 0 its mean the picture is different.</p>
<pre><code class="lang-auto">from PIL import Image
x1 = Image.open('A.PNG')
x2 = Image.open('X.PNG')
# Transform

x1 = transforms(x1)
x2 = transforms(x2)

x1 = torch.stack([x1])
x2 = torch.stack([x2])

model.eval()

# Get prediction
output = model(x1,x2)
output = (torch.sigmoid(output) &gt;= 0.5).float()
print (output)
</code></pre>
<p>Both pictures are different, so tensor give this output</p>
<pre><code class="lang-auto">tensor([[0.]])
</code></pre>
<p>I want to ask, I have 26 types of image that I use for this siamese neural network.</p>
<p>Can I insert 1 type of picture and then the image will be compared to the 26 data that I have, if it has a value of 1 (the same picture) can I display which image is the same?</p>
<p>Or can it display the closest range to the image that I entered after comparing into 26 of my data?</p>

username: Dipam_Vasani
cooked: <p>Hey, you can create your own data set in which the first image will be repeated 26 times and the other 26 images will remain as they are. Basically, you can create 26 tuples of the images you want to compare. Then you can do the standard Siamese network. Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Training a linear classifier in the middle layers

username: MUnique
cooked: <p>I have pre-trained a network on a dataset. I wanted to train a linear classifier on a middle layer of it while the weights of the former layers shouldn’t be updated. The new network is going to be trained on another dataset. Can anyone help me with that? I don’t know how to train the classifier in between and how to turn off the gradient update for the first layers.</p>

username: Dipam_Vasani
cooked: <p>You can freeze the first few layers. This means you will set the <code>requires_grad</code> parameter to False for those layers. Check <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">this link</a> for more information.</p>
<p>Hope this helps</p>
Diff between linear vs con2d

username: Lj_Fitness
cooked: <p>Can anyone explain the diff between <code>nn.Linear(in_features, out_features)</code> vs <code>nn.conv2d(in_channels, out_channels, kernel_size)</code></p>
<p>I believe the kernel_size is easy to understand as the size of the kernel window to compute content for next layer.</p>
<p>is the in_features same as in_channels, and out_features same as out_channels?</p>
<p>Thank you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Lj_Fitness" data-post="1" data-topic="73157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lj_fitness/48/21672_2.png" class="avatar"> Lj_Fitness:</div>
<blockquote>
<p>is the in_features same as in_channels, and out_features same as out_channels?</p>
</blockquote>
</aside>
<p>From a high level point of view you could probably say so.<br>
However, the operations in a convolution and linear layer are different regarding the connectivity etc.</p>
<p>Have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolution</a> for more information about conv layers.</p>
Why isn't output shape of Conv2d equal to input shape?

username: agt
cooked: <p>I have a 96x96 input image. My first layer is nn.Conv2d(1, 16, 3, padding = 1).<br>
The equation for the output shape in the docs is: <code>height + 2*padding - dilation * (kernel_size - 1) - 1 + 1</code> assuming a stride of 1.<br>
So my output should be <code>96 + 2 - 1 * (3 - 1) - 1 + 1</code> = <code>96</code>, the same dimension as the input.<br>
But when I <code>print(x.shape)</code> after applying the layer, it’s <code>torch.Size([1, 16, 48, 48])</code>.<br>
What’s going on, I thought a padding of 1 would make it the same dimensions, but somehow it split in two? It’s 48 instead of 96. Why is this?</p>

username: agt
cooked: <p>Nevermind, I forgot I was pooling (2, 2) in the same line I was convulsing.</p>
Understanding stride and filter size

username: Glerk
cooked: <p>Hello,</p>
<p>I am confused on how stride and filter size interact. Is the output of a convolution layer the largest tensor such that the filter can fit completely within the input?</p>
<p>Ex, consider A = [0,1,2,3,4,5,6,] and L = conv1d(1,1,3, stride=2)</p>
<p>What will the size of L(A) be?</p>
<p>Will the filter stop at 4 since applying it starting at A[7] will result in the filter going out of bounds?</p>
<p>So the dimension of your output will be 3?</p>

username: pramod.srinivasan
cooked: <p>From the function of definition for <code>nn.conv1d</code></p>
<pre><code class="lang-auto">torch.nn.Conv1d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')
</code></pre>
<p>C_in = 1,<br>
C_out = 1,<br>
K  = 3<br>
S = 1<br>
P = 0</p>
<p>From the slide 63 of lecture 5, cs231n, the size of the L(A) : 1 X 5 X 1</p>
<p>So, the filter stops cross correlation at A[5] ie. 4 because of it goes out of bounds in the next stride.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07_2_690x334.jpeg" alt="image" data-base62-sha1="fgh7efTGkg0V8CRYryKYVBrur0X" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07_2_690x334.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07_2_1035x501.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6af79c7d91ab11c9f5ddc59dd2df1237baacbe07_2_1380x668.jpeg 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1666×808 229 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Glerk
cooked: <p>thanks, so the convolution operation continues to slide until it can no longer be completed within the tensor?</p>
Why is the length of data and label in the read .mat dataset inconsistent?

username: ky_Pa
cooked: <p>The length of the data is 200, and the length of the label is 1. These two should correspond to each other! Why is this the case now? Please inform, thank you!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7998079aeb1eba93ae685b589cded8dee6ba55d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/7998079aeb1eba93ae685b589cded8dee6ba55d5" title="微信图片_20200313101822"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7998079aeb1eba93ae685b589cded8dee6ba55d5.png" alt="微信图片_20200313101822" data-base62-sha1="hlFuKTxdXNluYNQxmwSEw339A33" width="690" height="338" data-dominant-color="34352F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200313101822</span><span class="informations">798×391 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import h5py
import torch
from torch.utils.data import TensorDataset

tgt_dataset_root = h5py.File('E:\\ADDA\\pytorch-adda-master-lab\\datasets\\lab\\test_target_domain_sandy.mat','r')
data = tgt_dataset_root['data'][:]
label = tgt_dataset_root['label'][:]

data = torch.from_numpy(data)
label = torch.from_numpy(label)

print(tgt_dataset_root.keys())

print(data)
print(label)

print(len(data))
print(len(label))</code></pre>

username: ptrblck
cooked: <p><code>len(tensor)</code> uses the first dimension of the tensor to calculate the length.<br>
Your target seems to have the shape <code>[1, X]</code>.<br>
You could use <code>print(label.size())</code> to get the size of all dimensions.</p>

username: ky_Pa
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/487e191c28964539233b44120349645285596c2c.png" alt="微信图片_20200313122651" data-base62-sha1="alixAAfGjTYyjiHt53MrNLo8H9O" width="425" height="136"><br>
This is the case, how can I read the length of the next 2000? Or reverse the order of the four columns like the following, these two results show exactly the opposite.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a511f7f9caeff37fcca5afac1a81f4557f8325fe.png" alt="微信图片_20200313122655" data-base62-sha1="nyhhnr4tF9l6cZBxJdTvLsS6Eqy" width="330" height="115"></p>

username: ptrblck
cooked: <p>It depends how you’ve stored and how you access the HDF5 file.<br>
If the shape is not right, you could <code>permute</code> or <code>view</code> the tensor to get the right shape.</p>

username: ky_Pa
cooked: <p>I tried it. Thank you very much!</p>
Dataloader basic doubt

username: sandeep1
cooked: <p>Below is program of data loader.<br>
Doubt: x_data and y_data was already made a tensor by this statement:-<br>
self.x_data=from_numpy(xy[:,:-1])</p>
<p>Then why  again converting it to tensor<br>
inputs,labels=tensor(inputs),tensor(labels)</p>
<p>Full code:-</p>
<p>from torch.utils.data import Dataset,DataLoader<br>
from torch import from_numpy,tensor<br>
import numpy as np</p>
<p>class DiabetesDataset(Dataset):<br>
def <strong>init</strong>(self):<br>
xy=np.loadtxt(’/content/diabetes.csv’,delimiter=’,’,dtype=np.float32)<br>
self.len=xy.shape[0]<br>
self.x_data=from_numpy(xy[:,:-1])<br>
self.y_data=from_numpy(xy[:,[-1]])</p>
<p>def <strong>getitem</strong>(self,index):<br>
return self.x_data[index],self.y_data[index]<br>
def <strong>len</strong>(self):<br>
return self.len</p>
<p>dataset=DiabetesDataset()<br>
trainLoader=DataLoader(dataset=dataset,<br>
batch_size=32,<br>
shuffle=True,<br>
num_workers=2)</p>
<p>for epoch in range(2):<br>
for i,data in enumerate(train_loader,0):<br>
inputs,labels=data<br>
inputs,labels=tensor(inputs),tensor(labels)</p>
<pre><code>print(f'Epoch: {i}|Inputs:{inputs.data} | Labels:{labels.data} ')</code></pre>

username: pramod.srinivasan
cooked: <p>No, <code>x_data</code> and <code>y_data</code> are numpy arrays. They need to be converted to Torch Tensor</p>

username: sandeep1
cooked: <p>But what does from_numpy does in :-<br>
self.x_data=from_numpy(xy[:,:-1])</p>
<p>Syntax of from_numpy:-<br>
torch.from_numpy` ( <em>ndarray</em> ) → Tensor</p>
<p>Doesn’t it already converted it into tensor?</p>