akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using nn.CrossEntropyLoss(), how can I get softmax output?

username: wonchulSon
cooked: <p>As I know, nn.CrossEntropyLoss() automatically  apply logSoftmax using FC layer output.<br>
So then, how can I get logsoftmax/softmax output?</p>
<p>Thank you.</p>

username: ajhanwar
cooked: <p>The outputs would be the featurized data, you could simply apply a softmax layer to the output of a forward pass. Something like:</p>
<pre><code class="lang-auto">model = nn.Sequential(...)
probs = nn.Softmax(dim=1)

outputs = model(input)
probs(outputs)
</code></pre>

username: wonchulSon
cooked: <p>Yeah that’s one way to get softmax output.<br>
But there is problem.<br>
I want to use this loss function.<br>
<code>criterion = nn.CrossEntropyLoss().cuda()</code></p>
<pre><code class="lang-auto">outputs = model(input)
softmax_output = probs(outputs)
loss = criterion(softmax_output , labels)  ?? 
</code></pre>
<p>Then the loss is like nn.Softmax(nn.logsoftmax(outputs)), right?<br>
Because nn.CrossEntropyLoss() will apply logsoftmax itself.<br>
What should I do?</p>
<p>Thank you.</p>

username: ajhanwar
cooked: <p>From the official docs <a href="https://pytorch.org/docs/stable/nn.html:" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html:</a><br>
<code>nn.CrossEntropyLoss()</code> combines <code>nn.LogSoftmax()</code>  and  <code>nn.NLLLoss()</code>  together</p>

username: wonchulSon
cooked: <p>Thank you.<br>
But the link is dead.<br>
Can I seperate <code>nn.CrossEntropyLoss()</code>'s <code>nn.LogSoftmax()</code> or <code>nn.NLLLoss()</code> ?</p>

username: Oli
cooked: <p>Code says more than words <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<pre><code class="lang-auto">probs = nn.Softmax(dim=1) # Or logsoftmax
criterion = nn.CrossEntropyLoss() 

outputs = model(inputs)
softmax_output = probs(outputs)
loss = criterion(outputs , labels)
</code></pre>

username: wonchulSon
cooked: <p>Thank you for your help ^^.</p>
How does one decide if to pass the softmax or one-hot to the next cell of an RNN/LSTM/GRU?

username: pinocchio
cooked: <p>I have my RNN/LSTM/GRU outputing a softmax distribution over tokens. How do I decide if to pass the soft-vector or the thresholded/sampled one-hot tensor version of it as input to the next cell?</p>
<p>(teacher training is not possible in my application)</p>

username: Mazhar_Shaikh
cooked: <p>Hi pinocchio,<br>
You should pass the direct output of the rnn to the next cell. Sampling/threshold will cause a discontinuity in the graph and the gradients won’t back propagate through time.</p>
Why is interpolate() function so slow?

username: backseason
cooked: <p>Does anybody have any idea why the interpolate function in PyTorch is so slow?</p>
<p>The forward part of interpolate() is fast, however the backward part is really slow.<br>
If you only use 1 or 2 times of interpolate() you may not notice the speed, but if you use this function 10+ times you’ll find the whole training time doubles (e.g.,  training imagenet).</p>

username: xwang-icsi
cooked: <p>I also met similar problem recently. Have you find a method to fix it?</p>

username: Zhilin_Lu
cooked: <p>I met the same problem as well, the training become 6 times slower when adding tow Upsample module in each BottleNeck Block.</p>

username: Zhilin_Lu
cooked: <p>And the main problem is the backward indeed. If the backward path is blocked, the training will be four times faster</p>

username: backseason
cooked: <p>Nope, I haven’t found out a proper solution.</p>
Autograd graph computation interrupted

username: DeepLearner17
cooked: <p>Hello,<br>
l have a numpy layer within my network which is not concerned by back-propagation. When l put this numpy_layer(.) between the layers of my network, the backpropagation algorithm updates the weights of layer4 and layer5 but  not the layers prior to numpy_layer(.). I removed the numpy_layer(.) and the backpropagation indeed works for all the layers. What’s wrong ?<br>
Here is my code</p>
<pre><code class="lang-auto">def forward(x):
    x=layer1(x)
    x=layer2(x)
    x=layer3(x)
    x=numpy_layer(x) # not concerned by backpropagation
    x=layer4(x)
    x=layer5(x)
    return x
</code></pre>
<p>I think that the reason why backward doesn’t achieve the three first layers, is that  my computation graph is interrupted.<br>
One of the solution to avoid that is to implement the backward function manually since Autograd cannot create it automatically once you leave PyTorch.</p>
<p>l would like to backpropagate from layer4 to layer3 without backpropagating through numpy_layer(x) during the backward process.<br>
How to write correctly numpy_layer(x) so that it doesn’t interrupt the computation graph.</p>
<p>I’m thinking about the following solution:<br>
writing an autograd class for numpy_layer() without any update in the backward() :</p>
<pre><code class="lang-auto">class  numpy_layer(torch.autograd.Function):
    def forward(self,x):
           x=x.cpu().data.numpy()
           x=my_numpy(x)# numpy operations
          x=torch_from_numpy(x)
        return x
    def backward(grad_output):
          # nothing 
        
</code></pre>
<p>Is this implementation correct ?<br>
I am agnostic about it because, l don’t see clearly how  can l  backpropagate the weights from layer 4 to 3 without passing/updating  numpy_layer().</p>
<p>The shape of my layers is as follow :</p>
<pre><code class="lang-auto">  x=layer1(x) # shape : x \in R^{n}
  x=layer2(x) # shape : x \in R^{n}
  x=layer3(x) # shape : x \in R^{n}
  x=numpy_function(x) # not concerned by backpropagation, shape : x \in R^{m}, where n &lt;&lt; m
  x=layer4(x) # shape : x \in R^{m}, where n &lt;&lt; m
  x=layer5(x) # shape : x \in R^{m}, where n &lt;&lt; m
</code></pre>
<p>my numpy function just makes an expansion  (concatenation with respect to some operations) of x from x \in R^{n} to x \in R^{m} where   n &lt;&lt; m</p>
<p>Thank you</p>

username: Mazhar_Shaikh
cooked: <p>Hi DeepLearner17,<br>
Please go through the <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">link for a tutorial on creating a custom autograd function.</a><br>
The example you have provided need to return a grad_input which would be required for layer3 to calculate its gradient. In the sense that, grad_output would be in R^{m}. You will need to transform it to R^{n} and return that for gradient to pass through.</p>
<p>Alternatively, Pytorch was built to mimic Numpy to a great degree, so almost any method written using numpy can be converted to their equivalent in Pytorch, which will enable the gradient to flow automatically.</p>
Is there a way in Pytorch that has the same functionality as padding 'same' in TensorFlow?

username: woals2274
cooked: <p>I’m trying to match the input shape with the output shape, but since kernel_size is (4, 3, 3), it’s hard to match the output shape with the input shape.</p>
<p>For example,<br>
input_shape = [5, 256, 4, 64, 64]<br>
kernel_size = (4, 3, 3)</p>
<p>Applying kernel_size to inpus_shape gives me this:<br>
output_shape = [5, 128, 1, 62, 62]</p>
<p>Even if you apply multiple paddings to the above output, I have a hard time getting the same size as the input_shape.</p>
<p>However, in TensorFlow or Keras code, padding = ‘same’ makes it easy to scale input and output.</p>
<p>How can I implement padding=‘same’ in Pytorch?</p>

username: Mazhar_Shaikh
cooked: <p>Hi woals2274,<br>
The pytorch conv modules apply only symmetric padding to the input, thus can’t be used to create ‘same’ padding for even sized kernels.<br>
You could instead apply unsymmetric padding to the input using the following code</p>
<pre><code class="lang-auto">import torch
input_data = torch.rand(5,256,4,64,64)
pad = torch.nn.ConstantPad3d((1,1,1,1,2,1), 0)
conv = torch.nn.Conv3d(256, 128, kernel_size=(4,3,3), padding=0, stride=1)
print("Input Data size:{}\nOutput Data size:{}".format(conv(pad(input_data)).size()))
</code></pre>
Zero ing gradients makes the model very slow

username: RahimEntezari
cooked: <p>I am implementing “Deconstructing lottery ticket” paper.<br>
I am training Alexnet for several times.<br>
In the first epoch, it has exactly the same weight of Alexnet.<br>
In the second Epoch, I selected 20% of the weights(4M indexes in index_dict) and keep them as zero while training. To do this,<br>
I defined hooks for all layers, and grad_clone[some weights] =0:<br>
def my_hook4d_conv1(grad):<br>
grad_clone = grad.clone()<br>
for i in range(len(index_dict4d[“conv1”])):<br>
a,b,c,d = index_dict4d[“conv1”][i]<br>
grad_clone[a,b,c,d] = 0<br>
return grad_clone<br>
and then register hooks for all layers, before training started for the second epoch.</p>
<ol>
<li>This process makes the training very slow. (4M operation in the above for loop). Is there a way to speed up this process?</li>
</ol>
Semantic segmentation model (UNet) doesnt learn

username: Can_Keles
cooked: <p>I tried training a UNet model written in pytorch but i cant seem to make it work. I tried training on a single image (the dataset is Carvana) for 500 epochs but the output is pure black. Any help would be appreciated.</p>
<p>Here is the link to my Kaggle kernel: <a href="https://www.kaggle.com/cankls3/carvana-pytorch?scriptVersionId=18544207" rel="nofollow noopener">Carvana-Pytorch</a></p>

username: ptrblck
cooked: <p>Did you make sure the target tensor do represent the valid segmentation mask?<br>
Your current code is:</p>
<pre><code class="lang-python">

class MyDataset(Dataset):
    def __init__(self, image_paths, mask_paths, train=True):
        self.image_paths = image_paths
        self.mask_paths = mask_paths
        
    def transforms(self, image, mask):
        image = transforms.Resize(size=(256, 256))(image)
        mask = transforms.Resize(size=(256, 256))(mask)
        image = TF.to_tensor(image)
        mask = TF.to_tensor(mask)
        
        return image, mask
    
    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        mask = Image.open(self.mask_paths[index])
        
        x, y = self.transforms(image, mask)
        return x, y
    
    def __len__(self):
        return len(self.image_paths)
</code></pre>
<p>I’m a little bit concerned about the transformation applied on the target.<br>
If you are loading a valid segmentation mask containing only the class indices (in your case it should be zeros for background and ones for the car), the <code>Resize</code> and <code>ToTensor</code> transformations might mess up the target values.</p>
<p>For <code>Resize</code> you should use <code>PIL.Image.NEAREST</code> as the <code>interpolation</code>, since <code>PIL.Image.BILINEAR</code> is the default one and will interpolate to invalid class indices at the borders.<br>
Depending which image type you are passing to <code>ToTensor</code>, this might also normalize the image, which might further cause trouble.</p>
<p>Could you print some target tensors and just make a sanity check, that they only contain your desired class indices?</p>

username: Can_Keles
cooked: <p>I resized the images using PIL.Image.NEAREST and the result seems the same, i think the ToTensor method normalizez the image becouse the values of the image are betweem -1 and 1. I have one output channel, is this a problem?</p>

username: Can_Keles
cooked: <p>I also checked if the masks are matching, here is an updated version of the notebook: <a href="https://www.kaggle.com/cankls3/carvana-pytorch?scriptVersionId=18548775" rel="nofollow noopener">Updated</a></p>

username: Can_Keles
cooked: <p>I multiplied the values of image by 255 and it workes now, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
Propagating different losses

username: nandof
cooked: <p>Consider a network <code>(i) -&gt; (h) -&gt; (o)</code> where <code>i</code>, <code>h</code>, <code>o</code>, are input, hidden, and output layers, respectively.<br>
I would like to associate a loss <code>Lh</code> and a loss <code>Lo</code> to layers <code>h</code> and <code>o</code>, respectively. However, I wish to backpropagate <code>Lh</code> only from layer <code>h</code>, backward, and loss <code>Lo</code> from layer <code>o</code> backward.<br>
Could anyone please point me through the right direction to do so?<br>
Many thanks</p>

username: ptrblck
cooked: <p>Here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(10, 10)
        self.fc2 = nn.Linear(10, 10)
        self.act = nn.ReLU()
        
    def forward(self, x):
        x1 = self.act(self.fc1(x))
        x = self.fc2(x1)
        return x, x1
    

# Create model and execute forward pass
criterion = nn.MSELoss()
model = MyModel()
x = torch.randn(1, 10)
o, h = model(x)

# Calculate losses
loss_o = criterion(o, torch.rand_like(o))
loss_h = criterion(h, torch.rand_like(h))

# Backward loss_h and keep intermediate activations
loss_h.backward(retain_graph=True)

# Check that self.fc2 grads are empty
for name, param in model.named_parameters():
    print(name, param.grad)

# Backward loss_o
loss_o.backward()

# Gradient are accumulated in self.fc1 and newly populated in self.fc2
grads2 = []
for name, param in model.named_parameters():
    print(name, param.grad)
    grads2.append(param.grad.clone())
</code></pre>
<p>You would also get the same results if you sum both losses and call <code>.backward()</code> on the result tensor.</p>
Spike in cache memory on inference using no_grad()

username: josemz
cooked: <p>Hello, I’m using a pretrained resnet50 only to do inference and I noticed the cached memory explodes in the first pass through the network. I was wondering if someone could shed some light on what’s being stored and if there is a way to control it. Here is an example:</p>
<pre><code class="lang-python">import torch

from torchvision import models

def fmt_MB(alloc, cached):
    print(f'Alloc : {alloc:&gt;8,.1f} MB\n'
          f'Cached: {cached:&gt;8,.1f} MB')

def mem_usage():
    return torch.cuda.memory_allocated() / 2**20, torch.cuda.memory_cached() / 2**20

alloc0, cached0 = mem_usage()
fmt_MB(alloc0, cached0)
</code></pre>
<p>Alloc: 0.0 MB<br>
Cached: 0.0 MB</p>
<pre><code class="lang-python">model = models.resnet50(pretrained=True).cuda()

alloc1, cached1 = mem_usage()
fmt_MB(alloc1, cached1)
</code></pre>
<p>Alloc: 97.7 MB<br>
Cached: 118.0 MB</p>
<pre><code class="lang-python">x = torch.rand((1024, 3, 224, 224)).cuda()
x_size = x.numel() * 32 / (8 * 2**20)

alloc2, cached2 = mem_usage()
fmt_MB(alloc2, cached2)
alloc2 - alloc1, cached2 - cached1, x_size
</code></pre>
<p>Alloc: 685.7 MB<br>
Cached: 706.0 MB<br>
(588.0, 588.0, 588.0)</p>
<pre><code class="lang-python">model.eval()
with torch.no_grad():
    y = model(x)
y_size = y.numel() * 32 / (8 * 2**20)

alloc3, cached3 = mem_usage()
fmt_MB(alloc3, cached3)
alloc3 - alloc2, y_size
</code></pre>
<p>Alloc: 689.6 MB<br>
Cached: 13,250.0 MB<br>
(3.90625, 3.90625)</p>
<pre><code class="lang-python">torch.cuda.empty_cache()

alloc4, cached4 = mem_usage()
fmt_MB(alloc4, cached4)
</code></pre>
<p>Alloc: 689.6 MB<br>
Cached: 3,842.0 MB</p>

username: ptrblck
cooked: <p>I would assume the forward activations will use the memory and will be cleared after the forward pass, since you’ve used <code>no_grad()</code> and thus they are not needed anymore.</p>
Why does torch.backends.cudnn only deal with recurrent layers?

username: sidg54
cooked: <p>Hello!</p>
<p>I was reading through the documentation and was wondering why there seems to only be a file for rnn.py in the torch.backends.cudnn module? The reason I ask is because while attempting to create a language model with torch.nn.GRU as one of the layers I received the following error: “RuntimeError: cuDNN Error: CUDNN_STATUS_EXECUTION_FAILED” but this error goes away when I run the same code, setting torch.backends.cudnn.enabled = False. But then a new error occurs: “RuntimeError: Input and parameters tensors are not at the same device, found input tensor at cuda:0 and parameters tensor at cpu”. I am using the following GPU setup:</p>
<ul>
<li>RTX 2080ti</li>
<li>CUDA Version: ‘9.0.176’    (this is after running torch.version.cuda)</li>
<li>cuDNN: 7501      (after checking using torch.backends.cudnn.version())</li>
</ul>
<p>And using:</p>
<ul>
<li>torch 1.1.0</li>
<li>Ubuntu 18.04</li>
</ul>
<p>The first part of the forward method’s code is the following:</p>
<pre><code>    embed = self.embeddings(x)
    h0 = self.init_hidden(self.batch_size)
    embed = embed.permute(1, 0, 2)
    if self.cuda:
        embed = embed.cuda()
        h0 = h0.cuda()
    temp, hidden = self.gru(embed, h0)
</code></pre>
<p>Thank you!</p>

username: ptrblck
cooked: <p>cudnn is also used for e.g. convolutions as seen <a href="https://github.com/pytorch/pytorch/blob/7d207363bf10402cf161057f05545d9e96417763/aten/src/ATen/native/Convolution.cpp#L126" rel="nofollow noopener">here</a>.</p>
<p>It’s was a good idea to disable cudnn, since it can potentially hide some other errors.<br>
As you can see in your example, the actual error is a device mismatch.<br>
Make sure all parameters and inputs are on the same device.<br>
While <code>embed</code> and <code>h0</code> seem to be on the default GPU, <code>self.gru</code> or some other layers might still be on the CPU.</p>
<p>Try to run your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>, as this will point to the right line of code, which causes this error.</p>

username: sidg54
cooked: <p>Thank you, this helped greatly in debugging my issues. The issue was that my GRU layer wasn’t being loaded onto cuda:0 correctly, it remained on the cpu.</p>
Maximize function with the gradient (vector-jacobian product)

username: AlexiaJM
cooked: <p>This is a problem that I frequently encountered.</p>
<p>Let say I want to find a network G (with output of size k and m parameters) which minimize Loss = F(G(z)), where F(x) is a function. I know that the gradient of the Loss with respect to the parameters of G is equal to (Grad_x F) (Jacobian G) where Grad_x F is a vector of size k and Jacobian G is size k*m. Now if I know F and G, I can just maximize the loss and apply backward().</p>
<p>But, what if I already have Grad_x F, but I don’t have access to F directly. Is there a easy way to maximize the loss? Basically, I want to know how to do the vector-jacobian product (vjp) directly and have it added to the .grad, so that I can then use any optimizer to minimize F(G(z)).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/32c8dbaa89aad94c0780de42ec35f7d3e48f859d.gif" alt="CodeCogsEqn" data-base62-sha1="7fgd146g18AvvtG5opQBmdrQ08Z" width="531" height="85"></p>

username: SimonW
cooked: <p>See <code>torch.autograd.backward</code>: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward</a></p>

username: AlexiaJM
cooked: <p>Thanks! i didn’t expect it to be that easy <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. I will try it out tomorrow.</p>
Is pytorch 1.2 available for CUDA 9.0?

username: Saurabh_Kataria
cooked: <p>I am trying to install pytorch 1.2 (conda, python3.7, linux). However, using instructions on <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a>, when specifying cuda9.0, it still installs pytorch1.1.</p>
Effective value of LR

username: Intel_Novel
cooked: <p>There is one old and dusty article in here: <a href="https://distill.pub/2017/momentum/" class="inline-onebox" rel="noopener nofollow ugc">Why Momentum Really Works</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>And I think the alpha and beta are lr and momentum that corresponds to PyTorch:</p>
<pre><code> optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
</code></pre>
<p>Now, if I print the optimizer I will get:</p>
<blockquote>
<p>SGD (<br>
Parameter Group 0<br>
dampening: 0<br>
lr: 0.05<br>
momentum: 0.9<br>
nesterov: False<br>
weight_decay: 0<br>
)</p>
</blockquote>
<p>But what I would like is to get the effective learning rate (ELR) inside the forward.<br>
For instance if batchnumer=0 is the first batch, I assume ELR is 0.05.<br>
As the batches counter increases ELR may change. See how in that article step size (ELR) will vary in size.</p>
<p>Can you give me the idea how to do that?</p>
<hr>
<p>I assume this is one step (ELR):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d7065bb7fcd675d4e050d6c55eba0fa6e60577ac.png" alt="image" data-base62-sha1="uGchhhJtonBMscinxMdctc2HDmQ" width="364" height="237"></p>
Attention Is All You Need (Transformer)

username: arjung
cooked: <p>I was looking at the paper titled “Attention Is All You Need” (<a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1706.03762.pdf</a>) which introduces the Transformer, an encoder-decoder sequence model based solely on attention mechanisms.</p>
<p>Here’s a snippet of the code from a PyTorch implementation (<a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py" rel="nofollow noopener">https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py</a>):</p>
<pre><code class="lang-auto"># prepare data
src_seq, src_pos, tgt_seq, tgt_pos = map(lambda x: x.to(device), batch)
gold = tgt_seq[:, 1:]

# forward
optimizer.zero_grad()
pred = model(src_seq, src_pos, tgt_seq, tgt_pos)

# backward
loss, n_correct = cal_performance(pred, gold, smoothing=smoothing)
loss.backward()
</code></pre>
<p>According to <a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/issues/76" rel="nofollow noopener">this</a>, <code>tgt_seq</code> is one of the inputs to the decoder, while <code>gold</code> is the target of the decoder. How is it fair that one of the inputs to the model is <code>tgt_seq</code>, where <code>gold</code>, the variable used to compute the loss, is simply a slice of <code>tgt_seq</code> (an input to the model )?</p>
<p>Any ideas will be much appreciated, thanks in advance!</p>

username: arjung
cooked: <p>I think that they are using “teacher forcing” (the concept of using the real target outputs as each next input, instead of using the decoder’s guess as the next input) which is why they pass in the target output as input to the decoder.</p>
Mask-RCNN with custom AnchorGenerator

username: Cristian_23
cooked: <p>I am trying to train a Mask-RCNN on a custom data set for instance segmentation of a single class. My images are 600x600 and I know that the instances of my class can always be bounded by boxes with width and height in the range 60-120. I want to take advantage of this and generate Anchor boxes only in that range. Below is my initialisation code for Mask-RCNN (model_bb is a standard Resnet50 trained on a custom dataset):</p>
<p>########################################################<br>
backbone = BackboneWithFPN(model_bb, return_layers, in_channels_list, out_channels)</p>
<pre><code>anchor_generator = AnchorGenerator(sizes=((60, 80, 100, 120),),
                                   aspect_ratios=((0.8, 1.0, 1.2),))

roi_pooler = MultiScaleRoIAlign(featmap_names=in_channels_list,
                                output_size=7,
                                sampling_ratio=2)

mask_roi_pooler = MultiScaleRoIAlign(featmap_names=in_channels_list,
                                     output_size=14,
                                     sampling_ratio=2)

model = MaskRCNN(backbone,
                num_classes=num_clss,
                rpn_anchor_generator=anchor_generator,
                box_roi_pool=roi_pooler,
                mask_roi_pool=mask_roi_pooler)
</code></pre>
<p>#######################################################</p>
<p>The problem is I always get the following error:<br>
RuntimeError: shape ‘[1920000, -1]’ is invalid for input of size 10232448<br>
The training works fine if I don’t pass the rpn_anchor, box_roi_pool and mask_roi_pool arguments.<br>
What am I doing wrong?</p>
Best practices for seeding random numbers on gpu?

username: hughperkins
cooked: <p>The random numbers I get on the gpu are different than on the cpu:</p>
<pre><code>In [17]: torch.manual_seed(123); torch.rand(device='cuda', size=(2,3))
Out[17]:
tensor([[ 0.0474,  0.1951,  0.2235],
        [ 0.5574,  0.7839,  0.9306]], device='cuda:0')

In [18]: torch.manual_seed(123); torch.rand(device='cuda', size=(2,3))
Out[18]:
tensor([[ 0.0474,  0.1951,  0.2235],
        [ 0.5574,  0.7839,  0.9306]], device='cuda:0')

In [19]: torch.manual_seed(123); torch.rand(device='cpu', size=(2,3))
Out[19]:
tensor([[ 0.2961,  0.5166,  0.2517],
        [ 0.6886,  0.0740,  0.8665]])

In [20]: torch.manual_seed(123); torch.rand(device='cpu', size=(2,3))
Out[20]:
tensor([[ 0.2961,  0.5166,  0.2517],
        [ 0.6886,  0.0740,  0.8665]])
</code></pre>
<p>One option could be to always generate on the cpu, then copy to the gpu. But presumably this characteristic is going to apply to eg dropout masks too? Another option would be to choose that seed consistency is only guaranteed conditioned on choice of using gpu or cpu.</p>
<p>What are best practices that are generally used to ensure reproducibility given a random seed?</p>

username: pinocchio
cooked: <p>Did you find an answer?</p>
<hr>
<p>related:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/best-practices-for-seeding-random-numbers-on-gpu/18751" class="inline-onebox">Best practices for seeding random numbers on gpu?</a></li>
<li><a href="https://discuss.pytorch.org/t/the-random-seed/19516/4" class="inline-onebox">The Random Seed</a></li>
<li><a href="https://discuss.pytorch.org/t/best-practices-for-generating-a-random-seed-to-seed-pytorch/52894/2" class="inline-onebox">Best practices for generating a random seed to seed Pytorch?</a></li>
</ul>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/57416925/best-practices-for-generating-a-random-seeds-to-seed-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    <img alt="Pinocchio" src="https://www.gravatar.com/avatar/ba92064b084638cb903cbd271120be64?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/57416925/best-practices-for-generating-a-random-seeds-to-seed-pytorch" target="_blank" rel="nofollow noopener">Best practices for generating a random seeds to seed Pytorch?</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    Pinocchio
  </a>
  on <a href="https://stackoverflow.com/questions/57416925/best-practices-for-generating-a-random-seeds-to-seed-pytorch" target="_blank" rel="nofollow noopener">04:20PM - 08 Aug 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Train multiple models in parallel on the same GPU

username: pclucas
cooked: <p>Hi,</p>
<p>I’m trying to implement this <a href="https://arxiv.org/pdf/1802.10542.pdf" rel="nofollow noopener">paper</a>. Basically, at test time, <em>for every test image x_i,</em> I want to fetch images relevant to that test image <em>x_i</em>, finetune the model on the retrieved images, make a prediction on <em>x_i</em>, and then discard the finetuned weights.</p>
<p>Right now, everything is sequential, which basically means that my test batch size is 1, which is pretty slow. Is it possible to have multiple (independent) models on the same GPU that are updated in parallel ?</p>
<p>Thanks,<br>
Lucas</p>
Weighted Random Sample works

username: Jaideep_Valani
cooked: <p>How does the weighted random sampler selects the values</p>
<pre><code class="lang-auto">list(WeightedRandomSampler([10,5,10,5,10],10, replacement=True))
[0, 0, 0, 4, 4, 1, 2, 0, 4, 1]
</code></pre>
<p>why are there more no of  zeros and 4s</p>

username: ptrblck
cooked: <p><code>WeightedRandomSampler</code> will use <code>torch.multinomial</code> internally as shown <a href="https://github.com/pytorch/pytorch/blob/7d207363bf10402cf161057f05545d9e96417763/torch/utils/data/sampler.py#L160" rel="nofollow noopener">here</a>.</p>
<p>The passed weights will determine the weight to sample each index.<br>
E.g. you can see that the returned indices will approximate the weights:</p>
<pre><code class="lang-python">sampler = WeightedRandomSampler([10,5,10,5,10],1000, replacement=True)
indices = []
for idx in sampler:
    indices.append(idx)

print(torch.tensor(indices).unique(return_counts=True))
&gt; (tensor([0, 1, 2, 3, 4]), tensor([280, 124, 230, 137, 229]))
</code></pre>
Data loader Error - 'numpy.ndarray' object has no attribute 'keys'

username: miltos
cooked: <p>My Code is actually really simple just for testing but I get this error and I can’t find anything similar online</p>
<pre><code class="lang-auto">
class customDatasetflowGraph(Dataset):

    def __init__(self,path):

        self.x = np.load(path+"/"+"dataX.npy")
        self.infoSuperp = np.load(path+"/"+"infoSuperp.npy")
        self.data_len = self.infoSuperp.shape[0]


    def __getitem__(self,index):

        print(index)
        print(self.x.shape)
        
        vid = self.x


        return vid



    def __len__(self): # This need to exist from the documentation
        return self.data_len
</code></pre>
<p>And this is the main</p>
<pre><code class="lang-auto">train_flow = aDataLoaderClass.customDatasetflowGraph('../flowData')
# train_loader_flow = DataLoader(train_flow, batch_size=64, shuffle=True)
train_loader_flow = DataLoader(train_flow, batch_size=1, shuffle = True)#,  num_workers=1)

for vid in train_loader_flow:
    print(vid.shape)
    ink = input("next ")
</code></pre>
<p>I am stack for days now if anyone has any ideas I will be more than happy to descuss thank you for your time</p>

username: Nikronic
cooked: <p>Hi,<br>
Can your add the stacktrace of error and the output of the print function you have added?</p>
Cuda() & DataParallel

username: Pirate-Roberts
cooked: <p>Hi  everyone !</p>
<p>If I have a single GPU, are the following functions equivalent ?</p>
<pre><code class="lang-auto">my_model = model.cuda()
my_model = DataParallel(model, devices_ids = [torch.cuda.current_device()])
my_model = DataParallel(model, devices_ids = [torch.cuda.current_device()]).cuda()
</code></pre>
<p>Thank you!</p>

username: ptrblck
cooked: <p>If you are using a single GPU, these approaches should yield the same results.<br>
However, wrapping the model in an <code>nn.DataParallel</code> will also use the <code>.module</code> attribute for the original model:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
print(model)
&gt; Linear(in_features=1, out_features=1, bias=True)
model = nn.DataParallel(model, device_ids=[0])
print(model)
&gt; DataParallel(
  (module): Linear(in_features=1, out_features=1, bias=True)
)
</code></pre>
<p>Do you see any unexpected behavior?</p>

username: Pirate-Roberts
cooked: <p>Thanks for the response.</p>
<p>If I do <code>my_model = model.cuda()</code> I have to do <code>model_input = model_input.cuda()</code>, otherwise I get the following message:<br>
<code>RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'weight'</code></p>
<p>But if I do:</p>
<p><code>my_model = DataParallel(model, devices_ids = [torch.cuda.current_device()])</code><br>
or<br>
<code>my_model = DataParallel(model, devices_ids = [torch.cuda.current_device()]).cuda()</code></p>
<p>I don’t need to do <code>model_input = model_input.cuda()</code>, it works as well with  <code>model_input = model_input.cpu()</code>.</p>

username: Pirate-Roberts
cooked: <p>However for both <code>model.cuda()</code> and <code>DataParallel(model, ...)</code> I don’t have to specify the <code>map_location</code> in torch.load(path, map_location = …),  it can either be <code>'cpu'</code> or <code>'cuda'</code>.</p>
<p>I use torch.load() to load the weights of a trained model from a .pt file.</p>
RuntimeError: module mush have its paraeters and buffers on device cuda:0 (device_ids[0]) but found one of them on device:cpu

username: Eden_Park
cooked: <p>Hi.</p>
<p>I’m trying to use GPU with my code. This is the code about cuda.</p>
<pre><code class="lang-auto">model = model.cuda()
model = nn.DataParallel(model, device_ids=range(torch.cuda.device_count)))
</code></pre>
<p>But I get <code>RuntimeError: module mush have its paraeters and buffers on device cuda:0 (device_ids[0]) but found one of them on device:cpu</code> this error.</p>
<p>My Pytorch already know that I have GPU on my computer. I checked it with this code.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.device_cound()
1
</code></pre>

username: ptrblck
cooked: <p>It looks like you are passing the function <code>torch.cuda.device_count</code> to <code>device_ids</code> (note the missing parentheses).<br>
Try to call this method to get the device count: <code>device_ids=torch.cuda.device_count()</code>.</p>

username: Eden_Park
cooked: <p>Thanks for your advice but sadly it was just my mistake.<br>
My actual code is <code>device_ids=torch.cuda.device_count()</code>.<br>
I think I need another adivce.</p>

username: ptrblck
cooked: <p>Could you try to call <code>DataParallel</code> first, then push the model to the device?<br>
I don’t think it’ll get rid of the error, but it’s the recommended way, if I’m not mistaken.</p>
<p>If that doesn’t help, could you check, if you have <code>to(device)</code> or <code>.cuda()</code> calls inside your model?<br>
If so, these will mess up the <code>DataParallel</code> wrapper, as they force this particular tensor to be pushed to the specified device.</p>
Dataset class train/test split without second call to Dataset object?

username: kirk86
cooked: <p>Hi folks,<br>
the <code>Dataset()</code> object when first called with the <code>train=True</code>, (i.e.)<br>
<code>dset = torchvision.datasets.CIFAR10(...)</code><br>
option contains the following params:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3e2e54ea588634baad12568b122cd3d618614379.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e2e54ea588634baad12568b122cd3d618614379" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e2e54ea588634baad12568b122cd3d618614379_2_690x39.png" alt="image" data-base62-sha1="8S4RQ32UB8e98MvLR6QEvw83Pzb" width="690" height="39" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e2e54ea588634baad12568b122cd3d618614379_2_690x39.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e2e54ea588634baad12568b122cd3d618614379_2_1035x58.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e2e54ea588634baad12568b122cd3d618614379_2_1380x78.png 2x" data-dominant-color="212421"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1886×107 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can see that the <code>test_list</code> is also pointing to the test set in our dataset, would it be valuable to have the ability to load the test set with <code>dset.train=False</code> instead of having to call <code>torchvision.datasets.CIFAR10(..., train=False)</code> again, or is it completely wrong what I’m proposing?</p>

username: ptrblck
cooked: <p>Just setting the attribute won’t change anything, since <code>self.data</code> and <code>self.target</code> was already loaded as seen in <a href="https://github.com/pytorch/vision/blob/ccbb3221b7f0637f1706df29d2c2995e9d5171bf/torchvision/datasets/cifar.py#L77" rel="nofollow noopener">these lines of code</a>.<br>
You would therefore have to implement a new method which reloads the test/train data, which would basically be another call to <code>__init__</code>.</p>
<p>In my opinion, you should definitely create separate train and test datasets, as even using separate datasets you often encounter code with data leakage.<br>
Also the training and test transformation often differ, which you would also have to pass again to the custom method.</p>

username: kirk86
cooked: <p>Thanks for the clarifications!</p>
Custom operators (RoiPooling) with ONNX

username: Feynman27
cooked: <p>I have a model with a custom RoiPooling operator that inherits from <code>torch.autograd.Function</code>.  When trying to convert the pytorch model to caffe2, I’m getting the error:</p>
<pre><code class="lang-bash">  File "/home/thomasbalestri/PycharmProjects/pytorch-faster_rcnn/tools/../lib/nets/network.py", line 110, in _roi_pool_layer
    return RoIPoolFunction(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1. / 16.)(bottom, rois)
RuntimeError: Tracing of legacy functions is not supported
</code></pre>
<p>I see that there’s a list of supported operators <a href="http://pytorch.org/docs/master/onnx.html#" rel="nofollow noopener">here</a>. Does this mean that there’s no way to export pytorch models with custom operators to caffe2?</p>

username: houseroad
cooked: <p>Hi <a class="mention" href="/u/feynman27">@Feynman27</a>, right now, we don’t support neither custom operators nor legacy operators.</p>
<p>For custom operators and legacy operators, later we may expose APIs to allows users to define the translation rules.</p>

username: dizcza
cooked: <p>Is there any progress on this since Sep 2017? If I have a custom function with forward and backward passes, how can I export it to onnx?</p>
GLIBC Version Issue

username: James_Arambam
cooked: <p>Hi Guys,</p>
<p>I have GLIBC version issue.</p>
<p>Current linux system GLIBC is 2.12 and pytorch need 2.14.</p>
<p>I tried two ways:</p>
<ol>
<li>
<p>Install from the source - No success!</p>
</li>
<li>
<p>Installed GLIBC_2.14 in home and linked pytorch with it. I could import pytorch, but when I run gives segmentataion fault.</p>
</li>
</ol>
<p>I see similar GLIBC version issue with tensorflow as well when I install using pip. But <strong>no issues if installed using conda</strong>. I suspect that tensorflow is picking glibc from anaconda.</p>
<p>But, no luck with pytorch either ways (pip or conda).</p>
<p>Any commments ?</p>
<p>Thanks,</p>
I found the wrong description on Pytorch Docs

username: YeongHwa_Jin
cooked: <p>I found the wrong description on Pytorch Docs.<br>
So I wanna revise them.<br>
How can I fix it?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>The best way is to fork the documentation on GitHub, then make changes and send a pull request. Developers will reach you there for sure.</p>
<p>Good luck</p>

username: YeongHwa_Jin
cooked: <p>Thnaks to your reply.<br>
I forked the pytorch Github, but I didn’t find the where description is.</p>

username: Nikronic
cooked: <p>It is odd. Because you can access the GitHub page exactly at the same place the docs are. If it does not bother you, can you share the doc page you are considering? Maybe I can help you find the source code.</p>

username: YeongHwa_Jin
cooked: <p>Sure<br>
I want to revise this page<br>
<a href="https://pytorch.org/docs/stable/nn.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a></p>
<p>Oh, I see.<br>
The URL of the docs page is corresponding to the directory in Github!<br>
I didn’t know that…<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Thank you!</p>
Can I register a forward hook for nn.functional.interpolate?

username: FYTau
cooked: <p>I am setting up FLOPs counter for models using <code>register_forward_hook</code> to register a counter at each module. However, I found forward hooks can only be applied on modules defined in the <code>__init__</code> function of the model after some coding experiments, meaning we cannot register hooks on APIs like <code>nn.functional.interpolate</code> because they need to confirm the size of output which can be various if the input size is not fixed.</p>
<p>A post from <a href="https://discuss.pytorch.org/t/calculating-flops-of-a-given-pytorch-model/3711/6" class="inline-onebox">Calculating flops of a given pytorch model</a> also mentioned this problem.</p>
<p>Are there any workarounds for registering a forward hook for those functions? Or are there any workarounds for counting FLOPs of interpolate together with other modules?</p>

username: FYTau
cooked: <p>if <code>nn.functional.interpolate</code> is used for upsampling, then one can register forward hooks on <code>nn.Upsample</code>.</p>
<p>However, one can always wrap <code>nn.functional.interpolate</code> into a <code>nn.Module</code> inherited class to define a module containing <code>nn.functional.interpolate</code>. Then instantiate the defined module in the <code>__init__</code> function of the model, where the defined modules can be registered forward hooks.</p>
LSTM learning with batches seems to ignore hidden state

username: saschalippert
cooked: <p>Hi,</p>
<p>after playing around with lstm i realized the following:</p>
<p>If I train the network with a fresh initialized hidden state for each batch, I will not be able to feed the network single timesteps with the hidden state of the previous timestep during testing. The test results are getting worse in that case. But if i feed sequences and a fresh hidden state during testing, the results are as expected.</p>
<p>Does that mean the lstm did just learn to ignore <strong>ALL</strong> hidden states and will also not learn the hidden state “sequence internally”?</p>
<p>Is there a good way to fix this “issue” with having batches at the same time? Organizing the batches in the natural order (kind of) seems to help, but I am not sure if this is the correct approach.</p>
<p>Thanks</p>
<p>Sascha</p>

username: saschalippert
cooked: <p>ok, so i guess my assumption is correct and initializing the hidden layer for each batch will cause the network to become stateless.</p>
<p>so to have the network statful, i either need to order the batches appropriately or “get rid” of batches (eg batch size 1). if somebody has an alternative solution, i would much appreciate it</p>
<p>here a some informative resources i found on this topic:</p>
<p><a href="https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/" class="onebox" target="_blank" rel="nofollow noopener">https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/</a></p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3njjcbhbojbot.cloudfront.net/web/images/favicons/icon-194x194.png" class="site-icon" width="194" height="194">
      <a href="https://www.coursera.org/lecture/ai/stateful-vs-stateless-lstms-NJaKk" target="_blank" rel="nofollow noopener">Coursera</a>
  </header>
  <article class="onebox-body">
    <img src="https://d3c33hcgiwev3.cloudfront.net/imageAssetProxy.v1/QjnObvfnEeetIhICvTHYsg.processed/thumbnails/360p/0.jpg?expiry=1565308800000&amp;amp;hmac=MGxjC0ub1PwD7Tc4FgFkUV1ixZYAYYqJdYQmzcqRAZ8" class="thumbnail" width="" height="">

<h3><a href="https://www.coursera.org/lecture/ai/stateful-vs-stateless-lstms-NJaKk" target="_blank" rel="nofollow noopener">Stateful vs. Stateless LSTMs - DeepLearning Applications | Coursera</a></h3>

<p>Learn online and earn valuable credentials from top universities like Yale, Michigan, Stanford, and leading companies like Google and IBM. Join Coursera for free and transform your career with degrees, certificates, Specializations, &amp; MOOCs in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p><a href="https://www.quora.com/Im-having-a-hard-time-conceptualizing-the-difference-between-stateful-and-stateless-LSTMs-in-Keras-What-does-one-choose-stateful-vs-stateless" class="onebox" target="_blank" rel="nofollow noopener">https://www.quora.com/Im-having-a-hard-time-conceptualizing-the-difference-between-stateful-and-stateless-LSTMs-in-Keras-What-does-one-choose-stateful-vs-stateless</a></p>
What to understand/interpret from training/validation epoch loss, data

username: banikr
cooked: <p>Hello,<br>
I am training a network with 3D MR-CT images.<br>
right now I have the epoch wise loss being printed like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/67b17d99a5fdee5cc76eda664c188cf61f225adf.png" data-download-href="https://discuss.pytorch.org/uploads/default/67b17d99a5fdee5cc76eda664c188cf61f225adf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/67b17d99a5fdee5cc76eda664c188cf61f225adf.png" alt="image" data-base62-sha1="eNjrZ3EJJFkvedjvWde6Sa5WQhN" width="517" height="270" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1019×533 8.67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The overall train and validation loss looks so far:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4b4af71a012bb1a71e3ff393114a86ca89dc0f73.png" alt="image" data-base62-sha1="aK4qcmsnZX238RLVs18G6Veb2z9" width="306" height="204"></p>
<p>train and validation losses in last 15 epochs:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f7b9e33f076b84c43c787f189d3fa4378fa34105.png" alt="image" data-base62-sha1="zlucg27pM1LptsaIUnNHGDuxj37" width="312" height="207"></p>
<p>Overall validation loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/762fedef9678422da6e672886410d4a5cf85068c.png" alt="image" data-base62-sha1="gRx1QM4veJU6eJuU3zbMmLJAOTy" width="310" height="204"></p>
<p>The model saving criteria is minimum validation loss, so the model was last saved in epoch 30.<br>
As the network is still training and will be so untill epoch 120. What can I interpret from these data about the network being trained</p>
Access the weights of manually implemented backward via Autograd

username: DeepLearner17
cooked: <p>Hello,</p>
<p>l would like to print the weights of a specific layer which is implemented manually with autograd. It is called my_autograd_function().</p>
<p>I tried<br>
print(net.my_autograd_function.weight) but it doesn’t work, “net.my_autograd_function” is not recognized. Here is my code :</p>
<pre><code class="lang-auto">class my_autograd_function(torch.autograd.Function):
    def forward(self,W,x):  
        self.save_for_backward(W, x)
        y = torch.mm(W, x)
        return y
    
    def backward(self, grad_output):
        W, x = self.saved_tensors 
        grad_input = grad_output.clone()
        grad_input_dL_dW = torch.mm(grad_input, x.t()) 
        grad_input_dL_dx = torch.mm(W.t(), grad_input )
        return grad_input_dL_dW, grad_input_dL_dx



class model(nn.module):
	def __init__(self, model_parameters):
           # self.layer1
	   # self.layer2
           # self.layer3
         def forward(self,x):
            x=layer1(x)
            x=layer2(x)
	    x=my_autograd_function()(x,Z)
            x=layer3(x)
	    return x

if __name__ == '__main__':

net=model()

print(net.layer1.weight) # it works
print(net.layer2.weight) # it works
print(net.layer3.weight) # it works
print(net.my_autograd_function.weight) # it doesn't work   "net.my_autograd_function" is not recognized
</code></pre>