akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BatchNorm stats update leads to huge drop in accuracy during eval

Laurent_Dillard: <p>I’m currently trying to implement Virtual Adversarial Training (<a href="https://arxiv.org/abs/1704.03976" rel="noopener nofollow ugc">https://arxiv.org/abs/1704.03976</a>) method and test it on CIFAR100.</p>
<p>I came across a weird behavior working on that project. During training, an adversarial perturbation is generated, in both the TensorFlow and Chainer official implementations it is enforced that the batchnorm stats are not updated while generating the adversarial perturbation (which makes sense since those can deviate significantly from the true data distribution). Using PyTorch, what I thought was the best way to accomplish this was the following:</p>
<pre><code class="lang-auto">@staticmethod
def set_bn_eval(m):
   if isinstance(m, nn.modules.batchnorm._BatchNorm):
      m.eval()

@staticmethod
def set_bn_train(m):
   if isinstance(m, nn.modules.batchnorm._BatchNorm):
      m.train()
</code></pre>
<p>And then running</p>
<pre><code class="lang-auto">self.trainer.model.apply(self.set_bn_eval) # disable batch stats update
</code></pre>
<p>And</p>
<pre><code class="lang-auto">self.trainer.model.apply(self.set_bn_train)
</code></pre>
<p>Respectively before and after the adversarial perturbation is generated.</p>
<p>The training goes well and I obtain around 30% accuracy after the first epoch however when switching to eval mode and processing the validation set the accuracy drops massively (even to 1-2%). For further epochs training accuracy keeps increasing while validation remains extremely low.</p>
<p>First, I did not have that discrepancy between train and eval when not using VAT. I checked that simply the model in train mode mode during validation solves the issue which confirms it is related to a change between train and eval behavior. I also checked that when switching to eval mode but forcing BN layers to use batch statistics instead of the running ones (by setting them to None and setting track running stats to False) then the performance is again matching that of training.</p>
<p>Hence my only conclusion is that the BatchNorm layers are indeed updating their running stats during the adversarial perturbation generation which is extremely unsettling because I checked by hand that the set_bn_eval and set_bn_train were working as expected and that running stats were not updated when set_bn_eval had been apply.</p>
<p>The link to my repo is: <a href="https://github.com/laurent3577/Image-Classification" rel="noopener nofollow ugc">https://github.com/laurent3577/Image-Classification</a></p>
<p>I’m really puzzled as to what exactly is going on there and would appreciate any help.</p>
Setting requires_grad on gpu doesn't work

niata: <p>I want to set requires_grad to True on the input to find out what information is used.<br>
When doing that on the cpu everything is fine, but when using the gpu it doesn’t like that.</p>
<blockquote>
<p>RuntimeError: cudnn RNN backward can only be called in training model</p>
</blockquote>
<p>here is an example (the model is simplified):</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
sequencelength = 366
inputlength = 202
batchsize = 128
device="cuda:1"
class model(nn.Module):
    def __init__(self,inputlength,device):
        super(model, self).__init__()
        self.lstm=nn.LSTM(inputlength,inputlength).to(device)
        self.linear=nn.Linear(inputlength,1).to(device)
    def forward(self,x):
        x,_=self.lstm(x)
        return self.linear(x[-1])
net=model(inputlength,device)
net.eval()
sequences=torch.randn([sequencelength,batchsize, inputlength]).to(device)
sequences.requires_grad=True
out = net(sequences)
out[0].backward()
data=torch.mean(torch.mean(sequences.grad.data.abs(),dim=0),dim=1).cpu().numpy()
</code></pre>

albanD: <p>Hi,</p>
<p>You will have to disable cudnn during the rnn computation for this to work I’m afraid.<br>
cudnn does not support double backward. Hence the error you’re seeing.</p>
Speed up image loading in CPU before transferring to GPU

add023: <p>TLDR: I am unable to fully utilize GPU because a lot of time is spent loading images. What are some strategies to always keep the GPU utilized?</p>
<p>Say I have a batch of size 8. Each element in the batch is a video folder with Y number of frames, where Y is between 90 and 400. The number of frames differs in each video folder, so in a given batch I try to use video folders with similar number of frames to minimize padding. The images are usually resized to 224x224. The total training data size is greater than system RAM. I use <code>DataLoader</code> with <code>num_workers &gt; 1</code> and then transfer the batch tensor to <code>CUDA</code>. Right now I am using 1 GPU.</p>
<p>I’ve tried to tune <code>num_workers</code> and got some speed up. Regardless, the GPU is not fully utilized. There is delay of a few seconds when new batch is created.</p>
<p>What are approaches I can take? I think creating a forward looking queue of 2+ batches would ensure GPU always has 1 batch available. Can this be done in conjunction with <code>DataLoader</code>?</p>

ptrblck: <p>If you use <code>num_workers &gt; 1</code> you are already creating multiple workers, where each worker will load a batch and push it to the queue.<br>
If you are still seeing a slowdown in loading the data, it would mean that your current data loading pipeline is still not fast enough.<br>
<a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> gives some insights why data loading can be the bottleneck and some best practices.</p>

add023: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. The link is useful.</p>
How to do torch::nn::Linear = torch::nn::Sequential like python

Eokseob_Kim: <p>I want to use the pretrained resnet50 for transfer-learning and fine-tuning at C++.</p>
<pre><code>at python&gt;&gt;
resnet50 = models.resnet50(pretrained=True)

fc_inputs = resnet50.fc.in_features  #fc_inputs: 2048
resnet50-&gt;fc = nn.Sequential(
    nn.Linear(fc_inputs, 256),
    nn.ReLU(),
    nn.Dropout(0.4),
    nn.Linear(256, 2))
at C++ &gt;&gt;
resnet50.fc = torch::nn::Sequential()  &lt;- ERROR</code></pre>

a_d: <p>in libtorch,  you can do the following,</p>
<pre><code class="lang-auto">torch::nn::Sequential resnet50;
resnet50.push_back(nn::Conv2d(nn::Conv2dOptions(64, 256, 3)).stride(1).padding(1));
</code></pre>
<p>to run the forward<br>
<code>resnet50-&gt;forward(input);</code></p>
<p>Hope it helps</p>
Multiple node, Multiple worker Allreduce

vineeths: <p>How is Multiple node, Multiple worker Allreduce implemented in PyTorch?</p>
<p>I know that in a single node multi-worker setting, allreduce is implemented with a ring allreduce algorithm. How does this change in a multinode setting?</p>

mrshenli: <p>Hey <a class="mention" href="/u/vineeths">@vineeths</a>, PyTorch distributed <code>all_reduce</code> calls into the allreduce API provided by the communication backend (Gloo, NCCL, and MPI). Gloo uses ring allreduce. NCCL has both ring and tree allreduce. See this discussion: <a href="https://github.com/NVIDIA/nccl/issues/256" rel="noopener nofollow ugc">https://github.com/NVIDIA/nccl/issues/256</a></p>
Error occurs in rpc-pipeline code in tutorial

ConnollyLeon: <p>I am using torch 1.6. When I try to run the code, it tells me like below:</p>
<p>AttributeError: ‘torch.distributed.rpc.TensorPipeRpcBackendOptions’ object has no attribute ‘num_send_recv_threads’</p>
<p>Could anyone please tell me what possible reasons could be?</p>
<p>details are like below:<br>
File “rpc_pipeline.py”, line 291, in setup<br>
rpc_backend_options=options<br>
File “<em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/<strong>init</strong>.py", line 90, in init_rpc<br>
rpc_backend_options=options<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/<strong>init</strong>.py”, line 90, in init_rpc<br>
api._init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)<br>
File “<em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 299, in _init_rpc_backend<br>
api._init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py”, line 299, in _init_rpc_backend<br>
api._init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)<br>
File “<em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 299, in _init_rpc_backend<br>
rpc_backend_options=rpc_backend_options,<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py”, line 94, in init_backend<br>
rpc_backend_options=rpc_backend_options,<br>
File “<em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 94, in init_backend<br>
rpc_backend_options=rpc_backend_options,<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py”, line 94, in init_backend<br>
return backend.value.init_backend_handler(<em>args, **kwargs)<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 144, in _process_group_init_backend_handler<br>
return backend.value.init_backend_handler(<em>args, **kwargs)<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 144, in _process_group_init_backend_handler<br>
rpc_backend_options.num_send_recv_threads,<br>
AttributeError: ‘torch.distributed.rpc.TensorPipeRpcBackendOptions’ object has no attribute ‘num_send_recv_threads’<br>
rpc_backend_options.num_send_recv_threads,<br>
AttributeError: ‘torch.distributed.rpc.TensorPipeRpcBackendOptions’ object has no attribute ‘num_send_recv_threads’<br>
return backend.value.init_backend_handler(<em>args, **kwargs)<br>
File "</em>/conda/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 144, in _process_group_init_backend_handler<br>
rpc_backend_options.num_send_recv_threads,</p>

ConnollyLeon: <p>I solve this after adding this argument.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/037a56585139c67b25735853d41042b89b125842.png" alt="image" data-base62-sha1="uLxphG4iGA28VUnlk6otFmqnOq" width="658" height="288"></p>

mrshenli: <p>Yep, we are switching to TensorPipe as the default RPC backend in v1.7.</p>
Semantic segmentation evaluation metrics (from torchvision references)

tominator: <p>I train a DeepLabV3 and FCN (torchvision) with own dataset (2 classes: background and my object) and use the metrics calculated in the class <a href="https://github.com/pytorch/vision/blob/master/references/segmentation/utils.py" rel="noopener nofollow ugc">ConfusionMatrix</a> in the <a href="https://github.com/pytorch/vision/blob/master/references/segmentation/utils.py" rel="noopener nofollow ugc">utils.py</a>.</p>
<p>Can someone explain these metrics to me? I know the IoU, but what do “global correct” and “average row correct” mean?</p>
<pre><code class="lang-auto">Test:  [ 0/32]  eta: 0:00:18    time: 0.5879  data: 0.1397  max mem: 0
Test: Total time: 0:00:18
global correct: 99.5
average row correct: ['99.9', '88.2']
IoU: ['99.4', '86.9']
mean IoU: 93.2
</code></pre>

Caruso: <p>Hi,</p>
<p><code>average row correct</code> shows the accuracy per class - so the amount it predicted this class correct divided by the amount it predicted a pixel as this class. In the code this is given as <code> acc = torch.diag(h) / h.sum(1)</code>, the diagonal of h are the correctly predicted pixels per class and the h.sum(1) is just the sum of each row in the ConfusionMatrix - so the total amount this class got predicted.<br>
The <code>global correct</code> is your pixel accuracy, which tells you how much of your image got correctly classified - in the code: <code>acc_global  = torch.diag(h).sum() / h.sum()</code>.</p>
<p>Easy example</p>
<pre><code class="lang-auto">				       Actual Class
			       |	Cat	|	Dog	|
Predicted	Cat	   |	5	|	2	|
class		Dog	   |	3	|	4	|

average row correct:
cat accuracy: 5 / (5 + 2) = 0.71
dog accuracy: 4 / (3 + 4) = 0.57

global correct:
total acc = (5 + 4) / (5 + 2 + 3 + 4) =  0.64

IoU:
cat IoU = 5 / (2 + 5 + 3) = .5
dog IoU = 4 / (2 + 4 + 3) = .44
mIoU = .47 

</code></pre>

tominator: <p>Thank you very much <a class="mention" href="/u/caruso">@Caruso</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
View-wise operation?

zh7117: <p>I have two Tensors A and B,<br>
A.shape is (b,c,100,100),<br>
B.shape is (b,c,80,80),<br>
how can I get tensor C with shape (b,c,21,21) subject to<br>
C[:, :, i, j] = torch.mean(A[:, :, i:i+80, j:j+80] - B)?<br>
I wonder whether there’s an efficient way to solve this?<br>
Thanks very much.<br>
Sorry if it’s not a proper title.</p>

seungjun: <p>If I understand you correctly, would it be the way to subtract B from part of A?<br>
You can specify corresponding index ranges if the dimension lengths equal to each other.</p>
<pre><code class="lang-python">C = A[:,:,80:100, 80:100] - B
# C shape: b x c x 20 x 20
</code></pre>
<p>You can further apply <code>torch.mean</code> afterwards by your needs.</p>

zh7117: <p>Appreciate for your reply. But you did misunderstand my question.<br>
I hope B to be a filter, do torch.mean(slice_A - B), and slide B to next position, etc. And the result would has shape (b, c, A.shape[-1] - B.shape[-1] + 1,  A.shape[-1] - B.shape[-1] + 1).</p>

seungjun: <p>Ah, sorry for the misunderstanding.<br>
You can try below.</p>
<pre><code class="lang-python">C = (A.unfold(2,80,1).unfold(3,80,1) - B.expand(b,c,21,21,80,80)).mean(dim=(-2,-1))

# variable version
_, _, H, W = A.shape
_, _, h, w = B.shape
C = (A.unfold(2,h,1).unfold(3,w,1) - B.expand(b,c,H-h+1,W-w+1,h,w)).mean(dim=(-2,-1))
</code></pre>
<p>The keys are <a href="https://pytorch.org/docs/stable/tensors.html?highlight=unfold#torch.Tensor.unfold" rel="noopener nofollow ugc">torch.Tensor.unfold</a> and <a href="https://pytorch.org/docs/stable/tensors.html?highlight=torch%20tensor%20expand#torch.Tensor.expand" rel="noopener nofollow ugc">torch.Tensor.expand</a>.</p>
<p>The unfold function extracts sliding window and expand function views the tensor in expanded size without copying values.</p>

zh7117: <p>I asked the same question on stackoverflow. <a href="https://stackoverflow.com/questions/64313895/element-wise-operation-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/64313895/element-wise-operation-in-pytorch</a></p>
Weight.get_device() returns -1

sumanth9: <p>I got the following error <strong>RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1595629411241/work/aten/src/THC/THCCachingHostAllocator.cpp:278</strong> while running my code with cuda().</p>
<p>So I decided to check the device number for the variables. I printed following variables from forward() function<br>
<strong>input_ device no:  1</strong><br>
<strong>support device no:  1</strong><br>
<strong>weight device no:  -1</strong></p>
<p>Here is the code that caused the error.</p>
<p>import math<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
from torch.nn.parameter import Parameter<br>
from torch.nn.modules.module import Module</p>
<p>class GNNLinear(Module):</p>
<pre><code>def init(self, in_features, out_features, num_supports, act_fn, dropout=False, batch_norm=False):
 ¦   super(GNNLinear, self).init()
 ¦   self.num_supports = num_supports
 ¦   self.in_features = in_features
 ¦   self.out_features = out_features
 ¦   self.dropout = dropout
 ¦   if self.dropout:
 ¦   ¦   self.dropout_fn = nn.Dropout(p=0.5, inplace=True)
 ¦   self.batch_norm = batch_norm
 ¦   if self.batch_norm:
 ¦   ¦   self.batch_norm_fn = nn.BatchNorm1d(out_features)
 ¦   self.act_fn = act_fn
 ¦   self.weight = []
 ¦   for i in range(num_supports):
 ¦   ¦   self.weight.append(Parameter(torch.FloatTensor(in_features, out_features)))
 ¦   if bias:
 ¦   ¦   self.bias = Parameter(torch.FloatTensor(out_features))
 ¦   else:
 ¦   ¦   self.register_parameter('bias', None)
 ¦   self.reset_parameters()

 def reset_parameters(self):
 ¦   for i in range(self.num_supports):
 ¦   ¦   ¦bound = 1. / math.sqrt(self.weight[i].size(1))
 ¦   ¦   ¦self.weight[i].data.uniform_(-bound, bound)
 ¦   if self.bias is not None:
 ¦   ¦   self.bias = Parameter(torch.zeros(self.bias.shape, dtype=torch.float32))

 def forward(self, input_, support):
 ¦   if self.dropout:
 ¦   ¦   input_ = self.dropout_fn(input_)
 ¦   z = 0
 ¦   for i in range(self.num_supports):
 ¦   ¦   print('input_ device no: ', input_.get_device())
 ¦   ¦   print('support device no: ', support[i].get_device())
 ¦   ¦   print('weight device no: ', self.weight[i].get_device())
 ¦   ¦   h_i = torch.mm(input_, self.weight[i])
         print(h_i)
 ¦   ¦   z_i = torch.sparse.mm(support[i], h_i)
 ¦   ¦   z += z_i
 ¦   if self.bias is not None:
 ¦   ¦   z = z + self.bias
 ¦   if self.act_fn:
 ¦   ¦   z = self.act_fn(z)
 ¦   if self.batch_norm:
 ¦   ¦   z = self.batch_norm_fn(z)
 ¦   return z
 ¦

 def __repr__(self):
 ¦   return self.__class__.__name__ + ' (' \
 ¦   ¦   ¦  + str(self.in_features) + ' -&gt; ' \
 ¦   ¦   ¦  + str(self.out_features) + ')'</code></pre>

sumanth9: <p>Someone please help me with this</p>

ptrblck: <p><code>self.weight</code> seems to be on the CPU, since you’ve used a <code>list</code> for it.<br>
To properly push parameters to the device, either register them directly as <code>nn.Parameter</code>s or use <code>nn.ParameterList</code> instead of a Python <code>list</code>.</p>

sumanth9: <p>Worked,<br>
Thank you for your reply.</p>
Fintuning a pruned model, makes it 10x slower at inference

Shisho_Sama: <p>I dont know if I have encountered a bug or its something that I’m doing wrong.<br>
I pruned a resnet18 model and then trained it. during checkpoints, I save the model as a jit model so I dont need to have a model definition to run the pruned model.<br>
Everything seems fine, the model learns and performs as expected. however, what is not normal is the 10x times slow-down at inference.<br>
Below are the stats I recorded for 10 iterations of the pruned model forward time before and after being trained :</p>
<pre><code class="lang-auto">Before training : 7,405 ms  
After training :  75,480 ms
</code></pre>
<p>You can see and test the models yourself from here : <a href="https://gofile.io/d/sFkJkO" rel="noopener nofollow ugc">https://gofile.io/d/sFkJkO</a></p>
<p>Whats wrong here?<br>
Thanks a lot in advance</p>
Optimizing loops

Jean-Eric_Campagne: <p>Hello,<br>
I wander if someone can help me to find a better way to do the following things (ie. I’m not a python geek)<br>
Let say that in a forward method of my model I get a batch of image of size N,Cin,H,W, then I make a treatment that produce a tensor of dim; N, Cin, K, H, W let us call it ‘x’<br>
with K=1+J*L+L**2 J(J-1)/2  for some J and L values.</p>
<p>Now, I compute some means values on the two last dims (H,W)</p>
<pre><code class="lang-auto">meanCoeff= torch.mean(x,axis=(3,4))
</code></pre>
<p>Then, I proceed like that</p>
<pre><code class="lang-auto">xnew=torch.zeros_like(x)
#lvl 0
xnew[:,:,0,:,:] = x[:,:,0,:,:]
#lvl 1
for j1 in range(0,J):
     for t1 in range(0,L):
           i=adict[(j1,t1)]
           xnew[:,:,i,:,:] = x[:,:,i,:,:]-meanCoeffs[:,:,i,None,None]
#lvl 2
for j1 in range(0,J-1):
     for t1 in range(0,L):
           i1=adict[(j1,t1)]
          for j2 in range(j1+1,J):
               for t2 in range(0,L):
                    i12=adict[(j1,t1,j2,t2)]
                    xnew[:,:,i12,:,:] =x[:,:,i12,:,:]-meanCoeffs[:,:,i1,None,None] 
</code></pre>
<p>where in case of J=2, L=4 adict is a dictionary equals to</p>
<pre><code class="lang-auto">adict={(-1,): 0,
 (0, 0): 1,
 (0, 1): 2,
 (0, 2): 3,
 (0, 3): 4,
 (1, 0): 5,
 (1, 1): 6,
 (1, 2): 7,
 (1, 3): 8,
 (0, 0, 1, 0): 9,
 (0, 0, 1, 1): 10,
 (0, 0, 1, 2): 11,
 (0, 0, 1, 3): 12,
 (0, 1, 1, 0): 13,
 (0, 1, 1, 1): 14,
 (0, 1, 1, 2): 15,
 (0, 1, 1, 3): 16,
 (0, 2, 1, 0): 17,
 (0, 2, 1, 1): 18,
 (0, 2, 1, 2): 19,
 (0, 2, 1, 3): 20,
 (0, 3, 1, 0): 21,
 (0, 3, 1, 1): 22,
 (0, 3, 1, 2): 23,
 (0, 3, 1, 3): 24}
</code></pre>
<p>My question is : is there a better way to compute xnew (eg. vectorization) especially on cuda device (of course)</p>
<p>Thanks</p>
Can I use an nlp model to suggest the next word in a sequence, from a limited dictionary?

hellothere: <p>Hello, I’ve downloaded a GPT2 model with PyTorch, and I can use it to generate text. However, my use case is slightly different - I need it to only select the next word from a limited dictionary,</p>
<p>e.g. The black cat sat on the [rat, hat, mat, fat …]</p>
<p>So in this case I would want it to complete the sentence, only using the words rat, hat, mat or fat etc.</p>
<p>I’ve been playing around with PyTorch, but I’m only a beginner so I’m not sure where to begin with this. Perhaps GPT2 is not the right model for this use case - if so, could you recommend me a more suitable one?</p>
<p>Thank you for your help!</p>
Broadcast C++ API

mfkasim: <p>I am working on wrapping a function written in C++ to PyTorch. The C++ function takes 3 floats and produces 1 float.<br>
Is there a C++ PyTorch API to perform the broadcasting of the wrapped function?<br>
Probably something like np.nditer in numpy.</p>

ptrblck: <p>I think tensors will be automatically broadcasted in the C++ API in the same manner as in Python, won’t they?<br>
To trigger broadcasting you could <code>unsqueeze</code> dimensions if necessary.<br>
Let me know, if I misunderstood your use case.</p>

mfkasim: <p>Thanks for your answer.<br>
I have a C++ code with signature <code>double some_fcn(double a, double b)</code>, and I want to wrap it to PyTorch function <code>Tensor some_fcn_wrap(Tensor&amp; a, Tensor&amp; b)</code> with some broadcasting capability.<br>
The problem of wrapping it is I need to implement the broadcasting inside the function, but I’m sure there’s a function (or class) in PyTorch’s C++ source code to do that.<br>
That’s what I’m looking for. Something like <a href="https://numpy.org/doc/stable/reference/generated/numpy.nditer.html" rel="noopener nofollow ugc">numpy.nditer</a>.</p>
Model.to('cpu') memory leak issue

Omer_B: <p>Hey,<br>
I have a workload that has to take a model back and forth to cuda and cpu:</p>
<pre><code class="lang-python">for i in range(iterations):

    with torch.no_grad():
        model.to('cuda')

    ... operations ...

    with torch.no_grad():
        model.to('cpu')
</code></pre>
<p>This code leads to memory overflow on CPU depending on how many iterations there are.<br>
Am I missing something? or is it a bug?</p>

ptrblck: <p>If I understand the issue correctly your (CPU) RAM is increasing by pushing the model back and forth?<br>
Could you post an executable code snippet, which reproduces this issue so that we could debug it?</p>
Unable to read the masks from the getitem() in the custom dataset. Please help!

Varun_Tirupathi: <p>Hi, I implemented the custom dataset in which both features and labels are images. I’m unable to read the masks it is showing assertion error. Can someone help with this please I’m stuck here please find my code below:</p>
<p>class DirDataset(Dataset):<br>
def <strong>init</strong>(self, img_dir, mask_dir):<br>
self.img_dir = img_dir<br>
self.mask_dir = mask_dir</p>
<pre><code>    try:
        self.ids = [s.split('.')[0] for s in os.listdir(self.img_dir)]
    except FileNotFoundError:
        self.ids = []

def __len__(self):
    return len(self.ids)

def __getitem__(self, i):
    idx = self.ids[i]
    img_files = glob.glob(os.path.join(self.img_dir, idx+'.*'))
    mask_files = glob.glob(os.path.join(self.mask_dir, idx+'_mask.*'))

    assert len(img_files) == 1, f'{idx}: {img_files}'
    assert len(mask_files) == 1, f'{idx}: {mask_files}'

    # use Pillow's Image to read .gif mask
    # https://answers.opencv.org/question/185929/how-to-read-gif-in-python/
    img = Image.open(img_files[0])
    mask = Image.open(mask_files[0])
    assert img.size == mask.size, f'{img.shape} # {mask.shape}'

    # img = self.preprocess(img)
    # mask = self.preprocess(mask)

    return torch.from_numpy(img).float(), \
        torch.from_numpy(mask).float()</code></pre>

ptrblck: <p>What kind of error are you seeing?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Print shape of a tensor

avinash_m: <p>I am a beginner in pytorch. I have the following neural network<br>
class FirstNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(FirstNN,self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(784,hidden_size)<br>
self.act1 = nn.ReLU()<br>
self.fc2 = nn.Linear(hidden_size,num_classes)</p>
<pre><code>def forward(self,x):
    out = self.fc1(x)
    print(out.Size())
    out = self.act1(out)
    out = self.fc2(out)
    return out
</code></pre>
<p>model = FirstNN()</p>
<p>I wanted to know the shape of tensor after fc1. When I tried using print function it doesn’t print anything.How can I know the shape of tensor??</p>

ptrblck: <p>Did you pass a random sample through your model?</p>
<pre><code class="lang-python">x = torch.randn(1, 784)
model = FirstNN()
out = model(x)
</code></pre>

avinash_m: <p>No I haven’t. It’s working now. Thanks a lot.</p>

Shekhar_Gupta: <p><a class="mention" href="/u/avinash_m">@avinash_m</a>  How did you print shape of tensor after fc1?</p>

KarthikR: <p>The print statement is inside the ‘forward’ function. However, you need to call the function to see the result. You call the function as given above:</p>
<p>x = torch.randn(1, 784)<br>
model = FirstNN()<br>
<strong>out = model(x)</strong>  <span class="hashtag">#This</span> makes a call to forward</p>
Custom Dataset for Siamese CNN

simoloca: <p>Hi all,<br>
I’m building a particular kind of dataset for a Siamese neural network.<br>
I was inspired by this topic: <a href="https://discuss.pytorch.org/t/dataloader-for-a-siamese-model-with-concatdataset/66085" class="inline-onebox">Dataloader for a Siamese Model with ConcatDataset</a>, but my implementation is slightly different since I need to take the 2 images, passed to the net, from 2 different folders and I need that the first and the second one images are always taken from the respective dataset.<br>
So I tried the implementation linked but it doesn’t fit my problem because the Concat function don’t respect the constraint discussed.<br>
Thus, my question is: is there a way to tell the Concat function to respect these constraint?</p>
<p>P.s. The 2 dataset/folders are of different size so I think that the one with less images must recycle some of its images to be combined with others of the second bigger dataset</p>

simoloca: <p>Anyone got some tips for this task?</p>

ptrblck: <p>Since your use case seems to be a bit more complicated I would recommend to implement a custom <code>Dataset</code> as explained in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>.<br>
This would allow you to implement the loading logic (and correspondence of the image folders) manually.</p>
Getting this error while installing torch and torchvision!

Vinayaka_Shastri: <p>I’m trying to install pytorch and torch vision(pip3 install torch==1.2.0+cpu torchvision==0.4.0+cpu -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>) in my u<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/115afa245b3a8eeba227e1024fa77d822489d398.png" data-download-href="https://discuss.pytorch.org/uploads/default/115afa245b3a8eeba227e1024fa77d822489d398" title="ubuntu@ip-172-31-23-180_ ~ 10-10-2020 03_06_18"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/115afa245b3a8eeba227e1024fa77d822489d398.png" alt="ubuntu@ip-172-31-23-180_ ~ 10-10-2020 03_06_18" data-base62-sha1="2tx28TzpcjstsfcOAye4m7wcmJq" width="690" height="414" data-dominant-color="141111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ubuntu@ip-172-31-23-180_ ~ 10-10-2020 03_06_18</span><span class="informations">802×482 9.84 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> buntu AWS machine. But im getting this error.</p>

ptrblck: <p>Unsure, but maybe you could install the wheel directly from the URL (with <code>pipenv</code>?).<br>
Btw. what’s your use case that you want to install such an old PyTorch version?</p>
What does param.requires_grad = False or True do in the Pretrained model

Kalyan: <p>In the following code, what’s the role of param.requires_grad = True? what happens when param.requires_grad = False?</p>
<pre><code class="lang-auto">model = torchvision.models.resnet18(pretrained=True)

for param in model.parameters():
    param.requires_grad = True
    
model.fc  = nn.Sequential  (OrderedDict ([
                            ('fc1', nn.Linear (512, 256)),
                            ('relu1', nn.ReLU ()),
                            ('dropout1', nn.Dropout (p = 0.5)),
                            ('fc2', nn.Linear (256, 128)),
                            ('relu2', nn.ReLU ()),
                            ('dropout2', nn.Dropout (p = 0.5)),
                            ('fc3', nn.Linear (128, 10)), 
                            ('output', nn.LogSoftmax (dim =1))
                            ]))

</code></pre>

KarthikR: <p>If requires_grad is set to false, you are freezing the part of the model as no changes happen to its parameters.  In the example below, all layers have the parameters modified during training as requires_grad is set to true.</p>
<p>import torch, torchvision<br>
import torch.nn as nn<br>
from collections import OrderedDict</p>
<p>model = torchvision.models.resnet18(pretrained=True)<br>
for param in model.parameters():<br>
param.requires_grad =<strong>True</strong></p>
<p>model.fc  = nn.Sequential  (OrderedDict ([<br>
(‘fc1’, nn.Linear (512, 256)),<br>
(‘relu1’, nn.ReLU ()),<br>
(‘dropout1’, nn.Dropout (p = 0.5)),<br>
(‘fc2’, nn.Linear (256, 128)),<br>
(‘relu2’, nn.ReLU ()),<br>
(‘dropout2’, nn.Dropout (p = 0.5)),<br>
(‘fc3’, nn.Linear (128, 10)),<br>
(‘output’, nn.LogSoftmax (dim =1))<br>
]))</p>
<p>for name, param in model.named_parameters():<br>
print('Name: ', name, ’ Requires_Grad: ', param.requires_grad)</p>
<p>Try changing the highlighted bold text in above code to False and you will see the model is frozen except for the last ‘fc’ layer, which you have modified.</p>
PyTorch Index Error for Boolean Tensors

Lingyao_Zhang: <p>I saw this unexpected error for expanded boolean tensor.</p>
<pre><code class="lang-auto">expanded_mask shape: torch.Size([1105, 10, 2]), i: 0, j: 0
aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [22,0,0], thread: [96,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
</code></pre>
<p>I have the following code:</p>
<pre><code class="lang-auto">def create_mask(sizes: torch.Tensor, mask_dim: int) -&gt; torch.Tensor:
    batch_size = sizes.shape[0]
    mask = torch.arange(mask_dim, dtype=sizes.dtype, device=sizes.device).expand(
        batch_size, mask_dim
    ) &lt; sizes.unsqueeze(1)
    return mask


old_mask = create_mask(sizes, 10) # old_mask has shape (1105, 10)
batch_size, mask_dim = old_mask.shape
expanded_mask = old_mask.reshape(batch_size, mask_dim, 1).expand(batch_size, mask_dim, 2)

for i in range(batch_size):
    for j in range(mask_dim):
       print(float(expanded_mask[i][j][0]))-&gt; this will raise error about index
</code></pre>
<p>I am pretty sure it should not be out of bound. Not sure if this is the real error or some potential bug. This only happens when I convert other float tensors into half precision, but this should not touch anything about creating mask here. If I don’t do any conversion, then I don’t see this error which is very strange.</p>

ptrblck: <p>Based on the code snippet it looks indeed a bit fishy.<br>
Could you post an executable code snippet, so that we could reproduce this issue and debug it, please?</p>
<p>Also, which Pytorch and CUDA versions are you using?</p>
Label Layout in Multi-Class Classification

JustCunn: <p>I am a bit new to ML and PyTorch and I’ve been trying to get to grips with this all. I’m trying to train an LSTM that gives me probabilites for my multi-class data. For example, I have a sentence and each sentence can be given a label from 0 to 4 (and only one label, so it’s not multi-label classification).</p>
<p>Right now, I was working from an example (using my own data), but it seems that was for binary classification. When passing <code>y</code> to my model during training, it appears that it is only a tensor of labels corresponding to each sentence like <code>[1,4,2,0,2,2,3,1]</code> (so that would be a batch of 8 sentences, each sentence given one label). If I want to do multi-class, must I pass <code>y</code> so that it looks like <code>[0,0,1,0,0]</code>, which correspomds to label 2? I’m just not sure at all what way my labels must be passed for this.</p>
<p>Thanks in advance.</p>

ptrblck: <p>The first approach of passing the class indices to the criterion, which should be <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>, is correct. Both loss functions expect a target containing the class indices and are not expecting a one-hot encoded target (which would be the second approach).</p>
Why the output turns into “nan“ Afer a 3d convolution?

Pengzhangzhi: <p>Hi!<br>
Afer a 3d convolution ,all of the reslut of the output are nan.<br>
Here is the code in forward  function</p>
<pre><code class="lang-auto">            print("input",torch.isnan(input).any())
            
            conv = self.con3d1(input)

            print("conv1",torch.isnan(conv).any())
</code></pre>
<p>And the self.conv3d1 is</p>
<pre><code class="lang-auto">self.con3d1= nn.Conv3d(in_channels=2, out_channels=64, kernel_size=(6, 3, 3), padding=(2, 1, 1))
      
</code></pre>
<p>For your information, I pad the input tensor by hand (because torch don’t support “same padding”)</p>
<pre><code class="lang-auto">pad1 = torch.zeros((input.shape[0], 2, 1, 16, 8)).to(DEVICE)
input = torch.cat((input, pad1), dim=2)
</code></pre>
<p>Noted that the input data is absolutely clean and normalized.</p>
<p>Turns out it’s because the gradient is toooo large,so i implement  gradient clipping,then the problem sloved.</p>
<p>But I am wondering that why gradient explode would happend in pytorch?</p>
<p>I was trying to convert a keras code into a pytorch code, and the same 3d convolution layer in keras was ran perfectly.</p>
<pre><code class="lang-auto">conv = Conv3D(filters=64, kernel_size=(6, 3, 3), strides=(1, 1, 1), border_mode="same",
                      kernel_initializer='random_uniform')(input)
</code></pre>
<p>Any suggestion is helpful !<br>
If you want to know more  detailed information please let me konw!</p>

Pengzhangzhi: <p>I have finally solved this bug.The reason is nothing about the 3d convolution!</p>

ptrblck: <p>I’m curious to know what the reason for the NaN outputs was. Could you share the debugging steps and solution?</p>

Pengzhangzhi: <p>Thanks sir!  The reason is I didn’t initialize parameters of a special layer.</p>
How to use half2 in customer kernels?

hughperkins: <p>It looks like custom kernels use at::Half instead of half? And use of half itself doesnt seem to work? eg casts between half and float fail? I’m guessing these cast failures are something to do with the definition of <strong>CUDA_NO_HALF_OPERATORS</strong> and <strong>CUDA_NO_HALF2_OPERATORS</strong> in the compilation step?</p>
<p>Then, I figured that to use half2, I would use eg at::Half2, but this seems not to exist?</p>
<ul>
<li>why do we use at::Half instead of half?</li>
<li>why are half casts etc removed?</li>
<li>what should we do if we want to use half2? (or, is there some way to get the compiler to handle half2 vectorization for us, just as g++ handles SSE vecotrization automatically for us in cpu code?)</li>
</ul>

ptrblck: <p>Some questions are already answered in the <a href="https://discuss.pytorch.org/t/cuda-no-half2-operators-for-cuda-9-2/18365/4">cross post</a>.</p>
<aside class="quote no-group" data-username="hughperkins" data-post="1" data-topic="99008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hughperkins/48/915_2.png" class="avatar"> hughperkins:</div>
<blockquote>
<p>what should we do if we want to use half2?</p>
</blockquote>
</aside>
<p>You should be able to use <code>__half2</code> in your CUDA code as seen e.g. <a href="https://github.com/NVIDIA/apex/blob/325f5a0bec542701edba1628ad34f3b2ea47c556/csrc/layer_norm_cuda_kernel.cu#L174">here</a>.</p>

hughperkins: <p>sure, but there are no operators. Like eg for sigmoid I have to write:</p>
<pre><code class="lang-auto">__h2div(one, (__hadd2(one, h2exp(__hneg2(x)))))
</code></pre>
<p>instead of simply:</p>
<pre><code class="lang-auto">one / (one + hexp(-x))
</code></pre>
How to element-wise multiply two network yet still use autograd?

wasabi: <p>Hi guys. Let’s say I have two one-layer network w1 and w2, they have the same shape. I want the classification rule to be (w1\odot w2)^T x, where \odot just means element-wise multiplication. My question is, how can I write such code, s.t pytorch can still use autograd to update the weights? Thanks.</p>

KFrank: <p>Hi Wasabi!</p>
<aside class="quote no-group" data-username="wasabi" data-post="1" data-topic="99006" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasabi/48/13151_2.png" class="avatar"> wasabi:</div>
<blockquote>
<p>how can I write such code, s.t pytorch can still use autograd to update the weights?</p>
</blockquote>
</aside>
<p>Just write the code in a straightforward way (using pytorch tensor<br>
operations) and autograd will work as long as <code>w1</code> and <code>w2</code> have<br>
<code>requires_grad = True</code>.</p>
<p>Here’s an example script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

l1 = torch.nn.Linear (3, 1)
l2 = torch.nn.Linear (3, 1)
w = l1.weight * l2.weight
input = torch.randn (3)
loss = torch.matmul (w, input)
loss
loss.backward()
l1.weight.grad
l2.weight.grad
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt;
&gt;&gt;&gt; l1 = torch.nn.Linear (3, 1)
&gt;&gt;&gt; l2 = torch.nn.Linear (3, 1)
&gt;&gt;&gt; w = l1.weight * l2.weight
&gt;&gt;&gt; input = torch.randn (3)
&gt;&gt;&gt; loss = torch.matmul (w, input)
&gt;&gt;&gt; loss
tensor([-0.2095], grad_fn=&lt;MvBackward&gt;)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; l1.weight.grad
tensor([[-0.1094, -0.3962, -0.3002]])
&gt;&gt;&gt; l2.weight.grad
tensor([[-0.1291,  0.2055, -0.4207]])
</code></pre>
<p>(The <code>weight</code> property of a <code>Linear</code> has <code>requires_grad = True</code> by<br>
default, and <code>*</code> performs element-wise multiplication on tensors.)</p>
<p>Best.</p>
<p>K. Frank</p>
Training contour data (positions) instead of image pixel data (pixel values)

Ziri_Ziri: <p>I am trying to train an extracted contour data  to check if a certain shape has defects or not.<br>
Contours are vectors of points  but i am confused how to proceed . I am also curious if this approach is right .</p>
<p>Thank you.</p>

ptrblck: <p>If your contours have a static number of points you could try to fit your model to a regression task where the outputs would be these contour points.<br>
I don’t know what the best approach would be for a variable number of contour points. Maybe a recurrent architecture could work in this case.</p>

Ziri_Ziri: <p>Thank you for your reply.<br>
I will interpolate the contours to have same size and check results.</p>
Cuda tensors in forked subprocess

gshen: <p>Hi all,</p>
<p>My code works fine in PyTorch v1.0.0. However, in PyTorch v1.6.0, the line torch.set_default_tensor_type(torch.cuda.DoubleTensor) causes a Cannot re-initialize CUDA in forked subprocess error. I noticed that this is due to the fork call when using num_workers &gt; 0 in the PyTorch dataloader. My guess is that its because of the subprocesses loading data directly onto the GPU. So one possible fix I found on the forums was to use spawn or forkserver start methods instead, however these methods are quite slow compared to using fork. Is there a way to benefit from the speedup of num_workers &gt; 0 while also keep the default tensor type as Cuda tensors?</p>
<p>Code that can reproduce the error:</p>
<pre><code class="lang-python">import torch
import torchvision
import torchvision.transforms as transforms

torch.set_default_tensor_type(torch.cuda.DoubleTensor) 

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)
print(trainloader.__iter__().__next__())
</code></pre>
<p>Thanks!</p>

albanD: <p>I’m afraid this is not possible to do as setting the cuda type causes cuda to be initialized and the limitation with fork and cuda is due to the cuda driver itself, not pytorch. So there is little we can do.</p>
<p>Note that in general, I would advise against setting the default type on cuda as it might make simple ops slower than they should (in particular when default Tensors are created).</p>
Extract batch size and channels from torch tensor (multi modal network)

Rolpan: <p>I have a multi modal network with a forward function as below:</p>
<pre><code>def forward(self, x):    # x.shape torch.Size([1, 2, 200, 100])
    
    x1= x.data[0][0,:,:] # torch.Size([200, 100])
    x2= x.data[0][1,:,:]  # torch.Size([200, 100])
    out_x1 = self.conv(x1)
    out_x2 = self.conv(x2)
</code></pre>
<p>The input x is of shape torch.Size([1, 2, 200, 100]), i.e [batch=1, ch=2, height=200, width=100 ] and I want x1 to be first channel and x2 to be second channel. If I use:</p>
<p>x1= x.data[0][0,:,:]</p>
<p>then I only get height and width, torch.Size([200, 100]), which will not work in self.conv, which expect  torch.Size([1, 1, 200, 100]). How should I extract x1 and x2 from x to get the 4-dimensions that I need?</p>
<p>Thank you!</p>

pchandrasekaran: <p>You can use the view function to reshape it.</p>
<pre><code class="lang-auto">x1 = (x.data[0][0, :, :]).view(1, 1, 200, 100)
</code></pre>

Rolpan: <p>Thank you. But then I would not be able to change batch size. This is a nested network so it would be good if I could extract the batch size</p>

pchandrasekaran: <p>My mistake. I misunderstood what you were asking. I’ve tried some code below and got these results.</p>
<pre><code class="lang-auto">simulated_batch_size = 64
X = torch.rand((simulated_batch_size, 2, 200, 100), dtype=torch.float64)

x1 = X[:, 0, :, :]
print(x1.shape)  # [64, 200, 100]
x2 = X[:, 1, :, :]
print(x2.shape)  # [64, 200, 100]

#### Reshaped for Conv ####
x1 = x1.view(x1.shape[0], 1, 200, 100)
print(x1.shape) # [64, 1, 200, 100]
x2 = x2.view(x2.shape[0], 1, 200, 100)
print(x2.shape) # [64, 1, 200, 100]
</code></pre>
<p>So I believe in your case, it would be</p>
<pre><code class="lang-auto">x1= x.data[:, 0, :, :]
</code></pre>
<p>It should give you a tensor of shape [1, 200, 100]</p>

Rolpan: <p>Thank you! That works great</p>
Tracing backward pass in v1.1.0

singam-sanjay: <p>Hello All,</p>
<p>My team at Microsoft Research India has built an adaptive compiler framework for deep learning jobs and implemented it on PyTorch (<a href="https://dl.acm.org/citation.cfm?id=3304072&amp;preflayout=tabs" rel="nofollow noopener">ASPLOS '19 paper</a>). We used the tracing API in v0.3 to capture the compute graph representation of the network and its gradient computation, whose execution was then optimized by our framework.</p>
<p>We would like to port the implementation to v1.1.0, which entails capturing the compute graph through the tracing API. Since the backward pass doesn’t return any outputs, it’s not possible to use v1.1.0’s tracing API to build the computational graph for the backward pass. I tried the graph visualization package to build the computational graph for the backward pass, but the nodes were named “SelectBackward” or “ViewBackward”.</p>
<p>Could someone please help me with understanding the tracing API so that I can capture the graph for the backward pass as well ? Or please suggest a different methodology ?</p>
<p>Thank You,<br>
Sanjay</p>

ailzhang: <p>Hi <a class="mention" href="/u/singam-sanjay">@singam-sanjay</a>, could you give an example about how it worked in 0.3 and what you need from 1.1.0?<br>
“SelectBackward” are the autograd nodes and I’m not sure if they’re expected to show up in your backward pass. (Given that you capture the graph representation and do optimization through your compiler framework.)<br>
Happy to help if you can give more context. Thanks!</p>

singam-sanjay: <p>Hi <a class="mention" href="/u/ailzhang">@ailzhang</a>,</p>
<p>Thanks for responding !</p>
<p>We used the tracing API in 0.3 in the following way,</p>
<ol>
<li>Tracing the forward pass,</li>
</ol>
<pre><code class="lang-auto">trace, fwd_outputs = torch.jit.trace(model, fwd_args, nderivs = 1)
</code></pre>
<ol start="2">
<li>Tracing the backward pass,</li>
</ol>
<pre><code class="lang-auto">torch.autograd.backward(fwd_outputs, bwd_args)
</code></pre>
<p>After Step 2, the <code>trace.graph()</code> is updated with nodes from the backward pass.</p>
<p>Could you please suggest an approach to generate equivalent results in the v1.1.0 API ?</p>
<p>Thanks,<br>
Sanjay</p>

cjolivier01: <p>This is very interesting!<br>
I have a somewhat similar question along these lines.  If anyone feels I should address this in a different issue, I can create a new one.<br>
Is there a way to “intercept” the backward pass on-the-fly, as it runs in a normal fashion?  Specifically for pytorch/xla package, but I know ptxla doesn’t have this, so maybe it’s in autograd?  The reason being, i would very much like to be able to tell which forward op each backward op is associated with (i.e. this subgraph is, for the most part, the bwd pass of the dropout layer, for instance) and this be able to keep track of this in some sidebanded fashion to be processed during graph compile?</p>