akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dist_autograd.context only computes local gradients

MLbrain: <p>Hello,</p>
<p>I am new to pytorch distributed computing – using pytorch1.9.0. I have a large model that I have distributed over several processes on a single computer. The model is organized such that the master process has an input nn.Linear layer and an output nn.Linear layer that execute at the very start and the very end of the forward pass respectively.  Between those two layers are many RemoteModule objects that each execute an RNN model with the overall goal of predicting some time series data.  I have successfully obtained rrefs to the parameters associated with each of these RemoteModules as:</p>
<pre><code class="lang-auto">MyModules.append(RemoteModule("worker"+str(proc),MyRNN))
MyParameters = []
for module in MyModules:
      MyParameters.extend(module.remote_parameters())
</code></pre>
<p><code>MyParameters</code> contains 1000 Parameters</p>
<p>Then in the forward pass running on the master process:</p>
<pre><code class="lang-auto">for batch in DataLoader:
     with dist_autograd.context() as context_id: 
          input = InputLinear(batch)       
          for t in range(sequenceLength):
               ids =[]    
               for mod in MyModules:
                    ids.append(mod.forward_async(input,t)
               get_state = [id.wait() for id in ids]
               state[:,t,:] = torch.stack(get_state)
          out = OutputLinear(state)
          loss = nn.MSELoss(out,batch)
          dist_autograd.backward(context_id,[loss])
          grads = dist_autograd.get_gradients(context_id)
          dist_optim = DistributedOptimizer( optim.Adam, MyParameters ,lr=0.001,)
          dist_optim.step(context_id)
</code></pre>
<p>At this point <code>grads</code> contains only 4 items corresponding to the parameters associated with the <code>InputLinear</code> and <code>OutputLinear</code> layers only.  It does not contain gradients for any of the remote module parameters. If I understand properly, the <code>context_id</code> should be able to track computations across these remote modules. Can anyone tell me what I am doing wrong or how I can properly compute the gradients over these remote modules?</p>

wanchaol: <p>Hi <a class="mention" href="/u/mlbrain">@MLbrain</a> thanks for posting the question, as you can see it only have the local_gradients, this is because the gradients will only be visible in the rank where remote module located. When using RPC to do training, you can simply pass the parameter_rrefs into the distributed optimizer, and the distributed optimizer will figure out the remote module parameter gradients and sync it properly. You can take a look at this tutorial <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed RPC Framework — PyTorch Tutorials 1.9.1+cu102 documentation</a></p>
Custom loss function: NN not training between epochs

The_duke: <p>Hello,</p>
<p>I have a neural network. If I use nn.CrossEntropyLoss() as objective function, the network trains as intended.</p>
<p>I am trying to implement this: <a href="https://en.wikipedia.org/wiki/Geometric_mean" class="inline-onebox" rel="noopener nofollow ugc">Geometric mean - Wikipedia</a> on my own:</p>
<p>class Geometric_mean(nn.Module):<br>
def <strong>init</strong>(self, weight=None, size_average=True):<br>
super(Geometric_mean, self).<strong>init</strong>()</p>
<pre><code>def forward(self, output, target, smooth=1): 
    
    #neural network output
    def add(loss , output, target, u_left , no_of_uniques):
        if u_left &lt; 0:
            loss = loss**(1/no_of_uniques)
            print(loss)
            return loss
        else:
            #split NN output into two
            #print(f"this is NN output:{output}")
            a1,a2 = output.split(split_size = 1,dim=1) 
            pred = (a2-a1).floor() + 1 &lt;&lt;&lt;----------

            pred_target = torch.column_stack((pred,target))


            #for each values of output_target that are == to indexed member of unique
            class_rows = pred_target[(pred_target[:, 1] == target.unique()[u_left])]     &lt;&lt;&lt;----------
            pred,class_target = class_rows.split(1,dim=1)

            percentage_class_correct =torch.sum(1-(pred-class_target))/len(class_target) &lt;&lt;&lt;----------

            return add(loss + percentage_class_correct,output,target,u_left-1,no_of_uniques)

    no_of_unique = len(target.unique())
    #-1 is here because array starts with 0, while len() returns a normal count
    u_left = no_of_unique-1
    
    return add(0, output, target, u_left, no_of_unique)
</code></pre>
<p>I assume this is where I made my mistake. As this is the only thing I changed.</p>
<p>It does seem to train within an epoch, but kinds of reset itself and training for 10 epochs is like running the same epoch with the same losses 10 times rather than building on top of each other.</p>
<p>E.g. Epoch 001: | Train Loss: 1.21563 | Val Loss: 0.73585 %| Train Acc: 40.167 %| Val Acc: 49.057<br>
Epoch 002: | Train Loss: 1.21563 | Val Loss: 0.73585 %| Train Acc: 40.167 %| Val Acc: 49.057<br>
Epoch 003: | Train Loss: 1.21563 | Val Loss: 0.73585 %| Train Acc: 40.167 %| Val Acc: 49.057</p>
<p>…</p>
<p>etc</p>
<p>It does compute the loss and everything within an epoch, so at least that part is OK.<br>
I have another loss function I tried implementing that have the same problem also but this is the simpler of the two.</p>
<p>Earlier on, I had made a mistake of messing with the computation graph, but then python complained very loudly and the code would not run at all. This does not throw any error so I am not sure if this is the case.</p>
<p>Any suggestions as to what I can do to fix the code?</p>

The_duke: <p>Embarrassing but my mistake was due to using the floor function —</p>
<p>it has derivative zero almost everywhere, and thus whatever composite function it is part of will also have derivative zero by the chain rule</p>
Why the linear layer after Convolution layers for a classifier?

pumplerod: <p>I’ve been working on an image classification network, and all the examples I find show, as a final step, flattening the final convolution layer into at least one fully connected linear layer.  I’m hoping someone can elaborate on why this is useful, and if there are other methods.</p>
<p>For example, I’m currently attempting to classify images into 5 different classes.  So I am creating a 5 channel image with the same dimensions as my input, where-by the channel for the target class is a white square and all other channels are black squares.</p>
<p>My thought process here is that by comparing the output of my model’s predictions, overlaid with the target image, I could see certain feature details that the model thinks should be associated with a particular class.</p>
<p>once I have that final 5 channel image, it’s trivial enough to get the mean of each channel and using argmax can determine a final number to present.  But I’m wondering if there is any advantage during training if I only calculate the error against the 5 channel target images.</p>
<p>Hopefully this makes some sense.  But if anyone could point me to an explanation for why it is so common to flatten the final convolution layers I would really appreciate it.  Flattening all those images into a long line of numbers just doesn’t make logical sense to me and I hope someone can straighten men out.</p>

pumplerod: <p>To update for anyone also wondering about this.  I found some additional information regarding 1x1 kernels and how they can effectively serve as a flattening layer…</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://iamaaditya.github.io/2016/03/one-by-one-convolution/">
  <header class="source">

      <a href="https://iamaaditya.github.io/2016/03/one-by-one-convolution/" target="_blank" rel="noopener nofollow ugc">iamaaditya.github.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://iamaaditya.github.io/2016/03/one-by-one-convolution/" target="_blank" rel="noopener nofollow ugc">One by One [ 1 x 1 ] Convolution - counter-intuitively useful</a></h3>

  <p>Whenever I discuss or show GoogleNet architecture, one question always comes up -</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Which of the CPU implementations is used?

ValeriyFedyunin: <p>Good day!</p>
<p>I’m experimenting with simple networks and trying to figure out, which of the implementations was acutally used during training/inference.</p>
<p>I’m using simple net (embeddings-&gt;LSTM-&gt;Linear-&gt;Sigmoid), np.float32 data, and <code>pytorch 1.9.1+cu111</code> (which was obtained through pip).</p>
<p>The source code browsing showed that a lot of functions has 3 implementations: xnnpack, MKL and MKLDNN (ideep). And <code>torch.__config__.show()</code> mentions that both MKL and MKLDNN are used.</p>
<p>The question is: what is the easiest way to find out, which of the CPU implementations (e.g. of Linear operation) actually took place during training or during inference? Is there any explicit priorities between these implementations? Or some debug mode which will allow me to see the stack?</p>
Global average pooling excluding zeros

Honnang: <p>I am trying to find the average of a tensor (b, c, h, w), which would transform it to (b, c, 1, 1). It is the same as average pooling and can be performed with adaptivepool.<br>
However, I am trying to exclude all zeros in the average calculation. Is there any way to do this?</p>

JuanFMontesinos: <pre><code class="lang-auto">tensor=tensor.view(b,c,-1)
mask = tensor&gt;0
nelements = mask.sum(dim=-1)
pooling = tensor.sum(dim=-1)/nelements
</code></pre>

Honnang: <p>wow thank you very much</p>