akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pipeline Parallelism (Pipe) Assertion Fail

Rifle_Zhang: <p>I am running the torch.distributed.pipeline.sync.Pipe library using pytorch 3.8.1 (also tried nightly). I have 2 visible devices. Below is the example from doc.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributed.pipeline.sync import Pipe
from torchgpipe import GPipe

# Run with Pipe
fc1 = nn.Linear(16, 8).cuda(0)
fc2 = nn.Linear(8, 4).cuda(1)
model = nn.Sequential(fc1, fc2)
model = Pipe(model, chunks=8)
input = torch.rand(16, 16).cuda(0)
output_rref = model(input)

# Run with GPipe
fc1 = nn.Linear(16, 8)
fc2 = nn.Linear(8, 4)
model = nn.Sequential(fc1, fc2)
model = GPipe(model, balance=[1,1], chunks=8)
model = nn.DataParallel(model)
input = torch.rand(16, 16).cuda(0)
output_rref = model(input)
print(output_rref)
</code></pre>
<p>I am getting this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 12, in &lt;module&gt;
    output_rref = model(input)
  File "/usr0/home/ruohongz/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr0/home/ruohongz/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/distributed/pipeline/sync/pipe.py", line 366, in forward
    return RRef(output)
RuntimeError: agent INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp":247, please report a bug to PyTorch. Current RPC agent is not set!
</code></pre>
<p>However, the GPipe code works fine. What is the problem with the pytorch assertion?</p>

pritamdamania87: <p>You need to initialize the RPC framework, see the latest master docs: <a href="https://pytorch.org/docs/master/pipeline.html#torch.distributed.pipeline.sync.Pipe" class="inline-onebox" rel="noopener nofollow ugc">Pipeline Parallelism — PyTorch master documentation</a></p>
How to compute inner product between gradient of neural networks?

Mao_Tam: <p>For linear model (e.g logistic regression) the gradient of the loss function w.r.t model parameter over training data A is g1 which is a vector. Similarly the gradient of the loss function w.r.t model parameter over another data B is g2. We can compute the similarity between g1 and g2 by its inner product easily since they are both vectors.</p>
<p>How to extend inner product between two gradients for neural networks, since the gradient here is no longer a vector. Should we stack all model’s parameter in gradients together in a vector?</p>
TypeError: 'NoneType' object is not callable

S_M: <p>Hi all when I use cross-entropy loss <code>loss=self.cross_entropy_loss(embedded_x, target1)</code> I see the error <code>object is not callable</code>. Could anyone help me?</p>

JuanFMontesinos: <p>Well you have defined <code>self.cross_entropy_loss</code> in a wrong way. It equals None.</p>