Pytorch (>= 1.3) showing an error, perhaps for loss computed from paired outputs

Sudipto: <p>I am trying to use the code for IIC (Invariant Information Clustering) : <a href="https://github.com/xu-ji/IIC" rel="nofollow noopener">https://github.com/xu-ji/IIC</a></p>
<p>It basically calls the network twice on original input and modified input.</p>
<p>x_out = net(sample)    # Softmax output for original sample<br>
x_tf_out = net(noisy_sample)   # Softmax output for noisy sample</p>
<p>loss = IIC_loss(x_out, x_tf_out)</p>
<p>And then the loss wants the original sample and the noisy sample to be labeled to the same class. It is computed by the IIC loss :</p>
<p>def IIC_loss(x_out, x_tf_out, EPS=sys.float_info.epsilon):<br>
# has had softmax applied<br>
_, k = x_out.size()<br>
p_i_j = compute_joint(x_out, x_tf_out)<br>
assert (p_i_j.size() == (k, k))</p>
<p>p_i = p_i_j.sum(dim=1).view(k, 1).expand(k, k)<br>
p_j = p_i_j.sum(dim=0).view(1, k).expand(k, k)  # but should be same, symmetric</p>
<p># avoid NaN losses. Effect will get cancelled out by p_i_j tiny anyway<br>
p_i_j[(p_i_j &lt; EPS).data] = EPS<br>
p_j[(p_j &lt; EPS).data] = EPS<br>
p_i[(p_i &lt; EPS).data] = EPS</p>
<p>loss = - p_i_j * (torch.log(p_i_j) <br>
- torch.log(p_j) <br>
- torch.log(p_i))</p>
<p>loss = loss.sum()</p>
<p>return loss</p>
<p>def compute_joint(x_out, x_tf_out):<br>
# produces variable that requires grad (since args require grad)</p>
<p>bn, k = x_out.size()<br>
assert (x_tf_out.size(0) == bn and x_tf_out.size(1) == k)</p>
<p>p_i_j = x_out.unsqueeze(2) * x_tf_out.unsqueeze(1)  # bn, k, k<br>
p_i_j = p_i_j.sum(dim=0)  # k, k<br>
p_i_j = (p_i_j + p_i_j.t()) / 2.  # symmetrise<br>
p_i_j = p_i_j / p_i_j.sum()  # normalise</p>
<p>return p_i_j</p>
<p>When I now call backward() on the loss, it leads to the following error :</p>
<p>loss.backward()<br>
optimizer.step()</p>
<p><strong>in backward</strong><br>
<strong>#allow_unreachable=True)  # allow_unreachable flag</strong><br>
<strong>RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.</strong></p>
<p>What changes in the code are needed in new version of Pytorch ? The code ran perfectly even for PyTorch 1.0.</p>

albanD: <p>Hi,</p>
<p>Yes we added more checks for operations that were potentially invalid.<br>
Could you run your code with <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection">anomaly mode</a> by setting <code>autograd.set_detect_anomaly(True)</code> at the beginning of your script. That will let us know what is causing the error.</p>

Sudipto: <p>Hi albanD,</p>
<p>I figured out the issue yesterday. The inplace operations inside the function IIC_loss created the problem.</p>
<p>Instead of the original lines,<br>
p_i_j[(p_i_j &lt; EPS).data] = EPS<br>
p_j[(p_j &lt; EPS).data] = EPS<br>
p_i[(p_i &lt; EPS).data] = EPS</p>
<p>The correct version for new PyTorch (&gt;= 1.3) should be as follows :<br>
p_i_j = torch.where(p_i_j &lt; EPS, torch.tensor([EPS], device = p_i_j.device), p_i_j)<br>
p_j = torch.where(p_j &lt; EPS, torch.tensor([EPS], device = p_j.device), p_j)<br>
p_i = torch.where(p_i &lt; EPS, torch.tensor([EPS], device = p_i.device), p_i)</p>