Cant apply nn.Softmax() along each dimension or different parts of tensor

username: Mr_OG
cooked: <p>Hi,</p>
<p>I cant apply nn.Softmax() along each dimension separately.</p>
<p>I have a tensor in one dimension of size 4. I want to apply softmax on the first 2 values and the last 2 values separately.<br>
in each way I tried to do it I get:<br>
“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [6, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead.”</p>
<h2>adding my net architecture:</h2>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(4, 6)<br>
self.fc2 = nn.Linear(6, 8)<br>
self.fc3 = nn.Linear(8, 16)<br>
self.fc4 = nn.Linear(16, 32)<br>
self.fc5 = nn.Linear(32, 32)<br>
self.fc6 = nn.Linear(32, 16)<br>
self.fc7 = nn.Linear(16, 8)<br>
self.fc8 = nn.Linear(8, 6)<br>
self.fc9 = nn.Linear(6, 4)</p>
<pre><code>def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.relu(self.fc4(x))
    x = F.relu(self.fc5(x))
    x = F.relu(self.fc6(x))
    x = F.relu(self.fc7(x))
    x = F.relu(self.fc8(x))
    x = F.relu(self.fc9(x))

    # x = x.view(2, 2)
    # x = x.reshape(2, 2)
    # x = F.softmax(x, dim=1)
    # x = x.view(4)

    # x1 = F.softmax(x[0, 0:2], dim=0)
    # x2 = F.softmax(x[0, 2:4], dim=0)
    x1 = x[0:2]
    x2 = x[2:4]
    x1 = F.softmax(x1, dim=0)
    x2 = F.softmax(x2, dim=0)
    x = torch.cat((x1, x2), 0)

    # x = F.softmax(x, dim=1)
    return x
</code></pre>
<hr>
<p>can anyone please help me?</p>

username: KFrank
cooked: <p>Hello OG!</p>
<p>I think your root issue is that pytorch models operate on tensors that<br>
carry a leading <code>nBatch</code> dimension that is not directly referenced in<br>
the model architecture.</p>
<aside class="quote no-group quote-modified" data-username="Mr_OG" data-post="1" data-topic="97856" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/a8b319/48.png" class="avatar"> Mr_OG:</div>
<blockquote>
<p>I cant apply nn.Softmax() along each dimension separately.</p>
<p>…</p>
<h2>adding my net architecture:</h2>
<pre><code>    self.fc1 = nn.Linear(4, 6)
</code></pre>
</blockquote>
</aside>
<p>Such a model will operate on a tensor of shape <code>[nBatch, 4]</code>.<br>
(If you want to pass in a single sample of shape <code>[4]</code>, you have to<br>
<code>unsqueeze()</code> it to get a tensor with a leading singleton dimension,<br>
that is, of shape <code>[1, 4]</code>.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code>    self.fc9 = nn.Linear(6, 4)
</code></pre>
<p>…</p>
<pre><code>    x = F.relu(self.fc9(x))
</code></pre>
</blockquote>
</aside>
<p>The output of this model will have shape <code>[nBatch, 4]</code> (rather than<br>
shape <code>[4]</code>).</p>
<aside class="quote no-group">
<blockquote>
<pre><code>    # x = x.view(2, 2)
</code></pre>
</blockquote>
</aside>
<p>You probably commented out this line because it didn’t work.  It doesn’t<br>
work because your <code>view()</code> call doesn’t take into account <code>x</code>’s <code>nBatch</code><br>
dimension.</p>
<p>Regardless, your basic idea does work.  The sample script, below,<br>
shows, step by step, how you can use <code>view()</code> to group your four<br>
values into two sets of two, apply <code>softmax()</code> to the two sets of two<br>
separately, and then use <code>view()</code> again to reorganize your two sets<br>
of two back into one set of four.  But we have to account for the<br>
leading <code>nBatch</code> dimension.</p>
<p>Here is a pytorch 0.3.0 version of the script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

x = torch.autograd.Variable (torch.randn ((3, 4)))    # assume nBatch = 3
x
v = x.view (-1, 2, 2)   # use -1 to "infer" size of nBatch dimension
v
sm = torch.nn.functional.softmax (v, dim = -1)        # -1 to perform softmax over last dimension = 2
sm
y = sm.view (-1, 4)     # use -1 to "infer" size of nBatch dimension
y
sm.sum (dim = -1)       # check softmax sum (-1 to sum over last dimension = 2)
y.sum (dim = -1)        # check softmax sum (-1 to sum over last dimension = 1)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.autograd.Variable (torch.randn ((3, 4)))    # assume nBatch = 3
&gt;&gt;&gt; x
Variable containing:
-2.2293  2.3681  0.5411 -1.3360
 0.7701 -1.1548  0.2012 -0.6301
-0.0882 -0.1200 -0.4608  0.9952
[torch.FloatTensor of size 3x4]

&gt;&gt;&gt; v = x.view (-1, 2, 2)   # use -1 to "infer" size of nBatch dimension
&gt;&gt;&gt; v
Variable containing:
(0 ,.,.) =
 -2.2293  2.3681
  0.5411 -1.3360

(1 ,.,.) =
  0.7701 -1.1548
  0.2012 -0.6301

(2 ,.,.) =
 -0.0882 -0.1200
 -0.4608  0.9952
[torch.FloatTensor of size 3x2x2]

&gt;&gt;&gt; sm = torch.nn.functional.softmax (v, dim = -1)        # -1 to perform softmax over last dimension = 2
&gt;&gt;&gt; sm
Variable containing:
(0 ,.,.) =
  0.0100  0.9900
  0.8673  0.1327

(1 ,.,.) =
  0.8727  0.1273
  0.6966  0.3034

(2 ,.,.) =
  0.5079  0.4921
  0.1891  0.8109
[torch.FloatTensor of size 3x2x2]

&gt;&gt;&gt; y = sm.view (-1, 4)     # use -1 to "infer" size of nBatch dimension
&gt;&gt;&gt; y
Variable containing:
 0.0100  0.9900  0.8673  0.1327
 0.8727  0.1273  0.6966  0.3034
 0.5079  0.4921  0.1891  0.8109
[torch.FloatTensor of size 3x4]

&gt;&gt;&gt; sm.sum (dim = -1)       # check softmax sum (-1 to sum over last dimension = 2)
Variable containing:
 1  1
 1  1
 1  1
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; y.sum (dim = -1)        # check softmax sum (-1 to sum over last dimension = 1)
Variable containing:
 2
 2
 2
[torch.FloatTensor of size 3]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Mr_OG
cooked: <p>thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>i comment the  # x = x.view(2, 2) part because  it gave the same error. although you probably right, it wasn’t the source of my problem. my error is with the gradient computation which probably has been modified by an <strong>inplace</strong> operation that I cant find. maybe it is in my loss function?</p>
<h2>net architecture and train function:</h2>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from POC_loss import loss1, component3, numerical_mutual_Iinformation

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(4, 6)
        self.fc2 = nn.Linear(6, 8)
        self.fc3 = nn.Linear(8, 16)
        self.fc4 = nn.Linear(16, 32)
        self.fc5 = nn.Linear(32, 32)
        self.fc6 = nn.Linear(32, 16)
        self.fc7 = nn.Linear(16, 8)
        self.fc8 = nn.Linear(8, 6)
        self.fc9 = nn.Linear(6, 4)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        x = F.relu(self.fc6(x))
        x = F.relu(self.fc7(x))
        x = F.relu(self.fc8(x))
        x = F.relu(self.fc9(x))

        x = x.view(-1, 2, 2)
        x = torch.nn.functional.softmax (x, dim = -1)
        x = x.view(-1, 4)

        # x = x.view(2, 2)
        # x = x.reshape(2, 2)
        # x = F.softmax(x, dim=1)
        # x = x.view(4)

        # x1 = F.softmax(x[0, 0:2], dim=0)
        # x2 = F.softmax(x[0, 2:4], dim=0)

        # x1 = x[0:2]
        # x2 = x[2:4]
        # x1 = F.softmax(x1, dim=0)
        # x2 = F.softmax(x2, dim=0)
        # x = torch.cat((x1, x2), 0)

        # x = F.softmax(x, dim=1)
        return x

def train(train_loader, optimizer, net, my_loss, epochs, D, Px=0.5, alpha=1, toleranceParcent=10, toleranceLen: int=10):

    # initialise variables:
    tolerance = toleranceParcent / 100

    running_loss = 0.0
    running_Ed = 0.0
    running_RofD = 0.0
    running_outputs = 0.0

    lossList = []
    EdList = []
    RofDList = []
    outputsKeepwrList = []

    for epoch in range(epochs):

        if next(net.parameters()).is_cuda: outputs = torch.tensor([0.5, 0.5, 0.5, 0.5]).cuda()
        else: outputs = torch.tensor([0.5, 0.5, 0.5, 0.5])

        loss = 0
        Ed = 0
        RofD = 0
        outputsKeepwr = 0
        for batch_ndx, batch_features in enumerate(train_loader):

            # get the inputs; data is a list of [inputs, labels]
            # inputs_x1 = data

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize:
            # -----------------------------
            #   forward:
            outputs = net(outputs)
            #   compute training reconstruction loss
            train_loss = loss1(outputs, Px, alpha, D)
            #   compute accumulated gradients
            # train_loss.backward(retain_graph=True)
            train_loss.backward(retain_graph=True)
            #    perform parameter update based on current gradients
            optimizer.step()
            # add the mini-batch training loss to epoch loss
            loss += train_loss.item()
            add the mini-batch training loss to epoch loss:
            Ed += component3(outputs, 0, alpha=1, D=0) + component3(outputs, 1, alpha=1, D=0)
            RofD += numerical_mutual_Iinformation(outputs, Px)
            outputsKeepwr += outputs

        # average for current epoch:
        loss = loss / len(train_loader)
        Ed = Ed / len(train_loader)
        RofD = RofD / len(train_loader)
        outputsKeepwr = outputsKeepwr / len(train_loader)
        # insert to list:
        lossList.append(loss)
        EdList.append(loss)
        RofDList.append(loss)
        outputsKeepwrList.append(loss)

        # display progress (epoch training loss):
        print("epoch : {}/{}, loss = {:.6f}, E[d] = {:.6f}, R(D) = {:.6f}, out = {:.6f}".format(epoch + 1, epochs, loss, Ed, RofD, outputsKeepwr))

        # training efficiency control:
        if len(lossList) &gt;= toleranceLen:
            stopRunFlag = True
            for i in range(1, toleranceLen + 1):
                if not ((lossList[-toleranceLen] &lt;= lossList[-i] + tolerance * lossList[-toleranceLen]) and
                        ((lossList[-i] - tolerance * lossList[-toleranceLen] &lt;= lossList[-toleranceLen]))):
                    stopRunFlag = False
                    break
        if ("stopRunFlag" in locals()) and stopRunFlag:
            break

    dataDict = {"lossList": lossList, "EdList": EdList, "RofDList": RofDList, "outputsKeepwrList": outputsKeepwrList}
    return net, dataDict
</code></pre>
<h2>my loss function(loss1):</h2>
<pre><code class="lang-auto">import torch


def numerical_mutual_Iinformation(output, Px):
    comp1 = component1(output, 0)
    comp2 = component2(output, Px, 0)
    comp3 = component1(output, 1)
    comp4 = component2(output, Px, 1)
    NMI =  comp1 - comp2 + comp3 - comp4
    return NMI

def component1(P_yGIVENx, input_x1):
    if (input_x1 == 0):
        # P_yGIVENxt = P_yGIVENx[0, 0:2]
        P_yGIVENxt = P_yGIVENx[0:2]
    else:
        # P_yGIVENxt = P_yGIVENx[0, 2:4]
        P_yGIVENxt = P_yGIVENx[2:4]
    return torch.sum(torch.log2(P_yGIVENxt) * P_yGIVENxt)

def component2(P_yGIVENx, Px, input_x1):
    # return torch.sum(torch.log2(P_yGIVENx)*Px)
    # if P_yGIVENx.is_cuda: maskXis0 = torch.tensor([True, True, False, False]).cuda()
    # else: maskXis0 = torch.tensor([True, True, False, False])
    # P_yGIVENxIs0 = torch.masked_select(P_yGIVENx * Px, maskXis0)
    # if P_yGIVENx.is_cuda: maskXis1 = torch.tensor([False, False, True, True]).cuda()
    # else: maskXis1 = torch.tensor([False, False, True, True])
    # P_yGIVENxIs1 = torch.masked_select(P_yGIVENx * Px, maskXis1)

    multVal = P_yGIVENx * Px
    if P_yGIVENx.is_cuda: P_yGIVENxIs0 = multVal[0:2].cuda()
    else: P_yGIVENxIs0 = multVal[0:2]
    if P_yGIVENx.is_cuda: P_yGIVENxIs1 = multVal[2:4].cuda()
    else: P_yGIVENxIs1 = multVal[2:4]


    logSumOverX = torch.log2(P_yGIVENxIs0 + P_yGIVENxIs1)

    if (input_x1 == 0):
        # P_yGIVENxt = P_yGIVENx[0, 0:2]
        P_yGIVENxt = P_yGIVENx[0:2]
    else:
        # P_yGIVENxt = P_yGIVENx[0, 2:4]
        P_yGIVENxt = P_yGIVENx[2:4]
    return torch.sum(logSumOverX * P_yGIVENxt)

def d(x1):
    "returns vector of size 2 represent y=0 and y=1 given x1"
    if (x1 == 0):
        return [0, 1]
    else:
        return [1, 0]

def component3(P_yGIVENx, input_x1, alpha, D):
    if (input_x1 == 0):
        # P_yGIVENxt = P_yGIVENx[0, 0:2]
        P_yGIVENxt = P_yGIVENx[-1, 0:2]
    else:
        # P_yGIVENxt = P_yGIVENx[0, 2:4]
        P_yGIVENxt = P_yGIVENx[-1, 2:4]
    if P_yGIVENxt.is_cuda: out = alpha * (torch.sum(P_yGIVENxt * torch.FloatTensor(d(input_x1)).cuda()) - D)
    else: out = alpha * (torch.sum(P_yGIVENxt * torch.FloatTensor(d(input_x1))) - D)

    if P_yGIVENxt.is_cuda: returnVal = max(torch.tensor(0).cuda(), out)
    else: returnVal = max(torch.tensor(0), out)

    return returnVal
    # return out
    # return max(0, out.data.numpy())

def loss1(output, Px, alpha, D):
    comp1 = numerical_mutual_Iinformation(output, Px)
    comp2 = component3(output, 0, alpha, D)
    comp3 = component3(output, 1, alpha, D)
    loss = comp1 + comp2 + comp3
    return loss
</code></pre>
<p>Thanks</p>

username: KFrank
cooked: <p>Hello OG!</p>
<aside class="quote no-group" data-username="Mr_OG" data-post="3" data-topic="97856" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/a8b319/48.png" class="avatar"> Mr_OG:</div>
<blockquote>
<p>my error is with the gradient computation which probably has been modified by an <strong>inplace</strong> operation that I cant find. maybe it is in my loss function?</p>
</blockquote>
</aside>
<p>Please see if you can reproduce your error in a simple script of ten<br>
or twenty lines.  The idea would be that you generate some random<br>
example data, call your loss function, and call backward, at which<br>
point the error occurs.</p>
<p>Make sure that the script is complete, self-contained, and runnable.<br>
Leave out any unnecessary complications such as training loops, data<br>
loaders, models, etc.  Please don’t import any non-pytorch packages.<br>
For example, if your loss function comes from a third-party package,<br>
please just include a stripped-down version of the loss-function code<br>
in your self-contained script.</p>
<p>Then post the runnable script and also post the complete output you<br>
get when you run it, including the error.</p>
<p>Best.</p>
<p>K. Frank</p>