Grad can be implicitly created only for scalar outputs

username: Lighting_Up
cooked: <p>I am building a MLP with 2 outputs as mean and variance because, I am working  on quantifying uncertainty of the model. I have used a proper scoring for NLL for regression as metrics. My training function passed with MSE loss function but when I am applying  my proper scoring function , I am getting the following error : <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c95c47e7349764ab42e19934bfa5f18767e89c24.png" data-download-href="https://discuss.pytorch.org/uploads/default/c95c47e7349764ab42e19934bfa5f18767e89c24" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c95c47e7349764ab42e19934bfa5f18767e89c24_2_690x160.png" alt="image" data-base62-sha1="sJjGjcPEoOWs3sHHpVCj1FXCYkY" width="690" height="160" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c95c47e7349764ab42e19934bfa5f18767e89c24_2_690x160.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c95c47e7349764ab42e19934bfa5f18767e89c24_2_1035x240.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c95c47e7349764ab42e19934bfa5f18767e89c24_2_1380x320.png 2x" data-dominant-color="EEF4F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1508×351 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is a piece of my function:<br>
def train1(model, epoch, trainloader, criterion, sp, optimizer):<br>
model.train()<br>
for batch_idx, (data, target) in enumerate(trainloader):<br>
if torch.cuda.is_available():<br>
data, target = data.cuda(), target.cuda()<br>
optimizer.zero_grad()<br>
output = model(data)</p>
<pre><code>    print(output)
    
    mu, sig = output[0][0], sp(output[0][1])+(10)**-6
    loss = nll_criterion(mu, sig, target)
</code></pre>
<h1><a name="loss-criterionoutputtarget-1" class="anchor" href="#loss-criterionoutputtarget-1"></a>loss = criterion(output,target)</h1>
<pre><code>    loss.backward()
    optimizer.step()
</code></pre>
<p>One more thing. My code passed when I do not use dataloader or any pytorch method, like tensor.<br>
But when I am using Dataloader and convert my variables to tensors. I am having troubles - I would like to use pytorch for this project.<br>
def main():<br>
n<br>
n_hidden = 100<br>
batch_size = 20</p>
<pre><code>print("Setting up data")

dict_ = split_data(df,data_ratio=0.30)
X_train, Y_train, = dict_["train_x"].values.astype(np.float32), dict_["train_y"].values.astype(np.float32)
X_test, Y_test, = dict_["test_x"].values.astype(np.float32), dict_["test_y"].values.astype(np.float32)    

#dataloader
trainset = mydata(X_train, Y_train)
trainloader = DataLoader(trainset,shuffle=False,batch_size=20)
testset = mydata(X_test, Y_test)
testloader = DataLoader(testset,shuffle=False,batch_size=20)
</code></pre>
<h1><a name="trainset-torchutilsdatatensordatasettorchfrom_numpyx_traintorchfrom_numpyy_train-2" class="anchor" href="#trainset-torchutilsdatatensordatasettorchfrom_numpyx_traintorchfrom_numpyy_train-2"></a>trainset = torch.utils.data.TensorDataset(torch.from_numpy(X_train),torch.from_numpy(Y_train))</h1>
<h1><a name="trainloader-torchutilsdatadataloadertrainset-batch_size20-shuffletrue-num_workers2-3" class="anchor" href="#trainloader-torchutilsdatadataloadertrainset-batch_size20-shuffletrue-num_workers2-3"></a>trainloader = torch.utils.data.DataLoader(trainset, batch_size=20, shuffle=True, num_workers=2)</h1>
<h1><a name="testset-torchutilsdatatensordatasettorchfrom_numpyx_testtorchfrom_numpyy_test-4" class="anchor" href="#testset-torchutilsdatatensordatasettorchfrom_numpyx_testtorchfrom_numpyy_test-4"></a>testset = torch.utils.data.TensorDataset(torch.from_numpy(X_test),torch.from_numpy(Y_test))</h1>
<h1><a name="testloader-torchutilsdatadataloadertestset-batch_size1-shufflefalse-num_workers2-5" class="anchor" href="#testloader-torchutilsdatadataloadertestset-batch_size1-shufflefalse-num_workers2-5"></a>testloader = torch.utils.data.DataLoader(testset, batch_size=1, shuffle=False, num_workers=2)</h1>
<pre><code>model = make_model(X_train, n_hidden)
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.MSELoss()

# Proper scoring rule using negative log likelihood scoring rule
nll_criterion = lambda mu, sigma, y: torch.log(sigma)/2 + ((y-mu)**2)/(2*sigma)
sp = torch.nn.Softplus()

n_epochs= 1000
predict_every = 15

running_loss = []
for epoch in range(n_epochs):
    epoch_loss = 0
    print("(Start) Epoch ",epoch," of ",n_epochs,":")

    epoch_loss = train1(model, epoch, trainloader, criterion=nll_criterion,sp=sp, optimizer=optimiz
</code></pre>

username: ptrblck
cooked: <p>The error is raised, if you try to call <code>.backward()</code> on a non-scalar tensor (i.e. a tensor with more than a single element). If that’s the desired use case, you would have to provide the gradients manually e.g. via <code>.backward(gradient=torch.ones_like(loss))</code> or reduce the loss before (e.g. via <code>loss.mean().backward()</code>).<br>
<a class="mention" href="/u/alband">@albanD</a> explains it in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-pass-the-gradient-parameter-to-the-backward-function-in-pytorch/116079/4">this post</a> with more detail.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>