Train routine with arbitrary criterions

username: PhysicsIsFun
cooked: <p>Greetings,</p>
<p>I wrote a train routine that takes in an arbitrary model, data loader, and loss criterion, and which contains the following code:</p>
<pre><code class="lang-auto">        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            model.zero_grad()
            output = model(data)
            loss = criterion(output, target)*datasize
            loss.backward()
            optimizer.step()
</code></pre>
<p>Here, <code>output</code> is a 2D-tensor and <code>target</code> is a 1D tensor. The problem is that when using NLL loss as a criterion (say for MNIST classification), everything works fine as is, but if I use BCE loss (for some binary classification), Torch complains that the criterion requires both tensors to be of the same shape.</p>
<p>Since, for binary classification, <code>output</code> will be a 2D-tensor with size (B, 1) where B is the batch size, calling a simple <code>squeeze()</code> would be enough. But I want my code to work in both cases, i.e. BCE loss and NLL loss. I already tried to use <code>squeeze()</code> on the return value of the<code> forward( )</code> pass inside my model, but this did not work either. What is the usual way to do this?</p>
<p>Cheers!</p>

username: srishti-git1110
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="164092">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>Torch complains that the criterion requires both tensors to be of the same shape.</p>
</blockquote>
</aside>
<p>Yes, for BCE loss the <code>model(input)</code> and <code>target</code> tensor need to be of the same shape. <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" rel="noopener nofollow ugc">See</a></p>
<p>Please note that BCE loss is appropriate for a multi-class multi-label setting while NLLLoss is appropriate for a multi-class single-label setting.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="164092">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>But I want my code to work in both cases, i.e. BCE loss and NLL loss.</p>
</blockquote>
</aside>
<p>So, according to me only one of these shall be the appropriate one for the problem at hand. Is there anything that I am missing?<br>
Could you elaborate on your task and why you’d like to use both?</p>

username: PhysicsIsFun
cooked: <p>Well, I would like to write a routine that works on supervised classification tasks. I don’t understand what you mean by “multi-label”. If I have N classes, I have N labels, one for each class.</p>
<p>For binary classification (2 classes and labels 0 or 1), I need to use BCE. For something like MNIST (N classes, N labels with N&gt;2), I need to use NLL.</p>

username: PhysicsIsFun
cooked: <p>Is there really no way of handling this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> How does one write general training routines then that work with different criterions?</p>

username: ptrblck
cooked: <p>You could always add conditions and transform the outputs and targets to the desired <code>dtype</code> if needed. Usually, you don’t care about these kind of abstractions, since your use case defines the actual criterion and allowing other loss functions to work with your data often doesn’t make sense.</p>

username: srishti-git1110
cooked: <aside class="quote no-group" data-username="PhysicsIsFun" data-post="3" data-topic="164092">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>I don’t understand what you mean by “multi-label”.</p>
</blockquote>
</aside>
<p>A multi-label multi-class problem is the one where where there are more than two (multiple) classes and a data point can belong to more than one classes at a time. <code>BCEWithLogitsLoss</code> (with no <code>sigmoid()</code> or <code>softmax()</code> ) is the right loss function for such tasks.</p>
<p>A single-label multi-class problem is the one where there are more than two (multiple) classes and a data point can belong to only one class. <code>CrossEntropyLoss</code> is the right loss function for such tasks.</p>