[Code review] Combining BCE and MSE loss

fauxneticien: <p>Hi all — I’m new to deep learning and PyTorch and am playing with some small examples to learn the ropes. I just wanted to check that I’ve implemented something correctly.</p>
<p>I’m looking to use a CNN to detect whether there’s a quasi-diagonal line in an image of a distance matrix (1 = Yes, 0 = No), and if so where (t_start, t_end), like in the image below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f30e6a68a41f2b0975d5f592fa06a856fb1591c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4f30e6a68a41f2b0975d5f592fa06a856fb1591c" title="Screen Shot 2020-11-02 at 8.12.28 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f30e6a68a41f2b0975d5f592fa06a856fb1591c_2_517x367.jpeg" alt="Screen Shot 2020-11-02 at 8.12.28 PM" data-base62-sha1="biyunFFjjtqN0Qj8jcT3aG3k2Ac" width="517" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f30e6a68a41f2b0975d5f592fa06a856fb1591c_2_517x367.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f30e6a68a41f2b0975d5f592fa06a856fb1591c_2_775x550.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f30e6a68a41f2b0975d5f592fa06a856fb1591c.jpeg 2x" data-dominant-color="C5CF9E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-11-02 at 8.12.28 PM</span><span class="informations">787×559 209 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code I’ve written:</p>
<pre><code class="lang-auto">BCE      = torch.nn.BCELoss()
MSE      = torch.nn.MSELoss()

labels   = torch.tensor([
    # Class, t_start, t_end
    [0, 0, 0],
    [1, 0.25, 0.31],
    [0, 0, 0]
])

# Simulate output = model(distance_matrix)
output = torch.tensor([
    [0.1478, 0.1370, 0.2288],
    [0.7052, 0.3207, 0.5404],
    [0.4511, 0.2978, 0.5279]
])

# Binary cross-entropy error on class labels, mean-squared error on time points
# Use labels to zero-out irrelevant rows
loss = BCE(output[:,:1], labels[:,:1]) + MSE(labels[:,:1] * output[:,-2:], labels[:,-2:])

# etc ...
# loss.backward()
# optimizer.step()
</code></pre>
<p>I was wondering whether using the labels to zero out the rows of the output I wasn’t interested in (where class = 0) was the right way to go about this. Thanks!</p>

KFrank: <p>Hey Nay!</p>
<aside class="quote no-group quote-modified" data-username="fauxneticien" data-post="1" data-topic="101440" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fauxneticien/48/30230_2.png" class="avatar"> fauxneticien:</div>
<blockquote>
<pre><code class="lang-auto">labels   = torch.tensor([
    # Class, t_start, t_end
    [0, 0, 0],
    [1, 0.25, 0.31],
    [0, 0, 0]
])

# Simulate output = model(distance_matrix)
output = torch.tensor([
    [0.1478, 0.1370, 0.2288],
    [0.7052, 0.3207, 0.5404],
    [0.4511, 0.2978, 0.5279]
])

# Binary cross-entropy error on class labels, mean-squared error on time points
# Use labels to zero-out irrelevant rows
loss = BCE(output[:,:1], labels[:,:1]) + MSE(labels[:,:1] * output[:,-2:], labels[:,-2:])
</code></pre>
<p>I was wondering whether using the labels to zero out the rows of the output I wasn’t interested in (where class = 0) was the right way to go about this.</p>
</blockquote>
</aside>
<p>I don’t really understand your overall use case – “test utterance,”<br>
applying the model to a distance matrix – so I don’t have any<br>
thoughts on your “real” problem</p>
<p>Three lower-level comments:</p>
<p>Your code for <code>loss</code> looks sensible and correct.  (I’ll leave it up to<br>
you as to whether at the higher level it does anything useful or<br>
what you want it to do.)</p>
<p>As written, your loss doesn’t care at all about the values of<br>
<code>output[0, 1]</code>, <code>output[0, 2]</code>, <code>output[2, 1]</code>, and <code>output[2, 1]</code>.<br>
So, in principle, these values – and the weights that produce<br>
them – could drift off and become large.  My gut tells me that it<br>
might be beneficial (although hardly essential) to regulate these<br>
output values (that don’t logically matter).</p>
<p>Assuming that a “class-0” <code>labels</code> row – that is, a row <code>i</code> for which<br>
<code>labels[i, 0]</code> is <code>0</code> – is all zeros – that is, for that <code>i</code>, <code>labels[i, j]</code><br>
is <code>0.0</code> for all <code>j</code> – then simply using:</p>
<pre data-code-wrap="python"><code class="lang-python">MSE (output[:, -2:], labels[:, -2:])
</code></pre>
<p>(or <code>MSE (output[:, 1:], labels[:, 1:])</code>, see below), will<br>
provide that regularization.  (Weight decay or a weight-regularization<br>
term in your loss would presumably provide equivalent benefit.)</p>
<p>As a minor stylistic comment:  Your choice of slicing notation looks<br>
rather perverse to me.  For readability (at least for me), I would prefer,<br>
e.g., <code>output[:, 0]</code> to extract the first column, and <code>output[:, 1:]</code> to<br>
extract the remaining columns.  <code>output[:, :1]</code> and <code>output[:, 1:]</code><br>
could also work, as you could argue that using <code>:1</code> and <code>1:</code> together<br>
emphasizes that you are using all of the columns (but I still like <code>0</code><br>
better for the first column).</p>
<p>Best.</p>
<p>K. Frank</p>

fauxneticien: <p>Hi K. Frank!</p>
<p>Thanks so much for your response. Low level comments are super helpful because I’m at the stage where I’m mainly trying to discover/get used to what best practices are around for writing PyTorch code.</p>
<blockquote>
<p>I don’t really understand your overall use case – “test utterance,” applying the model to a distance matrix – so I don’t have any thoughts on your “real” problem</p>
</blockquote>
<p>Ah yes, sorry, I didn’t explain it at all. The general task is called ‘query-by-example spoken term detection’ (QbE-STD) where you search for a spoken query (e.g. Q = ‘coffee’) in a corpus of audio documents (D1 = ‘I had coffee today’, D2 = ‘Where is the car?’) and output how likely the query is to occur in each document. If you don’t have access to a speech-to-text system, you typically just try to match on the spectral features (e.g. MFCCs) extracted from the audio files.</p>
<p>So the distance matrix is for a query Q of length M with F features and document D of length N with F features is a matrix of size M x N (taking the standardised Euclidean distance between each pair of feature vectors in Q and D). If the query occurs in the document, you typically get a quasi-diagonal band showing high spectrotemporal correlation somewhere along the document. I’m slowly making my way through replicating <a href="http://publications.idiap.ch/downloads/papers/2018/Ram_CNNBASEDQUERYBYEXAMPLESPOKENTERMDETECTION_2018.pdf" rel="noopener nofollow ugc">this paper</a> while also learning PyTorch and looking at <a href="https://github.com/idiap/CNN_QbE_STD" rel="noopener nofollow ugc">the GitHub repo</a> associated with the paper.</p>
<p>While the approach in that paper is interesting, it occurred to me while learning about object detection in my deep learning class that I could try to extend this to the QbE-STD. My (naive) thought on this was to change the final layer from <code>nn.Linear(60, 1)</code> to <code>nn.Linear(60, 3)</code> (full model <a href="https://github.com/fauxneticien/bnf_cnn_qbe-std/blob/main/src/Models.py#L32" rel="noopener nofollow ugc">here</a>) and make the network estimate, if a query occurs in the document, then at which time points (where <code>t_start</code> and <code>t_end</code> are proportion to the document, so in [0, 1]). Happy to hear any thoughts/cautions about this if you have any.</p>
<blockquote>
<p>Assuming that a “class-0” <code>labels</code>  row – that is, a row <code>i</code>  for which <code>labels[i, 0]</code>  is <code>0</code>  – is all zeros – that is, for that <code>i</code>, <code>labels[i, j]</code> is <code>0.0</code>  for all <code>j</code>  – then simply using: <code>MSE (output[:, -2:], labels[:, -2:])</code> (or <code>MSE (output[:, 1:], labels[:, 1:])</code>, see below), will provide that regularization. (Weight decay or a weight-regularization term in your loss would presumably provide equivalent benefit.)</p>
</blockquote>
<p>Ah, I see. I don’t think I’ve grasped the entirety of what the loss function is doing then. I guess when I was playing around I was only looking at the output of MSE(), which is <code>torch.Size([])</code>. I’m guessing since MSE and BCE are instantiated as classes, e.g. <code>MSE = torch.nn.MSELoss()</code>, there are some internals in the object that keep track of the pairwise differences? So leaving the outputs as-is and doing the MSE will encourage the network to predict <code>t_begin = 0</code> and <code>t_end = 0</code> when <code>class = 0</code>, therefore constraining the weights that are associated with the 2nd and 3rd outputs when something like <code>loss.backwards()</code> is called? Please correct me if I’m wrong anywhere.</p>
<p>I was also wondering whether using <code>0</code> for values I wasn’t interested in was a good approach. One issue I see is confounding for <code>t_start</code> because for many cases where <code>class = 1</code>, the <code>t_start</code> is also 0, e.g. for ‘coffee’ in ‘Coffee’s the best!’ the <code>t_start</code> is 0 since the query ‘coffee’ is at the start of the document.</p>
<blockquote>
<p>As a minor stylistic comment: Your choice of slicing notation looks rather perverse to me. For readability (at least for me), I would prefer, e.g., <code>output[:, 0]</code>  to extract the first column, and <code>output[:, 1:]</code>  to extract the remaining columns. <code>output[:, :1]</code>  and <code>output[:, 1:]</code> could also work, as you could argue that using <code>:1</code>  and <code>1:</code>  together emphasizes that you are using all of the columns (but I still like <code>0</code> better for the first column).</p>
</blockquote>
<p>Ah yes, this is the result of me trying various things in the console till the slice gave me what I wanted. Something I haven’t fully grasped is how <code>:</code> interacts with indices. I see suffixing the <code>:</code> to an index <code>1:</code>, e.g.  <code>output[:, 1:]</code>, means return columns from 1 onwards. But when <code>:</code> is prefixed, as in <code>output[:, :1]</code>, then the index starts from 1 instead of 0? So <code>output[:, :2]</code> returns the 1st and 2nd columns (not columns 0, 1, 2). For small label/output matrices, I think I might stick to explicit indexing with <code>output[:, [0]]</code> for the first column and use <code>output[:, [0, 1]]</code> for the 2nd and 3rd columns (coming from an R background).</p>
<p>Thanks!<br>
Nay</p>

KFrank: <p>Hey Nay!</p>
<aside class="quote no-group" data-username="fauxneticien" data-post="3" data-topic="101440" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fauxneticien/48/30230_2.png" class="avatar"> fauxneticien:</div>
<blockquote>
<p>I’m guessing since MSE and BCE are instantiated as classes, e.g. <code>MSE = torch.nn.MSELoss()</code>, there are some internals in the object that keep track of the pairwise differences?</p>
</blockquote>
</aside>
<p>Just one clarifying comment on this point:</p>
<p>In general, <code>torch.nn.SomePytorchLoss</code> (the class version) and<br>
<code>torch.nn.functional.some_pytorch_loss()</code> (the function version)<br>
are equivalent.  Instantiating the class version gives an instance of<br>
a <em>function object</em> (that is, an object instance that has a “call” method<br>
(<code>instance.(args ...)</code>) that does essentially the same thing as the<br>
function version.</p>
<p>The class version doesn’t really have any “smart,” loss-specific<br>
internals.  You can typically pass the constructor some “control”<br>
parameters such as class weights or <code>reduction = 'sum'</code>.  These<br>
are stored internally to the specific instance constructed.  But the<br>
instance doesn’t remember anything from call to call about the actual<br>
loss values calculated.</p>
<p>Best.</p>
<p>K. Frank</p>

fauxneticien: <blockquote>
<p>The class version doesn’t really have any “smart,” loss-specific<br>
internals.</p>
</blockquote>
<p>Ah, I see. Hm.</p>
<blockquote>
<p>As written, your loss doesn’t care at all about the values of<br>
<code>output[0, 1]</code> , <code>output[0, 2]</code> , <code>output[2, 1]</code> , and <code>output[2, 1]</code> .<br>
So, in principle, these values – and the weights that produce<br>
them – could drift off and become large. My gut tells me that it<br>
might be beneficial (although hardly essential) to regulate these<br>
output values (that don’t logically matter).</p>
</blockquote>
<p>So including <code>output[0, 1]</code> , <code>output[0, 2]</code> , <code>output[2, 1]</code> , and <code>output[2, 1]</code> in the loss calculation even though they don’t matter has a regulating effect so that the network finds some weights that continually predict [0, 0] for [t_start, t_end] when class = 0, and penalises any changes that move away from that? I guess my original line of thinking was that having the network predict [t_start, t_end] for class = 0 would ‘distract’ from learning good weights for when I do care about [t_start, t_end] when class = 1. But I guess that’s theory vs. practice. Thanks for your help!</p>