Using torch.nn.L1Loss to do regularization

Iris: <p>Hello!</p>
<p>I’m doing a text classification project. Apart from cross-entropy loss, I also add one more regularization term to encourage the score of words given by the model (score(w)) to be close to the ideal scores(r).<br>
score(w) is between -1 and 1, r is either -1 or 1.</p>
<p>e.g. input: I had lunch.<br>
score(‘lunch’)=0.01<br>
r=1<br>
I try to use L1 loss to encourage the score of ‘lunch’ to be 1.</p>
<p>Below is the code:</p>
<p>L1_loss=torch.nn.L1Loss(size_average=False)<br>
r=torch.tensor([r]).float().reshape(-1,1)<br>
loss=reg_strength*L1_loss(score(w),r)<br>
loss.backward(retain_graph=True)</p>
<p>Here are the values of score(w) and r<br>
score(w): tensor(0.9046, grad_fn=)<br>
r: tensor([[1.]])<br>
loss: tensor(0.0095, grad_fn=)</p>
<p>score(w): tensor(0.8485, grad_fn=)<br>
r: tensor([[-1.]])<br>
loss: tensor(0.1849, grad_fn=)</p>
<p>Here is the question: although I tried to encourage some terms to be 1, some terms to be -1, after training using 700 instances, I found that almost all the terms are encouraged to be 1. I’m not sure whether the code and the way to encourage are correct  or not.<br>
I would appreciate if you could help!</p>
<p>Thank you!</p>

KFrank: <p>Hi Iris!</p>
<aside class="quote no-group" data-username="Iris" data-post="1" data-topic="151905" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/a5b964/48.png" class="avatar"> Iris:</div>
<blockquote>
<p>Apart from cross-entropy loss, I also add one more regularization term to encourage the score of words given by the model (score(w)) to be close to the ideal scores(r).</p>
</blockquote>
</aside>
<p>I’m skeptical that this is a good idea.</p>
<p>First, <code>CrossEntropyLoss</code> should be enough to train what I imagine<br>
your use case to be.  (By the way, what is your use case?)</p>
<p>Second, <code>CrossEntropyLoss</code> and <code>L1Loss</code> are rather different in<br>
character and I don’t see them as blending together very effectively.</p>
<aside class="quote no-group">
<blockquote>
<p>score(w) is between -1 and 1, r is either -1 or 1.</p>
</blockquote>
</aside>
<p>You say that your “ideal” scores are -1 or 1.  This sounds like you are<br>
performing a binary classification problem.  You can treat this as a<br>
two-class, multi-class classification problem and use <code>CrossEntropyLoss</code><br>
(but it would be slightly preferable to use <code>BCEWithLogitsLoss</code>), but you<br>
wouldn’t want to feed a “score” that runs from -1 to 1 to <code>CrossEntropyLoss</code>.</p>
<p>Where does <code>score</code> come from, and do you pass <code>score</code> or something else<br>
to <code>CrossEntropyLoss</code>?</p>
<aside class="quote no-group">
<blockquote>
<p>L1_loss=torch.nn.L1Loss(size_average=False)<br>
r=torch.tensor([r]).float().reshape(-1,1)<br>
loss=reg_strength*L1_loss(score(w),r)<br>
loss.backward(retain_graph=True)</p>
</blockquote>
</aside>
<p>You <em>could</em> use <code>L1Loss</code> to train a classifier, but some version of cross<br>
entropy is likely to work much better.</p>
<p>Also, what is the purpose of <code>retain_graph = True</code>?  If this so you can<br>
combine your “regular” loss with <code>L1_loss</code>, you’re better off just summing<br>
them together and calling <code>.backward()</code> once:</p>
<pre data-code-wrap="python"><code class="lang-python">total_loss = regular_loss + L1_loss
total_loss.backward()
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>Here is the question: although I tried to encourage some terms to be 1, some terms to be -1, after training using 700 instances, I found that almost all the terms are encouraged to be 1.</p>
</blockquote>
</aside>
<p>I don’t see any obvious error that would encourage scores that should<br>
have been near -1 to be near 1.</p>
<p>What fraction of your scores are positive vs. negative if you train just<br>
with <code>CrossEntropyLoss</code>?  What fraction if you train just with <code>L1Loss</code>?<br>
Do you get almost all scores becoming positive only when you combine<br>
the two losses together?</p>
<p>Best.</p>
<p>K. Frank</p>

Iris: <p>Thanks for your reply, Frank!</p>
<p>The use case is sentiment analysis, predicting a product review to be positive or negative.</p>
<p>Say the input is “This dress looks cute but uncomorfotable”, the label is negative. I have the following objective function:<br>
L=L(f(x),y)+L(score(w),r(w))</p>
<p>The first loss term is the cross entropy loss which encourages the model prediction to be close to the label.<br>
The second loss term is to encourage the score(w) to be close to r(w), the ideal score. The score(w) is obtained from an explanation method (a post-hoc method), which outputs  the feature importance for each token given the model, so that we would know how the the prediction model captures each token.<br>
In the example “This dress looks cute but uncomorfotable”, suppose score(‘cute’)=0.1, score(‘uncomorfotable’)=0.01, I want to use the second loss term to encourage score(‘cute’) to be 1 since it contribute to positive sentiment, and score(‘uncomorfotable’) to be -1.</p>
<p>The reason I use retain_graph = True is because currently I did loss.backward() for  selected words (e.g. ‘cute’, ‘uncomorfotable’) iteratively, so I did backward multiple times for one input text, so I need to retain the graph.</p>
<p>As for the fraction of the scores, there are a lot more ideal scores r(w) be 1 than -1, so this might be the reason why almost all the terms are encouraged to be 1.</p>
<p>Thank you!<br>
Iris</p>

KFrank: <p>Hi Iris!</p>
<aside class="quote no-group quote-modified" data-username="Iris" data-post="3" data-topic="151905" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/a5b964/48.png" class="avatar"> Iris:</div>
<blockquote>
<p>The use case is sentiment analysis, predicting a product review to be positive or negative.</p>
<p>Say the input is “This dress looks cute but uncomorfotable”, the label is negative. I have the following objective function:<br>
L=L(f(x),y)+L(score(w),r(w))<br>
…<br>
In the example “This dress looks cute but uncomorfotable”, suppose score(‘cute’)=0.1, score(‘uncomorfotable’)=0.01, I want to use the second loss term to encourage score(‘cute’) to be 1 since it contribute to positive sentiment, and score(‘uncomorfotable’) to be -1.</p>
</blockquote>
</aside>
<p>As I understand your use case – more or less – you have a “sentence,”<br>
<code>x</code>, made up of “words,” <code>w</code>.  You have a “sentence” model, <code>f (x)</code>, that<br>
predicts the sentiment of the whole sentence, <code>x</code>.  You also have a “word”<br>
model, <code>score (w)</code>, that predicts the sentiment of a given word, <code>w</code>.</p>
<p>(The word model might be complicated and depend on some of the<br>
processing performed by the sentence model, but that’s okay.)</p>
<p>Your ground-truth labels – for both the sentiment of sentences and of<br>
words – are “negative” vs. “positive.”</p>
<p>So I would look at this as a collection of binary classifications.  You<br>
classify your sentence, <code>x</code>, as “class-0” (negative) vs. “class-1” (positive).<br>
You also classify each “important” word in the sentence, <code>w</code>, as “class-0”<br>
vs. “class-1”.</p>
<p>You say you are using cross entropy for the sentence classifier.  This<br>
makes sense, but because you have a binary (two-class) classification<br>
problem, it would be slightly preferable to structure it as a binary problem<br>
and use <code>BCEWithLogitsLoss</code> as your loss criterion (if you are not already<br>
doing so).</p>
<p>But your word classifier is also performing a binary classification.  One<br>
could use <code>L1Loss</code> (or <code>MSELoss</code>, etc.) as a loss criterion, but experience<br>
shows that, as a general rule, cross entropy should be your first choice<br>
for classification problems.  So I would also use <code>BCEWithLogitsLoss</code><br>
as the loss criterion for your word classifier.</p>
<p>So for the sentence <code>s</code>, with <code>w_i</code> being the important words in <code>s</code>, I would<br>
structure the total loss something like:</p>
<pre><code class="lang-plaintext">total_loss = binary_cross_entropy (f (x), y) + a * sum_i (binary-cross-entropy (score (w_i), r (w_i)))
</code></pre>
<p>where <code>a</code> is some sort of weighting / normalization constant.  First, maybe<br>
you would want to average over the <code>w_i</code>, rather than sum.  So maybe<br>
<code>a</code> should look something like <code>1 / N</code>, where <code>N</code> is the number of words<br>
in the specific sentence <code>s</code>.  Second, what should the relative weight<br>
between the whole-sentence loss and the word losses be?  This relative<br>
weight would also be incorporated into <code>a</code>.</p>
<p>When structured like this, you can use a single backward pass, which<br>
you would perform by calling <code>total_loss.backward()</code>.  This will then<br>
compute the gradients for the parameters of the sentence model <code>f</code><br>
as well as for any parameters in the word model, <code>score</code>.  If the result<br>
of <code>score</code> depends on some of the processing performed in <code>f</code>, the<br>
gradients from that dependency will flow back properly to the parameters<br>
of <code>f</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>As for the fraction of the scores, there are a lot more ideal scores r(w) be 1 than -1, so this might be the reason why almost all the terms are encouraged to be 1.</p>
</blockquote>
</aside>
<p>This could well be why <code>score (w)</code> tends to predict mostly positive results.<br>
The technical term for this situation is <em>unbalanced data,</em> meaning that<br>
you have significantly more data samples in one class than in the other.</p>
<p><code>BCEWithLogitsLoss</code> has a <code>pos_weight</code> constructor argument that<br>
you can use to compensate for the class imbalance in your data.  (Note,<br>
if your sentence labels are also imbalanced – say, significantly more<br>
positive-sentiment sentences than negative – you could also use<br>
<code>pos_weight</code> in the sentence loss criterion.)</p>
<p>If you go the route of using <code>BCEWithLogitsLoss</code>, you will want both<br>
your sentence and word models to output “scores” that run from <code>-inf</code><br>
to <code>inf</code> (that would be converted to the probability of being in the “positive”<br>
class by passing them through <code>sigmoid()</code>, a step that you don’t need<br>
to do explicitly).  These “scores” would typically be the output of a final<br>
<code>Linear</code> layer of the model, and become the so-called <code>input</code> of<br>
<code>BCEWithLogitsLoss</code>.</p>
<p>Your labels – the so-called <code>target</code> – should be (floating-point) <code>0.0</code> for<br>
sentences and words labelled as having negative sentiment (that is, in<br>
the “negative” class), and <code>1.0</code> for sentences and words that have positive<br>
sentiment (that is, in the “positive” class).  (Note, these <code>target</code> labels could<br>
actually range from <code>0.0</code> to <code>1.0</code> if it makes sense in your use case to label<br>
your data with the strength of the sentiment, rather than just being “negative”<br>
or “positive.”)</p>
<p>Good luck!</p>
<p>K. Frank</p>