Convert a one hot vector into smooth vector - label smoothing

jpainam: <p>Hi,<br>
<a href="https://discuss.pytorch.org/t/calculate-embedding-from-smoothened-one-hot-vector/13721/5">This</a> and <a href="https://discuss.pytorch.org/t/fastest-way-of-converting-a-real-number-to-a-one-hot-vector-representing-a-bin/21578">this</a> didn’t really help.<br>
I want to use two losses. First loss uses smooth labels and the second one uses the one_hot vector. For the smooth label, i converted the one_hot vector <code>targets = torch.tensor([[1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0] ...)</code> into a smooth vector. Since pytorch doesn’t provide a ready function for smooth target, I did this and want to confirm I’m doing things right?</p>
<pre><code class="lang-python">outputs = torch.randn(6, 10)
targets_onehot = torch.zeros(outputs.size())
smooth_label = 1. / self.eps # 0.2
smooth_target = targets_onehot.scatter(1, targets.unsqueeze(1), smooth_label)
logfeat = F.log_softmax(outputs, dim=1)
smooth_loss = logfeat * smooth_target 
smooth_loss = torch.sum(smooth_loss, 1) / smooth_loss.size(1)

real_target = targets_onehot.scatter(1, targets.unsqueeze(1), float(1))
real_loss = logfeat * real_target 
real_loss = torch.sum(real_loss, 1) / real_loss.size(1)

loss = - real_loss * ( 1 - self.flags)  - self.flags * smooth_loss
</code></pre>
<p>I short, i would like to use a loss based on <code>torch.tensor([[0.2, 0, 0, 0, 0, 0, 0], [0, 0.2, 0, 0, 0, 0], [0, 0, 0.2, 0, 0, 0] ...)`` and another one based on </code>targets `</p>
<p>Looking at torch.scatter, I’m a little bit confused how the operation is done. Am I doing things right?<br>
Thank you.</p>

KFrank: <p>Hi Jean Paul!</p>
<aside class="quote no-group quote-modified" data-username="jpainam" data-post="1" data-topic="62067" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpainam/48/5870_2.png" class="avatar"> jpainam:</div>
<blockquote>
<p>i converted the one_hot vector <code>targets = torch.tensor([1, 0, 1, 0, 1, 0])</code> into a smooth vector. Since pytorch doesn’t provide a ready function for smooth target, I did this and want to confirm I’m doing thing right?</p>
</blockquote>
</aside>
<p>I haven’t worked through your code, so I can’t comment on it.</p>
<p>Two side comments, though:</p>
<p>First, I wouldn’t call your example,<br>
<code>targets = torch.tensor([1, 0, 1, 0, 1, 0])</code>,<br>
“one hot.”  You have multiple non-zero entries.  (I might see this<br>
as a “multi-label” target, where classes “0”, “2”, and “4” are active,<br>
and classes “1”, “3”, and “5” are not active.)   But this doesn’t really<br>
change the issue.</p>
<p>One way to smooth a one-hot vector (or a multi-label vector, or<br>
any binary vector made up of zeros and ones) is to run it through<br>
<code>torch.nn.functional.softmax (alpha * target)</code>.</p>
<p>(<code>alpha</code> is a smoothing parameter: larger alpha makes the result<br>
sharper, and smaller alpha makes it smoother.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

jpainam: <p>my bad… It’s actually a multi class problem. I really wanted to simply the <code>targets</code> but messed up… Thank you. I’ll edit the question. By the way, i managed to solve it this way, for someone interested in the future.</p>
<pre><code class="lang-python"> targets = targets.contiguous().view(-1)
 logfeat = F.log_softmax(inputs, dim=1)
 one_hot = torch.zeros_like(inputs).scatter(1, targets.view(-1, 1), float(1))
 smooth_target = torch.zeros_like(inputs).scatter(1, targets.view(-1, 1), self.smoothing)
 smooth_loss = (smooth_target * logfeat).sum(dim=1)
 return smooth_loss.mean()
</code></pre>