Convert int into one-hot format

Response777: <p>Hi all.</p>
<p>I’m trying to convert the <code>y</code> labels in <code>mnist</code> data into one-hot format.</p>
<p>Since I’m not quite familiar with PyTorch yet, for each iteration, I just convert the <code>y</code> to numpy format and reshape it into one-hot and then convert it back to PyTorch. Like that</p>
<pre><code class="lang-auto">for batch_idx, (x, y) in enumerate(train_loader):
    y_onehot = y.numpy()
    y_onehot = (np.arange(num_labels) == y_onehot[:,None]).astype(np.float32)
    y_onehot = torch.from_numpy(y_onehot)
</code></pre>
<p>However, I notice that the it gets slower each iteration, and I doubt it’s these code which might request new memory each iteration that makes the code slower.</p>
<p>So my question is, is there a more <strong>PyTorch</strong> way, which may help me avoid such conversion?</p>
<p>Thanks!</p>

moskomule: <p>HI, it depends on your loss function, but some PyTorch’s loss functions take class labels as their targets(e.g. <a href="http://pytorch.org/docs/nn.html#torch.nn.NLLLoss" rel="nofollow noopener">NLLloss</a>). So if you use them, you don’t need to convert targets into onehot vectors.</p>

Response777: <p>Thanks for advice, I have seen such solution in the <code>examples</code> branch. However, I use such <code>y</code> as <code>input</code>, so that can’t solve my case.</p>

albanD: <p>Hi,</p>
<p>You can use the <code>scatter_</code> method to achieve this.<br>
I would also advise to create the <code>y_onehot</code> tensor once and then just fill it:</p>
<pre><code class="lang-auto">import torch

batch_size = 5
nb_digits = 10
# Dummy input that HAS to be 2D for the scatter (you can use view(-1,1) if needed)
y = torch.LongTensor(batch_size,1).random_() % nb_digits
# One hot encoding buffer that you create out of the loop and just keep reusing
y_onehot = torch.FloatTensor(batch_size, nb_digits)

# In your for loop
y_onehot.zero_()
y_onehot.scatter_(1, y, 1)

print(y)
print(y_onehot)
</code></pre>

Response777: <aside class="quote no-group quote-modified" data-username="albanD" data-post="4" data-topic="507">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>batch_size = 5<br>
nb_digits = 10</p>
<h1>Dummy input that HAS to be 2D for the scatter (you can use view(-1,1) if needed)</h1>
<p>y = torch.LongTensor(batch_size,1).random_() % nb_digits</p>
<h1>One hot encoding buffer that you create out of the loop and just keep reusing</h1>
<p>y_onehot = torch.FloatTensor(batch_size, nb_digits)</p>
<h1>In your for loop</h1>
<p>y_onehot.zero_()<br>
y_onehot.scatter_(1, y, 1)</p>
<p>print(y)<br>
print(y_onehot)</p>
</blockquote>
</aside>
<p>Thanks, that is exactly what I need!</p>

Nadav_Bhonker: <p>Isn’t there a more efficient way to input a “sparse Tensor” or a vector of indices into the network (specifically RNNs)?<br>
I guess something similar to torch’s sparse linear (only for RNNs).</p>

Response777: <p>Yes, I was using <code>one_hot_encoding</code> layer in Tensorflow, and it seems that there is no equivalent choice in PyTorch contemporarily.</p>

apaszke: <p><a class="mention" href="/u/nadav_bhonker">@Nadav_Bhonker</a> we’re working on adding more and more support for sparse operations, but the our fastest RNN backend (i.e. cuDNN) doesn’t support sparse inputs anyway. I’d recommend using Embedding for that.</p>

ncullen93: <p>just a note (from my understanding… maybe it doesnt apply in this case) it is currently advised to NOT follow this approach of creating the variable once and filling it each time (see <a href="https://discuss.pytorch.org/t/how-to-use-batch-normalization-in-testing-model/731/11">How to use Batch normalization in testing model</a>)</p>
<p>And also for future readers just to reiterate what user moskomule says- cross entropy and neg. log-likelihood losses in pytorch do NOT require one-hot encodings, so you can just use the normal target vector.</p>

apaszke: <p><a class="mention" href="/u/ncullen93">@ncullen93</a> the whole thread was about converting tensors I think, so it doesn’t apply there. But both of your statements are correct and are should be followed. Thanks <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

rajarsheem: <p>How about overriding the default nn.Embedding weights data with torch.eye ?</p>
<pre><code>emb = nn.Embedding(10, 10) 
emb.weight.data = torch.eye(10)
</code></pre>
<p>Done! Now, pass your batch containing indices to it.<br>
<code>emb(Variable(torch.LongTensor([[1, 2], [3, 4]])))</code><br>
will give output as:<br>
<code>Variable containing:</code><br>
<code>(0 ,.,.) =</code><br>
<code>0 1 0 0 0 0 0 0 0 0</code><br>
<code>0 0 1 0 0 0 0 0 0 0</code></p>
<p><code>(1 ,.,.) =</code><br>
<code>0 0 0 1 0 0 0 0 0 0</code><br>
<code>0 0 0 0 1 0 0 0 0 0</code></p>

apaszke: <p><a class="mention" href="/u/rajarsheem">@rajarsheem</a> that’s not a very good idea if your vector dimensionality is large. You’ll end up storing a huge weight matrix in memory, and in your code <code>emb.weight</code> requires gradient and it might be subject to optimization if you don’t take care.</p>
<p>Additionaly, zero + scatter a few ones will be  much faster than copying whole rows, of which most values are 0 anyway.</p>

rogetrullo: <p>Hi <a class="mention" href="/u/alband">@albanD</a> adn <a class="mention" href="/u/apaszke">@apaszke</a> , I was trying to use the scatter function, but I am running into some troubles.<br>
in my case I have something like this:</p>
<pre><code>batch_size=10
y = torch.LongTensor(batch_size,5,5).random_() % 3#3 classes,5x5 img
y_onehot = torch.FloatTensor(batch_size,3, 5,5)#I want the one hot going through the chans dim
y_onehot.zero_()
ones=torch.ones(y.size())

y_onehot.scatter_(1,y,ones)
</code></pre>
<p>However, it gives me the following error<br>
<code>Index tensor must have same dimensions as output tensor at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:450</code></p>
<p>Could you help me with this? Thanks!</p>

rogetrullo: <p>oh never mind, I just found that it works if I add a singleton dimension so that y and y_onehot have the same NUMBER of dimensions…</p>

zeng: <p>It seems that the y can’t be a <strong><em>Variable</em></strong> or a <strong><em>cuda.FloatTensor</em></strong>.<br>
How do I solve this <strong><em>TypeError</em></strong>?</p>

albanD: <p>There is no reason for y to be a <code>Variable</code> here.<br>
And since <code>y</code> is used to index in a tensor, it needs to have a proper indexing type: <code>LongTensor</code>.<br>
So if you use cuda, y should be a <code>cuda.LongTensor</code>, not a <code>cuda.FloatTensor</code>.</p>

zeng: <p>Sorry, I write wrong, it is also not available for cuda.LongTensor.</p>

albanD: <p>is <code>y_onehot</code> a cuda tensor?</p>

zeng: <p>Thank you, my problem. I ignored the y_onehot type is not a cuda tensor.</p>

kakrafoon: <p>Thank you! I got my cvae implemented with this tip.</p>