Sequence multiclass classification

username: JustAGuysInThailand
cooked: <p>I have a dataset in the size of [88,498,20] which represent 88 samples with the length of 498, each Time-steps will be represent by 20 classes.<br>
My output is [88,498,3] so it’s the same as input only different is now I only have 3 classes to predict.</p>
<p>So this is my first time with Pytorch. With Keras, I just simply create Dense layers , using Categorical_CrossEntropy loss function.</p>
<p>But for Pytorch I tried to create like this :</p>
<blockquote>
<p>class MLP(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MLP, self).<strong>init</strong>()<br>
self.dense1 = torch.nn.Linear(20,128)<br>
self.relu = torch.nn.ReLU()<br>
self.dropout = nn.Dropout(0.5)<br>
self.dense2 = torch.nn.Linear(128, 3)<br>
self.softmax = torch.nn.Softmax()</p>
<pre><code>def forward(self, x):
    x = self.dense1(x)
    x = self.relu(x)
    x = self.dense2(x)
    x = self.dropout(x)
    x = self.softmax(x)
    return x
</code></pre>
</blockquote>
<p>and the training go like this:</p>
<p>model = MLP()<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)<br>
train_losses = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
val_losses = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<blockquote>
<p>for epoch in range(100):<br>
running_loss = 0.0<br>
optimizer.zero_grad()<br>
outputs = model(x_train)<br>
loss = criterion(outputs, y_train)<br>
loss.backward()<br>
optimizer.step()</p>
</blockquote>
<p>But I got this error with the Loss function :</p>
<blockquote>
<p>ValueError: Expected target size (88, 3), got torch.Size([88, 498, 3])</p>
</blockquote>
<p>So I don’t know what should I configure with the network and how to determine the size of each Linear layers.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits, so you should remove the <code>self.softmax</code> from the <code>forward</code> method.<br>
Assuming your current model output represents: <code>[batch_size=88, seq_len=498, nb_classes=3</code>], you would have to <code>permute</code> the output so that it has the shape <code>[batch_size, nb_classes, seq_len]</code> via <code>output = output.permute(0, 2, 1)</code>.<br>
Also the target shape should be <code>[batch_size, seq_len]</code> and it should contain class indices in the range <code>[0, nb_classes-1]</code>.</p>

username: JustAGuysInThailand
cooked: <p>Thank you for your answer.<br>
But here I don’t really understand.<br>
First about the input : My first layer is</p>
<blockquote>
<p>nn.Linear(input_size, hidden_size)</p>
</blockquote>
<p>so what is the input size here? Normally in Keras i just put<code> input_size = (batch, seq_len, no_classes</code>) but for Pytorch I have tried and the only satisfied value for input_size is 20 which is my no_classes? So normally with Pytorch we should pout as no_layers for input size?</p>
<p>Second one, I understand the permute() you so me so to fit with the loss function, I think I have to make the target_size as <code>[batch_size, seq_len]</code> so that mean I will not need to convert my level to one-hot encoding as Keras I believe. But can you so me how to determine the number of sample for each batch? With Keras I just declare the <code>batch_size = no_samples per batch</code> only.</p>
<p>Sorry for the newbie question, I’m really new with Pytorch , just move from Keras to this framework so my idea just cannot follow Pytorch properly and tutorial on the internet is not as detail as Keras ( I think because Pytorch is still new ).</p>

username: ptrblck
cooked: <p>The input size to an <code>nn.Linear</code> layer is defined in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear">docs</a> as:</p>
<blockquote>
<p>Input: (N,∗,H_in) where *∗ means any number of additional dimensions and H_in=in_features</p>
</blockquote>
<p>In your case you would thus pass the input as <code>[batch_size, seq_len, in_features]</code>.</p>
<aside class="quote no-group" data-username="JustAGuysInThailand" data-post="3" data-topic="117543">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecc23a/48.png" class="avatar"> JustAGuysInThailand:</div>
<blockquote>
<p>Normally in Keras i just put<code> input_size = (batch, seq_len, no_classes</code>)</p>
</blockquote>
</aside>
<p>Are you sure you are passing the <code>no_classes</code> into the first layer in Keras? This seems like an unneeded limitation unless your model only contains a single layer.</p>
<aside class="quote no-group" data-username="JustAGuysInThailand" data-post="3" data-topic="117543">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecc23a/48.png" class="avatar"> JustAGuysInThailand:</div>
<blockquote>
<p>I think I have to make the target_size as <code>[batch_size, seq_len]</code> so that mean I will not need to convert my level to one-hot encoding as Keras I believe.</p>
</blockquote>
</aside>
<p>Yes, the shape is correct and also one-hot encoded targets are not needed, but the target should contain the class indices.</p>
<aside class="quote no-group" data-username="JustAGuysInThailand" data-post="3" data-topic="117543">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecc23a/48.png" class="avatar"> JustAGuysInThailand:</div>
<blockquote>
<p>But can you so me how to determine the number of sample for each batch? With Keras I just declare the <code>batch_size = no_samples per batch</code> only.</p>
</blockquote>
</aside>
<p>The batch size is not determined in the model (as it’s not depending on the number of samples passed to it) and you could set the desired <code>batch_size</code> in the <code>DataLoader</code>.</p>