MLP with one hidden layer, dimensions modules?

Bart_Louwers: <p>I’m toying around with PyTorch and MNIST, trying to get a hang of the API. I want to create an MLP with one hidden layer. What should the dimensions of the modules be?</p>
<p>The input is a 784x1 vector, so I’d say two modules, hidden layer 781x100 (100 hidden nodes), output layer 100x10 (for classification). However, that gives “size mismatch, m1: [784 x 1], m2: [784 x 100] at /build/python-pytorch/src/”.</p>
<p>My code is below. I’m using the C++ API but this question should be answerable with those only familiar with the Python API as well. Thanks!</p>
<pre><code class="lang-cpp">void TorchMLP::init(std::vector&lt;size_t&gt; nodes_per_layer) {
    assert(nodes_per_layer.size() == 3);
    h = register_module("h",torch::nn::Linear(nodes_per_layer[0], nodes_per_layer[1]));
    out = register_module("out",torch::nn::Linear(nodes_per_layer[1],nodes_per_layer[2]));
    optimizer = std::make_shared&lt;torch::optim::SGD&gt;(
        parameters(), torch::optim::SGDOptions(d_learning_rate));
}

torch::Tensor TorchMLP::test(torch::Tensor x) {
    x = torch::relu(h-&gt;forward(x));
    x = torch::sigmoid(out-&gt;forward(x));
    return x;
}

void TorchMLP::train(std::pair&lt;torch::Tensor, torch::Tensor&gt; example) {
    optimizer-&gt;zero_grad();

    auto [input, expected_output] = example;
    auto output = test(input);
    auto loss = torch::mse_loss(output, expected_output);
    loss.backward();
    optimizer-&gt;step();
}
</code></pre>

KFrank: <p>Hi Bart!</p>
<aside class="quote no-group quote-modified" data-username="Bart_Louwers" data-post="1" data-topic="48841" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bart_louwers/48/13439_2.png" class="avatar"> Bart_Louwers:</div>
<blockquote>
<p>I’m toying around with PyTorch and MNIST, trying to get a hang of the API. I want to create an MLP with one hidden layer. What should the dimensions of the modules be?</p>
<p>The input is a 784x1 vector, so I’d say two modules, hidden layer 781x100 (100 hidden nodes), output layer 100x10 (for classification). However, that gives “size mismatch, m1: [784 x 1], m2: [784 x 100] at /build/python-pytorch/src/”.</p>
<p>My code is below. I’m using the C++ API but this question should be answerable with those only familiar with the Python API as well. Thanks!</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">void TorchMLP::init(std::vector&lt;size_t&gt; nodes_per_layer) {
    assert(nodes_per_layer.size() == 3);
    h = register_module("h",torch::nn::Linear(nodes_per_layer[0], nodes_per_layer[1]));
    out = register_module("out",torch::nn::Linear(nodes_per_layer[1],nodes_per_layer[2]));
    optimizer = std::make_shared&lt;torch::optim::SGD&gt;(
        parameters(), torch::optim::SGDOptions(d_learning_rate));
}
...
</code></pre>
</blockquote>
</aside>
<p>With the pytorch api (i.e., not c++), I think the following does what<br>
you want:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.nn.Sequential (
...     torch.nn.Linear (784, 100),
...     torch.nn.ReLU(),
...     torch.nn.Linear (100, 10)
... )
Sequential(
  (0): Linear(in_features=784, out_features=100, bias=True)
  (1): ReLU()
  (2): Linear(in_features=100, out_features=10, bias=True)
)
</code></pre>
<p>(Just to be clear, the input has 784 features, the hidden layer<br>
has 100 “neurons”, the output has 10 values – the logits for<br>
the 10 classes, and the batch size is deduced from the shape<br>
of the input tensor.)</p>
<p>I’m not familiar with the torch c++ api, so I can’t say where your<br>
problem might lie, but, on the surface, it looks like you are doing<br>
something sensible, and I don’t have any guess about your<br>
particular “size mismatch, m1: [784 x 1], m2: [784 x 100]" error.</p>
<p>(It would help if you would tell us – or print out – the values in<br>
your <code>nodes_per_layer</code> vector.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

Bart_Louwers: <p>Hey <a class="mention" href="/u/kfrank">@KFrank</a> My problem was that my ‘batch’ wasn’t really a batch (it was a single training example), as I’m doing online learning.</p>
<p>The following seemed to have fixed that, (although I’m not exactly clear on the usage of view and unqueeze yet):</p>
<pre><code class="lang-auto">torch::Tensor TorchMLP::test(torch::Tensor x) {
    x = x.view(784);
    x = torch::relu(h-&gt;forward(x));
    x = torch::sigmoid(out-&gt;forward(x));
    return x.unsqueeze(-1);
}
</code></pre>
<p>But I got that very unhelpful error, so I’ll file a bug report for it. Thanks for your help.</p>