Model design choice suggestions

_joker: <p>Hi All,</p>
<p>I have a very simple use case where I would like to predict a floating number (y), given four floating numbers (x1, x2, x3, x4) that denotes the property of a process. I would like to have some pointers or insights on different supervised and unsupervised approaches to solve this problem and which one among the two would be a better choice?</p>
<p>y - doesn’t have any range so far, and can be any floating value.</p>
<p>Example:</p>
<pre><code class="lang-auto">Training data:
x1 = 3758687 (throughput)
x2 = 2.8678 (latency)
x3 = 0.098 (loss)
x4 = 0 or 1 (binary)
y = {1.28,2.28,4.019,9.8,13,2}
</code></pre>
<p>now using this I need to create a model to be able to predict ‘y’.</p>

KFrank: <p>Hi Joker!</p>
<aside class="quote no-group" data-username="_joker" data-post="1" data-topic="126111" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p>Example:</p>
<pre><code class="lang-auto">Training data:
x1 = 3758687 (throughput)
x2 = 2.8678 (latency)
x3 = 0.098 (loss)
x4 = 0 or 1 (binary)
y = {1.28,2.28,4.019,9.8,13,2}
</code></pre>
</blockquote>
</aside>
<p>First, especially because your <code>x</code>s differ in scale by several orders<br>
of magnitude, you should most likely normalize your input data to<br>
be or order one.  This makes it easier for your training to get started,<br>
and your model doesn’t have to “learn” to deal with these very different<br>
scales.</p>
<p>Given that your input data has no temporal or image-like structure,<br>
I would suggest that you start with a so-call <em>multi-layer perceptron</em><br>
(MLP) that consists just of fully-connected <code>Linear</code> layers and<br>
activations.</p>
<p>Because you have four input variables, your first <code>Linear</code> will have<br>
<code>in_features = 4</code>, and because you are predicting a single output<br>
value, your final <code>Linear</code> layer will have <code>out_features = 1</code>.  You<br>
might start with “hidden” layers with 64 “neurons.”  I like to start with<br>
<code>Sigmoid</code> as my non-linear activation (although <code>ReLU</code> is another<br>
popular choice that has some possible theoretical advantages and<br>
shows experimental benefits in some applications).</p>
<p>Thus, something like:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn. Sequential (
    torch.nn.Linear (4, 64),
    torch.nn.Sigmoid(),
    torch.nn.Linear (64, 64),
    torch.nn.Sigmoid(),
    torch.nn.Linear (64, 1)
)
</code></pre>
<p>You would then typically experiment a little bit by making the network<br>
<em>shallower</em> – go down to a single hidden layer; <em>deeper</em> – go up to<br>
perhaps three or four hidden layers; <em>narrower</em> – fewer “neurons” per<br>
layer; and <em>wider</em> – more “neurons” per layer.  You might also try switching<br>
all or some of your activations to <code>ReLU</code>.</p>
<p>I assume that your predicted value, <code>y</code>, is a continuous scalar value<br>
where it’s meaningful to say that a predicted value is closer to or further<br>
away from your ground-truth <code>target</code> value (and that closer is better).</p>
<p>For this situation, <code>MSELoss</code> would likely be the most appropriate<br>
loss criterion.</p>
<aside class="quote no-group">
<blockquote>
<p>supervised and unsupervised approaches to solve this problem and which one among the two would be a better choice?</p>
</blockquote>
</aside>
<p>If you have good training data – that is, enough samples for which you<br>
have the input <code>x</code>s as well as the ground-truth value for <code>y</code> – supervised<br>
learning will be the clearly preferable approach.  Unsupervised learning<br>
is a distinctly more difficult problem.</p>
<p>Best.</p>
<p>K. Frank</p>

_joker: <p><a class="mention" href="/u/kfrank">@KFrank</a> thanks a lot, that really helped me start from scratch.</p>