Mutli variate time series prediction - Conv1D and loss function

Barak_Gahtan: <p>I have a couple of questions.</p>
<p>I have data of the following shape:</p>
<pre><code>(32, 64, 11) 
</code></pre>
<p>where 32 is the batch size, 64 is a sequence length and 11 is the number of features. each sample of mine is 64X11, and has a label of 0 or 1.</p>
<p>I’d like to predict when a sequence has a label of “1”.</p>
<p>I’m trying to use a simple architecture with</p>
<pre><code>conv1D → ReLU → flatten → linear → sigmoid.
</code></pre>
<p>For the <code>Conv1D</code> I thought that since it is a multi variate time series prediction, and each row in my data is a second, I think that the number of in channels should be the number of features, since that way it will process all of the features concurrently, (I don’t have any spatial things in my data, it doesn’t matter if a column is in index 0 or 9, as it is important in image with pixels.</p>
<p>I can’t get to decide how to “initialize” the <code>conv1D</code> parameters. Currently I think the number of channels should be the number of features and not 1, as the reason I just explained, but unsure of it.</p>
<p>Secondly, should the loss function be BCELOSS or something else? assuming that my labels are 0 or 1, and the prediction for me is I want the model to provide a probability of belonging to class with label 1.</p>
<p>A lot of thanks.</p>

KFrank: <p>Hi Barak!</p>
<aside class="quote no-group" data-username="Barak_Gahtan" data-post="1" data-topic="171732" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barak_gahtan/48/56989_2.png" class="avatar"> Barak_Gahtan:</div>
<blockquote>
<p>I’m trying to use a simple architecture with<br>
conv1D → ReLU → flatten → linear → sigmoid.</p>
</blockquote>
</aside>
<p>This is reasonable (for a very simple architecture), but get rid of the<br>
<code>sigmoid()</code> (see below).</p>
<aside class="quote no-group">
<blockquote>
<p>I think that the number of in channels should be the number of features,</p>
</blockquote>
</aside>
<p>Yes, this makes sense.  Note that <code>Conv1d</code> expects its input to have<br>
shape <code>[nBatch, nChannels, length]</code>, so you will want to swap<br>
the last and second-to-last dimensions of your data before feeding<br>
it to <code>Conv1d</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>I can’t get to decide how to “intialize” the conv1D paramters.</p>
</blockquote>
</aside>
<p>When you instantiate <code>Conv1d</code>, pytorch initializes its <code>weight</code> and <code>bias</code><br>
randomly.  Start with that, an only do something different if you have<br>
a good reason.</p>
<p>[quote]<br>
currently I think the number of channels should be the number of features and not 1, as the reason I just explained, but unsure of it.<br>
[\quote]</p>
<p>Your <code>in_channels</code> should be your number of input features (11).</p>
<p>Assuming that you are asking about <code>out_channels</code>, that’s more of<br>
a judgment call.  Consider using something larger than <code>in_channels</code><br>
(and then let your final <code>Linear</code> figure out to combine them back<br>
together).</p>
<aside class="quote no-group">
<blockquote>
<p>should the loss function be BCELOSS or something else?</p>
</blockquote>
</aside>
<p>You have a binary-classification problem, so binary cross entropy is<br>
appropriate.  However, for reasons of numerical stability, you should<br>
use <code>BCEWithLogitsLoss</code>, but <em>without</em> a subsequent <code>sigmoid()</code> (as<br>
<code>BCEWithLogitsLoss</code> computes <code>log_sigmoid()</code> internally).</p>
<aside class="quote no-group">
<blockquote>
<p>the prediction for me is I want the model to provide a probability of belonging to class with label 1.</p>
</blockquote>
</aside>
<p>Without the final <code>sigmoid()</code>, your model will predict the <em>logit</em> that<br>
corresponds to your desired probability.  You probably don’t actually<br>
need the probability.  (Logits are the correct input to <code>BCEWithLogitsLoss</code>.)<br>
If you need the actual probabilities (and you probably don’t), convert<br>
your logits to probabilities by passing them through <code>sigmoid()</code>, but<br>
do this <em>after</em> you compute your loss function (and detached from the<br>
computation graph).</p>
<p>Best.</p>
<p>K. Frank</p>

Barak_Gahtan: <p>Regarding the last paragraph.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/245ed3529141db4324f3201cdf94f725815d1f84.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/245ed3529141db4324f3201cdf94f725815d1f84" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/245ed3529141db4324f3201cdf94f725815d1f84_2_690x125.jpeg" alt="image" data-base62-sha1="5bKlNv3T5RSys5IjTng1dbkfDjC" width="690" height="125" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/245ed3529141db4324f3201cdf94f725815d1f84_2_690x125.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/245ed3529141db4324f3201cdf94f725815d1f84_2_1035x187.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/245ed3529141db4324f3201cdf94f725815d1f84_2_1380x250.jpeg 2x" data-dominant-color="313432"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×350 38.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Right now, the criterion is BCEwithlogits, which I compute after the model is built without the sigmoid, and the labels are 0 or 1.</p>
<p>Did you mean “compute your loss function and detached from the computation graph” when you wrote that?</p>
<p>What I want is for the model, given a sequence, to give me a probability for class 1.</p>

KFrank: <p>Hi Barak!</p>
<aside class="quote no-group" data-username="Barak_Gahtan" data-post="3" data-topic="171732" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barak_gahtan/48/56989_2.png" class="avatar"> Barak_Gahtan:</div>
<blockquote>
<p>Right now, the criterion is BCEwithlogits, which I compute after the model is built without the sigmoid, and the labels are 0 or 1.</p>
</blockquote>
</aside>
<p>Good.</p>
<p>(As an aside, please don’t post screenshots of textual information. It breaks<br>
accessibility, searchability, and copy-paste.)</p>
<aside class="quote no-group">
<blockquote>
<p>Did you mean “compute your loss function and detached from the computation graph” when you wrote that?</p>
</blockquote>
</aside>
<p>Quoting from my previous post:</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="171732">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you need the actual probabilities (and you probably don’t), convert<br>
your logits to probabilities by passing them through <code>sigmoid()</code>, but<br>
do this <em>after</em> you compute your loss function (and detached from the<br>
computation graph).</p>
</blockquote>
</aside>
<p>What I meant by this is</p>
<ol>
<li>
<p>Pass the logits (no <code>sigmoid()</code>) to your loss function.  If you then need<br>
to convert the logits to probabilities, do so (by applying <code>sigmoid()</code>) <em>after</em><br>
calling your loss function.</p>
</li>
<li>
<p>Since you won’t be backpropagating through the <code>sigmoid()</code> call (because<br>
it comes after your loss function), you can gain a little efficiency by turning<br>
off autograd tracking by using <code>.detach()</code> or by wrapping the <code>sigmoid()</code><br>
(and any subsequent processing) in a <code>with torch.no_grad():</code> block.</p>
</li>
</ol>
<aside class="quote no-group">
<blockquote>
<p>What I want is for the model, given a sequence, to give me a probability for class 1.</p>
</blockquote>
</aside>
<p>Logits contain the same information as probabilities (but are numerically<br>
better behaved).  What you want – at least for training purposes – is for<br>
your model to give you a logit.  You do <em>not</em> need a probability for calling<br>
your loss function.  If you need a probability for some other reason – and<br>
ofttimes you don’t – take the logit that is output by your model and pass it<br>
through a <code>sigmoid()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>