Important pitfall! Scaling labels in Regression? Labels as dtype long or float32?

Pytorch-Noob: <p>My question is generic in nature but an important pitfall!</p>
<p>Do I need to scale my regression problem’s labels? Can labels be used as torch.long dtype for regression problems or we need to use only float32/float64.</p>
<p>For those interested in specific context: I played with load_diabetes dataset</p>
<p>from the sklearn package. This dataset has all the labels are from 25 - 346 as int type.</p>
<p>When I do not scale the labels, my model predicts and outputs values that seems to be in floats</p>
<p>from 0 and onwards. So error is huge.</p>
<p>However, when I scale the labels, (using MinMaxScaler); I have a scaled output and then I inverse_transform it to get fairly accurate results. So why does the scaling works and non-scaling does not work? Is their a way to get outputs with non-scaled case as values that are in tens or hundreds?</p>
<p>**My data is defined as:</p>
<h1><a name="p-474816-torchifying-via-dataset-class-1" class="anchor" href="#p-474816-torchifying-via-dataset-class-1"></a>torchifying via dataset class**</h1>
<p><code>class dataset(Dataset):</code><br>
<code>  def __</code><strong><code>init__</code></strong><code>(self, input_features, output_labels):</code><br>
<code>      self.input_features = torch.tensor(input_features, dtype=    torch.float32).to(device)</code><br>
<code>      self.output_labels = torch.tensor(output_labels, dtype= torch.float32).to(device)</code><br>
<code>  def __</code><strong><code>len__</code></strong><code>(self):</code><br>
<code>      return len(self.input_features)</code><br>
<code> def</code><strong><code>__getitem__</code></strong><code>(self,index): # gives complete data for that row</code><br>
<code>      return self.input_features[index], self.output_labels[index]</code></p>
<h2><a name="p-474816-model-2" class="anchor" href="#p-474816-model-2"></a>Model</h2>
<p><code>class MyModel(nn.Module):</code><br>
<code>  def</code><strong><code>__init__</code></strong><code>(self):</code><br>
<code>      super(MyModel, self).</code><strong><code>__init__</code></strong><code>()</code><br>
<code>      self.first = nn.Linear(X.shape[1], 192) # .shape[1] no. of features</code><br>
<code>      self.first_relu = nn.ReLU()</code><br>
<code>      self.second = nn.Linear(192,96)</code><br>
<code>      self.second_relu = nn.ReLU()</code><br>
<code>      self.third = nn.Linear(96,48)</code><br>
<code>      self.third_relu = nn.ReLU()</code><br>
<code>      self.fourth = nn.Linear(48, 12)</code><br>
<code>      self.fourth_relu = nn.ReLU()</code><br>
<code>      self.fifth = nn.Linear(12, 6)</code><br>
<code>      self.fifth_relu = nn.ReLU()</code><br>
<code>      self.sixth = nn.Linear(6, 1)</code></p>
<pre><code class="lang-auto"># regression has only one output

   def forward(self, xx):
      xx = self.first(xx)
      xx = self.first_relu(xx)
      xx = self.second(xx)
      xx = self.second_relu(xx)
      xx = self.third(xx)
      xx = self.third_relu(xx)
      xx = self.fourth(xx)
      xx = self.fourth_relu(xx)
      xx = self.fifth(xx)
      xx = self.fifth_relu(xx)
      xx = self.sixth(xx)
      return xx
</code></pre>
<p><code>model = MyModel().to(device) # initialize the class and then give to device</code><br>
<code>summary(model, input_size = (X.shape[1],))</code></p>

KFrank: <p>Hi Noob!</p>
<aside class="quote no-group" data-username="Pytorch-Noob" data-post="1" data-topic="222731" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorch-noob/48/78054_2.png" class="avatar"> Pytorch-Noob:</div>
<blockquote>
<p>Do I need to scale my regression problem’s labels?</p>
</blockquote>
</aside>
<p>The short story is that you should follow the standard advice and scale your<br>
data (input data as well labels).</p>
<p>You don’t <em>need</em> to, but you generally want to.</p>
<p>Why?  Because with pytorch’s default initializations, a randomly initialized<br>
model will start out making predictions that are, roughly speaking, of order one.<br>
So your model not only has to learn the real substance of your problem, it also<br>
has to learn to make predictions that are not (roughly speaking) of order one.<br>
This can slow training and possibly cause instability in early training.</p>
<p>(If your labels are large compared to the finite-precision or dynamic range of<br>
floating-point arithmetic, additional precision / range issues can arise.)</p>
<aside class="quote no-group">
<blockquote>
<p>Can labels be used as torch.long dtype for regression problems</p>
</blockquote>
</aside>
<p>Yes, but …</p>
<p>Your prediction has to be some sort of float (in order for autograd to be able<br>
to backpropagate).  Your loss criterion (typically something like <code>MSELoss</code>)<br>
will generally upcast the type of your labels to match that of your predictions.</p>
<p>As a matter of programming style (“say what you mean”), I would generally have<br>
the type of my labels (what the pytorch documentation usually calls <code>target</code>)<br>
explicitly match that of the predictions – but this is just style.</p>
<aside class="quote no-group">
<blockquote>
<p>or we need to use only float32/float64.</p>
</blockquote>
</aside>
<p>Generally, no, although you could certainly write a loss criterion that requires<br>
the prediction and label types to match.</p>
<aside class="quote no-group">
<blockquote>
<p>So error is huge.<br>
…<br>
So why does the scaling works and non-scaling does not work?</p>
</blockquote>
</aside>
<p>What do you mean by “error is huge”?</p>
<p>Let’s say that your labels start out of order 100, but you scale them down to<br>
order 1 and you successfully train your model to give predictions of order 1<br>
that achieve a <code>MSELoss</code> of, say, 0.3.  Now let’s say that you don’t scale<br>
your labels and again successfully train your model.  Your predictions will<br>
now be of order 100, and let’s assume that you’re in fact getting the same<br>
predictions as before, just scaled by a factor of 100.  Your <code>MSELoss</code> will<br>
now get scaled by a factor of 100**2, so will therefore be 3000.  Naively<br>
3000 looks like a big number.  Is that what you mean by huge?  If so, it’s<br>
not really huge – it’s the same as before, just scaled up.</p>
<p>On the other hand, if your unscaled model doesn’t train successfully, you<br>
could get a <code>MSELoss</code> of, say, 1.e7 that would be “huge” even relative to<br>
your unscaled predictions and labels.</p>
<aside class="quote no-group">
<blockquote>
<p>Is their a way to get outputs with non-scaled case as values that are in tens or hundreds?</p>
</blockquote>
</aside>
<p>You should be able to.  As mentioned above, unscaled labels could slow training.<br>
So start by training longer.  Also, unscaled labels could cause instability in<br>
early training.  So start by training with a small (or very small) learning rate<br>
until you get past such instabilities and your model starts making predictions<br>
closer in scale to your predictions.  Then you can increase your learning rate<br>
back up to a more typical value.</p>
<p>Note, however, that if your unscaled labels (and predictions) are a factor of<br>
100 larger than your scaled values, a <code>MSELoss</code> loss value will be 1.e4 times<br>
larger (as will the gradients produced by backpropagation).  For many optimizers,<br>
e.g., <code>SGD</code>, this will have the same effect as training with the scaled data, but<br>
using a learning rate that is, figuratively speaking, 1.e4 times as large.</p>
<p>Best.</p>
<p>K. Frank</p>

Pytorch-Noob: <p>Thank you Frank! To add regarding:</p>
<p><code>Do I need to scale my regression problem’s labels?</code></p>
<p>I have realized that scaling, like MinMaxScaler, should be done on all the subsets of X (features) and y (label) data. This means that each X_train, X_test, X_val, and same for y (label) subsets; we should have seperate scaling class initialized. This helps in preventing <strong>“leakage of info”</strong> from the training subsets to testing and validation subsets, since a single scaling class on the whole X implemented can cause info like “<strong>mean</strong>“, “<strong>standard deviation”</strong> etc to leak to subsets!</p>
<p>Correct this Noob if wrong!</p>