Advice on implementing input and output data scaling

username: billtubbs
cooked: <p>I’ve searched for a while and I can’t find any examples or conclusive guidance on how to implement input or output scaling.</p>
<p>Situation:<br>
I am training an RNN on sequence input data to predict outputs (many-to-many).  Both the inputs and outputs are continuous-valued so I should scale them (to zero mean and unit variance).</p>
<p>Obviously there is <a href="https://discuss.pytorch.org/t/pytorch-tensor-scaling/38576">no built-in function to do scaling in Pytorch</a>.  I thought <code>transforms.Normalize</code> might be suitable but every time I try to use it on my data it says <code>TypeError: tensor is not a torch image.</code> Is it only designed for images?</p>
<p>I can easily <a href="https://discuss.pytorch.org/t/pytorch-tensor-scaling/38576/2">implement scaling by hand</a> or <a href="https://discuss.pytorch.org/t/normalization-of-input-data-to-qnetwork/14800/2">as a custom transformer as others have</a> so this is not the main issue.  The bigger question is where to put the code.</p>
<ol>
<li>
<p>One way is to add this to my custom dataset.  I could initialize the scaler mean and variance when the dataset is initialized and then <a href="https://discuss.pytorch.org/t/how-to-normalize-a-tensor-to-0-mean-and-1-variance/18766/2">add the transformation to the dataset’s <code>__getitem__</code> method</a>  Since I’m also doing output scaling then I will have to ‘hang on’ to the transform method so I can do the inverse operation to the predictions before comparing them with the targets.</p>
</li>
<li>
<p>Another option is to add it to the data loader.</p>
</li>
<li>
<p>Some people think it should be <a href="https://discuss.pytorch.org/t/input-normalization-inside-the-model/56260">added to the estimator model</a> (RNN in this case).</p>
</li>
<li>
<p>Keep it out of all objects and manually add it to the high-level code running the training and prediction tasks.</p>
</li>
</ol>
<p>Please advise.</p>

username: billtubbs
cooked: <p>In the absence of any responses to my question I developed a custom transform (based on <a href="https://discuss.pytorch.org/t/pytorch-tensor-scaling/38576/2">this example</a>) and pass two instances of it (one as <code>input transform</code>, one as <code>output_transform</code>) to the custom dataset at intialisation (option 1).  Then I will use the <code>output_transform.inverse_transform()</code> method to “decode” the predictions later when using the estimator.</p>
<p>Here is the code if anyone is interested:</p>
<pre><code class="lang-auto">class StandardScaler():
    """Standardize data by removing the mean and scaling to
    unit variance.  This object can be used as a transform
    in PyTorch data loaders.

    Args:
        mean (FloatTensor): The mean value for each feature in the data.
        scale (FloatTensor): Per-feature relative scaling.
    """

    def __init__(self, mean=None, scale=None):
        if mean is not None:
            mean = torch.FloatTensor(mean)
        if scale is not None:
            scale = torch.FloatTensor(scale)
        self.mean_ = mean
        self.scale_ = scale

    def fit(self, sample):
        """Set the mean and scale values based on the sample data.
        """
        self.mean_ = sample.mean(0, keepdim=True)
        self.scale_ = sample.std(0, unbiased=False, keepdim=True)
        return self

    def __call__(self, sample):
        return (sample - self.mean_)/self.scale_

    def inverse_transform(self, sample):
        """Scale the data back to the original representation
        """
        return sample * self.scale_ + self.mean_
</code></pre>