Using scikit-learn's scalers for torchvision

username: lorenzo_fabbri
cooked: <p>I noticed an improvement by doing per-channel normalization (6-channel images). It would be nice to simply use scikit-learn’s scalers like <code>MinMaxScaler</code>, but I noticed it’s much slower. The code for doing it is (inside <code>__getitem__</code>):</p>
<pre><code class="lang-auto">scaler = MinMaxScaler()
for i in range(img.size()[0]):
    img[i] = torch.tensor(scaler.fit_transform(img[i]))
</code></pre>
<p>I tried to code it myself using PyTorch. For the <code>MinMaxScaler</code> I wrote:</p>
<pre><code class="lang-auto">class MinMaxScaler(object):
    """
    Transforms each channel to the range [0, 1].
    """
    
    def __call__(self, tensor):
        
        for ch in tensor:
            scale = 1.0 / (ch.max() - ch.min())
            ch.mul_(scale).sub_(ch.min().mul_(scale))
        
        return tensor
</code></pre>
<p>The problem is that I do not obtain the same results as the <em>original</em> scaler. Do you spot anything wrong in my code? Is per-channel scaling implemented already in torchvision?</p>

username: ptrblck
cooked: <p>Since you are working inplace on <code>ch</code>, you don’t need the second multiplication with <code>scale</code> in your custom implementation. <code>ch.min()</code> will give you the new minimal value, which doesn’t need to be scaled again.</p>
<p>Also, you would need to get the max and min values in dim0 as done in the sklearn implementation.</p>
<p>This implementation should work:</p>
<pre><code class="lang-python">class PyTMinMaxScaler(object):
    """
    Transforms each channel to the range [0, 1].
    """    
    def __call__(self, tensor):
        for ch in tensor:
            scale = 1.0 / (ch.max(dim=0)[0] - ch.min(dim=0)[0])        
            ch.mul_(scale).sub_(ch.min(dim=0)[0])        
        return tensor
</code></pre>
<p>However, the loop will slow down your code.<br>
To get the most out of PyTorch, you should use vectorized code:</p>
<pre><code class="lang-python">class PyTMinMaxScalerVectorized(object):
    """
    Transforms each channel to the range [0, 1].
    """
    def __call__(self, tensor):
        scale = 1.0 / (tensor.max(dim=1, keepdim=True)[0] - tensor.min(dim=1, keepdim=True)[0]) 
        tensor.mul_(scale).sub_(tensor.min(dim=1, keepdim=True)[0])
        return tensor
</code></pre>
<p>Let’s check, if we get the same values:</p>
<pre><code class="lang-python">img1 = torch.randn(6, 100, 100)
img2 = img1.clone()
img3 = img1.clone()

# sklearn
scaler = MinMaxScaler()
for i in range(img1.size()[0]):
    img1[i] = torch.tensor(scaler.fit_transform(img1[i]))

# PyTorch manual
scaler = PyTMinMaxScaler()
out2 = scaler(img2)

# PyTorch fast
scaler_fast = PyTMinMaxScalerVectorized()
out3 = scaler_fast(img3)

print((img1 - out2).abs().max())
&gt; tensor(1.1921e-07)
print((img1 - out3).abs().max())
&gt; tensor(1.1921e-07)
print((out2 == out3).all())
&gt; tensor(True)
</code></pre>
<p>That looks good! The small differences are due to the limited floating point precision.</p>
<p>Let’s see, how fast the PyTorch versions are on the CPU via <code>%timeit</code> (on my old laptop):</p>
<pre><code class="lang-python">%timeit scaler(img2)
&gt; 1.85 ms ± 208 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit scaler_fast(img3)
&gt; 529 µs ± 44.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>
<p>If you are using a modern CPU, your code should be way faster. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: lorenzo_fabbri
cooked: <p>Thanks for the answer. Unfortunately, I’m having some issues with your solution.<br>
I just changed GPU and using the same code but with your scaler implementation, I kept getting NaN for the training loss. I thought I had some issues with the new GPU but after a day of experiments I finally realized that if I do not use your scaler, I get the <em>right</em> losses! Moreover, it’s much faster if I do not scale the images (from roughly 19 minutes to 12).</p>

username: ptrblck
cooked: <p>Do you only get the NaN values using one of my implementations or also your original one using sklearn?<br>
If just using mine, could you post an input tensor, which creates these NaN values?<br>
It’s a bit weird, as I’ve tested your approach to mine and got the same results (up to floating point precision).<br>
So if the scaling creates NaNs in your training, both approaches should behave the same.</p>
<aside class="quote no-group" data-username="lorenzo_fabbri" data-post="3" data-topic="53455">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lorenzo_fabbri/48/7696_2.png" class="avatar"> lorenzo_fabbri:</div>
<blockquote>
<p>I finally realized that if I do not use your scaler, I get the <em>right</em> losses!</p>
</blockquote>
</aside>
<p>Are you getting the right losses using the sklearn approach now?</p>
<aside class="quote no-group" data-username="lorenzo_fabbri" data-post="3" data-topic="53455">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lorenzo_fabbri/48/7696_2.png" class="avatar"> lorenzo_fabbri:</div>
<blockquote>
<p>Moreover, it’s much faster if I do not scale the images (from roughly 19 minutes to 12).</p>
</blockquote>
</aside>
<p>This is expected, as you will save some operations or am I missing something?</p>

username: lorenzo_fabbri
cooked: <p>Using my implementation (with for loop over the channels, the <code>MinMaxScaler</code> class) I get something (not NaN). With yours, I get NaN after the first epoch already. Everything else is exactly the same.<br>
I’m working with images, I wouldn’t know how to post an example: is just 1 tensor (512x512) enough?</p>

username: ptrblck
cooked: <p>This should be caused by dividing by zero, if the max and min values are equal.<br>
I naively implemented sklearn’s approach without checks.<br>
Add this check to your implementation:</p>
<pre><code class="lang-python">class PyTMinMaxScalerVectorized(object):
    """
    Transforms each channel to the range [0, 1].
    """
    def __call__(self, tensor):
        dist = (tensor.max(dim=1, keepdim=True)[0] - tensor.min(dim=1, keepdim=True)[0])
        dist[dist==0.] = 1.
        scale = 1.0 /  dist
        tensor.mul_(scale).sub_(tensor.min(dim=1, keepdim=True)[0])
        return tensor
</code></pre>

username: lorenzo_fabbri
cooked: <p>It works perfectly, thank you very much!</p>

username: muammar
cooked: <p>For those who want the general case:</p>
<pre><code class="lang-auto">class MinMaxScalerVectorized(object):
    """MinMax Scaler

    Transforms each channel to the range [a, b].

    Parameters
    ----------
    feature_range : tuple
        Desired range of transformed data.
    """

    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

    def __call__(self, tensor):
        """Fit features

        Parameters
        ----------
        stacked_features : tuple, list
            List of stacked features.

        Returns
        -------
        tensor 
            A tensor with scaled features using requested preprocessor.
        """

        tensor = torch.stack(tensor)

        # Feature range
        a, b = self.feature_range

        dist = tensor.max(dim=0, keepdim=True)[0] - tensor.min(dim=0, keepdim=True)[0]
        dist[dist == 0.0] = 1.0
        scale = 1.0 / dist
        tensor.mul_(scale).sub_(tensor.min(dim=0, keepdim=True)[0])
        tensor.mul_(b - a).add_(a)

        return tensor
</code></pre>
<p>Usage:</p>
<pre><code class="lang-auto">scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_data = scaler(data)
</code></pre>

username: SimplyAdi
cooked: <p>Hi, I could not understand the order of addition and multiplication in</p>
<pre><code class="lang-auto">tensor.mul_(scale).sub_(tensor.min(dim=0, keepdim=True)[0])
</code></pre>
<p>My understanding of the standard min-max scaling is (x-xmin)*scaler where scaler = 1/(xmax-xmin). So, could you pls. help me understand why the mul_() comes before the subtraction.</p>

username: AndhikaWB
cooked: <p>Another alternative based on Sklearn implementation [<a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html" rel="noopener nofollow ugc">1</a>, <a href="https://github.com/scikit-learn/scikit-learn/blob/ddc170d41/sklearn/preprocessing/_data.py#L291" rel="noopener nofollow ugc">2</a>]:</p>
<pre data-code-wrap="python"><code class="lang-python">class MinMax(torch.nn.Module):
    def __init__(self, fit_tensor: torch.Tensor, feature_range = (0, 1), inplace = False):
        super().__init__()
        self.feature_range = feature_range
        self.inplace = inplace
        self.is_inverse_ = False
        self.fit_(fit_tensor)

    def fit_(self, tensor: torch.Tensor):
        # Get min and max values for each variable
        data_min = tensor.min(dim = 0, keepdim = True)[0]
        data_max = tensor.max(dim = 0, keepdim = True)[0]

        data_range = data_max - data_min
        # Prevent division by zero in line below
        data_range[data_range == 0.0] = 1.0

        # See the Sklearn implementation for reference
        self.scale_ = (self.feature_range[1] - self.feature_range[0]) / data_range
        self.min_ = self.feature_range[0] - data_min * self.scale_

        self.data_min_ = data_min
        self.data_max_ = data_max
        self.data_range_ = data_range

    def forward(self, tensor: torch.Tensor):
        if self.is_inverse_:
            return self.backward(tensor)

        if not self.inplace:
            tensor = tensor.clone()

        tensor.mul_(self.scale_)
        tensor.add_(self.min_)

        return tensor
    
    def backward(self, tensor: torch.Tensor):
        if not self.inplace:
            tensor = tensor.clone()

        tensor.sub_(self.min_)
        tensor.div_(self.scale_)

        return tensor

    def inverse(self):
        # Return an inverse copy of this class
        clone = copy.deepcopy(self)
        clone.is_inverse_ = True

        return clone
</code></pre>
<p>Seems to work well after a simple testing:</p>
<pre data-code-wrap="python"><code class="lang-python">arr = torch.Tensor([
    [7,2,3],
    [4,5,9],
    [1,8,6]
])

# Default way
temp = MinMax(fit_tensor = arr)
fwd_arr = temp(arr)
bwd_arr = temp.backward(fwd_arr)

print(arr)
print(fwd_arr)
print(bwd_arr)

# Alternative way without calling backward
inv_temp = temp.inverse()
bwd_arr = inv_temp(fwd_arr)

print(bwd_arr)
</code></pre>
<pre><code class="lang-auto">tensor([[7., 2., 3.],
        [4., 5., 9.],
        [1., 8., 6.]])
tensor([[1.0000, 0.0000, 0.0000],
        [0.5000, 0.5000, 1.0000],
        [0.0000, 1.0000, 0.5000]])
tensor([[7.0000, 2.0000, 3.0000],
        [4.0000, 5.0000, 9.0000],
        [1.0000, 8.0000, 6.0000]])
tensor([[7.0000, 2.0000, 3.0000],
        [4.0000, 5.0000, 9.0000],
        [1.0000, 8.0000, 6.0000]])
</code></pre>
<p>Not sure if this follows PyTorch style (I’m new to PyTorch), but just want to share in case someone need a quick solution</p>