Why is this transform resulting in a divide by zero error?

Wesley_Neill: <p>So I’m following along this <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial in the docs</a> on custom datasets. I’m using the MNIST dataset instead of the fancy one in the tutorial. This is the extension of the <code>Dataset</code> class I wrote:</p>
<pre><code class="lang-auto">class KaggleMNIST(Dataset):

    def __init__(self, csv_file, transform=None):
        self.pixel_frame = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.pixel_frame)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.tolist()

        image = self.pixel_frame.iloc[index, 1:]
        image = np.array([image])

        if self.transform:
            image = self.transform(image)

        return image
</code></pre>
<p>It works, until I try to use a transform on it:</p>
<pre><code class="lang-auto">tsf = transforms.Compose([transforms.ToTensor(), 
                          transforms.Normalize((0.5,), (0.5,))
                         ])
                          
trainset = KaggleMNIST('train/train.csv', transform=tsf)

image0 = trainset[0]
</code></pre>
<p>I’ve looked at the stack trace, and it seems like the normalization is happening in this line of code:</p>
<pre><code class="lang-auto">c:\program files\python38\lib\site-packages\torchvision\transforms\functional.py in normalize(tensor, mean, std, inplace)
--&gt; 218     tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
</code></pre>
<p>So I don’t get why there is divide by zero since <code>std</code> should be 0.5, nowhere remotely close to a small value.</p>
<p>Thanks for your help!</p>

Wesley_Neill: <p>I do not know why the problem exists, exactly, but I found that casting to <code>float64</code> seems to have solved it. I still wouldn’t mind a more detailed explanation as to what happened.</p>
<p>my solution:</p>
<pre><code class="lang-auto">image = self.pixel_frame.iloc[index, 1:].to_numpy(dtype='float64').reshape(1, -1)
</code></pre>

Nikronic: <p>Hi,</p>
<p>When you are loading your images, values are integer and has type of <code>int64</code>.<br>
This is source code of <code>normalize</code>.</p>
<pre><code class="lang-auto">
def normalize(tensor, mean, std, inplace=False):
    """Normalize a tensor image with mean and standard deviation.

    .. note::
        This transform acts out of place by default, i.e., it does not mutates the input tensor.

    See :class:`~torchvision.transforms.Normalize` for more details.

    Args:
        tensor (Tensor): Tensor image of size (C, H, W) to be normalized.
        mean (sequence): Sequence of means for each channel.
        std (sequence): Sequence of standard deviations for each channel.
        inplace(bool,optional): Bool to make this operation inplace.

    Returns:
        Tensor: Normalized Tensor image.
    """
    if not torch.is_tensor(tensor):
        raise TypeError('tensor should be a torch tensor. Got {}.'.format(type(tensor)))

    if tensor.ndimension() != 3:
        raise ValueError('Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = '
                         '{}.'.format(tensor.size()))

    if not inplace:
        tensor = tensor.clone()

    dtype = tensor.dtype  ############### here
    mean = torch.as_tensor(mean, dtype=dtype, device=tensor.device)
    std = torch.as_tensor(std, dtype=dtype, device=tensor.device)
    if (std == 0).any():
        raise ValueError('std evaluated to zero after conversion to {}, leading to division by zero.'.format(dtype))
    if mean.ndim == 1:
        mean = mean[:, None, None]
    if std.ndim == 1:
        std = std[:, None, None]
    tensor.sub_(mean).div_(std)
    return tensor
</code></pre>
<p>As you can see in the line that <code>mean</code> and <code>std</code> are converted to tensors instead of tuples or arrays, they will have same dtype as <code>input</code> tensor and as your input is already <code>int64</code>, <code>mean</code> and <code>std</code>  will be <code>int64</code> which in case of <code>0.5</code>, both will be <code>0</code>.</p>
<p>The reason is that <code>Normalize</code> or <code>Transforms</code> are defined for images and base library for image processing spcificly image loading is <code>PIL</code> which uses <code>ToPILImage</code> as first line of transforms. This method will automatically normalize data to <code>[0, 1]</code> range so what so ever mean and std values are, they will have same values as they will be converted to float.</p>
<p>For your case, a simple solution would be adding <code>/255.</code> where you convert data to array.</p>
<pre><code class="lang-auto">image = np.array([image])/255.
</code></pre>
<p>Bests</p>

Wesley_Neill: <p>That’s a really good explanation! Thank you so much!</p>