Not Normal Normalization?

username: 3DTOPO
cooked: <p>I am having issues that are almost certainly due to values being out of range of what is expected.</p>
<p>I computed the mean and std for the dataset and used them for training:</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.90346843004226, 0.488738864660263],
							      std=[0.10251837968826, 0.247761115431785])
</code></pre>
<p>However, based on a random sample taken, that doesn’t seem correct. For instance a random sample taken from the 2nd channel is:</p>
<pre><code class="lang-auto">sample=0.3415
</code></pre>
<p>Using:</p>
<pre><code class="lang-auto">mean=0.488738864660263
std=0.247761115431785
</code></pre>
<p>Considering:</p>
<pre><code class="lang-auto">(sample - mean) / std
</code></pre>
<p>Expanded:</p>
<pre><code class="lang-auto">(0.3415 - 0.488738864660263) / 0.247761115431785 = -0.5942775338400
</code></pre>
<p>I was under the impression that the standard practice with PyTorch normalization is to use the range of 0.0 to 1.0. Could -0.5942775338400 be correct?</p>
<p>How can I check if my mean and standard are correct?</p>
<p>This is how I calculated the mean and standard:</p>
<pre><code class="lang-auto">for i, (images, target) in enumerate(train_loader):
	images = images.to(device)
	for image in images:
		image = image.squeeze()
		channel_mean[0] += torch.sum(torch.mean(image[0, :, :]).to('cpu'))
		channel_mean[1] += torch.sum(torch.mean(image[1, :, :]).to('cpu'))		
		channel_std[0] += torch.std(image[0, :, :]).sum(0).item()
		channel_std[1] += torch.std(image[1, :, :]).sum(0).item()
		nb_samples += 1
		
channel_mean /= nb_samples
channel_std /= nb_samples
</code></pre>

username: KFrank
cooked: <p>Hi Jeshua!</p>
<aside class="quote no-group" data-username="3DTOPO" data-post="1" data-topic="96882" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/3dtopo/48/27560_2.png" class="avatar"> 3DTOPO:</div>
<blockquote>
<p>Using:</p>
<pre><code class="lang-auto">mean=0.488738864660263
std=0.247761115431785
</code></pre>
<p>Considering:</p>
<pre><code class="lang-auto">(sample - mean) / std
</code></pre>
<p>Expanded:</p>
<pre><code class="lang-auto">(0.3415 - 0.488738864660263) / 0.247761115431785 = -0.5942775338400
</code></pre>
</blockquote>
</aside>
<p>Yes, this is what <code>Normalize</code> does (and how it is supposed to work).</p>
<aside class="quote no-group">
<blockquote>
<p>I was under the impression that the standard practice with PyTorch normalization is to use the range of 0.0 to 1.0 (or some other range<br>
of order 1).</p>
</blockquote>
</aside>
<p>Not necessarily.</p>
<p>If your input values were approximately uniformly distributed over<br>
the range 100 to 200, you might choose to transform them so that<br>
they would lie in the range 0 to 1.</p>
<p>But if your values were approximately from a Gaussian distribution<br>
with a mean of 150 and a standard deviation of 50, you might<br>
choose to normalize them so they had a mean of 0 and a standard<br>
deviation of 1.</p>
<aside class="quote no-group">
<blockquote>
<p>Could -0.5942775338400 be correct?</p>
</blockquote>
</aside>
<p>Yes.  After normalization (to a mean of 0 and standard deviation of 1),<br>
one of your sample values is <code>0.59</code> standard deviations away from the<br>
mean.  This is perfectly reasonable, and, indeed, to be expected.</p>
<p>Best.</p>
<p>K. Frank</p>

username: 3DTOPO
cooked: <p>Thanks for the explanation Frank. With your vote of confidence I was able to get the network working. Thanks again!</p>