Normalization of input image

username: Jake_Pan
cooked: <p>I am a beginner to pytorch here. As I read the tutorial,  I always see such expression to normalization the input data.</p>
<pre><code class="lang-auto">transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) 
</code></pre>
<p>However, if I understand correctly, this step basically do</p>
<blockquote>
<p>input[channel] = (input[channel] - mean[channel]) / std[channel]</p>
</blockquote>
<p>according to the documentation.</p>
<p>So the question is, in order to normalize an input image, why we just say the mean and std is 0.5 and 0.5? Shouldn’t we first calculate the mean &amp; std of one whole image and then use that value to normalize? I mean, it doesn’t have to be the case that every image has the same mean and std. Do I miss something here?</p>
<p>Any comments and idea are highly appreciated! Thank you!</p>

username: JuanFMontesinos
cooked: <p>Well, yeh. You have to compute it or find it. For imagenet you can just google those numbers but if you work with a custom dataset it’d be good you to compute it.</p>

username: Jake_Pan
cooked: <p>Sorry, I am a beginner here. Maybe I am asking silly questions. Could you please tell me how to pass the sample-dependent mean and std to the <code>transforms.Normalize(mean=, std=)]</code>? Because what I find is all like assigning a fixed value for it. But how can I pass the specific data point and its np.mean() and np.std() to the <code>transforms.Normalize</code>?</p>

username: JuanFMontesinos
cooked: <p>Oh sorry for the confusion. It’s not sample-dependent but dataset-dependent. For example here you can find the parameters for imagenet. In short people computes an average of all the images of the dataset (assuming you are working with pictures and no other kind of data)<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L195-196" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L195-196" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/imagenet/main.py#L195-196</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="185" style="counter-reset: li-counter 184 ;">
<li>        print("=&gt; loaded checkpoint '{}' (epoch {})"</li>
<li>              .format(args.resume, checkpoint['epoch']))</li>
<li>    else:</li>
<li>        print("=&gt; no checkpoint found at '{}'".format(args.resume))</li>
<li>
</li>
<li>cudnn.benchmark = True</li>
<li>
</li>
<li># Data loading code</li>
<li>traindir = os.path.join(args.data, 'train')</li>
<li>valdir = os.path.join(args.data, 'val')</li>
<li class="selected">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],</li>
<li>                                 std=[0.229, 0.224, 0.225])</li>
<li>
</li>
<li>train_dataset = datasets.ImageFolder(</li>
<li>    traindir,</li>
<li>    transforms.Compose([</li>
<li>        transforms.RandomResizedCrop(224),</li>
<li>        transforms.RandomHorizontalFlip(),</li>
<li>        transforms.ToTensor(),</li>
<li>        normalize,</li>
<li>    ]))</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
in lines 195-196 provided for torchvision models.</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
</code></pre>
<p>You may find this interesting<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stats.stackexchange.com/questions/211436/why-normalize-images-by-subtracting-datasets-image-mean-instead-of-the-current" target="_blank" rel="nofollow noopener">stats.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/5429/max-gordon" target="_blank" rel="nofollow noopener">
    <img alt="Max Gordon" src="https://www.gravatar.com/avatar/a10fcf5472cf73b75540d4bf49dc8511?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stats.stackexchange.com/questions/211436/why-normalize-images-by-subtracting-datasets-image-mean-instead-of-the-current" target="_blank" rel="nofollow noopener">Why normalize images by subtracting dataset's image mean, instead of the current image mean in deep learning?</a>
</h4>

<div class="tags">
  <strong>deep-learning, image-processing</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/5429/max-gordon" target="_blank" rel="nofollow noopener">
    Max Gordon
  </a>
  on <a href="https://stats.stackexchange.com/questions/211436/why-normalize-images-by-subtracting-datasets-image-mean-instead-of-the-current" target="_blank" rel="nofollow noopener">11:11AM - 08 May 16 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Jake_Pan
cooked: <p>Oh, I see!<br>
Yes, that’s my mistake. Thank you!</p>

username: BarryBA
cooked: <p>Hi. May I ask, how to define the mean value and std value for each image channel? My dataset consists of the natural staellite images, just use 0.5? Or compute the mean and std value for each channel of each image?<br>
Moreover, can we set a parameter to make the CNN find the optimal parameter (mean value, std value or other weights/biases used in each channel) for the image processing? If so, can you tell me how to set the parameter?</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
You should compute mean and std for each channel for all the images in the dataset (or a representative subset if your dataset is very big).</p>
<p>Setting that preprocessing as a learnable parameter is not very practical. It’s just an statistical normalization.</p>
<p>In  the worst case you can set everything to 0.5, which is an approximation</p>

username: BarryBA
cooked: <p>Thank you very much. I have tested the results between different image processing strageies, i.e. set all the mean and std value to 0.5, or compute mean and std for each channel for all the images in the dataset, the results indicate that the 0.5 setting is more conductive to the improvement of final accuracy.</p>

username: Hong_Cheng
cooked: <p>should we always do the input image normalization?</p>

username: JuanFMontesinos
cooked: <p>It is strongly recommended. Networks may be able to fit any range of values but it’s been proved that normalization improves performance.</p>

username: Alec_young
cooked: <p>May I know how you are able to calculate the mean and standard deviation for the  datasets?</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
First of all, if the dataset is “huge” you pick a representative subset.<br>
Then you just need to iterate over that set and:<br>
-Ideally you would save all the pixel values (per channel) and compute the mean and std of those.</p>
<p>Since this may be memory demanding you can just save the mean of each image (per channel). If all the images are the same size, it is equivalent to the previous formulation. Otherwise it’s an approximation.</p>
<p>Once you have the mean you can compute the std in the analogous way.</p>