Data/image normalization

username: Dirk10001
cooked: <p>Hi all,<br>
if I try normalizing gray images with mean/standard deviation, calculated from 9000 images, a few images fall off the range from -1…+1. What to do with them?<br>
A normalization with min/max woprks stable, but sensitive to outliers. Mean/Stddev ist stable, summed over all images, not only over one batch, but outliers now fall off -1…+1. Just bring them to -1 or +1?</p>
<p>Many thanks for your help.</p>

username: ptrblck
cooked: <p>Why would values outside of <code>[-1, 1]</code> be problematic?</p>

username: Dirk10001
cooked: <p>The mean of the images rages from 77…118, the deviation from 3,0…13. The images look cloudy, or like wool, a slight texture, all gray images. I try to build an autoencoder trained only with errorfree images, to build a defect detection.</p>
<pre><code class="lang-auto">conv1(torch::nn::Conv2dOptions(1, CINV1_MASK_COUNT, /*kernel_size=*/3).padding(1)),
    bn1(torch::nn::BatchNorm2d(CINV1_MASK_COUNT)),
    conv2(torch::nn::Conv2dOptions(CINV1_MASK_COUNT, CINV2_MASK_COUNT, /*kernel_size=*/3).padding(1)),
    bn2(torch::nn::BatchNorm2d(CINV2_MASK_COUNT)),
    conv3(torch::nn::Conv2dOptions(CINV2_MASK_COUNT, CINV3_MASK_COUNT, /*kernel_size=*/3).padding(1)),
    bn3(torch::nn::BatchNorm2d(CINV3_MASK_COUNT)),

    fc1(32*32* CINV3_MASK_COUNT, 4096),
    fc2(4096, 32 * 32 * CINV3_MASK_COUNT),

    convT1(torch::nn::ConvTranspose2dOptions(CINV3_MASK_COUNT, CINV2_MASK_COUNT, 3).padding(1)),
    bnt1(torch::nn::BatchNorm2d(CINV2_MASK_COUNT)),

    convT2(torch::nn::ConvTranspose2dOptions(CINV2_MASK_COUNT, CINV1_MASK_COUNT, 3).padding(1)),
    bnt2(torch::nn::BatchNorm2d(CINV1_MASK_COUNT)),

    convT3(torch::nn::ConvTranspose2dOptions(CINV1_MASK_COUNT, 1, 3).padding(1))
</code></pre>
<p>3 convolutional layer in encoder/decoder.</p>
<pre><code class="lang-auto">x = conv1-&gt;forward(x);          // 15 x 256x256
    x = torch::max_pool2d(x, 2);    // 15 x 128x128
    x = torch::leaky_relu(x);             // 15 x 128x128
    //x = torch::tanh(x);             // 15 x 128x128
    x = torch::batch_norm(bn1-&gt;forward(x), bn1W, bnBias1W, bnmean1W, bnvar1W, true, 0.9, 0.001, true);
    

    x = conv2-&gt;forward(x);          // 30 x 128x128
    x = torch::max_pool2d(x, 2);    // 30 x 64x64
    x = torch::leaky_relu(x);             // 30 x 64x64
    //x = torch::tanh(x);             // 15 x 128x128
    x = torch::batch_norm(bn2-&gt;forward(x), bn2W, bnBias2W, bnmean2W, bnvar2W, true, 0.9, 0.001, true);

    x = conv3-&gt;forward(x);          // 30 x 64x764
    x = torch::max_pool2d(x, 2);    // 30 x 32x32
    x = torch::leaky_relu(x);             // 30 x 32x32
    //x = torch::tanh(x);             // 15 x 128x128
    x = torch::batch_norm(bn3-&gt;forward(x), bn3W, bnBias3W, bnmean3W, bnvar3W, true, 0.9, 0.001, true);

    // linear
    x = x.view({-1, 32 * 32 * CINV3_MASK_COUNT});

    x = fc1-&gt;forward(x);
    x = torch::leaky_relu(x);
    x = fc2-&gt;forward(x);
    x = torch::leaky_relu(x);
    
    // wieder Bilder
    x = x.view({ -1, CINV3_MASK_COUNT, 32, 32 });

    x = convT1-&gt;forward(x);
    x = torch::upsample_nearest2d(x, c10::IntArrayRef{ 64,64 });// 2);
    x = torch::leaky_relu(x);
    //x = torch::tanh(x);             // 15 x 128x128
    x = torch::batch_norm(bnt1-&gt;forward(x), bnt1W, bntBias1W, bntmean1W, bntvar1W, true, 0.9, 0.001, true);
    
    x = convT2-&gt;forward(x);
    x = torch::upsample_nearest2d(x, c10::IntArrayRef{ 128,128 });// 2);
    x = torch::leaky_relu(x);
    //x = torch::tanh(x);             // 15 x 128x128
    x = torch::batch_norm(bnt2-&gt;forward(x), bnt2W, bntBias2W, bntmean2W, bntvar2W, true, 0.9, 0.001, true);
    
    x = convT3-&gt;forward(x);
    x = torch::upsample_nearest2d(x, c10::IntArrayRef{ 256,256 });// 2);
    //x = torch::sigmoid(x);
    //x = torch::tanh(x);
</code></pre>
<p>I tried tanh and relu for activation, with and without batch_norm, MSE or L1 loss. All converges to a point, but the reconstruction is not very well. A few defect images, run by the network and than subtracted, show the defect areas, a few not.</p>
<p>MinMax and just divided by 255 converges. Mean/Stddev and the network does not converge.</p>

username: ptrblck
cooked: <p>It’s an interesting observation and maybe your use case would indeed benefit from another normalization approach.</p>
<p>I would also recommend to double check the batchnorm usage:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch::batch_norm(bnt1-&gt;forward(x), bnt1W, bntBias1W, bntmean1W, bntvar1W, true, 0.9, 0.001, true);
</code></pre>
<p>as it seems you are applying it twice. The first time via <code>bnt1-&gt;forward(x)</code> and then via the functional <code>torch::batch_norm</code> call.</p>

username: Dirk10001
cooked: <p>Thank you for your advice, I corrected the batch_norm call. Perhaps ZCA normalization will help.</p>