Enchantment on the Image Enhancement
By: Nico Aguila
Tags: Image Processing, Python, Image Enhancement, Histogram Manipulation, Data Science

Image enhancements can Fourier Transforms, White Balancing, Histogram Manipulation, and Contrast Stretching. I will discuss what I have learned here in this post.

An example application of image enhancement. Image from https://petapixel.com/2017/11/01/photo-enhancement-starting-get-crazy/

1.Fourier Transforms

Since images can be transformed into numbers, it can also be thought that these numbers are a superposition of wave patterns. The resulting wave patterns, just like signals, can have frequencies that can be transformed using Fourier Transforms. As a result, Fourier Transforms can help data scientists remove artifacts from images.

Here, we use Marty from Back to the Future. On the left side is the grayscale image and to its right is its corresponding logarithmic Fourier transform. We shall be using the codes below to show how it’s done:

#importing libraries

import numpy as np

import matplotlib.pyplot as plt

from skimage.io import imread, imshow

from skimage.color import rgb2gray

marty = rgb2gray(imread('bttf.png'))

marty_fft = np.fft.fftshift(np.fft.fft2(marty)) fig, ax = plt.subplots(1, 3, gridspec_kw=dict(width_ratios=[16,16,1]),figsize=(16,8))

img = ax[0].imshow(marty, cmap='gray')

ax[1].imshow(np.log(abs(marty_fft)), cmap='gray')

fig.colorbar(img, cax=ax[2]);

What do you notice from the image?

Upon careful inspection of the logarithmic Fourier transformed image, we can see that there are distinct white lines, where two of them are dominant vertical and horizontal lines at the middle of the Fourier transform. Removing the white lines on the image can be done by replacing the vertical line with a small value and then transforming the new values back into the image domain.

marty_fft2 = marty_fft.copy()

marty_fft2[:1250,marty_fft.shape[1]//2] = 1

marty_fft2[-750:,marty_fft.shape[1]//2] = 1

imshow(np.log(abs(marty_fft2)), cmap='gray');

The vertical line removed from the log Fourier transformed image

imshow(abs(np.fft.ifft2(marty_fft2)), cmap='gray');

Final processed image

As you can see, the image had a lot more contrast after the modification, with the items on the dashboard gaining more detail after the processing. This is one of the potential uses of Fourier transforms in image processing.

2.White Balancing

White Balancing, on the other hand, corrects images from the color range, where whites or neutral colored images appear more white, depending on the editing done. There are different approaches to White Balancing:

White Patch Algorithm — The color white is present when RGB channels are at its maximum. We can correct images based on the color white by either normalizing or rescaling based on the maximum value for the corresponding channel.

Let’s try the algorithm out

import numpy as np

import skimage.io as skio

from skimage import img_as_ubyte, img_as_float marty = skio.imread('bttf.png')

skio.imshow(marty);

Where we’re going, we don’t need Fourier transforms

Since white is observed as a maximum value in the RGB channel, let’s try to divide each channel by it’s maximum value to bring about white balance

marty_wp = img_as_ubyte(marty*1.0 / marty.max(axis=(0,1)))

skio.imshow(marty_wp);

White-balanced picture

Oddly enough, the image looks like the earlier image displayed. Let’s check on the maximum values already contained within each channel

print('Max value for red channel is: '+str(np.max(marty[:,:,0])))

print('Max value for green channel is: '+str(np.max(marty[:,:,1])))

print('Max value for blue channel is: '+str(np.max(marty[:,:,2]))) Max value for red channel is: 255

Max value for green channel is: 255

Max value for blue channel is: 251

This means that both red and green channels are already at maximum, while the blue channel is nearly there. This makes a straightforward white balance approach impractical. In order to properly correct the image color, we must first look at the histogram of the pixel values along the color range and take the any given percentile as our maximum value. For this case, we’ll be using the 95th percentile.

for channel, color in enumerate('rgb'):

print(channel)

channel_values = marty[:,:,channel]

plt.step(np.arange(256),

np.bincount(channel_values.flatten(), minlength=256)*1.0/

channel_values.size,

c=color)

plt.xlim(0, 255)

plt.axvline(np.percentile(channel_values, 95), ls='--', c=color)

plt.xlabel('channel value')

plt.ylabel('fraction of pixels');

Histogram of RGB channel

Using this piece of code lets us determine the exact numerical value of the 99th percentile of each channel

for channel, color in enumerate('RGB'):

print('99th percentile of %s channel:'%channel,

np.percentile(marty[:,:,channel], 95)) 99th percentile of 0 channel: 147.0

99th percentile of 1 channel: 148.0

99th percentile of 2 channel: 157.0

#channel 0 = Red

#channel 1 = Green

#channel 2 = Blue

Using this information, we then apply the 95th percentile to the previous approach of White-patch algorithm

marty_wp2 = img_as_ubyte((marty*1.0 / np.percentile(marty, 95, axis=(0, 1)))

.clip(0, 1))

skio.imshow(marty_wp2)

Such vibrant colors

Original image for reference

Gray-world Algorithm — Gray is essentially a neutral color (a mix of black and white). With this concept, this algorithm also assumes that pixels are gray on average and that users should be able to adjust each color channel such that their mean values are the same.

Translating the concept into code gets you this output

marty_gw = ((marty * (marty.mean() / marty.mean(axis=(0, 1))))

.clip(0, 255).astype(int))

skio.imshow(marty_gw);

Gray-world Algorithm result

Ground-truth algorithm — Based from the observed image, users can set a specific part of an image as the reference when enhancing the entire canvas. With a set reference, users can then either apply the White Patch or the Gray-world Algorithm depending on the set reference that the ground truth is based on.

Let’s try it out with Marty and his companions. First, we set “ground truths”, where we set reference colors for the correction of the entire image to be based on. For this, we’ll be trying out the skin tones of Dr. Brown, Jennifer, and Marty himself. Let’s see the code in action:

fig, ax = plt.subplots(1,1,dpi=100)

gt1 = Rectangle((1700,600),20,20,edgecolor='r',facecolor='none')

gt2 = Rectangle((1000,350),20,20,edgecolor='g',facecolor='none')

gt3 = Rectangle((500,700),20,20,edgecolor='b',facecolor='none')

ax.add_patch(gt1)

ax.add_patch(gt2)

ax.add_patch(gt3)

ax.imshow(marty)

Red: Doc Brown’s wrist, Green: Jen’s forehead, Blue: Marty’s cheek

Now let’s take a closer look at how these references look like

fig, ax = plt.subplots(1,3,figsize=(12,10))

ax[0].imshow(marty[int(xy1[0][1]):int(xy1[1][1]),

int(xy1[0][0]):int(xy1[1][0])])

ax[0].set_title('Doc')

ax[1].imshow(marty[int(xy2[0][1]):int(xy2[1][1]),

int(xy2[0][0]):int(xy2[1][0])])

ax[1].set_title('Jen')

ax[2].imshow(marty[int(xy3[0][1]):int(xy3[1][1]),

int(xy3[0][0]):int(xy3[1][0])])

ax[2].set_title('Marty')

fig.tight_layout()

Closer look at the ground-truth references we established

Now that we have the ground truths, we can do at least two approaches on image correction: either to normalize each color channel to the maximum value of each channel of the ground truth, or to make the mean values of each color channel match those of the mean value of the ground truth. We can show that with the following code below:

doc_patch = marty[int(xy1[0][1]):int(xy1[1][1]),

int(xy1[0][0]):int(xy1[1][0])]

jen_patch = marty[int(xy2[0][1]):int(xy2[1][1]),

int(xy2[0][0]):int(xy2[1][0])]

marty_patch = marty[int(xy3[0][1]):int(xy3[1][1]),

int(xy3[0][0]):int(xy3[1][0])] patches = [doc_patch, jen_patch, marty_patch] for patch in patches:

image_max = (marty / patch.max(axis=(0,1))).clip(0,1)

image_mean = ((marty * patch.mean())

/ marty.mean(axis=(0,1))).clip(0,255).astype(int)

fig, ax = plt.subplots(1, 2, dpi=100)

ax[0].imshow(image_max)

ax[0].set_title('Max Method')

ax[0].set_axis_off()

ax[1].imshow(image_mean)

ax[1].set_title('Mean Method')

ax[1].set_axis_off()

fig.tight_layout()

Doc Brown’s ground truth

Jen’s ground truth

Marty’s ground truth

3.Histogram Manipulation

Given that images are superposition of wave patterns, the numerical distribution generated by these images can be further analyzed, which can be seen through a histogram. Manipulating the histogram means that the cumulative distribution function of the original image will be matched to a target cumulative distribution function as well, wherein all the values from the image will be replaced with the value that matches from the target. This will be done across all corresponding intensity values, using interpolation to lookup their respective replacement values.

Let’s start with color channel exploration for now with Marty.

def rgb_splitter(image):

rgb_list = ['Reds','Greens','Blues']

fig, ax = plt.subplots(1, 3, figsize=(17,7), sharey = True)

for i in range(3):

ax[i].imshow(image[:,:,i], cmap = rgb_list[i])

ax[i].set_title(rgb_list[i], fontsize = 22)

ax[i].axis('off')

fig.tight_layout()

rgb_splitter(marty)

How Marty and friends look like per color channel

Taking a look at each color’s intensity values give us the following visualization

def df_plotter(image):

freq_bins = [cumulative_distribution(image[:,:,i]) for i in

range(3)]

target_bins = np.arange(255)

target_freq = np.linspace(0, 1, len(target_bins))

names = ['Red', 'Green', 'Blue']

line_color = ['red','green','blue']

f_size = 20



fig, ax = plt.subplots(1, 3, figsize=(17,5))

for n, ax in enumerate(ax.flatten()):

ax.set_title(f'{names[n]}', fontsize = f_size)

ax.step(freq_bins[n][1], freq_bins[n][0], c=line_color[n],

label='Actual CDF')

ax.plot(target_bins,

target_freq,

c='gray',

label='Target CDF',

linestyle = '--')

df_plotter(marty)

Beautiful squiggly lines

As expected, there are a lot of pixels already have high intensity values. Correcting this image would mean that we want to make the distribution of the intensity values uniform, generally making the plots above look linear as the broken line shown in the plots.

#linear adjuster function

def rgb_adjuster_lin(image):



target_bins = np.arange(255)

target_freq = np.linspace(0, 1, len(target_bins))

freq_bins = [cumulative_distribution(image[:,:,i]) for i in

range(3)]

names = ['Reds', 'Greens', 'Blues']

line_color = ['red','green','blue']

adjusted_figures = []

f_size = 20



fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

ax.set_title(f'{names[n]}', fontsize = f_size)

ax.imshow(adjusted_image, cmap = names[n])

adjusted_figures.append([adjusted_image])

fig.tight_layout()

fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

freq_adj, bins_adj = cumulative_distribution(adjusted_image)



ax.set_title(f'{names[n]}', fontsize = f_size)

ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')

ax.plot(target_bins,

target_freq,

c='gray',

label='Target CDF',

linestyle = '--')

fig.tight_layout()

return adjusted_figures rgb_adjuster_lin(marty)

Linearly adjusted image

Corresponding normalized pixel intensity values

The final adjusted image can then be generated by the following codes

target_bins = np.arange(255)

target_freq = np.linspace(0, 1, len(target_bins))

freq_bins = [cumulative_distribution(marty[:,:,i]) for i in

range(3)]

adjusted_figures = [] for n in range(3):



interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[marty[:,:,

n]].astype(int))

adjusted_figures.append([adjusted_image])

channel_figures = adjusted_figures imshow(np.dstack((channel_figures[0][0],

channel_figures[1][0],

channel_figures[2][0])));

Suddenly, the image had more detail and color!

Here’s the original image for reference

See the difference?

Aside from a linear behavior, we can also try out different probabilistic distributions to apply to our image. For this case, we’ll try out a Gaussian distribution, as well as a Sigmoid one.

Gaussian adjustment:

#Gaussian adjuster

gaussian = norm(128,64)

def rgb_adjuster_gauss(image):



target_bins = np.arange(255)

target_freq = gaussian.cdf(np.arange(255))

freq_bins = [cumulative_distribution(image[:,:,i]) for i in

range(3)]

names = ['Reds', 'Greens', 'Blues']

line_color = ['red','green','blue']

adjusted_figures = []

f_size = 20



fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

ax.set_title(f'{names[n]}', fontsize = f_size)

ax.imshow(adjusted_image, cmap = names[n])

adjusted_figures.append([adjusted_image])

fig.tight_layout()

fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

freq_adj, bins_adj = cumulative_distribution(adjusted_image)



ax.set_title(f'{names[n]}', fontsize = f_size)

ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')

ax.plot(target_bins,

target_freq,

c='gray',

label='Target CDF',

linestyle = '--')

fig.tight_layout()

return adjusted_figures rgb_adjuster_gauss(marty)

Gaussian RGB

Gaussian distribution adjusted values

We can change some part the code from the linear adjustment to extract the gaussian adjusted image

target_bins = np.arange(255)

target_freq = gaussian.cdf(np.arange(255))

freq_bins = [cumulative_distribution(marty[:,:,i]) for i in

range(3)]

adjusted_figures = [] for n in range(3):



interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[marty[:,:,

n]].astype(int))

adjusted_figures.append([adjusted_image])

channel_figures = adjusted_figures imshow(np.dstack((channel_figures[0][0],

channel_figures[1][0],

channel_figures[2][0])));

Gaussian adjusted image

Sigmoid adjustment:

#Sigmoid adjuster

from scipy.stats import logistic

def rgb_adjuster_sigmoid(image):



target_bins = np.arange(255)

target_freq = logistic.cdf(np.arange(255))

freq_bins = [cumulative_distribution(image[:,:,i]) for i in

range(3)]

names = ['Reds', 'Greens', 'Blues']

line_color = ['red','green','blue']

adjusted_figures = []

f_size = 20



fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

ax.set_title(f'{names[n]}', fontsize = f_size)

ax.imshow(adjusted_image, cmap = names[n])

adjusted_figures.append([adjusted_image])

fig.tight_layout()

fig, ax = plt.subplots(1,3, figsize=[15,5])

for n, ax in enumerate(ax.flatten()):

interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[image[:,:,

n]].astype(int))

freq_adj, bins_adj = cumulative_distribution(adjusted_image)



ax.set_title(f'{names[n]}', fontsize = f_size)

ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')

ax.plot(target_bins,

target_freq,

c='gray',

label='Target CDF',

linestyle = '--')

fig.tight_layout()

return adjusted_figures rgb_adjuster_sigmoid(marty)

Sigmoid RGB

Sigmoid distribution adjusted values

We can’t really see much of the image already when applied with the Sigmoid function. Let’s see how this works out

target_bins = np.arange(255)

target_freq = logistic.cdf(np.arange(255))

freq_bins = [cumulative_distribution(marty[:,:,i]) for i in

range(3)]

adjusted_figures = [] for n in range(3):



interpolation = np.interp(freq_bins[n][0], target_freq,

target_bins)

adjusted_image = img_as_ubyte(interpolation[marty[:,:,

n]].astype(int))

adjusted_figures.append([adjusted_image])

channel_figures = adjusted_figures imshow(np.dstack((channel_figures[0][0],

channel_figures[1][0],

channel_figures[2][0])));

Guess they ran past 88 on this one

Hmmmm. There seems to be no detail seen here on the Sigmoid one. That could be expected since the intensity values of a lot of the pixels are at max value. This means there is no difference between intensities of pixels to show any detail at all.

4.Contrast Stretching

Unlike Histogram Manipulation, Contrast Stretching rescales the intensities of each color space that are within the target percentile range. The resulting image only has specific elements enhanced, and not the entire canvas. This can be beneficial for those who wish to emphasize only certain parts of an image.

Let’s show the impact of contrast stretching by putting in a high contrast image. To do that, we’ll be converting Marty and his friends into a grayscale image with the following code

from skimage.color import rgb2gray dark_marty = rgb2gray(marty)

imshow(dark_marty)

Look at that sweet black and white tones

Now we change the contrast. We’ll be using skimage.exposure.rescale_intensity to perform contrast stretching

from skimage.exposure import rescale_intensity

from skimage import img_as_ubyte

dark_marty_intensity = img_as_ubyte(rgb2gray(dark_marty)) #stretching the values between the 5th and 95th percentile

dark_marty_contrast = rescale_intensity(dark_marty_intensity,

in_range=tuple(np.percentile(dark_marty_intensity, (5, 95))))

imshow(dark_marty_contrast);

The image looks sharper now

Congratulations! Now you have a better idea of how image enhancements work.