Engineer-to-Engineer Note

EE-358

Technical notes on using Analog Devices DSPs, processors and development tools
Visit our Web resources http://www.analog.com/ee-notes and http://www.analog.com/processors ore-mail processor.support@analog.com or processor.tools.support@analog.com for technical support.

Interfacing Red/Clear Sensors to ADSP-BF609® Blackfin Processors
Contributed by Gopal Karanam

Rev 1 – May 28, 2013

Introduction
TheADSP-BF609
dual-core
Blackfin®
processor series is optimized for embedded
vision and video analytics applications, such asautomotive advanced driver assistance systems(ADAS), machine vision and robotics forindustrial manufacturing systems, and security
and surveillance system analytics. This Engineerto-Engineer Note discusses the advanced features
available for interfacing application specificcameras to the ADSP-BF609 Blackfin processor.
Vision cameras have photosensors that convert
light intensity into an electronic signal. Typicalphotosensors have little or no sensitivity to
wavelength of incident light and therefore cannot
separate color information. In order to capturecolor information, a mosaic of tiny color filters isplaced over the pixel sensors of an image sensor.
One of the most common color filters used inconsumer cameras is the Bayer filter, which
gives information about the intensity of light inred, green and blue wavelength regions. One
possible arrangement of the color filter is shownin Figure 1. The raw image data captured by theimage sensor is then converted to full-color
image by a demosaicing algorithm withintensities of all three primary colors (red, green,blue) represented at each pixel. Naturally, thedemosaicing algorithm has to be tailored for eachtype of color filter.

Figure 1. Bayer Color Filter Array (CFA)

Vision based ADAS use multiple cameras tocapture the scene around the car to assist during
the driving process. Typical machine visionalgorithms use only the intensity of the light(Luma). Hence, sensor manufactures produce
monochrome sensors exclusively for ADAS
applications. But, in order to meet stringent
quality requirements the application needs accessto some color information as well. One of thekey color information is the red channel, which
will help localize the region of interest (ROI) ofthe image, such as traffic light, car rear-light, etc.
For example, Traffic Sign Recognition (TSR)
applications can identify possible locations of redcolored speed limit signs in the captured image.
Hence, sensor manufacturers have designed
Red/Clear (RCCC) color filter array specificallyfor automotive use. Unlike Bayer sensors, RCCC
sensors use clear filters instead of the blue andthe two green filters in the 2x2 pixel
neighborhood. The red filter remains the same. A“clear filter” is the same concept as monochrome

Copyright 2013, Analog Devices, Inc. All rights reserved. Analog Devices assumes no responsibility for customer product design or the use or application ofcustomers’ products or for any infringements of patents or rights of others which may result from Analog Devices assistance. All trademarks and logos are property of
their respective holders. Information furnished by Analog Devices applications and development tools engineers is believed to be accurate and reliable, however noresponsibility is assumed by Analog Devices regarding technical accuracy and topicality of the content provided in Analog Devices Engineer-to-Engineer Notes.

sensors. Hence, the output generated by RCCC
sensors is almost as detailed as a monochrome
output and still provides the red color
information as shown in Figure 2.

The image has red, green, and blue values foreach pixel and is converted to reference
monochrome image using Equation 1:
𝑌 = 0.59 ∗ 𝐺 + 0.3 ∗ 𝑅 + 0.11 ∗ 𝐵

Equation 1. Conversion to Monochrome Image

Figure 2. Red/Clear Color Filter Array

A Red/Clear color filter is simulated by replacing
1(one) luma pixel out of 2x2 pixel region in thereference monochrome image with a red pixel.
The demosaicing algorithm will then generate
the missing clear pixel. The resulting image iscompared visually as well as using peak-signalto-noise ratio (PSNR) metric.
Legacy De-Mosaicing Method

De-Mosaicing Algorithm
The red pixels form a quarter (width/2 xheight/2) of the full resolution of the sensor. On
the other hand, the clear pixels form 3/4th of thesensor resolution. In order to generate the fullresolution clear or monochrome pixels, a demosaicing algorithm has to generate the missing“C” pixels at “R” locations, commonly referred
to as the red pixel substitution.

The clear pixel is generated by a convolution
operation. The filter kernel is only applied at “R”
locations to compute the missing “C” pixel. A
popular method to compute the missing clear
pixel is to use the mean of the neighboring clear
pixels. The filter kernel for such an algorithm isshown in Figure 4.

1/8 1/8 1/8

The quality of the algorithm can be estimated bysimulating a Red/Clear color filter, applying thede-mosaicing algorithm and then comparing
against the reference image. A test image shownin Figure 3 with lots of detail in color, texture
and edges is chosen for analysis.

1/8

1/8

1/8 1/8 1/8

1/4
1/4

1/4
1/4

Figure 4. 8-Pixel and 4-Pixel Averaging Kernel

Proposed Method

Figure 3. Test Image[2]The legacy methods discard the red value at thered pixel locations. It has most valuable
information and can be used to correct the meanvalue of the computed clear pixel. The estimateof the red pixel can be computed from theneighboring red pixels, and the difference withthe center value represents the luminance change
at the location. A fraction of this estimate error
can be added to the mean of the clear pixels as
described in Figure 5.

Interfacing Red/Clear Sensors to ADSP-BF609® Blackfin Processors (EE-358)

Page 2 of 2

Filter Kernel

-1/4

st1/4
1/4

1/4+ ζ * -1/4

1-1/4

1/4
-1/4

Figure 5. 4-Pixel Averaging Kernel & Correction Term

The parameter zeta (ζ) controls the amount ofcorrection. Zeta is varied from 0 (4-clear pixelaveraging kernel) to 1 in steps of 0.05. The
PSNR of the resulting images is plotted in
MATLAB as shown in Figure 6.

PSNR (dB)

Mean (1 kernel in Figure 4)

43.4

Mean (2nd kernel in Figure 4)

44.4

Mean + Correction term (Figure 7)

48

Table 1. PSNR table for different demosaic kernels

The visual quality improves as shown inFigure 8.

Figure 8. Section of De-Mosaiced image using mean
kernel (Top) & optimal kernel (bottom)
Figure 6. PSNR Plot of De-Mosaiced Images

It is clear from the graph that adding a correction
term improves PSNR significantly and it peaks
around zeta equals 0.5. The optimal kernel isshown in Figure 7.
-1/8
1/4-1/8 1/4 1/2 1/4 -1/8
1/4-1/8

Figure 7. Optimal Red/Clear De-Mosaicing Kernel

Implementation of the De-Mosaicing
Algorithm on the ADSP-BF609 Blackfin
Processor
The Pipelined Video Processor (PVP) hardware
engine on ADSP-BF609/8 derivatives[5] isdesigned to process images from Red/Clear color
filter sensors.
The PVP contains four Convolution blocks witheach of them supporting any arbitrary 5x5coefficient matrix. One of the Convolution
blocks, CNV1, provides a unique feature, which
performs convolution operation, shift andsaturate only at the red pixel location. Theconvolution and normalization coefficients arefreely programmable.

Table 1 demonstrates that adding the correction
term results in a PSNR improvement of 3.6dB:

Interfacing Red/Clear Sensors to ADSP-BF609® Blackfin Processors (EE-358)

Page 3 of 3

The following parameters are proposed for thediscussed de-mosaicing algorithm:
0
0Coeff CNV 1 =  − 1

0
0


0 −10 22 40 20 00 0
2 − 1
0 0
0 − 1 0 0 The output is normalized by setting thePVP_CNV1_CTL.SHIFT value to 3. If the output
needs to be clipped between a particular range,
then the Threshold-Histogram-Compression(THCx) block can be used.
The quarter resolution red image can beseparated using the Input Formatter IPF0. Color
separation grants concurrent access to multiple
color components. It is enabled by setting thePVP_IPF0_CTL.BFRMT0 bit. Color separation mode
supports two arrangements of color filter arrays
as controlled by PVP_IPF0_BFRMT1 bit. The two
types are shown in Figure 9.

Figure 9. Color Separation for Type 1 & 2In color separation mode, IPF0 output port 2outputs the red pixels.
Naturally, the PVP performs the operation on incoming data real time, without consuming any
MIPS (Millions of Instructions Per Second) on
the Blackfin cores, available for further signal
processing tasks.

References[1] Wikipedia (http://en.wikipedia.org/wiki/Color_filter_array). May 2013.
[2] Sigma SD1 Merrill Thumbnails (http://www.imaging-resource.com/PRODS/SD1/SD1THMB.HTM). May 2013.
[3]

H.S. Malvar, L.-W. He, and R. Cutler, High-quality linear interpolation for demosaicing of Bayer-patterned color
images, IEEE Int Conf (ICASSP '04), Proceedings on Acoustics, Speech, and Signal Processing 3, 2004, 485-8.

[4] ADSP-BF60x Blackfin Processor Hardware Reference. Rev 0.5, February 2013. Analog Devices, Inc.
[5] ADSP-BF606/607/608/609 Blackfin Dual Core Embedded Processors Data Sheet. Rev PrG, April 2013.
Analog Devices, Inc.

Document History
Revision

Description

Rev 1 – May 28, 2013by Gopal Karanam

Initial release.

Interfacing Red/Clear Sensors to ADSP-BF609® Blackfin Processors (EE-358)

Page 4 of 4