Engineer-to-Engineer NoteaEE-301

Technical notes on using Analog Devices DSPs, processors and development tools
Visit our Web resources http://www.analog.com/ee-notes and http://www.analog.com/processors ore-mail processor.support@analog.com or processor.tools.support@analog.com for technical support.

Video Templates for Developing Multimedia Applications on Blackfin®
Processors
Contributed by Kaushal Sanghai

Rev 1 – September 28, 2006

Introduction

Video Templates Overview

Blackfin® processors enable a variety of ways toefficiently manage data for high-performance
applications. This document describes a set of“templates” that can be used to efficiently
manage multimedia data on Blackfin processors.

A video sequence can be regarded as a threedimensional (3-D) signal, comprising twodimensions in the spatial domain (i.e., a single 2D frame) and one dimension in the temporal
domain (i.e., a temporal sequence of 2-Dframes). The data flow of the video stream canthus be partitioned in one of the two dimensions:
1) a temporal dimension, or 2) a spatial
dimension. In the temporal dimension, the data
flow can be managed at the granularity of agroup of pictures (GOP) level or at the framelevel. In the spatial dimension, the data
movement can be managed at the granularity of aline or a macro-block. These various levels ofgranularity in a video signal are called subprocessing blocks. Note that the granularity ofthe sub-processing decreases from GOP level toline level.

The templates provided with this EE-Note canhelp users get started in implementing
customized applications, as they can be modified
to fit specific application needs. The first part of
the EE-Note discusses the types of templates
available, and the second part describes specificexamples for each template.

BackgroundThe size of data buffers involved in multimedia
applications exceeds the processor’s internal
memory space. To take advantage of the lowlatency access of the processor’s on-chip
memory (L1/L2), some example templates havebeen created that can be used as a starting point
to move peripheral data directly into L1 or L2
memory. The tradeoff to consider is between thesize of the memory and the processing
granularity of the image block within anapplication. The templates also exploit thepredictable data access pattern inherent in
multimedia applications to minimize thetransfers required to move data between different
levels of memory, thus improving resourceutilization.

The templates discussed in this document exploit
the temporal and spatial characteristics inherent
in video signals described above. The templates
take advantage of the predictable data accesspattern to hide the memory latencies and reduce
the number of memory transfers between
different levels of memory.
In most image processing algorithms, the data
access pattern of the sub-processing block ispredictable. For example, in JPEG compression,
the macro-blocks of an image are accessedsequentially across the rows. Table 1 shows

Copyright 2006, Analog Devices, Inc. All rights reserved. Analog Devices assumes no responsibility for customer product design or the use or application ofcustomers’ products or for any infringements of patents or rights of others which may result from Analog Devices assistance. All trademarks and logos are propertyof their respective holders. Information furnished by Analog Devices applications and development tools engineers is believed to be accurate and reliable, however
no responsibility is assumed by Analog Devices regarding technical accuracy and topicality of the content provided in Analog Devices Engineer-to-Engineer Notes.

aexamples in which the access pattern of the subprocessing block is predictable.
Example Application

Comments

Color Conversion (e.g.,
YUV to RGB)

The image is accessed line
by lineHistogram Analysis

Image data is accessed line
by lineEdge Detection

Macro-blocks are accessedsequentially row wise

JPEG/Motion JPEG

Macro-blocks are accessedsequentially row wise

Motion Detection

Macro-blocks are accessedsequentially row wise;
dependent on past frameMPEG2/MPEG4

Motion window
sequentially across macroblocks; dependent on pastand future frames

Table 1. Predictable Access Patterns

If the data access pattern of the image is known
in advance, the data can be transferred to L1
memory before it is requested by the core. Thisavoids cycles consumed due to the core being
held off for a memory request. Also, to hide thelatency of the memory transfer, DMA can beeffectively used in the background for any datatransfer. Using DMA instead of cache helps tosave core cycles consumed by cache misses.
To further save system resources, the templates
utilize the smaller and faster L1/L2 memory
spaces by storing image data from the peripheral
into on-chip memory, thus eliminating the delays
that would be incurred during accesses toexternal SDRAM. This applies to algorithms that
operate on finer sub-processing block
granularities (i.e., a line or a macro-block).
The templates are designed such that once theapplication’s sub-processing block is identified
and the real time constrains are met, the image
processing algorithm can be dropped into one of
the templates to obtain an efficient data layout.

This is demonstrated by taking specific examples
for each of the templates.
In certain algorithms, data dependency can exist
between the sub-processing blocks in thetemporal domain (i.e., current frame of reference
and past or future frames). In these cases,
managing data is more complicated. One such
scenario is discussed in the Inter-Frame
Processing template.
The proposed video templates are discussed in
the next few sections, and example applications
are provided to demonstrate the recommended
approach to using the templates.

Proposed Video Templates
In this section, three different templates, basedon the granularity of the sub-processing block,are broadly described. They include the subprocessing blocks in the spatial domain (line andmacro-blocks), as well as templates forapplications where data dependencies exist in thetemporal domain.
Line Processing
For line processing, one approach is to collectone frame at a time in external memory from theperipheral. After each frame is collected, a lineby-line memory transfer can then be performed
to L1 data memory, using either cache or DMA.
A more efficient approach involves transferring aline directly from the peripheral into L1 data
memory and subsequently transferring theprocessed line directly from L1 to a peripheral
interface or external memory1. This decreases thenumber of memory transfers required and saves
valuable external bus bandwidth.
1On ADSP-BF56x processors, two PPIs can be used for
I/O video. On ADSP-BF53x processors, one PPI interface
and one other peripheral interface (i.e., SPORT, Ethernetetc.) can be used for I/O. For sending/reading compressedimages, the USB or LAN interface can be used on ADSPBF56x or ADSP-BF53x processors.

Video Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 2 of 7

aFigure 1 shows the data flow implemented
within the line-based template. DMA is used totransfer the image directly to L1 data memory
from the peripheral. The image is then processedand sent out to the peripheral from L1 via DMA.
Double-buffering is maintained in L1 memory toallow concurrent DMA and core accesses, inaddition to avoiding L1 sub-bank contention.

Figure 1. Data Flow Diagram for Line Processing Template

Macro-Block Processing Template
For macro-block processing, the frame isprocessed in sections of dimension (n x m),
where n is the height and m is the width of themacro-block. The L1 data memory would beinsufficient to place the entire n rows of animage at a time, so either L22 or external
memory must be used. However, to conserve theexternal bus bandwidth, using only L2 memory
will help.

L2 memory would be insufficient to place anentire image, so only n rows of an image aretransferred from the Parallel Peripheral Interface(PPI) to L2 memory at a time. Macro-blocks arethen transferred from L2 to L1 data memory.
Double-buffering is used in L1 and L2
memories, and DMA channels are used for all
memory transfers. The dataflow diagram for thistemplate is shown in Figure 2. Also, in thistemplate, external memory can be used for an
input or output frame buffer to ensure more
uniform use of resources. In Figure 2, either theTx or Rx buffers can be placed in externalmemory.
Inter-Frame Processing Template

Figure 2. Data Flow Diagram for Macro-BlockProcessing Template

2L2 memory is available on ADSP-BF56x processors. OnADSP-BF53x processors, external memory can be used for
placing buffers.

This template can be used for applications inwhich dependencies exist between subprocessing blocks in the temporal dimension(i.e., between past and future frames). The L1and L2 memory spaces are insufficient to place
the dependent frames in memory. Thus, theexternal SDRAM memory is used to map thedependent frames.
Sub-processing blocks of the dependent frames
are then transferred from external memory to L1memory. If L1 memory space is insufficient, L2

Video Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 3 of 7

amemory can be used for placing internal buffers.
The data flow diagram in Figure 3 shows anapplication in which dependencies exist on amacro-block basis over a past frame.

Figure 3. Data Flow Diagram for Inter-Frame Processing Template

InFigure 3,

thecurrent_frame
andpointers are interchanged
every other frame. For applications wheredependencies exist on a line basis, line transfers
can be made between L1 and external memory.

SDRAM bank partitioning: The SDRAM ispartitioned into four banks to ensure
simultaneous access to multiple frame buffers
and minimal turn-around times. For example,
the ADSP-BF561 EZ-KIT Lite® board has
64 Mbytes of SDRAM, which can beconfigured as four 16-Mbyte internal
SDRAM banks. To take advantage of thebank structure, no two frame buffers that areaccessed simultaneously are mapped to thesame SDRAM bank.

DMA traffic control: DMA traffic control
registers are used for efficient system
bandwidth utilization.

reference_frame

Dependencies can also exist between several
sub-processing blocks. In these cases, a slice of aframe (set of lines) can be transferred to internalmemory.
Optimizations Specific to Blackfin Processors
To further improve performance and system
bandwidth utilization, several optimizations
within the templates have been adopted. The
following optimizations are included in thetemplates:

16/32-bit transfers: Maximum bus width isused for all peripheral DMA and memory
DMA transfers. 16-bit transfers are initiatedfor ADSP-BF53x processors, and 32-bit
transfers are initiated for ADSP-BF56xprocessors.
Efficient use of DMA channels: No two
simultaneous memory transfers are initiatedon the same DMA channel.

Refer to Embedded Media Processing[3] andVideo Framework Considerations for Image
Processing on Blackfin Processors (EE-276)[5] tolearn more about developing optimized
multimedia applications on Blackfin processors.

Using the Templates
To use the templates for specific applications,first identify the following items:
The granularity of the sub-processing blockin the image processing algorithm. This isVideo Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 4 of 7

arequired to choose a specific template for theapplication.
The available L1 and L2 data memory, asrequired by the specific templates.

An estimate for the computation cycles
required per sub-processing block. This will
indicate whether the application meets thereal-time constraints within the template.

The spatial and temporal dependencies
between the sub-processing blocks. If
dependencies exist, modify the templates toaccount for data dependencies.

Table 2 lists the core cycles available per subprocessing block for each of the templates. The
table lists the cycles for use with Debug andRelease modes of VisualDSP++® builds. Fewer
core cycles are available in macro-block
processing because it requires additional pointermanipulation. For inter-frame processing,multiple transfers are required between
dependent frames, reducing the available core
cycles as compared to line or macro-block
processing. Note that the cycles shown for interframe processing are only for a transfer of a sliceof a frame (set of lines) to internal memory.

Template

Approx.
CCLK/Pixel(Debug)

Approx.
CCLK/Pixel(Release)

L1 Data MemoryRequired

L2 Data Memory
Required

LineProcessing

36

42

2 * Line Size

N/AComments

Buffer chaining withloopback
Double-buffering in L1

MacroBlock
Processing

30

InterFrame
Processing

30

36

2 * Macro-Block Size
Macro-Block Size = n * m

35

Sub-Processing Block
Size * # of Dependent
BlocksMacro-Block Height * 2 *Line Size

Buffer chaining withloopback(Size of Frame Slice)

Double-buffering in L1
and L2

Sub-Processing Block Size
* # of Dependent Blocks

Only L1 or L2 can be used
Buffer chaining withloopback
Double-buffering in L1 orL2

Table 2. Specification for Each Template

Since most data accesses are managed from L1
memory, the number of cycles lost due tomemory latency is minimized. The cycles shown
can therefore be entirely budgeted for the corecomputation cycles required by an algorithm.
The core processing cycles required for analgorithm should be computed theoretically orobtained by using the cycle counters available on
Blackfin processors. A simulator or emulator
session in VisualDSP++ can be used to obtain
this information.

Table 2 also shows the required data memory
space for L1 and L2 memories and the bufferchaining mechanism used within the templates.
For more information on buffer chaining, refer tothe DMA drivers in the VisualDSP++ Device
Drivers and System Services Manual for Blackfin
Processors[6].

Video Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 5 of 7

aExample Applications
Each of the templates is evaluated with an
example application. Table 3 lists the examples
provided for each of the templates.
Template

Application

Line Processing

Color conversion,
YUV4:2:2 to RGB

Macro-Block
Processing

Edge detection

Inter-Frame Processing

Motion detection

Table 3. Example Applications for Each Template

The templates and the examples for each areincluded in the associated ZIP file [7] for this EENote. The examples are located in the directory:<template_name>/BF561/examples

These examples were tested on the ADSP-BF561
EZ-KIT Lite evaluation board with an input
camera source and output video configured for
ITU-R-656 frame format. For details on ITU-R656 format, refer to the appropriate Hardware
Reference Manual [1,2]. The example programs
operate on a D1-sized (720x480) image.

Combining Templates
Some applications are likely to have more than
one image-processing algorithm with different
sub-processing block granularities or data
dependencies between the sub-processing blocks.
In that case, multiple templates can be combined
into one project. This section discusses thecombination of templates for such applications.
Multiple Sub-Processing Blocks
Consider an application that may involve somepre-processing on a line basis, along with macroblock processing. In this case, the line processing
template and the macro-block processing

template can be combined into one framework.
Line processing should be modified to store theprocessed image to L2 or external memory, andthen the macro-block template should be invoked
to process the stored image.
Multiple Access to a Sub-Processing Block
Consider another example, such as histogram
equalization, where the line processing templateis invoked twice. In this case, the image isaccessed twice⎯once to compute the cumulativegrayscale values, and a second time to apply theequalization. For this application, one line at atime from L1 can be processed, stored back toexternal memory, and then brought back into L1.
Data Dependency in the Spatial Dimension
Data dependencies may exist between several
lines or macro-blocks of an image, and access tomultiple sub-processing blocks at one time may
be required. An example use for this template is
the motion estimation algorithm, which accesses
several macro-blocks.
Dual-Core/Multiprocessor Applications
The templates can be combined to develop
applications for the ADSP-BF561 dual-coreprocessor in a multiprocessor system. Based on
the dual-core programming model, different
templates can be executed on either core.

Conclusions
Managing data efficiently is critical to increaseperformance and improve system bandwidth
utilization on resource-constraining embedded
platforms. To help address this, efficient data
management techniques have been incorporated
within a set of templates for developing
multimedia applications on Blackfin processors.
The templates will help to produce optimized
data layouts and reduce development time.

Video Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 6 of 7

aReferences[1] ADSP-BF533 Blackfin Processor Hardware Reference. Rev 3.2, July 2006. Analog Devices, Inc.
[2] ADSP-BF561 Blackfin Processor Hardware Reference. Rev 1.0, July 2005. Analog Devices, Inc.
[3] Embedded Media Processing. David Katz and Rick Gentile. Newnes Publishers., Burlington, MA, USA, 2005.
[4] Digital Video and HDTV. Charles Poynton. Morgan Kaufmann Publishers Inc., San Francisco, CA,USA, 2003.
[5] Video Framework Considerations for Image Processing on Blackfin Processors (EE-276). Rev 1, September 2005.
Analog Devices Inc.
[6] VisualDSP++ 4.5 Device Drivers and System Services Manual for Blackfin Processors. Rev 2.0, March 2006.
Analog Devices, Inc.
[7] Associated ZIP File. Rev 1, September 2006. Analog Devices, Inc.

Readings[8] ADSP-BF53x/ADSP-BF56x Programming Reference. Rev 1. May 2005. Analog Devices, Inc.

Document History
RevisionDescription

Rev 1 – September 28, 2006
by Kaushal Sanghai

Initial Release

Video Templates for Developing Multimedia Applications on Blackfin® Processors (EE-301)

Page 7 of 7