. The popular method for addressing these deficiencies is through the
use of large multi-level caches, which attempt to increase performance
by leveraging temporal and spatial locality characteristics of the
application. However, since many scientific applications lack such
localities, cache technologies can deleteriously affect performance
while occupying a significant fraction of the available chip real estate
and increasing power consumption. Another approach for obtaining
high-performance on conventional systems is through the use of
superscalar microprocessors that attempt to dynamically extract
instruction level parallelism from the user code. This methodology can
have limited performance benefits if the underlying application is not
expressed in a manner amenable to fine-grained low-level parallelism. In
addition, these techniques increase design complexity and power
consumption.

**Motivation:** (2) technology drivers -- consumer-level systems drive
technology due to their large volume and high-profit margin (then
scientific and engineering). Media processing such as video processing,
speech rec, and 3D graphics will soon dominate the processing cycles in
computer-based systems \[reference 11 of christofois thesis\]. We at the
scienfic computing end of the spectrum need to check out whats going on
and leverage that technology where possible and influence it to make
changes to help benefit scienfic dudes.

\(1\) Failure of cache based systems (see crago -- pim-based multiproc
paper) . Moore law, gap...

\(3\) Mani class paper -\> histrorically embedded multimedia processors
have been handled by custom-designed ASIC's, but high cost of relatively
slow design cycles of custom ASIC design is a problem.... See manis
paper...

say something about floorplan and power consumption: If current trends
are not reversed then the physical size of supercomputers and their
power consumption are going to be limiting factors in procurements. The
cost of constructing or leasing sufficient machine room space and the
ongoing operational costs of powering and cooling future supercomputers
cannot be ignored.

The tight integration of logic and memory in PIM-based architectures can
significantly reduce power and cooling requirements and allow for higher
spatial densities. (ldrd?)

Since execution units are relatively cheap, duplicating them is a very
attractive idea. However the question is how to feed them from the
memory units, we explore two technologies, vector processing (takes
vectors as input operands and performs the same primitive operation on
each element of the vector) and stream processing similar to vector
processor, but can perform complex operations (or subroutine) on the
vectors (now called streams).

Vector and stream processor do the job well when there's enough
parallelism to exploit, because data parallel instructions have no
dependencies almost by definition, it obviates the need for complex
control hardware require to detect and prevent data dependent
instructions from violating program behavior. It is noted that its
suitable for certain programs.

**Imagine Issues**:

**Negatives:**

If a record is needed multiple times during the course of a kernel
(example neighboring elements in an unstructured grid computation), it
needs to be read from memory repetedly, wasting memory bandwidth
(imagine has a difficult time extracting performance for kernels that
keep a large number of persistent intermediate values)

A record which is read multiple times in one of more index loads will be
replicated in the SRF

The overhead of point chasing using index streams will increase as the
depth of the pointer chain increases -- for each level an index load
from mem to srf is required

For applications where records have variable sizes, the lock-step
execution of the SIMD architecture is limited by the longest record (or
the longest required record comp).

In unstructured mesh computation: irregular record size due to varying
number of neighbors per node. Also node visited several times in the
course of computation by its neighbors, therefore requiring multiple
reads from main memory

Scientific codes often exhibit a large degree of data parallelism and
may be good candidates for SIMD streaming applications. Although does
not work well with frequent data reuses nad irregular data access

Intercluster communication may serialize the computation rate

**Performance characterization**:

Break down time into: Loading streams into SRF, loading micro-code, and
kernel execution (micro-code loading overhead only significant for short
streams and/or computationally non-intensive kernels)

**Attributes of Stream programs** (multimedia)

High computation rate, high comp to mem ratio, producer-consumer
locality with little global data reuse, Parallelism instruction (many
ops concurrently for each data element), data (multiple data records
processed concurrently), task level (multiple streams can be pipelined
or work can be divided across several stream processors). Can be applied
to applications with complex control and irregular data access patterns.

Stream programming model (spm): the data primitive is a stream: an
ordered set of data of an arbitrary (but homogenous) data-type.
Operations in the spm are expressed as ops on entire streams ex.
Load/store from mem, stream transfers over mult-node network,
computational kernels

Kernels perform computation on entire streams, by applying a function to
each element in the stream in order. Take one or more streams as input
and generate one or more streams as output.. Kernels cannot make general
memory references, and are limited to the LRF and intercluster comm.
Kernels operate only on local data. Advantage speed cause kernel data
always close to functional units, but limited application space. .
Kernel microcode stored in microcontroller

Streams can be constructed from other streams through append, truncate
or extract operations. A stream's elements can also index into another
stream

Peak 20 GOPS 32-bit floating point.

Overlaps stream operations -- exploits latency tolerance

Two levels of programming model level -- streaming and kernels. Streams
control data flow, kernels control computation

3 level mem hierarchy: main mem for large 2.1GB/s, infrequently accessed
data 25.6 GB/s, intermediate level for capturing on-chip locality of
data (working set) 435GB/s, local level for temporary use during
computation.

SRF -- 128KB. SRF connected to 8-SIMD-controlled VLIW arithmetic
clusters controlled by a single microcontroller, a mem-system interface
to off-chip DRAM, and NI to connect to other nodes of a multi-Imagine
system. All modules are controlled by an on-chip stream controller under
the direction of an external host processor.SDRAM clock 133 MHz

Stream-load stores occur between mem system and SRF, network
sends/receive occur between NI and SRF. Only contiguous blocks of Mem is
the SRF

8clusters fed by local reg file (distributed). The stream level is
handled by a stream scheduler and the kernel is handled by a kernel
scheduler.

Ideal processing long streams of independent data elements (same
functional applied to each element in stream, no element is dependent on
another element).

Conditional input: only fetch next elem in data stream if some local
cond is true,

Conditional output: onlt append data to output steam if local condition
relating to the data elem is true. Allows streams to be processed
efficiently on the same data type. Allows merging of streams. Allows
limited load balancing, when an additional element can be read in while
other clusters are processing others (so long as computation allows this
patterns of processing). Can reorder the data, requiring an additional
pass (kernel) to restore the correct order

Stripmining necessary -- so data can fit in SRF. (data processed in
batces). Ideal: long enough for kernels to achieve high efficiency,
short enough to capture SRF consumer-producer locality (without spilling
to memory)

Short streams bad for kenerls cause: kernels have prologue and epilogue
blocks of code that are run once per kernel exec, software-pipelining is
critical for achiving high rates on kenerls -- but short streams don't
amortize the software-pipelining overhead

Stream scheduler coordinates both the execution of the kernels and the
movement of streams in order to minimze program execution time. Stream
scheduler addresses 3 issues: allocation of SRF, exploint c-p locality
to avoid off-chip bandwidth, (3) maximize concurrency

Double buffering: if stream is too large to fit in SRF: while kernel is
reading stream from SRF, next part of stream is being loaded into the
SRF. Once kernel finished, it can start reading part 2, while part 3 is
coming in from main mem. (streams are atomic -- if it is being written
into SRF, it cannot be read from until finished)

Software pipelining (both in kernel and stream)

Strip-mining is not automatic, cause kernels have states that are
required by streams and cannot be arbitrarility subdivided. Scheduler
suggests to programmer ideal SRF usage and the programmer can manually

Imagine arch realizes thaht there is typically low reuse of data
(temporal locality) in typical media applications -- large data sets w/
high demand on memory bandwidth. Proposes a hierarchy where the bdwith
is served from where its least expensive, while the expensive external
memory access is minimized

SRF -- similar to cache, difference is it can read many words in same
cycle, since the data access is guaranteed to be sequential. When a
stram is read from main mem, only useful data is read and in a
particular order so SRF is not wasted for unuseful fetches.

\-

**Vector issues**:

Cross pipe/lane communication

Cannot take advantage of consumer-producer locality

Conditionals: Is iram performance based on VL-time (vector-length - same
as unconditional statements) or is it Density time?

Chip has virtual processor width (vpw) register, which sets the size of
data types that are operated on. 6,4 GB/second (or 32 bytes/cycle). The
number of vector lanes doubles as the data size halves, with 4 vector
lanes for 64 bit data, etc... Each lane contains 2 floating point units
(ine 1 can do 64-bit floating point). Single 8 KB register file with 32
registers is partitioned across the lanes. MIPS proc issues single
instruction at a time to vector unit, and lanes work together to execute
same instruction in SIMD mode. VIRAM uses delayed pipeline model to
carefully match the speed of the memory system.

32 Byte data path between processing units and DRAM, the DRAM is
partitioned into two wings each with 4 banks.

32 general-purpose registers each holding up to 32 64-bit elemtns

All vector functional units have multiple parallel data paths to process
multiple vector elements per cycle. 2 int and 1 Fp unit -\> peak
1.6/3.2/6.4 gops for 64/32/16-bit data and 1.6Gflop on 32 bit floting
point data

Strided access: only 4 address generators (factor of 2 for 32-bit) and
increased probability of dram conflicts

Vector processors have been expensivem largely because it for a
performance driven niche market, and because of exotic packaging
designed for supplying high bandwidth frfom the memory to the proc.

Vectorizing compilers well researched.