ACPMAPS 3 is a TeraFlop-class supercomputer developed for lattice gauge
research at Fermilab. It is based on the successful (patented)
Crossbar-Backplane technology that was used in previous generations of
ACPMAPS. Improvements in processor speed, communications, software, and
architecture give a 40-fold increase in performance over ACPMAPS2.

The machine is modular and scaleable from small 10-processor systems up
to 1-2 thousand processors. It is built, tinker-toy style, from three
basic modules:

Backplanes: 64 slot crossbar backplane. 400MB/sec/slot.

Processor boards: 3 CPUs, 512MB, and bus interface processor.

Links: Parallel optical fiber links between crates. 400MB/sec.

A typical large system consists of 8 crates of processors connected in a
star via a ninth crate. Each Crate has 57 processor boards and 7 links.
The central crate has 56 links to other crates and 8 links reserved for
I/O to terminals and mass storage.

The cylindrical physical architecture allows tight packing of processing
elements with short, simple communication links. Each crate is a
cylinder that is 32 inches in diameter and about 16 inches high. Three
crates are stacked together with a common power supply and cooling to
form a rack. Three racks make an entire system. The system is water
cooled.

**Backplane**

The backplane is a 64-port crossbar switch. It has a 32-bit data path
and 10ns cycle time to give a bandwidth of 400 Megabytes per second per
port. Its unique 3-dimensional physical architecture allows the entire
switch to be put into a cylinder that is 5 inches in diameter and 4
inches tall. CPU cards plug into the port connectors surrounding the
core.

Since the CPU cards radiate from the center of the crate, they are wedge
shaped. They are narrow at the connector and wide at the front panel,
allowing plenty of space for connectors, displays, daughter boards, and
tall chips.

The backplane uses custom crossbar switch chips. Each chip is a 2-bit
wide 64-way switch. Connections are made by specifying a 6-bit source
port, 6-bit destination port, and 2-bit control word. A pipelined
arbiter can make or break a connection every 20ns with a latency from
request to connect of 80ns.

As connections are made or broken, they are broadcast on a common bus to
all 64 ports. A smart controller on each CPU card uses this information
to keep a list of which ports are busy and which are free. The
controller also has a list of pending transfers. By matching pending
transfers against free destination ports, the controller can almost
eliminate arbitration overhead.

**CPU Card**

The CPU card consists of three floating point processors, a
communications processor, and 4 banks of memory. The memory is connected
to the processors via a crossbar switch to allow 4 independent data
paths. The address bus is common to all 4 processors so that cache
coherence can be maintained via bus snooping. Typically, a processor
will transfer entire cache lines which can be as large as 16 transfers.
This means that activity on the address bus will be much slower than
activity on each data bus. This allows all 4 processors to share the
address bus at nearly full speed.

The communications processor is a smart DMA controller that is shared by
the three CPUs. Transfers are queued on a 2-dimensional que that sorts
requests by destination and by thread number. Requests for the same
destination are coalesced into a single transfer. When all requests for
a thread have been satisfied, a message is sent back to the requesting
processor.

If the communications processor is not busy, it will begin a transfer
within about 200ns of a request. If it is busy, it will sort the
transfer into one of its queues.

In addition to managing requests, the communications processor takes
care of system security, memory heap management, and performance
monitoring. With minor modifications, it could also implement Canopy
primitives such as *get_field*, maintain Canopy lattice definitions, or
perform field caching.

The floating-point CPUs will execute the user's code. Ideally, they run
a very simple operating system that doesn't have to do things like
virtual address translation and complicated interrupt and process
management.

**Inter-Crate Links**

Connections between crates are done via Dense Optical Interconnect
Modules (DOIM). These are 10-bit wide parallel optical links that can
run at over 400 Megahertz. Two DOIMs are used to make a bidirectional
link. While data is being transferred in one direction, the bits in the
reverse direction are used for handshake and status.

**Variations**

-   The number of CPUs per card may vary from 1 to 3 depending on board
    space, heat dissipation, and CPU speed.

-   The basic link data rate can be doubled by using a wider data path
    or faster cycles.

-   The number of inter-crate links can vary. Depending on simulation
    results, there may be 24 links and 40 CPU boards per crate.

**Research Efforts** (Things that need on order of a person-year of
effort)

-   Cooling: It is estimated that we will need to dissipate as much as
    6000 watts per crate. This is not a trivial problem.

-   Mechanical: The cylindrical architecture presents a challenge to
    mechanical designers. If it isn't done well, it could be a big
    complaint for the entire life of the machine. If done well, it will
    be a major selling point.

-   Switching: The 64-way crossbar switch, arbiter, and protocol require
    significant design effort.

-   CPU: The CPU and memory system require significant design effort.

-   Communication Processor.

-   Inter-Crate link.

-   Software

-   Software

-   Software

Simulation