# Presentation: 586932

## Clustered Display AG

- Chris Willing
- QPSF
- willing@itee.uq.edu.au

## Today ...

- Larger display systems, not smaller
- Traditional display layouts
- Distributed Multihead X (DMX)
- Clustered display layouts
- Single Board Computers (SBC)
- Construction
- Uses

**Notes:**
In the early days of the Access Grid, the standard display configuration consisted of 3 projected outputs along with an operator's console output. In more recent times there has been some movement to smaller single machine AG nodes. However, today I'll be discussing methods of enlarging the graphics capabilities of an AG display, rather than reducing them.

Firstly we'll look at some traditional display configurations and the limitations they can impose on really large displays.

Then we'll look at DMX as a way to generate a display from a cluster of computers and some layouts which this enables.

Single board computers will be introduced as the basis of constructing a display cluster.

Finally some examples will demonstrate the utility of the larger displays which are possible with this technique.

## Normal Display Layout

- (all PCI bus)

- Projected displays

- Console

- PCI bus

**Notes:**
The original design of an Access Grid room specified four video outputs, made up of three projected displays and an operator display. Initially, this was most easily achieved using a Matrox G200, a single four output PCI plugin card. This was what was used in our first all-Linux node, built for SCGlobal 2001. It was made possible by a new feature at that time, named Xinerama, of XFree86 (then the dominant graphical window system for Linux). Xinerama is the feature which allows a single desktop to be spread across multiple graphics card outputs.

One of the original reasons for having such large displays in an AG node was the large number of video streams possible in an AG session. Three streams from each of ten sites require a lot of screen real estate.

In time, it became apparent that the bandwidth of the PCI bus was something of a hinderance to smooth perfrormance. In fact the AGP bus had already been developed because the PCI bus was considered too slow for even single monitors, let alone multiple ouput displays.

## Bus speeds

- PCI					   133 MHz
- AGP 1x			   266
- AGP 2x			   532
- AGP 4x			1.05   GHz
- AGP 8x			2.1
- PCI-Express 16x	4.0 (8.0) GHz

**Notes:**
This table shows speeds of  various bus architectures.

## Normal Display Layout

- (mixed PCI & AGP)

- Projected displays

- Console

- PCI bus

- AGP bus

**Notes:**
Node builders started to use various mixes of AGP and PCI cards in an attempt to take advantage of the higher throughput of the AGP bus. 

## Matrox QID Display Layout

- Projected displays

- Console

- AGP bus

**Notes:**
Recently an AGP card with 4 outputs has become available (Matrox QID).

## Slide 7

**Notes:**
At the same time it had become apparent that these large displays, built initially for the AG, were also very useful as displays in their own right.

More and more powerful computers were generating more and more output, much of which is best comprehended graphically. Sometimes only very large displays are capable of displaying such large ouput data. This image shows Bernard Pailthorpe working on a rear  projected 3x1 display at the Sydney Vislab. Since this was also our experimental AG node at the time, audio was somewhat less highly configured than usual ($20 speakers).

What happens if we want add to the number of display outputs e.g. a 3x2 array. 

## Extended Display Layout

- Projected displays

- Display  Video  Audio

- AGP bus

- PCI bus

**Notes:**
One method is to just add more output cards, but this inevitably means using more and more of the slower PCI cards, since there was only one AGP card per computer 

## Slide 9

**Notes:**
Its this mixed bus approach that we've taken for a rear projected 3x2 projector array we've built at the UQ Vislab.

## number of card slots per machine
(esp. AGP slots)
bandwidth imbalance between AGP/PCI
(when sufficient slots exist)

=> multiple machines?

- (esp. AGP slots)
- bandwidth imbalance between AGP/PCI
- (when sufficient slots exist)
- => multiple machines?

- Single machine limited by

**Notes:**
However, the number of display ouputs available in a single machine is ulimately limited by the number spare slots for video cards. Even if ordinary PCI bus cards are included, the difference in bandwidth between the AGP and PCI bus detracts from the performance.

Since there is limit on the number of video outputs from a single machine, would it be possible to combine the outputs of multiple machines? The answer is yes and, more importantly, it is now built in as  part of the X windowing system shipped with every distribution of Linux; it is named Distributed Multihead X (DMX). 

While DMX allows (in principle) AG nodes with as many screens as may be desired to be built, a standard node would look like this:

## Room node with DMX

- Projected displays

- Console

- Display  Video
- +Audio

- AGP bus

- AGP bus

**Notes:**
In this case dual head cards in two of the node machines are being used to provide the display output.

## Room node with DMX

- Projected displays

- Console

- Display  Video
- +Audio

**Notes:**
This is how a fully clustered node would look like; each display is provided by a separate computer.

## DMX
enables OpenGL across all tiles
Chromium + DMX also possible

Xinerama
OpenGL only on primary display

- enables OpenGL across all tiles
- Chromium + DMX also possible
- Xinerama
- OpenGL only on primary display

**Notes:**
As well as output scalability, DMX solves another problem; the use of OpenGL graphics. A problem with the old system of multiple output cards in a single machine was that accelerated OpenGL graphics were only possible  in the "primary display" i.e. only in one of the graphics cards.

## Open GL with Xinerama

- AGP Head 1

- AGP Head 2

- PCI

- PCI

- Projected

- Console

**Notes:**
That would lead to situations where only part of of the projected display could be used for OpenGL applications, such as Performer.

On the other hand, in a clustered display, accelerated graphics performance is available in all of the output screens.

## Stereo room node with DMX

- Projected displays

- Console

- Display  Video
- +Audio

**Notes:**
Furthermore, if the graphics card in each machine has dual outputs, then the possibility of stereo projection arises. Thus, some or all of the display outputs could be 3D.

## GeoWall/AGAVE augments AG, not integral

- GeoWall Image by Brenda Lopez,
- Electronic Visualization Laboratory,
- University of Illinois at Chicago

- _[www.evl.uic.edu.au/cavern/agave/docs/Introduction.htm#geowall](http://www.evl.uic.edu.au/cavern/agave/docs/Introduction.htm%22%20%5Cl%20%22eowall)_

**Notes:**
This is somewhat similar to the AG Augmented Video Environment (AGAVE), the main difference being that the stereo displays  are integral to the AG node, rather than separate entities which just happen to be in the same physical room.

## Room node with DMX

- Projected displays

- Console

- Display  Video
- +Audio

**Notes:**
Building an AG node with a clustered display can be quite simple - just take 4 PC's (only 2 PC's if using dual output cards), configure them to run DMX and install the AG toolkit.

## Construction Problem

- 20 display cluster => 20 PC's
- integrated graphics not A-grade
- add on cards => 4RU per PC
- i.e. nearly 2 complete racks

**Notes:**
However, some physical space issues arise if we want to build a display with, say,  20 displays, rather than 4. The actual housing of 20x PC's  is itself a problem. Rack mount cases will require almost two full height racks if normal construction methods are used. Graphics chips built into motherboards are usually not of sufficiently high quality to provide accelerated or stereo output.

 This rules out the possibility of using 1RU rack mounted computers with commodity motherboards since it will almost certainly be necessary to use additional plugin graphics cards. Addition of suitable graphics cards into  conventional motherboards means that 4RU cases will be required.

## Single  Board Computer (SBC)

- PCI Industrial Manufacturers Group
- Passive backplane for PCI, ISA
- CPU on plug in card, not motherboard
- New PCI, ISA, AGP format (PIAGP)
- enables plug in AGP cards

**Notes:**
An alternative construction method, which minimises space requirements, uses Single Board Computers (SBC's). In normal construcions, the motherboard is densely populated with the bulk of components necessary for operation, but containing slots to accommodate plugin boards to perform additional functions.

In SBC constructions, what was the motherboard is now itself just a plugin board, which we know call a CPU card. It plugs into a passive backplane, which is essentially just an array of slots for various plugin boards.

These backplanes also have a variety of bus architectures and the PCI Industrial Manufacturers Group (PICMG) defines specifications for industrial PCI/ISA passive backplane and CPU card interfaces. Various backplanes are available, some of which can even host multiple plugin CPU cards. These backplanes have always been for PCI and ISA cards.

## AGP card with 2RU passive backplane

- 2RU backplane with CPU card

**Notes:**
More recently, a variation on this specification, known as PIAGP (for PCI, ISA, AGP), has become available. It features a new ability to also accomodates AGP plugin cards.

## Slide 21

**Notes:**
Another difference, shown in these examples, is that the passive backplane can be made double sided. With the double sided backplane, the plug in boards are 180° opposed instead of being at right angles.

## 1RU version

**Notes:**
Now, a backplane with a width of just 1RU can host a CPU board from one side, together with an AGP graphics card from the other side. 

## 1RU vs 4RU

**Notes:**
This results in a solution which is only a quarter of the usual size (1RU vs 4RU).

## New size calculation

- 20 display cluster => 20 SBC's
- 1RU backplane holds CPU + AGP cards
- New size = 20RU
- 1⁄4 of previous usage
- saves 1 1⁄2 racks
- For AG, need video capture
- at least one 2RU SBC
- i.e. 21 RU total

**Notes:**
The imagined 20 screen cluster, which would have required nearly two racks of equipment, now need less than half of a rack.

In reality, additional plugin cards may be needed for video capture, unless USB or Firewire cameras are chosen. Additional PCI cards can be accommodated using 2RU backplane and case for one of the machines.

## Empty chassis with card cage

**Notes:**
These images show some of the construction of a 4 machine cluster using 2RU backplanes.

In the empty chassis, note the support bracket for the passive back plane.

## Backplane with CPU & AGP cards

- in mounting cage

**Notes:**
The backplane with CPU and VGA cards are assembled in a cage which drops into the chassis.

## Loading OS with temporary CDROM drive

**Notes:**
The OS is installed with a temporarily connected CDROM drive.

## Completed cluster in rack

**Notes:**
The final result.

## Minor differences

- Logins
- don't want 20 separate logins
- KVM would need VGA splitter (x20)
- Networking
- mostly on private network
- only media Producers/Consumers need external network

**Notes:**
Apart from the construction method itself and the superior graphics performance available, there is little else different to a traditional multi-machine node. Some of these differences are
Logins: DMX requires a user to be logged in under X when it starts. This is similar to the situation in normal multi-machine nodes. However when the number of machines in the cluster becomes larger, say 20, then the requirement to switch to each machine to login is extremely tedious.

## VGA Splitter

- from PC

- to KVM

- to projector

**Notes:**
It also means that 20 VGA splitters would be needed, since  each PC's VGA outputs must feed into the KVM as well as a projector.

For these reasons, automatic login to a generic user account is enabled at boot time on all the cluster machines. Since all the machines are housed in a machine room, this is not considered to be a security risk.

## Network Layout

- Switch

- Internal network

- Switch

- External network

- Audio

- Video

**Notes:**
Networking: all machines in the display cluster are connected by an internal gigabit network. Only machines running service producers or consumers need access to the external network.

## Additional network & video capture cards

**Notes:**
The CPU cards have built in gigabit and 100BaseT network interfaces. If gigabit speed is required for connections to both the internal and external networks, then an additional gigabit plugin card is needed. The 2RU backplane, with its additional PCI sockets, is required for these additional cards.

## Uses

- Obvious
- Mainly hi-res images
- Accelerated 3D graphics, models
- Stereo images
- Non-obvious
- compute cluster?

**Notes:**
Apart from accelerated graphics and 3D projection, the most obvious use of clustered displays is for high resolution images. Using 1280x1024 projectors, a display made up of 3 projectors in a "traditional" AG node enables images of up to 3840x1024 to be viewed without scaling. 

A less obvious use for clustered display is that of a small compute cluster. A display cluster of 20 machines can provide a moderate amount of compute power along with its display capability.

## What hi-res images?

- AG Map
- _[http://www.ap-accessgrid.org/agmap/](http://www.ap-accessgrid.org/agmap/)_
- IVS map is 22852x783
- Remember AG as interface to "the grid",
- (not just conferencing on steroids)
- GridMultiplier example:
- APAG->Service Lobby->Grid Multiplier Room
- real apps will generate huge results
- results need to be visualized