RPM : A RAPID PROTOTYPING ENGINE FOR MULTIPROCESSOR SYSTEMS1 Luiz A. Barroso , Sasan Iman , Jaeheon Jeong , Koray ? ner , Krishnan Ramamurthy and Michel Dubois Department of Electrical Engineering - Systems University of Southern California Los Angeles , CA 90089-2562 -LRB-213-RRB-740-4475 dubois@paris.usc.edu Abstract In multiprocessor systems , processing nodes contain a processor , some cache and a share of the system memory , and are connected through a scalable interconnect . The system memory partitions may be shared -LRB- shared-memory systems -RRB- or disjoint -LRB- messagepassing systems -RRB- . Within each class of systems many architectural variations are possible . Fair comparisons among systems are difficult because of the lack of a common hardware platform to implement the different architectures . RPM -LRB- Rapid Prototyping engine for Multiprocessors -RRB- is a hardware emulator for the rapid prototyping of various multiprocessor architectures . In RPM , the hardware of the target machine is emulated by reprogrammable controllers implemented with Field-Programmable Gate Arrays -LRB- FPGAs -RRB- . The processors , memories and interconnect are off-theshelf and their relative speeds can be modified to emulate various component technologies . Every emulation is an actual incarnation of the target machine and therefore software written for the target machine can be easily ported on it with little modification and without instrumentation of the code . In this paper , we describe the architecture of RPM , its performance and the prototyping methodology . We also compare our approach with simulation and breadboard prototyping . 1 . This work is supported by the National Science Foundation under Grant MIP-9223812 Future developments on this project will be posted on WWW at http://www.usc.edu/dept/ceng/dubois/ RPM.html To appear in the IEEE Computer Magazine , February 1995 . 1 . INTRODUCTION Multiprocessor systems are becoming common place in the computer industry . The consensus among machine designers is to favor asynchronous MIMD -LRB- Multiple Instructions Multiple Data streams -RRB- systems , in which processors execute their own instructions and run on different clocks . In these systems , processing elements contain a processor , some cache and a share of the system memory , and are connected through a scalable interconnect which facilitates machine packaging , such as a bus or a mesh -LRB- see Fig. 1 -RRB- . Whereas this physical model dominates , disagreement exists as to the interprocessor communication mechanism . There are two dominant models . One is based on disjoint memories and message-passing and the other is based on shared-memory . In a message-passing system , processors communicate by exchanging explicit messages through send and receive primitives . In the shared-memory model , processors communicate through load and store instructions and some form of explicit synchronization among processors is required at times to avoid data access races -LSB- 7 -RSB- . FIGURE 1 . Physical organization of most MIMD systems The shared-memory model facilitates fine grain -LRB- word level -RRB- communication but requires a large number of instructions to transmit large chunks of data , whereas the message-passing model can transmit large amounts of data in a single message . In terms of ease of programming , the shared-memory model has been so far the favored transition path from uniprocessors to multiprocessors . On the other hand , message-passing systems are generally perceived as more scalable than shared-memory systems . One concern in both kinds of systems is the growing disparity between the processor speed and the speed of communication . In message-passing systems , mes - System Memory Caches Processors Interconnection sage-passing primitives have traditionally suffered from a large software overhead . In sharedmemory systems , the large latency of loads and stores on shared data is usually solved by complex shared-memory access mechanisms . Some researchers advocate private caches -LSB- 15 -RSB- with hardware - or software-based consistency maintenance , such as in the Stanford DASH -LSB- 12 -RSB- prototype . The consistency protocol , the constraints on the ordering of memory accesses -LSB- 7 -RSB- , the cache parameters , as well as the interconnect latency and bandwidth are all factors affecting the performance and programming ease of multiprocessors . Machines such as the DASH have been called CC-NUMAs -LRB- for Cache-Coherent Non-Uniform Memory Access -RRB- architectures to differentiate them from COMAs -LRB- Cache-Only Memory Architectures -RRB- exemplified by the Data Diffusion Machine -LRB- or DDM -RRB- -LSB- 9 -RSB- . A COMA has the same architecture as the one shown in Fig. 1 and communication is done through shared variables ; however , no main memory in each processor node is replaced by a huge cache called attraction memory . For some forms of communication , message-passing is more efficient than shared-memory and there is a trend today towards integrating the message-passing and shared-memory paradigms in order to draw from the strengths of both -LSB- 11 -RSB- . To understand which form such an integrated system should take comparisons among systems are required . Presently , comparisons are difficult to make and hard to validate because of the lack of a common hardware platform to implement the different models . Because multiprocessors are complex and powerful the correctness of a design and its expected performance are very difficult to evaluate before the machine is built . Traditionally two approaches have been taken to verify a design : breadboard prototyping and software simulation . A breadboard prototype is costly , takes years to build and explores a single or a few design points . Discrete-event simulation is very flexible but very slow if the design is simulated in details and it is subject to validity problems because the target system must be considerably abstracted in order to keep simulation times reasonable . In some industrial projects where a detailed and faithful sim - ulation of a target system has been done in software , the simulation runs at the speed of a few cycles of the target system per second of simulation . The parallelization of discrete-event simulation is an ad-hoc procedure and usually exhibits low speedup -LSB- 8 -RSB- . Most simulators -LSB- 6 -RSB- -LSB- 14 -RSB- rely on the direct execution of each target instruction on the host and , because the code -LRB- either source or binary -RRB- must be instrumented , it is difficult to simulate efficiently the execution of interesting workloads , other than scientific programs with little or no I/O . The major objective of the RPM project is to develop a common , configurable hardware platform to emulate faithfully the different models of MIMD systems with up to eight execution processors . Emulation is orders-of-magnitude faster than simulation and therefore an emulator can run problems with large data set sizes , more representative of the workloads for which the target machine is designed . An emulation is closer to the target implementation than an abstracted simulation and therefore more reliable performance evaluation and design verification can be done . Finally , an emulator is a real computer with its own I/O and the code running on the emulator is not instrumented . As a result , the emulator ? looks ? exactly like the target machine to the programmer and a variety of workloads can be ported on the emulator , including code from production compilers , operating systems , database systems , and software utilities . In the following , we first introduce the hardware emulation approach for multiprocessors . Then we describe the architecture of RPM . In Section 4 we explain the methodology to keep track of emulated time , to measure performance , and to program an emulation . In Sections 5 and 6 we show the expected performance of RPM and we compare the emulation approach to simulation and breadboard prototyping . Current status and concluding remarks follow . 2 . THE HARDWARE EMULATION APPROACH The approach to system prototyping in RPM is based on hardware emulation . Emulators have been used in the past to experiment with instruction sets , and more recently in the pre-silicon validation of complex VLSI circuits . To the best of our knowledge , no attempt to apply the emulation approach to the design and verification of multiprocessor systems has ever been made . Several technologies , including FPGAs -LRB- Field-Programmable Gate Arrays -RRB- , and efficient Computer-Aided Design -LRB- CAD -RRB- tools , are currently converging , making it possible to build and program flexible multiprocessor emulators . Field-Programmable Gate Arrays -LRB- FPGAs -RRB- -LSB- 16 -RSB- are high-density , user configurable ASIC devices that are in-circuit programmable by software . The programming of FPGAs is greatly facilitated by the emergence of sophisticated design automation tools allowing designers to specify their design at the Register Transfer Level -LRB- RTL -RRB- using High-level Description Languages -LRB- HDL -RRB- such as VHDL and Verilog . With such tools designers can complete more complex designs in a much shorter time frame . RPM is built from off-the-shelf components except for the cache , memory , coherence and communication controllers which are implemented with FPGAs . The emulation of a particular machine model is done through the FPGAs and a part of the memory to which they are attached . This approach enables the emulation of an entire multiprocessor system at low cost . The clock rate is 10MHz , which is about 10 times slower than the rate permitted by current board and PLD technologies -LRB- estimated at 100MHz -RRB- . This compromise on the emulation efficiency results from two trade-offs . First , the design and fabrication of the PC-boards are greatly simplified . Second , the lower clock rate facilitates the configuration of the FPGAs . FPGAs are slower than other programmable logic devices or custom circuits ? especially when they are programmed with VHDL synthesizers ? and clocking them at lower speed promotes the mapping of more complex circuits . Additionally , in order to further simplify the design and to have the flexibility to emulate complex mechanisms , each processor clock -LRB- pclock -RRB- is emulated in several consecutive clocks . Therefore , overall , the processing speed is 10MHz divided by the number of clocks in a pclock . This relatively low rate allows us to use a standard interconnection fabric and still have enough bandwidth to emulate useful interconnections . Timings are preserved in RPM by scaling down the speed of all components -LRB- time-scaling -RRB- and performance data are collected in a set of counter stored in each memory -LRB- count memory -RRB- . Time scaling and count memory are explained in Section 4 . In the following we overview RPM?s architecture . 3 . ARCHITECTURE OF RPM The architecture of RPM has been geared towards the evaluation of multiprocessors with the general architecture shown in Fig. 1 . The interconnections of the possible targets are limited to FIFO -LRB- First-In-First-Out -RRB- interconnections with uniform access latencies such as crossbars or busses ; however , other interconnections , such as rings , can be modeled approximately . A FIFO interconnection is an interconnection such that messages sent between two nodes arrive at the destination in the same order in which they are sent . FIGURE 2 . RPM is made of nine processor boards connected to a standard Futurebus + backplane 3.1 . Hardware Organization Fig. 2 shows the RPM emulator , packaged in a standard MUPACTM Futurebus + cage . The hardware organization is depicted in Fig. 3 . Eight identical boards , each with one SPARC processor , are connected to a 64-bit wide -LRB- data -RRB- Futurebus + used as a medium to transmit packets among processors as well as for data broadcasting and interprocessor interrupts . An I/O processor board identical to the other boards is connected to a SUN SPARCStation 2 through a SCSI interface -LSB- 4 -RSB- . This workstation serves as the console for RPM and , additionally , executes its I/O requests . With a clock rate of 10 MHz and a pclock currently equal to eight clocks , the emulation THIS FIGURE IS THE PICTURE OF THE OVERALL MACHINE rate is 1.25 million cycles of the target per second -LRB- or a maximum of 10 MIPS of the target machine emulated per second -RRB- . The peak I/O bandwidth is 1.25 Mbytes per second . FIGURE 3 . Overall configuration of RPM The architecture of each processor board is shown in Fig. 4 . The processor is an LSI Logic L64831 SPARC IU/FPU -LSB- 4 -RSB- . This single-chip processor can be clocked at up to 40 MHz and executes both integer and floating-point instructions . It has no on-chip cache and therefore all instruction fetches and data accesses are visible on the pins of the chip . There are three memories on each board , each controlled by a set of Xilinx FPGAs : ? MC1/RAM1 : Each processor is attached to 2 Mbytes of static RAM -LRB- RAM1 -RRB- controlled by two Xilinx XC4013 -LRB- called MC1 -RRB- . The major function of MC1 is to control the cycle-by-cycle execution of the processor . At the beginning of each processor access , MC1 blocks the processor , executes the sequence of steps needed to satisfy the access and unblocks the processor when the access is completed . MC1 also manages RAM1 as a cache and interacts with the second-level memory . MC1 implements extensions to the SPARC instruction set such as shared and exclusive M HOST -LRB- SUN SPARC STATION II -RRB- FUTUREBUS + SCSI P0 FUTUREBUS + INTERFACE SYSTEM MEMORY LOCAL MEMORIES / EXECUTION PROCESSORS IO PROCESSOR SCSI M IO M P7 MODULES -LRB- DRAMs -RRB- CACHES/BUFFERS INTERFACE prefetches . These extension are possible through SPARC?s ASI -LRB- Alternate Space Identifiers -RRB- or through unused address bits . Finally , MC1 can also remap addresses in many different ways , including a full virtual-to-physical translation through a TLB . FIGURE 4 . Block diagram of a processor node ? MC2/RAM2 : An additional 8 Mbytes of static RAM -LRB- RAM2 -RRB- controlled by three Xilinx XC4013 called MC2 makes up the second-level memory . MC2 interfaces the processor to the rest of the system and usually acts as a second-level cache controller . ? MC3/RAM3 : System memory is emulated by 96 Mbytes of dynamic RAM -LRB- RAM3 -RRB- controlled by two Xilinx XC4013 and one Cypress CYM7232 DRAM controller . MC3 supports the virtual interleaving of on-board memory . This interleaving effect is obtained by multiplexing in time the memory controller . Virtual interleaving of memory relies on the fact that a large number of cycles are available in the emulator to emulate memory transactions and is supported by a set of eight interleaving registers -LRB- for up to eight interleaved banks -RRB- and some buffer space in RAM3 . Briefly , the interleaving register corresponding to a memory SI MC3 RAM3 INTERNAL BUS SCSI DU FIFOs FUTUREBUS + INTERFACEFUTUREBUS + CPU MC2 MC1 RAM1 RAM2 Interleaving Registers bank contains a counter which is decremented at every pclock . When the counter reaches zero the memory bank is free . A request to a busy bank is queued at the controller . A typical memory transaction in RPM has three phases : prelude , suspension and completion . During the prelude the packet is received and decoded . The request is then suspended by storing a transaction completion record -LRB- containing all the information needed to resume and complete the request -RRB- in a memory location associated with the virtual memory bank and by filling the interleaving register with a value in pclocks equal to the suspension time . While the request is suspended , accesses to a different virtual memory bank can be serviced . When the interleaving register reaches zero , the memory controller is interrupted ; it then fetches the transaction completion record in memory and completes the transaction by -LRB- possibly -RRB- sending some messages -LRB- completion phase -RRB- . Virtual memory interleaving is further illustrated in Section 4 . The internal bus is a synchronous bus with a protocol similar to the Sun Microsystems MBUS protocol -LSB- 4 -RSB- . It is a 32-bit wide packet-switched bus which transfers packets of size between 16 and 128 bytes . Controllers MC2 and MC3 connect to the internal bus through very large , two-way FIFO buffers , which are there to prevent deadlocks and to relieve the controllers from managing the data transfers . All on-board datapaths are 32-bit wide . The Delay Unit -LRB- DU -RRB- is a programmable unit which emulates variable interconnection delays . It is built with a FIFO controlled by one AMD MACH 210 chip . The FIFO -LRB- 16 Kbytes -RRB- contains blocks and messages which are sent to the bus interface after a programmable delay depending on the target machine?s interconnect latencies and packet size . This delay is computed by the formula : The Futurebus + interface is made of off-the-shelf chip sets . It includes bus transceivers , plus the LIFE chip from Newbridge and a distributed arbiter chip from National Semiconductors . More details on the hardware design of RPM can be found in -LSB- 13 -RSB- . Latency Tstart # of words in packet -LRB- -RRB- Tword ? + = 3.2 . Emulation of CC-NUMAs with Central Directory Protocol -LRB- CC-NUMA-CD -RRB- Our first emulator is a system with hardware-enforced cache-coherence under strong ordering of memory accesses , to enforce sequential consistency -LSB- 7 -RSB- . In this emulator , MC1/RAM1 is a firstlevel write-through cache -LRB- containing both data and instructions -RRB- , MC2/RAM2 is a second-level write-back cache and MC3/RAM3 is the main memory . The protocol is directory-based and each memory block has a home node where a directory records the presence and state of copies in every cache -LSB- 12 -RSB- -LSB- 15 -RSB- . The memory and the cache directories have pending states so that transactions for different blocks are executed concurrently . The protocol is a pure-write invalidate protocol but write-update and competitive-update protocols as described in -LSB- 5 -RSB- are easy extensions . The SRAM implementing the first-level cache -LRB- FLC -RRB- is divided into five parts : the data memory -LRB- up to 1 Mbyte -RRB- , the cache directory , the Translation Lookaside Buffer -LRB- TLB -RRB- needed for virtual memory support -LSB- 4 -RSB- , the space for the emulation of prefetch and write buffers , and the space dedicated to the collection of performance statistics . The controller is partitioned across two FPGAs : one for the control unit and the other for the data unit . Currently the first-level cache is write-through and direct-mapped with a block size of 16 bytes . A store issued by the processor is always propagated to the second-level cache . The first-level cache and the processor block on every write access that misses or requires coherence activity in the second-level cache . A typical data read cycle in the first-level cache consists of receiving an address from the processor , translating the address in the TLB space , accessing the cache directory space , fetching the data from the cache data space , fetching the counter in count memory for that event , updating the counter and returning the data to the processor -LRB- all this is done in eight cycles -RRB- . The second-level cache -LRB- SLC -RRB- is implemented by MC2/RAM2 . The current configuration is a two-way set-associative write-back cache and a block size of 16 bytes . Half of the memory -LRB- up to 4 Mbytes -RRB- is for the second-level cache data memory . The other half is dedicated to the cache directory , various buffers and performance counters . The second-level cache controller is by far the most complex controller of the machine . It is implemented by three FPGAs : the data unit contains the hardware resources needed by the controller , including the buses to the RAM ; the control unit implements the basic cache control functions and the consistency unit is currently unused and is reserved for implementing sophisticated memory consistency models -LSB- 5 -RSB- . A virtually unlimited number of prefetches are supported in the second-level cache . These non-blocking accesses are issued by the compiler to direct the second-level cache to prefetch cache blocks before they are needed . The main memory is implemented in MC3/RAM3 . 64 Mbytes are devoted to the data memory , 16 Mbytes to the directory -LRB- one 32-bit word per 16-byte block -RRB- , 8 Mbytes for performance counters and 8 Mbytes to support the emulation of various hardware mechanisms , including virtual memory interleaving . 3.3 . Emulation of Other Architectures Given its generic board architecture , RPM is capable of emulating in full detail very complex target multiprocessors with very different architectures , provided they fit the generic block diagram of Fig. 1 . Examples are : ? CC-NUMAs with centralized directories -LRB- CC-NUMA-CD -RRB- under weakly-ordered memory models : Under weak ordering of memory accesses -LSB- 7 -RSB- there can be a write buffer between the firstand second-level caches and between the second-level cache and the internal bus . Write buffers can be assisted by a write cache -LRB- WC -RRB- , which is a small cache keeping track of partially modified blocks -LSB- 5 -RSB- . Additionally , prefetching hardware may be added . ? CC-NUMAs with distributed directories -LRB- CC-NUMA-DD -RRB- : Instead of a centralized directory located at the home memory , the directory is distributed by linking caches containing a copy of the block through a set of hardware pointers in each cache entry . This organization is adopted in the Scalable Coherent Interface -LRB- SCI -RRB- standard -LSB- 10 -RSB- . ? COMAs -LRB- Cache-Only Memory Architectures -RRB- : This is the architecture of machines such as the Data Diffusion Machine -LRB- DDM -RRB- -LSB- 9 -RSB- . In this case there is no system memory and MC3/RAM3 acts as a huge cache -LRB- also called attraction memory -RRB- . The DRAM directory is replaced by the attraction memory state . MC1/RAM1 and/or MC2/RAM2 may be configured as caches . ? MPS -LRB- Message-Passing Systems -RRB- : In this configuration MC3/RAM3 acts as the local -LRB- private -RRB- memory of the processor , MC2/RAM2 acts as a Message Passing Controller -LRB- MPC -RRB- , and MC1 / RAM1 acts as the processor cache . The MPC buffers messages sent or received by the processor , formats out-going packets according to the protocol in the target , decodes the received messages , and , if needed , interrupts the processor when messages are received . ? Mixed Shared-Memory and Message Passing Systems : Every shared-memory organization can be augmented with a message-passing facility for bulk transfers of data among processors , as was done in Alewife -LSB- 11 -RSB- . ? Virtual Shared Memory : Shared memory can be implemented on top of a distributed , messagepassing system through virtual memory mechanisms . To promote efficiency , hardware support is often needed -LSB- 2 -RSB- and RPM is an ideal vehicle to experiment with such hardware . 4 . EMULATION METHODOLOGY The success of the RPM project relies on several novel approaches for measuring time and collecting performance data . In this Section we briefly describe the performance evaluation methodology as well as the methodology to program the emulator . 4.1 . Keeping Track of Time : Time Scaling Since the speeds of the hardware emulation and of the target system are different , the timings measured on the emulator must be related to the timings in the target machine . Rather than keeping track of simulated time through event-driven mechanisms and timestamping -LRB- as is done in software simulators -LSB- 8 -RSB- -RRB- , time is scaled . Time scaling preserves the relative timing of components in the emulator and in the target , and absolute times in the target are derived from executions on RPM by simple scaling arguments . For example , it is intuitively obvious that the processor utilization in a system with processors running at 100 MHz and with average memory latencies of 100 nanoseconds is equal to the processor utilization in a system with the same architecture but with processors running at 1 MHz and with average memory latencies of 10 microseconds . Therefore we do not have to build the system with the most up-to-date and fastest technology provided we scale memory , interconnection and processor speeds appropriately . Every component -LRB- interconnect , cache , memory and I/O processor -RRB- is characterized by two fundamental performance measures : latency and bandwidth . These two measures can be independent . For example , two networks can have the same latency but one may have more bandwidth because it has more links ; similarly , the bandwidth of a memory can be increased -LRB- while its latency remains the same -RRB- by interleaving it . A convenient unit for all timings is the pclock ? the clock period of the processor . If the latencies of all components are expressed in terms of pclocks and if all component bandwidths are expressed in terms of bytes per pclock , then systems with components of equal latencies and bandwidths are equivalent . In RPM , a pclock is currently eight cycles . This gives the emulator eight cycles to simulate all the activities occurring in one pclock in the target system . To simulate variable latencies , we delay requests . To simulate variable bandwidth of a given resource , we must vary the number of cycles that each request keeps the resource busy . The flexibility of this adjustment varies with the resource . All on-board data paths in RPM are 32-bit wide and the system bus -LRB- Futurebus + -RRB- is 64-bit wide . We can emulate one cycle of a data path with 64 , 128 or even 256 bits by 2 , 4 , or 8 cycles of the 32-bit data paths in RPM . The latencies and bandwidths of the first-level cache and of the internal bus are not adjusted . The implication is that their speed scales up proportionally with the speed of processors in all target systems . The Delay Unit -LRB- DU -RRB- -LRB- see Fig. 4 -RRB- simulates variable latencies in the interconnection -LRB- emulated by the Futurebus + -RRB- . As configured , the bandwidth available on the Futurebus + is very large , in effect implementing an interconnect with infinite bandwidth . To run experiments under limited interconnect bandwidth , we can reduce the width of the bus -LRB- by reprogramming the Futurebus + controllers -RRB- and artificially increase the size of each packet . To increase the interconnect bandwidth the number of clocks per pclock can be doubled or even quadrupled . To illustrate time scaling and virtual memory interleaving , consider the simple case of a miss in the second-level cache such that the block address maps to the local on-board memory -LRB- local miss -RRB- , and the block is uncached elsewhere . Let Tm be the latency of the miss and let n be the degree of memory interleaving so that the memory can deliver n blocks every Tm pclocks . RPM has a monolithic memory controller , which must emulate the operations of the n parallel memory controllers of the target system . In RPM the memory transaction must busy the controller for Tm/n pclocks by idling it before suspending the miss request . On the other hand , the total latency of the miss in RPM must be Tm pclocks , the same as in the target system . Let Tp , Ti , Ts and Tc be the prelude time , the idle time , the suspension time , and the completion time of the miss request in RPM , respectively . -LRB- All times are in pclocks . -RRB- The various phases of a memory access in RPM are illustrated in Fig. 5 . The memory controller is busy during the whole access except during Ts , when an access to another memory bank can be accepted by the controller . FIGURE 5 . Phases of a memory access in RPM Tp and Tc are known from the implementation of the emulation . To find Ti and Ts we have the following constraints : Tm = Tp + Ti + Ts + Tc and -LRB- 1 -RRB- Tm/n = Tp + Ti + Tc -LRB- 2 -RRB- -LRB- 1 -RRB- enforces the same latency in RPM and in the target whereas -LRB- 2 -RRB- enforces the same utilization Time Prelude Idle Suspend Complete Tm Tp Ti Ts Tc of the controllers in both systems . Unknown Ti and Ts are then given by : Ti = Tm/n - Tp - Tc and Ts = Tm -LRB- n - 1 -RRB- / n -LRB- 3 -RRB- Consider , for example , a target system with a pclock of 5 nsec in which a memory access to the 16 byte block takes 140nsec . Because the pclock in RPM is 800 nsec , a 140nsec block access time in the target translates into 140/5 pclocks or 28x8 = 224 clocks in RPM . This is a huge number of cycles during which the memory controller can emulate very complex directory mechanisms . Assume now that , based on the testbed implementation , the prelude time and the completion time for a block access in memory are both 30 clocks . If the memory is two-way interleaved , the idle time is 224/2 -30 -30 = 52 clocks and the suspension time is 224/2 = 112 . The timings are usually tighter in the second-level cache , which is usually built with SRAMs in the target system . For example , a block transfer from the second - to the first-level caches taking 4 pclocks must be emulated in 32 clocks . As the block size increases , the number of extra cycles available for emulation increases . The response time of the second-level cache can be adapted by adding dummy cycles to each VHDL hardware sequence of its controller . The scaling of I/O is automatic in RPM . When we simulate target systems with different processor speeds , I/O bandwidth is not adjusted because RPM always runs at the same speed , i.e. 10 MIPS peak , whatever the target system speed is -LRB- provided one pclock = eight clocks -RRB- . I/O latency must be scaled . The service of I/O requests must be delayed as faster processors are emulated . This is done in software , with the support of an interrupt timer . 4.2 . Collecting Performance Data The primary mechanism to collect performance data involves event counters stored in a special area of each memory called count memory . In each of the three on-board memories , a set of counters keeping track of the occurrence of mutually exclusive events are updated every time a transaction is completed in the controller . Addresses of event counters are formed automatically by merging signals corresponding to basic events . These events are mutually exclusive , so that only one counter is updated at a time in each memory . Thousands of counters are present in each memory , meaning that thousands of different events can be counted . At the end of the emulation run , the counters are uploaded and post-processed -LRB- basically they are selectively added together -RRB- to obtain the required performance data . This counting mechanism can be started and stopped under software control . FIGURE 6 . Generation of addresses for event counting in the first-level cache?s count memory Fig. 6 illustrates how the addresses in count memory are generated for mutually exclusive events in the first-level cache . In this simple example , three signals ? programmed in MC1 ? correspond each to one property of an access in the first-level cache . One signal indicates whether the access is to private or to shared data . The second line is the read/write signal from the processor and the third signal is high when the access hits and low when the access misses in the firstlevel cache . The combination of these signals forms a three-bit address , which can be used to address a counter in the area of RAM1 allocated to count memory . In this example there are only eight addresses , but in a practical situation up to 20 signals can be defined in each controller . For instance one signal may also distinguish between instructions and data , and , in the case of instructions , the opcode could also be a field of the address of the performance counters . Several other special purpose monitoring strategies can be implemented in all levels of the memory hierarchy . They can be added to the basic count memory mechanism or they can replace it , depending on the amount of hardware required . Counter Address Private / Shared Read / Write Hit / Miss Basic Event Shared-Write-Miss 1 1 Shared-Write-Hit 2 1 Shared-Read-Miss 3 1 1 Shared-Read-Hit 4 1 Private-Write-Miss 5 1 1 Private-Write-Hit 6 1 1 Private-Read-Miss 7 1 1 1 Private-Read-Hit 4.3 . Programming RPM RPM is programmed by mapping the controllers of the target design into the FPGAs . Currently , we do not have tools to partition automatically a design across multiple FPGAs . The data path and the RTL description of each controller are specified separately in VHDL . Design parameters such as block sizes , cache sizes , latencies and bandwidth are specified as constants in the VHDL descriptions . FIGURE 7 . The three dimensions of flexibility in RPM Using the current approach , modifications to the functionality of each FPGA require the design and specification of the RTL description of each controller in VHDL . In reality , however , a large number of common functions are shared by all the possible designs for a given FPGA . Nevertheless , the task of re-programming the FPGAs is error-prone and increases the turnaround time for emulating different machines . In the future , we expect that the turnaround time for this design stage will be reduced as behavioral compilers become more widely available . A behavioral compiler is a high-level synthesis tool which accepts an algorithmic description of a circuit to cre - interconnect BW Hardware Parameters Hardware Mechanisms message passing CC-NUMA-CD CC-NUMA-DD COMA cache size cache block size cache organization cache access time memory access time memory bandwidth interconnect latency inst . set extensions write cache prefetch delayed write buffers directory structure cache protocol consistency System Architecture virtual shared memory ate the hardware . This algorithmic description could be derived directly from the high-level description of each protocol . In the short term we will develop libraries of parameterized designs for every possible configuration of each FPGA . Fig. 7 illustrates the three dimensions of flexibility in RPM . Changing hardware parameters such as cache size , block size or latencies is done by modifying constants in the VHDL programs , which must then be recompiled -LRB- The only exception is the interconnect latency and bandwidth , which are adjusted at system configuration time . -RRB- Within a class of systems , changes to basic hardware mechanisms , such as the cache protocol , require a moderate amount of reprogramming . Designing the FPGA programs for different system architectures is a major endeavor . 5 . PERFORMANCE OF RPM Table 1 shows the slowdown factor , ie , the speed ratio between the target system and the emulator , for various uniprocessor technologies in the target systems . These slowdown factors can be predicted accurately in our emulation approach and they are independent of the number of processors -LRB- from 2 to 8 -RRB- since the number of processors is the same in RPM and in the target system . Table 1 also shows the time taken by RPM for different execution times of the target . We can reasonably expect to obtain experimental points for realistic workloads for systems with processors of up to 1 GIPS . Some of these experiments would take months to run on a current software simulator and the simulation would have to be significantly simplified and abstracted . 6 . COMPARISON WITH OTHER APPROACHES Recent breakthroughs in simulation methodology have opened the possibility of efficient , detailed Table 1 : Slowdown factors between target and RPM Target Uniprocessor Speed 50 MIPS 100 MIPS 200 MIPS 500 MIPS 1 GIPS Slowdown 40 80 160 400 800 Time on RPM per Second of Target Execution 40 seconds 1 minute and 20 seconds 2 minutes and 40 seconds 6 minutes and 40 seconds 13 minutes and 20 seconds Time on RPM per Minute of Target Execution 40 minutes 1 hour and 20 minutes 3 hours and 40 minutes 6 hours and 40 minutes 13 hours and 30 minutes and flexible evaluations . In an execution-driven simulator such as Tango -LSB- 6 -RSB- , or the Wisconsin Wind Tunnel -LRB- W.W.T. -RRB- -LSB- 14 -RSB- each instruction is run directly on the host machine and only some types of accesses generate simulation events . The processors and the components of the memory systems are simulated as processes or threads , and each event requires a context switch . Instrumentation code is added at basic block boundaries and at each ? sensitive ? data access -LRB- usually shared-data accesses and synchronization primitives -RRB- . By contrast , program-driven simulators such as Cache-Mire -LSB- 3 -RSB- interpret each instruction in software . Software simulation , whether it is execution - or program-driven , is slow . A simulator such as Tango or Cache-Mire can execute in the order of 10,000 instructions of a target multiprocessor with a complex memory model per second on a 50 MIPS workstation . This low performance is due to multiple factors : the overhead due to event scheduling -LRB- e.g. , context switching and event list management or activity scanning -RRB- , the code expansion due to code instrumentation to keep track of target instruction execution times in execution-driven simulators -LRB- which was reported to be between 2 and 3 in -LSB- 6 -RSB- -RRB- or the overhead of decoding and executing instructions in programdriven simulations , the management of timestamps associated with events , the collection of performance data , the semantic gap between hardware mechanisms and their execution on the simulator -LRB- the fact that each basic activity which takes one cycle in the hardware of the target takes several instructions to simulate on the host -RRB- , and the speedup of the target multiprocessor . To keep simulation times reasonable , the data set sizes of the workload must be drastically reduced . Observations made on the small data set sizes must then be extrapolated to the workload with the actual data set size , a difficult task which has never really been validated . A common drawback of all simulations is that they abstract the behavior of the target multiprocessor . Many effects are ignored or approximated , on the premise that they are negligible . In some cases some key hardware components and physical events are totally removed from the simulation . For example , it is not uncommon that a simulator avoids simulating the caches and the data transfers among them . The validity of these simplifications is usually not verified and relies on the experience and judgement of the evaluator . To predict the speedup of RPM with respect to a simulator such as Cache-Mire or Tango we run the simulation of the target system and measure the simulation time as well as the execution time of the target in pclocks -LRB- same as RPM?s -RRB- . Based on such measurements we have observed speedups of between 100 and 1,000 over execution-driven simulations . However , the level of implementation details in RPM is actually closer to a cycle-by-cycle , register-transfer level simulation . In practical cases , such simulations run at the rate of a few cycles per second and the speed up of RPM over these detailed simulators can be up to one million . Besides simulation , breadboard prototypes are also extensively used in industry to validate a new architecture and in academic research projects to explore novel architectures . Prototypes are fast and faithful to the target system , but their cost is high and they provide little research information besides proving one point . In an academic environment , the building of a prototype is a valuable experience for students in architecture and it helps direct the simulation experiments . In most projects , however , most of the research results are still derived through simulations . 7 . STATUS AND FUTURE PLANS Currently , the hardware is up and running and our first emulator -LRB- a CC-NUMA-CD under sequential consistency -RRB- is in the late debugging phase . We are replacing the Xilinx XC4013 in the controllers with the XC4025 . This upgrade will double the number of equivalent gates in each controller . It is possible because the XC4025 is pin-to-pin compatible with the XC4013 . FPGA programs for a CC-NUMA with release consistency , for the Scalable Coherent Interface -LRB- SCI -RRB- protocol and for a COMA are being developped . The I/O capability of RPM is being upgraded with two Gbytes of disk connected to each processor node . As soon as this is done , the total disk space will be 16 Gbytes and we will be able to port applications with distributed I/O . On the software side , we are currently porting some of the SPLASH benchmarks on the first emulator . In parallel , we are starting an effort to port an operating system kernel as well as a database engine to run database applications . 8 . CONCLUDING REMARKS The emulation methodology described in this paper is very cost-effective for the prototyping of various multiprocessors . In this sense it is an extremely valuable research tool in architecture , especially in an academic environment . Students working on the project gain valuable experience in complex chip designs as well as in system architecture . Each system architecture mapped onto RPM is an actual system design , which a graduate student can complete in a matter of months by re-using the same hardware platform and thus concentrating on the essential parts of the design . For all its advantages , there are however some weaknesses in the current emulator . The first weakness is the relatively low pclock rate . We were somewhat conservative in our design because of our lack of experience with FPGAs and with FPGA CAD tools . The low clock rate and the simplicity of the board-level design have facilitated the construction of RPM in an academic environment . The total time taken by the design and the construction was 15 months . The second limitation of the current emulator is the number of processors . This is due to our limited budget . Unfortunately , we have no technical solution to this problem : To investigate larger machines , an emulator with more processors will have to be built . RPM?s approach to hardware emulation differs from other FPGA-based emulators in which circuits are emulated by arrays of FPGAs . Because of cost constraints and for performance reasons , we restrict the use of FPGAs to the controllers in the various levels of the memory hierarchy . While this approach makes it feasible to emulate a whole multiprocessor system , it limits the flexibility of RPM to the emulation of machines with the overall organization of Fig. 1 . In the future , programmable interconnect technology might be used to increase the flexibility of emulators . Like crossbar switches , Field Programmable Interconnect Components -LRB- FPICs -RRB- at the chip level allow multiple input signals to be directed to multiple output pins . It may become possible to build emulators with FPGAs and FPICs and these emulators will have greater flexibility to be configured for different target systems . Finally , as for any piece of hardware , the efficiency advantage of an emulator over simula - tion erodes every year , as faster workstations and personal computers are introduced . Nonetheless , given the current speed of RPM , we expect that it will remain competitive with software simulators for at least ten years . There are several ways to improve on the speed of the emulator . With current CAD tools and FPGA technologies a more aggressive design could raise the clock speed to 20MHz . The number of clocks in each pclock could also be cut in half if a processor with an on-chip instruction cache was used to avoid emulating instruction fetches . Such an emulator would emulate 5 Millions cycles of the target per second and large speedups over software simulation could be expected for large-scale emulators with 128 or 256 processors . Acknowledgments . Funding for this project has been provided through a grant from the National Science Foundation . Besides the authors , several individuals have contributed to the design of RPM . In particular , we want to thank Per Stenstr?m from Lund University -LRB- Sweden -RRB- , Massoud Pedram from EE-Systems , U.S.C , and Jacqueline Chame , also from U.S.C. Through grants , gifts or rebates , several companies helped reduce the cost of the hardware and software needed for the project . These companies are AMD , Synopsys , Viewlogic , Axil Workstations , and Xilinx . Finally , John Granacki from ISI offered the services of EZFAB , which is part of the ARPA-sponsored Systems Assembly Project . 9 . Bibliography -LSB- 1 -RSB- Arnold , J. , Buell , D. , and Davis , E. , ? SPLASH-2 , ? Proceedings of the 4th Annual Symposium on Parallel Algorithms and Architectures , pp . 316-322 , June 1992 . -LSB- 2 -RSB- Blumrich , M.A. , Li , K. , Alpert , R. , Dubnicki , C. , Felten , E.W. , and Sandberg , J. , ? Virtual Memory Mapped Network Interface for the SHRIMP Multicomputer , ? Proceedings of the International Symposium on Computer Architecture , pp. 142-153 , April 1994 . -LSB- 3 -RSB- Brorsson , M. , Dahlgren F. , Nilsson , H. , and Stenstr?m , P. ? The CacheMire Test Bench : A Flexible and Effective Approach for Simulation of Multiprocessors ? , In Proceedings of the 26th Annual Simulation Symposium , pp. 41-49 , March 1993 . -LSB- 4 -RSB- Catanzaro , B. , Multiprocessor System Architectures . Prentice-Hall , 1994 . -LSB- 5 -RSB- Dahlgren , F. , Dubois , M. , and Stenstr?m , P. , ? Combined Performance Gains of Simple Cache Protocol Extensions , ? Proceedings of the International Symposium on Computer Architecture , pp. 187-199 , April 1994 . -LSB- 6 -RSB- Davis , H. , Goldschmidt , S.R. , and Hennessy , J. , ? Multiprocessor Simulation and Tracing Using Tango , ? Proceedings of the Parallel Processing Conference , pp . II99-II107 , August 1991 . -LSB- 7 -RSB- Dubois , M. , Scheurich , C. , and Briggs , F.A. , ? Synchronization , Coherence , and Event Ordering in Multiprocessors , ? IEEE Computer , Vol . 21 , No. 2 , pp. 9-21 , February 1988 . -LSB- 8 -RSB- Fujimoto , R.M. , ? Parallel Discrete Event Simulation , ? Communications of the ACM , Vol . 33 , No. 10 , pp. 30-53 , October 1990 . -LSB- 9 -RSB- Hagersten , E. , Landin , A. , and Haridi , S. , ? DDM -- A Cache-Only Memory Architecture , ? IEEE Computer , Vol . 25 , No. 9 , pp. 44-54 , September 1992 . -LSB- 10 -RSB- Gustavson , D. , ? The Scalable Coherent Interface and Related Standards Projects , ? IEEE Micro , Vol . 12 , No. 1 , February 1992 . -LSB- 11 -RSB- Kranz , D. , Johnson , K. , Agarwal , A. , Kubiatowicz , J. , and Lim , B. , ? Integrating MessagePassing and Shared-Memory : Early Experience , ? Proceedings of the 4th ACM Sigplan Symposium on Principles and Practice of Parallel Programming -LRB- PPoPP -RRB- , pp.54-63 , May 1993 . -LSB- 12 -RSB- Lenoski , D. , Laudon , J. , Gharachorloo , K. , Weber , W.-D. , Gupta , A. , Hennessy , J. , Horowitz , M. , and Lam , M.S. , ? The Stanford DASH Multiprocessor , ? IEEE Computer , pp. 63-79 , Vol . 25 , No. 3 , March 1992 . -LSB- 13 -RSB- Oner , K. , Barroso , L. , Iman , S. , Jeong , J. , Ramamurthy , K. , and Dubois , M. , ? The Design of RPM : An FPGA-based Multiprocessor Emulator , ? Proceedings of the 3rd ACM International Symposium on Field-Programmable Gate Arrays -LRB- FPGA ? 95 -RRB- , February 1995 . -LSB- 14 -RSB- Reinhardt , S. , Hill , M. , Larus , J. , Lebeck , A. , Lewis , J. , and Wood , D. , ? The Wisconsin Wind Tunnel : Virtual Prototyping of Parallel Computers , ? Proceedings of the ACM Sigmetrics Conf . on Measurements and Modeling of Computer Systems , pp. 48-60 , May 1993 . -LSB- 15 -RSB- Stenstr?m , P. , ? A Survey of Cache Coherence Schemes for Multiprocessors , ? IEEE Computer , Vol . 23 , No. 6 , pp. 12-24 , June 1990 . -LSB- 16 -RSB- Trimberger , S. , ? A Reprogrammable Gate Array and Applications , ? Proceedings of the IEEE , Vol . 81 , No. 7 , pp. 1030-1041 , July 1993 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

design verification
fpgas
performance evaluation
simulation

field-programmable gate arrays
message-passing multicomputers
shared-memory multiprocessors


Information Brokers : Sharing Knowledge in a Heterogeneous Distributed System Daniel Barbar ? Chris Clifton Matsushita Information Technology Lab Northwestern University 182 Nassau Street Department of EECS Princeton , NJ 08542 2145 Sheridan Road -LRB-609-RRB-497-4600 Evanston , IL 60208-3118 daniel@mitl.com -LRB-708-RRB-491-7642 clifton@eecs.nwu.edu Abstract Alarge percentage of valuable electronic information is not stored in conventional database systems . Very often , applications have to deal with heterogeneous services such as electronic libraries , airline reservation systems or weather information services . Many times , ? nding these services and information can be an overwhelming task . In this paper , wedescribe a tool to facilitate the ? nding of such information . The cornerstone of the service is the idea of a broker.Abroker indexes services and objects by their describing properties . A query to a broker returns enough information to contact the service and get the object or information needed . The type and amount of information indexed by the broker depends on the information that is known and available about the service or object . Wepresent a simple but powerful design for the broker that is ? exible enough to accommodate a wide variety of information providers . Wealso present a complete architecture for our distributed information system . 1 . Introduction There has been a growing interest in recent years in developing techniques to provide interoperability for data stored in conventional database management systems . -LRB- For a good survey , see -LSB- Elma90 -RSB- . -RRB- However , in practice a large percentage of valuable information is not stored in such systems , but as a wide variety of unstructured and semi-structured data such as electronic mail , documents , ? les and spreadsheets . In addition , applications must deal with heterogeneous services such as an electronic library , anairline reservation system or a weather information service . Many times , ? nding these services can be itself an overwhelming task . Consider for example an application for organizing a convention . This application might involve ? nding out possible locations -LRB- hotels , convention halls -RRB- , setting up arrangements with airline companies , catering services and so on . A ? rst step could involve ? nding out information sources that contain addresses for the possible service providers . A source containing information about hotels , for instance may be managed by a relational database system . Then the application will issue an SQL query to retrieve the data . In other cases , this information can be found by some special server that understands a ? xed set of commands . It is clear that in this environment interoperation will surely involve human intervention . However , the system can provide a set of tools for interoperation including tools for resource discovery , data translation , activity management and automated processing . At the Matsushita Information Technology Laboratory -LRB- MITL -RRB- we are developing some of this interoperation tools . These tools operate in an environment with objects and services.Objects and services can be located anywhere on the network . The internals of an object or service can be arbitrary ; however , objects and services have one or more properties that describe their contents and functionality.For instance an hotel information service may be described by keywords -LRB- hotel , accommodations -RRB- , the type of interface required to access it -LRB- SQL -RRB- , and its location -LRB- host and port -RRB- . This paper focuses in describing one of such tools : the resource and information discovery system . In the recent past , various research efforts -LRB- e.g. , -LSB- Kahl89 , Danz91 -RSB- -RRB- have aimed to address the problem of distributing and retrieving information electronically.However , the common denominator in their work is the belief that every provider of information will abide by their protocols and architecture , thereby enforcing a standard in information sharing . We believe that it is unlikely that this standardization will become areality.There is simply too great a diversity of autonomous organizations involved in providing information to hope for any kind of universal agreement . Heterogeneity in accessing methods and architectures is a fact of life which any useful system will have to deal with . As an example , in addition to WAIS information is currently being distributed over the internet using ? le transfer protocol -LRB- FTP -RRB- 1 , the NeXT Digital Librarian , and gopher2 . 1 There is an indexing service , archie , for ? les available via anonymous FTP.Information on Archie is available through the internet ; ? rlogin quiche.cs.mcgill.ca - l archie ? . 2 Gopher is an information server developed at the University of Minnesota which has limited indexing capabilities , however it does support connections to other servers and can make use of WAIS indexes . In this paper we present a novel architecture that addresses the problem of global , autonomous , shared information exchange , with three principles in mind : ? Noexplicit cooperation from information providers is assumed . The system can adapt to a spectrum of conditions from no cooperation at all to a very tight coupling with information providers . In other words , we should be able to support information providers without change . ? The system must be scalable ; i.e. the addition of a new information provider or a new user should not affect other components of the system or require major modi?cations of the software . ? Itisbetter to accept incomplete or partial results in the information search than to require all of the information system components to maintain a perfect , consistent view . This architecture is currently under development at MITL , in the context of the Gold project . This paper is divided as follows . In Section 2 , we review previous research efforts in the area and the drawbacks of these solutions . In Section 3 we present the architecture overview of our solution . Sections 4and 5 describe two key components of our architecture , the Information Brokers and the Translators . We devote Section 6 to discussing some of the security issues with which we have to deal when sharing information . Section 7summarizes the paper and describes some of the efforts currently under development . 2 . Related Work Considerable work has been done in the area of heterogeneous databases -LRB- -LSB- Litw90 , Shet90 -RSB- -RRB- . Much of this work tries to create a single multidatabase out of a heterogeneous collection of databases -LSB- Smit81 , Ahme91 -RSB- . The goal of projects such as these is to allow users to think of the collection of databases as a single global database , thus hiding the heterogeneity.The problem with these methods is that considerable work must be done up-front to build a global schema -LSB- Bati86 , Shet88 , Czej87 -RSB- , and this global schema must be updated every time a new information provider -LRB- database -RRB- is added . This does not scale well to the number and diversity of information providers we envision . Federated Databases -LSB- Heim85 -RSB- are not as intent on providing the illusion of a single global database . This eases the up-front work involved in adding new servers , but still does not provide the level of ? exibility we desire . We face many of the same issues as these systems ; the difference is in the degree to which we try to solve them . Rather than trying to provide the illusion of a homogeneous system , we provide resources that strive only to allow the user to obtain the needed information . Another area in which this difference shows up is transaction management : Existing work tries to approximate the transaction power of homogeneous database systems -LSB- Brei90 , Sopa91 -RSB- ; we will investigate less strict transaction models that will not compromise the heterogeneity and autonomy of the users and providers in this project . Pro?le -LSB- Pete88 -RSB- is an attribute-based naming service for large networks . It allows users to retrieve information about resources associated with the attributes . Aname server keeps links to a collections of resources administered by a user or an organization . Name servers are viewed as forming a loosely coupled federation . In this sense , the architecture of Pro?le resembles our own architecture . However , there are two main differences . First , all involved servers follow the same set of protocols . Second , attributes are of the form tag = value -LRB- for instance , name = Peter -RRB- . As we describe later , brokers provide a more ? exible de?nition for attributes . The systems that come the closest to our goals are WAIS -LSB- Kahl89 -RSB- and Distributed Indexing -LSB- Danz91 -RSB- . WAIS is also designed as a global-scale service . Adirectory of databases is used to ? nd the appropriate database , this is similar to our broker.WAIS , however , relies on a ? xed protocol -LRB- based on ANSI Z39 .50 -RRB- to connect servers and consumers . This has many disadvantages , not the least of which is that The success of this system lies in a `` critical mass '' of users and servers . This protocol , then , could be used on any electronic network from digital networks to phone lines -LSB- Kahl89 -RSB- . Distributed Indexing talks less about the database/application interface , however the index brokers are much more closely tied to the databases -LRB- and are not allowed to be out of date . -RRB- This approach violates our goal of supporting existing servers without change.The information providers are assumed to run a sitebroker responsible for remembering the queries that the brokers used to build their indexes . When an update occurs , the site broker applies the stored queries against the updates and forwards the appropriate changes to the index brokers . This schema obviously does not scale well . Finally , -LSB- Gold92 -RSB- describes an architecture to provide good scalability and availability of an information service . The paper emphasizes the issue of how to maintain information servers implemented by distributed databases using weak-consistency protocols . It brie?y addresses the issue of discovering information in the Internet , by suggesting that a database location server can be built using their protocols . Again , this solution requires collaboration among the sites that support this database location server . Although this solution can be easily weaved into our own architecture , we do not require that this to be the case . 3 . ArchitectureOverview The Gold distributed information system consists of four types of processes : information providers , information users , information brokers , and translators . Providers and users are the ? core ? of the system , brokers and translators are the ? glue ? that ties the -LRB- autonomous , heterogeneous -RRB- users and providers . However , wewish to place as few a-priori constraints on providers and users as possible , so as not to limit the capability of future providers and users that may wish to join the system . As a result information brokers act as a ? lowest common denominator ? that can communicate with all users and providers . Brokers act as an index that allows users to specify simple queries and responds with the provider -LRB- and often the speci?c object on that provider -RRB- that contains the requested information . The user then goes directly to the provider -LRB- possibly using the services of a translator -RRB- to obtain the information . Asample network is shown in Figure 1 . This paper concentrates on the brokers and translators . We also describe the minimum requirements of users and providers ; although this is a small part of this paper -LRB- as it should be , since if we spent a signi ? - cant amount of time discussing the requirements of information providers and users , we would be placing too many constraints on them . -RRB- 3.1 . Information Providers Information providers are a widely varied set of ? databases . ? Note that database is used in a loose sense ; we expect providers to be based on DBMS 's , ? le systems , on-line information systems , and perhaps platforms of which we have not yet conceived . The level of indexing support needed by the provider varies with the provider type . ADBMS may provide an address , interface -LRB- such as SQL -RRB- , and schema synopsis . The user then issues a query to the DBMS to get the speci?c information desired . A?le system accessed via FTP -LRB- File Transfer Protocol -RRB- , on the other hand , may wish to index each ? le individually so that the broker provides the complete path name -LRB- as the provider has no ef?cient mechanism to allow the user to determine which ? le contains the desired information . -RRB- Providers may support a variety of interfaces . For example , a relational database may support a variety of query languages . In addition , certain ? canned queries ? may be made available through the broker , for users who do not understand the query language . These might return a complete table rather than just the tuples desired , but at least the user is able to access the information .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

distributed databases
indexing
information services
naming
resource discovery