The Data Diffusion Machine with a Scalable Point-to-Point Network Henk L. Muller Paul W. A. Stallard David H. D. Warren October 1993 CSTR-93-17 University of Bristol Computer Science Department The Data Diffusion Machine with a Scalable Point-to-Point Network ? Henk L. Mullery Paul W.A. Stallard David H.D. Warren Department of Computer Science , University of Bristol , UK . Working at : Partnership in Advanced Computing Technologies , Science Research Foundation , 10 Priory Road , Bristol . BS8 1TU , UK . email : -LCB- henkm , paul , warren -RCB- @pact . srf.ac.uk Abstract The Data Diffusion Machine -LRB- DDM -RRB- is a virtual shared memory architecture . Data items in the machine have no fixed home location but instead migrate to the processors that are actually using the data . Coherency is maintained with a hierarchical directory scheme . In an uncongested machine , all memory operations complete in a time bounded by the logarithm of the number of processors . As the machine load increases , the natural combining of the DDM hierarchy allows better network utilisation and helps maintain performance . This paper proposes an implementation of the DDM with split directories interconnected with a point-to-point network . This is easier to construct and scales better than previous designs . The paper also defines the protocol that is needed to maintain consistency on this split tree -LRB- even though the network provides a number of paths between pairs of processors -RRB- . Conflictingrequests -LRB- writes on different processors to the same data item -RRB- are resolved in a time bounded by the number of processors , without chance of starvation . The protocol has been fully implemented , and we show some results for up to 72 processors . 1 Introduction The traditional distinction between shared and distributed memory machines is vanishing . With the introduction of virtual shared memory -LRB- VSM -RRB- , programmers can be given a shared memory abstraction above a -LRB- physically -RRB- distributed memory machine . Although the programmer must still take care of locality of reference , the need for explicit data transport is removed . The programmer thus has the convenience of the shared memory programming model , while much larger machines can be implemented cost effectively because of the distributed nature of the memory . The shared memory abstraction can be supported at any layer of the machine ? hardware , operating system or programming language -LRB- or a combination of these -RRB- . ? Work supported by ESPRIT OMI/HORN P7249 . yAuthors are ordered alphabetically . Orca -LSB- 1 -RSB- is a good example of a compiler that can provide a shared memory abstraction for a specific language . High Performance Fortran -LSB- 2 -RSB- also offers a kind of shared memory model . Using a special language requires reimplementation of application software , but it gives better -LRB- compile time -RRB- control over the locality . The lower level solutions provide a general shared memory abstraction available to all application programs . At operating system level , shared memory can be provided by means of a coherent paging scheme -LSB- 3 -RSB- . At the hardware level , a coherency protocol can be used to provide a shared memory view that is indistinguishablefrom a physical shared memory system . This is the approach taken in the DDM -LSB- 4 -RSB- , MIT - Alewife -LSB- 5 -RSB- , DASH -LSB- 6 -RSB- , KSR-1 -LSB- 7 -RSB- , and machines based on the Futurebus -LSB- 8 -RSB- or the SCI -LSB- 9 -RSB- . Although these machines all provide shared memory in hardware , the manner in which the protocols function are very different , affecting the performance and scalability of the machines . Many machines provide a sequentially consistent memory model . This means that the machine behaves -LRB- from the programmer 's viewpoint -RRB- identically to a machine with one single-ported shared memory . This consistency model allows the programmer to rely on new data becoming visible to all nodes at the same time . Processor consistency does not guarantee this , but still guarantees that the order of updates is preserved . Processor consistency allows write operations to be performed asynchronously , thereby potentially increasing the performance . Weaker consistency models like release consistency do not guarantee the order of updates , and consequently allow even more optimisations . The cost of these performance improvements is that the programmer is burdened with the extra duties of ensuring memory is consistent at the right times . This extra burden is considerable and will not always provide the desired increase in performance . Until compiler technology is able to assist the programmer in this task , it may be unreasonable to expect programmers to deal with this . The DDM therefore provides a sufficiently strong consistency model that existing shared memory programs execute correctly . At this moment we support a sequential consistency model , but we may weaken this to processor consistency if the extra programming effort turns out to be minimal . The DDM is easily adapted to run such a memory model without changing the protocol . A main design aim of the DDM is to be scalable . Many architectures claim some form of scalability , in various senses . The DDM is scalable in the sense that the cost of a machine with p processors is O -LRB- p log p -RRB- and the performance per processor for increasing p decreases at most with O -LRB- log p -RRB- -LRB- until for huge p the wires become so long that transmission delays are significant -RRB- . This means that larger applications with sufficient parallelism will continue to benefit from extra processors added to the system . Note that the cost function does not necessarily mean that adding one more processor incurs a fixed cost : adding a single node to a machine might require the addition of an extra layer in the hierarchy . The DDM and KSR-1 are different from other VSM architectures in that data does not have a home location . The data moves to the node -LRB- s -RRB- where it is referenced , and is not kept where it is not needed . This strategy has a number of advantages : the memory is potentially used more efficiently , dynamic relocation of working sets is naturally accounted for , and high degrees of sharing can be created and destroyed without a central bottleneck around the home location . The cost of such benefits is that each node must be able to locate the data , requiring some directory structure . 6 6 6 ? ? ? 6 6 6 ? ? ? 66 ?? ? ? ? Bus ' $ oe oe - ? Ring : : : ? ? ? ? ? ? ? ? ? ss ??? * ? 6HHHjH H HY Point-to-Point ?? ?? ???? ?? ??? ? ? 6 6 6@R @R @R @I @I @I Figure 1 : Interconnection schemes for hierarchical architectures . Both the DDM and the KSR-1 use a hierarchical directory structure . A number of interconnection options exist for hierarchical architectures as illustrated in Figure 1 . Busses -LRB- used in a prototype of the DDM being built in Sweden -LSB- 10 -RSB- , and also in the Futurebus -RRB- are popular because broadcasting and snooping are directly supported , simplifying the cache-coherency protocol . The disadvantage of a bus is that the construction of large parallel machines is difficult and expensive . On a ring network -LRB- as for example used in the KSR-1 and the SCI -RRB- , broadcast with snooping is still possible because each message travels around the entire ring . This has a higher latency than a bus but the implementation is simplified and the available bandwidth scales with the number of processors . The third option is to use direct communication between a node in the hierarchy and each of its children . It is this interconnect that has been chosen for the architecture described in this paper . Although broadcasting -LRB- and hence snooping -RRB- is a useful feature when designing protocols , we sacrificed it to provide low latency communication between the layers of the hierarchy . This implies that when a transaction should go to multiple receivers , the sender must communicate explicitly with each child . In this article we propose a DDM that uses a full duplex point-to-point interconnect . Unlike DDM designs previously published -LSB- 10 , 11 -RSB- , the scheme proposed here is fully scalable and overcomes starvation problems . The architecture and protocol are presented in Section 2 . The protocol has been fully implemented in an emulator . Section 3 presents this emulator , an estimation of the hardware overhead needed to support the DDM tree , and some preliminary performance measurements taken from the emulator . 2 Link-based DDM architecture The DDM -LSB- 4 -RSB- is organised hierarchically , using a tree of directories . Each directory registers the presence of all data items -LRB- an item is a small number of bytes -RRB- in the subtree below . The leaf nodes of the tree contain the actual storage for the data items . Both leaf nodes and directories store the item information in a set-associative manner . In this way , the shared data can freely migrate over the system . A leaf node that needs data posts a request transaction that is propagated to the nearest copy . The data is sent back along the same path . The directories update their bookkeeping information when the request and the copy passes . Because of the hierarchical organisation , -LRB- multiple -RRB- reads or a single write can be executed in a time bounded by the logarithm of the number of processors . The general DDM architecture leaves open particular design choices concerning the implementation . Designs have been studied with -LRB- amongst other things -RRB- different interconnects . They all preserve the features of a hierarchical network , no fixed home location and set-associative directories . The original protocol was elaborated for a bus-based implementation . SICS in Sweden is currently working on a hardware implementation of such a DDM -LSB- 10 -RSB- . Building large bus-based machines is relatively hard because of the quantity of wires and the electrical characteristics . An alternative DDM design based on a point-to-point communication network has been developed at the University of Bristol -LSB- 11 -RSB- . Both the SICS DDM and the Bristol DDM suffer however from the use of a centralised top directory that will eventually limit the performance . Like the Bristol DDM , the DDM proposed in this paper , referred to as the HORN DDM , uses links as the communication mechanism . The communication network can be either one of physical links , or a scalable packet switching network that allows point-to-point communication between arbitrary pairs of processors . In contrast with the Bristol DDM and SICS DDM , but as foreshadowed in the original DDM proposal -LSB- 4 -RSB- , the protocol presented here uses a split directory scheme -LRB- as described later -RRB- to improve scalability . The protocol is extended to retain sequential consistency on the split directories , and ensures forward progress even under extreme behaviour . To accomplish this , both write and read transactions are combined , while previously only read transactions were combined . Multiple reads are serviced in a time of O -LRB- log p -RRB- . Multiple writes of different processors to the same data item are serialised by the protocol in order to maintain sequential consistency . 2.1 Directory structure and interconnect As broadcasting is not available , the directories must maintain state information -LRB- introduced in this and the following section and summarised in Section 2.3 -RRB- for each branch individually . Therefore , the directories keep track of which branches below contain the data for any data item . The simplest and most flexible way to store this is by means of a bit mask , called the compass , for each item . The compass has one bit for each branch of the tree . If the bit is set then the data is present in that branch . An additional bit states if the data is present anywhere outside the subtree . This means that at least one bit of any compass is set : either the data is in the branches of the tree , or the data is outside the tree . Note that this last compass setting -LRB- data not here but somewhere else -RRB- can be seen as the ? Invalid ? state of conventional schemes . Since a directory has to maintain a state for every item present in the tree below , the directories higher up in the hierarchy must be larger than those lower in the hierarchy . Besides the difficulties of implementing a huge top level directory , the single node in the top of the tree will become a bottleneck when more processors are added . For these two reasons the directories at each level are split into smaller directories that maintain only part of the state information . For example , directories may be split so that at each level of the hierarchy there are as many directories as leaf nodes in the machine . Imagine a two level hierarchy with a fanout of 4 and hence 16 leaf nodes . The top level directory is split into 16 parts and each of the level 1 directories is split into 4 parts , resulting in the architecture depicted in Figure 2 . In practice , this level of splitting may be more than is really necessary but w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w A A A AA A A A AA A A A AA A A A AA @ @ @ @@ @ @ @ @@ @ @ @ @@ @ @ @ @@ QQQQQQQQ QQQQQQQQ QQQQQQQQ QQQQQQQQ A A A AA A A A AA A A A AA A A A AA @ @ @ @@ @ @ @ @@ @ @ @ @@ @ @ @ @@ ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ?? A A A AA A A A AA A A A AA A A A AA ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ???????????????????? ???????????????????? ???????????????????? ???????????????????? ????????????????????????????? ????????????????????????????? ????????????????????????????? ????????????????????????????? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ???????????????????? ???????????????????? ???????????????????? ???????????????????? @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ ? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ? Figure 2 : HORN DDM structure , the circles are directories , the squares are leaf memories . it remains as future work to determine the optimum splitting and branching factors . The directories are split according to the least significant bits of the item address . The leftmost top directory has the compasses of all items whose address ends with ` 0000 ' , the next one contains the compasses of ` ? ? ? 0001 ' , and so forth , the last one maintains the items with addresses ` ? ? ? 1111 ' . The leftmost middle directory stores the compasses of all items with an address ` ? ? ? 00 ' for the leftmost subtree -LRB- nodes 0 , 1 , 2 and 3 -RRB- . This directory therefore has links with top level directories 0000 , 0100 , 1000 and 1100 . Directory 4 in the middle is also linked to these top directories ; it has the compasses of all items ending with ` 00 ' for the subtree encompassing nodes 4 , 5 , 6 and 7 . The resulting interconnect between the directories is a Banyan graph -LSB- 12 -RSB- . This Banyan interconnect is only logical . The entire network may be built physically , but this will require a large number of node to node interconnections and hence could prove expensive . A less complex solution will be to collapse the structure according to the vertical boxes in Figure 2 . A leaf memory , an application processor and several directories can be placed on a single node . As we expect that the directories are idle for most of the time , we may combine the directory controllers to reduce the hardware overhead . Various possibleimplementations of this network are discussed and compared in a separate article -LSB- 13 -RSB- . Notice that in any case the directories have a task comparable to that of the router in an ordinary packet switching network . A directory lookup is more complex than a straightforward routing operation , but there is essentially no routing between directories . Also , directories can themselves use wormhole routing , only waiting for the header of a message before taking a protocol decision . Division of the directories using the lower order address bits is simple to implement but can lead to hot-spots in the network caused by certain memory access patterns . If this turns out to be a serious problem , we will hash the addresses at the leaf nodes before inserting them in the hierarchy -LSB- 14 -RSB- . 2.2 Coherency protocol The coherency protocol is transaction driven . The protocol has evolved from those developed at SICS -LSB- 10 -RSB- and Bristol -LSB- 11 -RSB- . The main novel features are the scalability with split directories , the prevention of starvation and the combination of multiple writes and of multiple reads and writes -LRB- previously only multiple reads could be combined -RRB- . The protocol has three basic transactions to obtain data : Read , Erase and Fetch . A Read transaction means that a node wants to obtain a shared copy of the item . The transaction will be forwarded to one of the nodes that has the data , selected from the compass . An Erase transaction is a request to destroy all outstanding copies in the system : the originator wishes to write to an item but has a shared copy that must first become exclusive . This transaction is propagated to every branch in the compass except the originating branch . A Fetch transaction indicates that a node does not have the data but wants an exclusive copy . Writing to data that has not been read locally since the last write by some other node is uncommon , but it occurs under false-sharing or when data is initialised . The Fetch is propagated to one of the branches in the compass to obtain the data , and Erase transactions are posted to all other branches in the compass to destroy their copies . When one of these three transactions is in progress , the item is in a TRANSIENT state . For this purpose , a single state bit is stored with each item that indicates if it is STABLE -LRB- no outstanding requests -RRB- or TRANSIENT -LRB- there is an outstanding request -RRB- . An item that is TRANSIENT goes back to STABLE when the request is fulfilled . There are four transactions to answer a request , called Copy , Data , Xclusiv and Inject . A Read is typically answered with a Copy containing a copy of the requested item . An Erase is normally answered with an Xclusiv meaning that the data has been erased successfully in that branch . A Fetch is either answered by two subsequent transactions Data , Xclusiv -LRB- where the Data carries a copy of the data soon to become exclusive -RRB- , or by a single Inject -LRB- carrying an exclusive copy -RRB- . These answers are propagated back to the node that performed the original request . To store this return path a TRANSIENT item remembers where the originating request came from using two bit-masks . These bit masks are called the shared and exclusive request masks and register which branches have sent which type of request . Like the compass , each bit of the request masks is associated with a branch of the tree . If the bit is set , that particular branch has an outstanding request , otherwise the branch has no outstanding request . As will be discussed in Section 2.3 these bit masks are only stored if an item is TRANSIENT , so the space overhead is small . Combining requests : Conflicts and Crossing transactions Single transactions are processed according to the description above . Many times , however , multiple Read and Fetch or Erase transactions for the same item may arrive at a directory . These transactions are combined by setting the appropriate bits in the shared and the exclusive request masks . Under certain conditions , the newly arrived transaction will continue . If the transaction is suspended , it is restarted when previous transactions on the item have been handled completely . In this way , Read transactions for example are combined as they propagate up the hierarchy and the resulting Copy transactions propagate in parallel back down the hierarchy . Because we have chosen an invalidating protocol and allow an item size that is larger than the smallest writable unit , multiple write operations must be handled sequentially to maintain consistency . To prevent starvation , the protocol takes care that all transactions are eventually answered . The rules that determine how directories behave are implemented in a state machine , but for clarity we present them here in the form of three invariants . Invariant 1 The compass has a bit set for each branch that has a copy of the data . The exclusive request mask has a bit set for each branch that wants exclusive access to the data . At most one branch may be present in both the compass and the exclusive request mask . The overlap between the exclusive request mask and the compass is reserved to denote a processor that has the data and that wants to erase all other copies . At most one such overlap must exist at any level , otherwise the machine will end up with two exclusive copies . The consequence of this is that when an Erase comes in while there is already an overlap , one of the Erase transactions must be cancelled . The Erase that has travelled the longest path always continues ; that choice is the most efficient and guarantees progress . It is easy to determine the longest path : an Erase that comes from above always wins , Erase transactions from below always lose . Notice that when such a conflict occurs , the same conflict will occur simultaneously at the directory from which the conflicting Erase originated . The directory highest in the hierarchy wins and ignores the second Erase ; the directory lower in the hierarchy loses and lets the second Erase transaction overrule the first . Invariant 2 If a bit in the exclusive request mask is set , then all branches pointed to by the compass will eventually post an Xclusiv except for the at most one branch that has an overlap between the compass and the exclusive request mask . This means that if an item has a bit set in the exclusive mask and a Fetch is received from a branch , this branch can simply be added to the exclusive request mask . If a directory has only a shared bit set when a Fetch comes in , then the Fetch has to be initiated as described in the beginning of this section . This invariant also implies that when a shared request is satisfied by a Copy , the branches receiving the copy -LRB- which are thus being added to the compass -RRB- should receive an Erase to destroy their copy1 . Likewise , if an exclusive request is satisfied and there are other exclusive requests waiting then a Fetch should be posted immediately after the Xclusiv to obtain the data for the next node . If there is more than one bit set in the exclusive mask , or if both the exclusive and the shared mask have bits set , then the next branch that will receive the data is selected on a round robin basis to prevent starvation . In this case , round robin means that the branches -LRB- below and the ? branch ? above -RRB- are selected clockwise starting 1It is tempting to optimise this by not sending the copy in the first place . This optimisation can however lead to starvation and non termination . For the same reason , a Fetch is not simply an optimisation replacing a Read followed by an Erase , the Fetch guarantees progress . w w w w w w w w w w w g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g 1 2 5 6 12 14 T P Q R ???????????????????? ?????????? H H H H H H H H H H A A A AA @ @ @ @ @ A A A AA @ @ @ @ @ @ @ @ @ @ s x x x x s s x x x x s s x x ? ? 9 AAU ?? * H HY 6 ? @ @I AAK @ @I 6 @ @I Figure 3 : Example : the arrows indicate that that branch appears in the compass , the x and s labels show bits set in the exclusive and shared request masks . next to the branch that provided the data . Because each branch that gets the data will eventually return it , all branches are serviced in turn . This results in a breadth first journey of the data along the nodes that want to have it . An example of this behaviour is shown in Figure 3 . Node 1 has the sole copy of a data item . Nodes 0 , 2 , 12 and 14 want the data exclusively while nodes 5 and 6 want a shared copy . As there were exclusive requests outstanding , Directory P has sent a Fetch to node 1 and will receive an Inject containing the data . The directory propagates the Inject to node -LRB- clockwise the first next to 1 -RRB- . As further requests are outstanding , the Inject is followed by a Fetch . When node returns the data , it is sent to directory T -LRB- again clockwise -RRB- and propagated via directory R to node 14 and then 12 before it is returned back to directory T . This time the data is sent to directory Q which in turn passes copies to nodes 5 and 6 simultaneously . The Inject to directory Q was followed by a Fetch -LRB- directory P still wants the data for node 2 -RRB- so after visiting nodes 5 and 6 it is returned to directory T and finally propagated to node 2 via directory P. Invariant 3 If a bit in the shared mask is set , then either the directory will receive a shared copy or it will first receive an exclusive copy . Consequently , when a Read is received for a TRANSIENT item , no other action has to be taken except that the branch should be added to the shared bit mask . If an exclusive request is satisfied while there are still shared requests outstanding then the Xclusiv will be followed by a Read to obtain a shared copy . Notice the asymmetry with Invariant 2 : because the compasses are not reliable when an Erase is in progress , a Read must be suspended . An Erase can safely continue during a Read , though the copies that are created as a result must be destroyed before the Erase can complete . This process terminates because the number of directories expecting a shared copy only decreases . Given these three invariants , the whole protocol is formulated . In our software implementation -LSB- 15 -RSB- , the protocol specification is expressed in not more than 300 lines . The specification is written as a state transition table that is converted automatically into the C code that forms the heart of each directory processor . The same description could equally be transformed into code for a protocol processor -LSB- 16 , 17 -RSB- or directly into a hardware description language . Replacement The directories store the state information set-associatively . When there is no more space to store the bookkeeping for an item , one of the other items in the set has to be evacuated . The directory selects an item to be replaced and acts as if a Fetch for that item came from above . For a leaf node this will result in the data being Injected to the directory above . In an interior directory , the Fetch will be sent to one of the branches containing the item , and Erase transactions are sent to the other branches in the compass , eventually resulting in an Inject , carrying the evacuated data to above . The directory above receives the Inject typically while the item is STABLE . It will update the compass as the data is no longer in that branch . If at any level of the tree there is another copy of the data below , the Inject is discarded as it concerns a duplicate . If the only other copies are above , the Inject is propagated to the directory higher in the hierarchy . If there are no other copies at all -LRB- and the transaction thus carries the only copy of the data -RRB- , the Inject is sent roundrobin through the hierarchy to find a memory with either space for it , or where it can replace a shared data item . Note that because Inject transactions can emerge spontaneously , the protocol must cope with the possibility that the Inject came from a node which has just been sent a Read , Fetch or Erase . Synchronisation operations Programs running on parallel machines must be able to synchronise and protect data against mutual access . This is ensured with the provision of primitives for locking and barriers . As synchronisation is a forced serialisation of at least two processors , it should be as efficient as possible . In most parallel machines this requires hardware support , but in the DDM the network and its combining allow us to do the synchronisation efficiently in software . Processors are forced to wait either by spinning on a locally cached variable -LRB- single threaded -RRB- or by suspending on an event if a cheap event scheduling mechanism is available as in the transputer . Using the network allows us to achieve efficient synchronisation and a simple programming model while keeping a simple protocol . A barrier between n nodes of a p processor system is constructed with O -LRB- n log p -RRB- transactions , while it is resolved in a time of O -LRB- logp -RRB- . A free lock can be obtained in a time of O -LRB- log p -RRB- worst case , if a lock is requested by multiple nodes they are serviced in turn . Both locks and barriers migrate to the node that needs them , in accordance with the DDM philosophy . This means that uncontested locks that are mostly used on the same processor are serviced in O -LRB- 1 -RRB- time . 2.3 Implementing the protocol The major part of the protocol can be implemented with a simple state machine . The expensive part of the implementation in terms of chip space is the storage of the bookkeeping information . A na ?? ve implementation would require a compass , two request masks , a state and a key for each item . This is a waste of chip space since only TRANSIENT items require the shared and exclusive masks , and experiments have shown that less than 1 % of the items are TRANSIENT at any time . : - - - ffl oe ppp ffi ae From above From below Input Queue From Type Address Data bytes W ? ? 6 ? ppp ss To above To below Output Queue To Type Address Data bytes State Machine X S Key Key C T X : Exclusive req mask S : Shared req mask C : Compass T : Transient state-bit Data Memory -LRB- only on the leaf nodes -RRB- Processor Cache TLB Proc - oe 6 6 ? ? 6 ? ? ? ? - oe ? ? ? ?? ? ? ? 6 6 oe oe - - Figure 4 : Architecture of the HORN DDM directory The directory bookkeeping is therefore stored in two set-associative memories : one for the compasses and state of all items located below this directory , and one for the request masks of the TRANSIENT items . In the exceptional case that the transient storage is not large enough , excessive requests will simply be answered by a Fail transaction , forcing them to back off temporarily . Since a Fail will bring TRANSIENT items back to STABLE , the machine will not deadlock . A block diagram of the directory showing the major components and the data flow is sketched in Figure 4 . The state machine takes messages from the input queue and handles them in FIFO order , with each message generating zero or more output messages . Note that the figure is not to scale , as the registers and the state machine will only consume a small portion of the chip area , while the major part will be needed to store the keys and compasses . In Section 3.2 the overhead is estimated in more detail . The directory at the leaf nodes has a slightly different implementation . First the compasses are only two bits wide -LRB- one bit denoting that the data is valid here , and one bit denoting if the data is valid elsewhere -RRB- . Secondly , the actual data storage is present . The processor/directory interface has not yet been designed as much will depend on the memory interface of the processor -LRB- we plan to use the successor of the Inmos T9000 -RRB- . Another point is whether to combine the directory state-machines on different levels . This may be worthwhile to reduce the hardware overhead , but before deciding on this we want to have insight into the utilisation of the directories and the extra complexity that is introduced in the design . 3 A complete implementation We do not yet have a hardware implementation of the architecture . To assess the performance and to get a feeling for the complexity of the protocol , an emulator has been developed -LSB- 15 -RSB- . The emulator is not a simplified model but a full implementation of the HORN DDM , in software , on a multiprocessor transputer system . Although slow compared with a hardware implementation it is fast enough to measure the performance of real programs , even on machines with many processors . The emulator has a parametrisation scheme that allows us to evaluate the HORN DDM for given hardware parameters . This includes the topology , splitting factor , Program Parameters 1 8 24 32 48 72 Wang 100000 1.15 0.084 0.031 0.023 0.016 0.011 Water 343 2 16.01 2.144 0.836 0.617 0.462 0.302 Wave 240 2.03 0.267 0.094 0.071 0.054 0.040 Barnes 1000 4 11.18 1.658 0.642 0.512 0.368 0.295 Aurora Tina 8.39 1.232 0.662 0.634 0.539 0.535 Cholesky bcsstk14 1.71 0.326 0.196 0.177 0.159 0.175 Mp3d 30000 5 0.79 0.320 0.190 0.159 0.119 0.140 Locus Primary1 2.17 0.445 0.430 0.461 0.564 0.653 Table 1 : Program timings in seconds . level of prefetching , level of multithreading , and all timing parameters . The timing parameters are used by the emulator to calibrate all components so that the emulator gives the timings that we can expect on a real implementation . The performance figures for 1 to 72 processor systems are presented in Section 3.1 . Although we have emulated an architecture with 81 leaf nodes , our hardware resources did not allow any splitting of the directories for this topology . As such , these results are not included as they are limited by a central bottleneck at the top of the machine . Because we want to know if a split directory scheme gives acceptable performance , we have disabled multithreading , as that would -LRB- partially -RRB- hide latencies , and would complicate interpretation of the results . The extra costs of implementing the HORN DDM in comparison with simpler VSM machines are estimated in Section 3.2 . 3.1 Performance evaluation The emulator has parameters to specify the hardware configuration . As we do not yet know the specification of our target processor , the figures presented below have been measured with parameter settings that we consider feasible within the next few years . The processor runs at 100 MIPS and 25 MFLOPS , the network introduces a delay of 400 ns per hop and has a bandwidth of 100 MBytes/s , and the protocol engine handles a transaction in 200 ns . Consequently , in an empty network a single hop for a transaction without data takes 600 ns -LRB- equivalent to 60 instructions on the processor -RRB- , while a transaction with 64 bytes of data -LRB- an item -RRB- takes 1240 ns . This assume a simple design?a more complex design could begin processing transactions while they were still arriving and hence reduce these latency figures . These components should represent parts that will be commodity components in the stated time scale , rather than state-of-the-art parts . The optimum item size and prefetch strategy for these settings depend on the application . An item size of 64 bytes with three prefetched items on a read miss has been found to be within a few percent optimal for all the benchmarks we have tested so far -LSB- 15 -RSB- . The programs that we run to test the performance of the system include the entire SPLASH benchmark suite -LSB- 18 -RSB- , the Aurora parallel Prolog system -LSB- 19 -RSB- running a journey planner program , Wang -LRB- an equation solver -RRB- , and Wave -LRB- a water height simulation -RRB- . For the purposes of this article , we include only a subset of the SPLASH benchmarks , including examples with both good and bad performance . The emulator has been used to evaluate the chosen benchmarks on a -LRB- hypothetical -RRB- DDM . The emulator was run on a machine equipped with 131 transputers , allowing us to run programs on DDM 's with up to 72 nodes . The configurations we have chosen to evaluate vary from a single node to a 72 node architecture , as sketched in Figure 5 . The largest DDM we can configure is actually an 81 node machine -LRB- using 121 transputers -RRB- , but this is a pure tree topology and has consequently a central node in the top giving rise to congestion in the DDM ; the performance of this 81-node architecture is therefore limited and less interesting . The execution times of the parallel phases as measured on the HORNDDM are presented in Table 1 . The corresponding speedup figures are depicted in Figure 6 . Some of the programs -LRB- such as Mp3d -RRB- have a low speedup while others show good performance . Wang , Wave and Water all perform very well , Barnes shows good performance up to about 32 nodes before starting to level off . This is due to the relatively small data set used for the experiment . Aurora , the parallel Prolog system , along with Cholesky are limited by the data sets used . The Prolog benchmark , Tina , and the matrix used for Cholesky do not have sufficient parallelism for more than about 16 nodes . Mp3d and Locus demonstrate poor locality and as a result poor performance . In the case of Locus , a different input circuit may give better performance but Mp3d will not perform better for even very large data sets . Notice that in Mp3d performance does improve steadily to a speedup of 6.6 on 48 nodes . The reason it levels off at 72 nodes is that our emulation platform was not large enough to model a machine with sufficient directory splitting to cope with Mp3d 's high miss ratios . The applications with sufficient parallelism are still improving and it seems likely that their performance on a split directory DDM will scale well beyond 100 nodes . Further experiments on larger machines will have to conform this . All the topologies used in these experiments have split directories higher in the network . We have designed the machine to use split directories to avoid bottlenecks in the network and hence poor performance . To demonstrate the effects of splitting , we have run Wang on two 72 node architectures , one with no splitting and one with splitting at the top node . These machines are pictured in Figure 7 along with the 1 node 8 nodes 24 nodes 32 nodes 48 nodes 72 nodes Figure 5 : Topologies of the experimental configurations . 10 20 30 40 50 60 70 10 20 30 40 50 60 70 Speedup of application Nodes aurora.tina water .343 barnes .1000 locus.Primary1.grin mp3d .30000 wang .100000 cholesky.bcsstk14 wave .240 Figure 6 : Program speedups . Run time = 14 msecs Run time = 11 msecs Figure 7 : Run time of Wang on two 72 node machines . run-times of Wang on each machine . It is clear from the results that splitting has a clear advantage and we have seen similar behaviour from other benchmarks . 3.2 Costs evaluation The most expensive element of a hardware HORN DDM directory will be the setassociative memory that is needed to store the compasses and state bits . The state machine is quite small : 10 input transactions with two states plus cases depending on the presence of bits in the shared and exclusive request masks . The memory storing the exclusive and shared request masks need be only a few percent of the size of the memory storing the compasses according to our experimental results . To show the level of overhead required , the size of the set-associative directories has been quantified in Table 2 for two example machines . The first is a small , 64 node machine and the other is a large 4096 node machine . In each case the number of bits needed to store the state bits , keys , and compasses at each level is shown . Because the fanout clearly affects the size and number of bit masks required , the overhead is estimated for various fanouts . We assume that the address space for the keys is large enough to address all items in the machine uniquely -LRB- a 13 Configuration Bits needed for : Total % Overhead nodes fanout levels State Key Compass Bits 64-byte item 2 6 7 35 20 62 12 % 64 4 3 4 20 17 41 8 % 8 2 3 15 20 38 7 % 2 12 13 104 38 155 30 % 4 6 7 56 32 95 18 % 4096 8 4 5 40 38 83 16 % 16 3 4 32 53 89 17 % 64 2 3 24 132 159 31 % Table 2 : Space overhead for maintaining keys and compasses . worst case assumption since one normally assumes a certain degree of sharing -RRB- , and also assume four-way set-associativity . Note that the overhead in bits per item is independent of the size of an item , the number of items in the machine and the degree of splitting . The overhead is very acceptable : with a 64-byte data item the overhead is less than 20 % on the large machine and less than 10 % on the small machine . Note that the overhead is not the only criterion in finding the appropriate value for the fanout . A low fanout increases the number of levels required in the tree and hence the latency whereas a high fanout is inefficient because of the forced serialisation of messages especially when invalidating a widely shared data item . A suitable balance between these two has to be found . In these cost estimates we have ignored the overhead of the protocol state machines . Splitting directories requires the addition of extra state machines so this overhead becomes more significant -LRB- although we may be able to share this overhead vertically as denoted in Figure 2 -RRB- . A more precise estimation of the hardware overhead is currently in progress . 4 Conclusion A new coherency algorithm has been presented for a link-based Data Diffusion Machine . The architecture supports split directories -LRB- improving the scalability -RRB- without losing the strong consistency model . A shared copy can be obtained in a time bounded by O -LRB- log p -RRB- ; data can be fetched exclusively in a time bounded by O -LRB- log p -RRB- when only one node wants an exclusive copy and by O -LRB- p -RRB- when all nodes simultaneously want the data exclusively . In the latter case it is guaranteed that all requests are eventually answered so starvation can not occur . In contrast to previous DDM protocols , our protocol is able to combine reads , writes and combinations of reads and writes . The protocol has been completely implemented , in software , as part of the DDM emulator . As well as demonstrating the feasibility of the protocol , we have been able to calibrate the emulator to measure the machine 's performance . We have run the SPLASH suite , a parallel Prolog system , Wang -LRB- an equation solver -RRB- and Wave -LRB- an ocean simulation -RRB- as application benchmarks . For many programs the performance figures are very satisfactory . We can expect some of the applications to perform better in a multi-threaded machine but some of these benchmarks have fundamentally limited parallelism . We have seen the benefit of splitting the higher level directories . Splitting reduces the size of each directory and helps to prevent bottlenecks in the network . With some applications , the introduction of splitting can dramatically improve performance . We have presented a cost evaluation of the DDM that shows that the cost of a DDM is not excessive . The communication network looks complex , but it does not need to be implemented physically . Several simple implementations are discussed elsewhere -LSB- 13 -RSB- . The extra costs in terms of memory for storing the directory are no more than 10 or 20 % . Additionally extra hardware is needed to implement the state machines and the key-matching . In exchange for this investment the programmer gets a strongly consistent machine with a uniform memory access model . Future performance studies have to show if the benefits of an architecture with freely migrating data do indeed justify the overheads . References -LSB- 1 -RSB- H.E. Bal , M.F. Kaashoek , and A.S. Tanenbaum . Orca : A Language for Parallel Programming of Distributed Systems . IEEETransactionson Software Engineering , 18 -LRB- 3 -RRB- :190 ? 205 , March 1992 . -LSB- 2 -RSB- HPF Forum . High Performance Fortran Language Specification . Scientific Programming , 2 -LRB- 1 -RRB- , June 1993 . -LSB- 3 -RSB- Kai Li . Shared Virtual Memory on Loosely Coupled Multiprocessors . PhD thesis , Yale University , September 1986 . -LSB- 4 -RSB- David H. D. Warren and Seif Haridi . The Data Diffusion Machine?A Scalable Shared Virtual Memory Multiprocessor . In Proceedings of the 1988 International Conference on Fifth Generation Computer Systems , pages 943 ? 952 , Tokyo , Japan , December 1988 . -LSB- 5 -RSB- D. Chaiken , J. Kubiatowicz , and A. Agarwal . LimitLESS Directories : A Scalable Cache Coherence Scheme . In Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS-IV , pages 224 ? 234 , April 1991 . -LSB- 6 -RSB- D. Lenoski , J. Laudon , T. Joe , D. Nakahira , L. Stevens , A. Gupta , and J. Hennessy . The DASH Prototype : Logic Overhead and Performance . IEEE Transactions on Parallel and Distributed Systems , 4 -LRB- 1 -RRB- :41 ? 61 , January 1993 . -LSB- 7 -RSB- KSR . KSR Technical Summary . Kendall Square Research , Waltham , MA , 1992 . -LSB- 8 -RSB- IEEE . Draft Standard : Futurebus P896 .1 . IEEE Computer Society Press , Los Alamitos , CA , 1990 . -LSB- 9 -RSB- IEEE . Scalable Coherent Interface IEEE P1596 - SCI Coherence Protocol . IEEE Computer , March 1989 . -LSB- 10 -RSB- Erik Hagersten . Toward Scalable Cache Only Memory Architectures . PhD thesis , Swedish Institute of Computer Science , 1992 . -LSB- 11 -RSB- Sanjay Raina and David H. D. Warren . Traffic Patterns in a Scalable Multiprocessor through Transputer Emulation . In Proceedings of the 25th Hawaii International Conference on System Sciences , pages 267 ? 276 , 1992 . -LSB- 12 -RSB- Kai Hwang and Fay?e A. Briggs . Computer Architecture and Parallel Processing . McGraw?Hill Book Company , 1984 . -LSB- 13 -RSB- Henk L. Muller , Paul W. A. Stallard , and David H. D. Warren . Implementing the Data Diffusion Machine network using C104 style routers . Technical Report CSTR-95 - . . , Department of Computer Science , University of Bristol , February 1995 . -LSB- 14 -RSB- Abhiram G. Ranade . How to Emulate Shared Memory . In 28th IEEE Symposium on Foundations of Computer Science , pages 185 ? 194 , 1987 . -LSB- 15 -RSB- Henk L. Muller , Paul W. A. Stallard , David H. D. Warren , and Sanjay Raina . Parallel Evaluation of a Parallel Architecture by means of Calibrated Emulation . In Proceedings of the 8th International Parallel Processing Symposium , pages 260 ? 267 , Cancun , Mexico , April 1994 . IEEE Computer Society Press . -LSB- 16 -RSB- Ian Page . Parameterised Processor Generation . In W. Moore and W. Luk , editors , More FPGAs , pages 225 ? 237 . Abingdon EE&CS Books , Oxford , England , 1994 . -LSB- 17 -RSB- Jeffrey Kuskin , David Ofelt , Mark Heinrich , John Heinlein , Richard Simoni , Kouroush Ghararchorloo , John Chapin , David Nakahira , Joel Baxter , Mark Horowitz , Anoop Gupta , Mendel Rosenblum , and John Hennessy . The Stanford FLASH Multiprocessor . In Proceedings of the 21st Annual International Symposium on Computer Architecture , pages 302 ? 313 , Chicago , Illinois , April 1994 . IEEE Computer Society Press . -LSB- 18 -RSB- Jaswinder Pal Singh , Wolf-Dietrich Weber , and Anoop Gupta . SPLASH : Stanford Parallel Applications for Shared-Memory . Technical report , Computer Systems Laboratory , Stanford University , 1991 . -LSB- 19 -RSB- E. Lusk , D.H.D. Warren , and S. Haridi et . al. . The Aurora Or-Parallel Prolog System . New Generation Computing , 7:243 ? 271 , 1990 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

memory architectures
scalability
virtual shared memory