Deterministic distributed multicache coherence protocol

Abstract: An efficient streamlined cache coherent protocol for a multi-processor multi-cache computing system. Each subsystem includes at least one processor and an associated cache and directory. The subsystems are coupled to a global interconnect via global interfaces. In one embodiment, each global interface includes a request agent (RA), a directory agent (DA) and a slave agent (SA). The RA provides a subsystem with a mechanism for sending read and write request to the DA of another subsystem. The DA is responsible for accessing and updating its home directory. The SA is responsible for responding to requests from the DA of another subsystem. Each subsystem also includes a blocker coupled to a DA and associated with a home directory. All requests for a cache line are screened by the blocker associated with each home directory. Blockers are responsible for blocking new request(s) for a cache line until an outstanding request for that cache line has been serviced. A "locked" state managed by the blocker greatly reduces corner cases and simplifies solutions in the few remaining corner cases.

Description: This invention relates to caches. In particular, this invention relates to a cache coherency scheme for multiple caches in a multiprocessor system. With the shift of computing technology to the "network is the computer" paradigm, the need for a shared global memory address space and a coherent caching system in a networked computing system becomes increasingly important. Figure 1A is a block diagram showing one such networked computer system 100 with a conventional non-uniform memory architecture (NUMA). System 100 includes a plurality of subsystems 110, 120, ... 180, coupled to each other via a global interconnect 190. Each subsystem is assigned a unique network node address. Each subsystem includes one or more processors, a corresponding number of memory management units (MMUs) and caches, a main memory assigned with a portion of a global memory address space, a global interface and a local subsystem interconnect. For example, subsystem 110 includes processors 111a, 111b ... 111i, MMUs 112a, 112b, ... 112i, caches 113a, 113b, ... 113i, main memory 114, global interface 115 and subsystem interconnect 119. Data from main memories 114, 124, ... 184 may be stored in one or more of caches 113a ... 113i, 123a ... 123i, and 183a ... 183i. Thus, cache coherency among caches 113a ... 113i, 123a ... 123i, and 183a ... 183i is maintained in order for system 100 to execute shared-memory programs correctly. In order to support a conventional directory-based cache coherency scheme, subsystems 110, 120, ... 180 also include directories 116, 126, ... 186 coupled to global interfaces 115, 125, ... 185, respectively. Referring now to Figure 1B, each global interface, e.g., interface 115 includes a slave agent ("SA"), a request agent ("RA") and a directory agent ("DA"), e.g, SA 115a, RA 115b and DA 115c. Each DA is responsible for updating its associated directory with the status of all cached copies of its (home) main memory, including copies cached in other subsystems. The status of cached copies in each node are recorded in directories 116, 126, ... 186 as one of four states per node. An invalid ("I") state indicates that the node, i.e., subsystem, does not have a copy of the data line of interest. A shared ("S") state indicates that the node has an S copy, and that possibly other nodes may have S copies. An owned ("O") state indicates that the node has an O copy, and that possibly other nodes may have S copies. Note that the node with the O copy is required to perform a write-back upon replacement. Finally, a modified ("M") state indicates that the node is the sole owner of the data line, i.e., there are no S copies in the other nodes. A RA provides a subsystem with a mechanism for sending read and write requests to the other subsystems. A DA provides access to and is responsible for updating its associated home directory. An SA is responsible for responding to requests from the DA of another subsystem. Requests for data and responses are exchanged by the respective agents between subsystems 110, 120, ... 180 in the form of data/control packets, thereby enabling subsystems to keep track of the states of their caches 113a ... 113i, 123a ... 123i, and 183a ... 183i in directories 116, 126, ... 186, respectively. These data/control packets are transported between subsystems via global interconnect 190. Unfortunately, since global interconnect 190 may be based on any one of a number of conventional networking protocols, e.g., a collision sense multiple access (CSMA) protocol, from the timing viewpoint, subsystems 110, 120, ... 180 may be loosely coupled to each other at the network layer of the protocol. As such, while the arrival of packets end-to-end is guaranteed, the order of arrival of the packets is not necessarily guaranteed. The out-of-order arrival of packets at subsystems 110, 120, ... 180 is problematic because they can result in "corner cases" which, if not detected and resolved, can disrupt cache coherency. One such corner case is illustrated by Figures 2A-2D in which a data packet associated with an earlier-in-time read-to-share request (RTS_req) arrives after the cache line is prematurely invalidated as a result of the arrival of a later-in-time read-to-own request (RTO_req) initiated by another subsystem. In this example, initially, subsystem 110, subsystem 120 and a fourth subsystem (not shown in Figure 1A) have shared ("S") copies of a data line from the memory space of subsystem 180. Referring first to Figure 2A, RA1 of global interface 115 of subsystem 110 sends a RTS_req packet to DA8 of global interface 185 of subsystem 180. As shown in Figure 2B, DA8 responds by initiating the transfer of a data packet to the requesting RA1. Next, as shown in Figure 2C, before the data packet arrives at RA1, RA2 of global interface 125 of subsystem 120 sends a read-to-own request (RTO_req) packet to DA8. Figure 2D shows DA8 respond by initiating the transfer of a data packet to RA2. In addition, DA8 sends invalidate (Invld) packets to SA1 and SA4, the slave agents of subsystem 110 and the fourth subsystem, respectively. Unfortunately, the later-in-time Invld packet arrives at SA1 before the earlier-in-time data packet arrives at RA1. As a result, SA1 receives the Invld packet first and proceeds to invalidate the old S copy of the data line of interest. Subsequently, RA1 receives the data packet, but is unable to update the value of its S copy because it has been erroneously and prematurely marked Invld. Several conventional brute-force handshaking protocols for resolving corner cases do exist. Figures 3A-3F illustrate one prior art solution to the corner case described above. Again, using the same starting conditions as the example illustrated by Figures 2A-2D, subsystem 110, subsystem 120 and the fourth subsystem have S copies of a data line from the memory space of subsystem 180. Referring first to Figure 3A, RA1 of subsystem 110 sends a RTS_req packet to DA8 of subsystem 180. As shown in Figure 3B, DA8 responds by initiating the transfer of a data packet to the requesting RA1. DA8 then idles while waiting for a read-acknowledgment (RTS_ack) packet from RA1. Next, as shown in Figure 3C, RA2 sends a RTO_req packet to DA8. However, DA8 is idle because it is waiting for a RTS_ack packet from RA1 to arrive, and hence is unresponsive. As shown in Figure 3D, after receiving the RTS_ack packet from RA1, DA8 is no longer idle and is now able to respond to the RTO_req packet from RA2. Accordingly, as shown in Figure 3E, DA8 sends Invld packet(s) to any SAs of subsystems with S copies of the data line of interest. In this example, DA8 sends Invld packets to SA1 and SA4. DA8 is also responsible for sending a data packet together with the #_Invld to RA2. Subsequently, as shown in Figure 3F, RA2 counts the number of incoming Invld_ack from SA1 and SA4 thereby avoiding the corner case illustrated by Figures 2A-2D. Unfortunately, the above-described brute-force hand-shaking solution for handling and/or reducing corner cases is inefficient because of the excessive number of handshaking control packets. These extra control packets substantially increase the network traffic. In other words, the "cure" for the infrequent but disastrous corner cases substantially degrade the efficiency of the network. US Patent 5,297,269, discloses a cache coherency protocol for a multi-node computer system. The nodes are central processing units, each being connected to one or more memories by a memory/node coupling mechanism. The memory/node coupling mechanism is a cross-bar switch coupled point-to-point to one or more main memory modules and nodes and wherein each node (central processing unit or other device) of the system has a right-back cache. Cache coherency throughout the system is maintained by each memory module through a memory directory resident on the module. Transitional states tell the main memory module whether an outstanding data transfer operation regarding a particular data block is being executed so that the memory module can block or inhibit a subsequent read request for that data block until the already commenced operation has been completed. "Cache coherency protocol for multiprocessor computer system", provides in the memory directory for each data block a field for identifying the processor which owns the data block in question. Data block ownership information is used to determine where requested owned data is located. This reference does not however send an invalidate acknowledgement from the third subsystem to the second subsystem in response to the invalidate request, nor terminate blocking when the second subsystem receives a completion acknowledgement from the first subsystem indicating that the data and invalidate acknowledgement have arrived at the first subsystem, thus queuing subsequent requests for a cache line at the cache lines directory agent. An article by David Chaiken, et al, entitled "LimitLESS Directories: A Scalable Cache Coherence System" published in Computer Architecture News, April 1991, at page 224 describes the Alewife for a scalable limitless cache coherence scheme. The cache coherence protocol realises the performance of a full-map directory protocol, with a memory overhead of a limited directory, but without excessive sensitivity to software optimization. The protocol is supported by a large-scale distributed-memory multiprocessor. Each processing node contains a processor, a floating-point unit, a cache and portions of the system's global shared memory and directory. A small set of pointers is implemented in the memory modules. When necessary, however, a memory module is able to interrupt a processor for software emulation of a full-map directory. The Alewife machine has an add counter to sum the invalidate acknowledgements in the directory of the cache line. The Alewife machine then, after adding all acknowledgements informs the requester that all the acknowledgements have been received, which process is inefficient. An article by Andreas Nowatzyk, entitled "The S3.mp Scalable Shared Memory Multiprocessor" published in the proceedings of the Twenty-Seventh Annual Hawaii International Conference on System Sciences, 1994 at page 144 describes a Scalable Shared Memory Multiprocessor. The scalable shared memory multiprocessor system described in this article is unlike conventional CC-NUMA MPs which strive to deliver the most MFLOPS to one scientific application. The scalable shared multiplexer system described in this article is optimized for a large collection of independent and cooperating parallel applications that share common computing resources which may be spatially distributed. The nodes may be separated by up to 200 metres, which means that a system could be distributed over an entire building. The described system relies on a non-deterministic protocol which means that transactions only interact unpredictably with each other. The described system implements distributed directories and point-to-point messages that are sent over a packet switched interconnect fabric. Each node in the multiprocessor system may have several processors, each equipped with caches that are kept consistent via snooping. When a processor accesses a remote memory location, the memory controller translates the bus transaction into a message that is sent to the remote memory controller, which replies with the data and maintains a directory of externally cached blocks. Where two local nodes are both requiring write access to a same memory block, a write-invalidate protocol is employed, whereby a first request to arrive at a home node will be granted exclusive access. When the second request arrives, it causes an ownership transfer sequence. As the interconnect system described does not preserve order, an ownership transfer message may arrive at a node before it receives an exclusive ownership grant. In such a case, it is assumed that an exclusive grant is in transit and the transfer is queued until it arrives. Hence, there is a need for a simple and streamlined cache coherency protocol which handles and/or reduces corner cases without substantially increasing network traffic. Advantages of the present invention include reduction of complicated race conditions resulting from the corner cases, ease of formal verification of the protocol due to the reduction of the race conditions, and increased reliability of the resulting cache coherent computer system. Aspects of the present invention are set out in the accompanying independent claims. Preferred features are set out in the accompanying independent claims. In one aspect of the invention, there is provided a directory-based cache accessing method for a multiprocessing computer system as set forth in claim 1. In another aspect of the invention, there is provided a multiprocessing computer system as set forth in claim 23. Embodiments of the present invention provide an efficient streamlined cache coherent protocol for a multi-processor multi-cache computing system. Each subsystem can include at least one processor and an associated cache and directory. The subsystems can be coupled to a global interconnect via global interfaces. In one embodiment, each global interface includes a request agent (RA), a directory agent (DA) and a slave agent (SA). The RA provides a subsystem with a mechanism for sending read and write requests to the DA of another subsystem. The DA is responsible for accessing and updating its home directory. The SA is responsible for responding to requests from the DA of another subsystem. Further, in accordance with an embodiment of the invention, each subsystem also includes a blocker. In this embodiment, each blocker is coupled to a DA and is associated with a home directory. All requests for a cache line are screened by the blocker associated with each home directory. Blockers are responsible for blocking new request(s) for a cache line until an outstanding request for that cache line has been serviced. Although counterintuitive, since blocking causes the new requests to be processed sequentially, a "locked" state managed by the blocker simplifies solutions by removing the few remaining corner cases. In one embodiment, the blockers also include queues for storing pending requests. Variations and modifications to the blocker are also possible. For example, write-backs and read-to-own requests may be given priority for service over read-to-share requests. Exemplary embodiments of the invention are described hereinafter, by way of example only, with reference to the accompanying drawings, in which: Figure 1A is a block diagram showing a networked computer system 100 with a conventional non-uniform memory architecture (NUMA). Figure 1B is a block diagram of the global interface of the computer system of Figure 1A which includes a directory agent ("DA"), a request agent ("RA") and a slave agent ("SA"). Figures 2A-2D illustrate a corner case in which a data packet associated with an earlier-in-time read request arrives after the cache line is prematurely invalidated by a later-in-time read-to-own request. Figures 3A-3F illustrate a conventional handshaking protocol for solving the corner case of Figures 2A-2D. Figure 4A is a block diagram showing an exemplary cache coherent networked computer system 400 of the present invention. Figure 4B is a block diagram of the global interface of the computer system of the present invention which includes a blocker, a directory agent ("DA"), a request agent ("RA") and a slave agent ("SA"). Figures 5A-5D illustrate an exemplary blocking of requests for the directory agent of Figure 4B until an outstanding read-to-share (RTS) request has been serviced. Figures 6A-6B and 6C-6D illustrate an exemplary blocking of requests for the directory agent of Figure 4B until an outstanding read-to-own (RTO) request has been serviced. Figures 6A-6B and 6E-6G illustrate an alternative way of blocking requests by the directory agent of Figure 4B until an outstanding read-to-own (RTO) request has been serviced. Figures 7A-7C illustrate an exemplary blocking of requests for the directory agent of Figure 4B until an outstanding write-back (WB) request has been serviced. The following notations and nomenclature are used herein: An invalid ("I") state indicates that a node/subsystem does not have a (cached) copy of a data line of interest. A shared ("S") state indicates that the node/subsystem, and possibly other nodes, have a shared (cached) copy of the data line of interest. An owned ("O") state indicates that the node/subsystem is the owner, i.e. this node has a "master copy". As such, this node must write the "master copy" to another node before the cache line can be reused. It is possible for other nodes to have a shared copy of the data line of interest. A modified ("M") state indicates that the node/subsystem has the only (cached) copy of the data line of interest. A blocked ("B") state indicates that a read/write request is outstanding for the data line of interest. A request agent ("RA") provides a subsystem with a mechanism for requesting access to a cached data line of interest from another subsystem. A directory agent ("DA") provides a subsystem with a mechanism for accessing its directory to track the status of copies of its main memory. A slave agent ("SA") provides a subsystem with a mechanism for responding to a data request from another subsystem. In the following description, numerous details provide a thorough understanding of embodiments of the invention. These details include functional blocks and an exemplary cache directory to assist a designer in implementing a cost-effective cache coherent computer system. In addition, while embodiments of the present invention are described with reference to a specific cache coherent scheme for an exemplary multi-cache multi-processor computer system, the invention is applicable to a wide range of caches and network architectures. In other instances, well-known circuits and structures are not described in detail so as not to obscure the invention unnecessarily. Figure 4A is a block diagram showing an exemplary embodiment of a cache coherent networked computer system 400 according to the present invention. System 400 includes a plurality of subsystems 410, 420, ... 480, coupled to each other via a global interconnect 490. Each subsystem includes one or more processors, a corresponding number of memory management units (MMUs) and caches, a main memory assigned with portion of a global memory address space, a global interface and a subsystem interconnect. For example, subsystem 410 includes processors 411a, 411b ...411i, MMUs 412a, 412b, ...412i, caches 413a, 413b, ...413i, main memory 414, global interface 415 and subsystem interconnect 419. Subsystems 410, 420, ...480 also include directories 416, 426, ... 486 coupled to global interfaces 415, 425, ...485, respectively. In the cache coherency scheme of the present embodiment, as shown in Figure 4B, each global interface, e.g., interface 415, includes a slave agent ("SA"), a request agent ("RA"), a directory agent ("DA") and a blocker, e.g., SA 415a, RA 415b, DA 415c and blocker 415d. Thus there is a one-to-one correspondence between each DA and each directory. Blockers, DAs, RAs and SAs can also be incorporated in circuits separate from the respective global interfaces. Each blocker is coupled to a DA and is responsible for holding pending request(s) for cache lines while outstanding request(s) are being serviced. Note that the "blocking" functionality can be provided by directories 416, 426, ... 486 as described below or alternatively by adding dedicated blocking logic. The dedicated blocking logic is responsible for queuing outstanding transactions and storing a cache-line address for each of the outstanding transactions. Subsequently, the dedicated blocking logic compares the address of each outstanding transaction with all the old cache-line addresses before allowing transactions to pass to the corresponding DAs. In this alternative implementation, each completion signal only needs to clear its respective address. DAs of each subsystem are responsible for updating its associated-directory with the status of each cached-line sized portion of its (home) main memory. Accordingly, for each such portion of the main memory, the directory includes a status indicating which subsystem(s), if any, have cached copies of that particular portion. In this implementation, each directory is a home directory, i.e., local directory, for a subsystem. Thus, each directory includes entries for cached copies of data lines associated with the subsystem, describing the state of cached copies in all subsystems. DAs are also responsible for forwarding the appropriate request packets to the respective SAs. Figures 5A-5D illustrate example I in which subsystem 410 needs read permission, i.e., a shared (S) copy, from a data line within the home address space of subsystem 480, but is "owned" by subsystem 420. First, as shown in Figure 5A, RA1 of global interface 415 of subsystem 410 sends a read-to-share request (RTS_req) packet to blocker 485d of global interface 485 of subsystem 480. Blocker 485d responds by causing DA8 to enter a blocked (B) state (denoted by the "dotted" circle), thereby freezing new requests to DA8 for the data line of interest. Next, as shown in Figure 5B, DA8, now in the B state, marks its home directory 486 to reflect the requesting RA1's new status as a sharer, and forwards the RTS_req packet to slave agent SA2 of global interface 425 of subsystem 420. The blocked ("B") state indicates that there is a write/read request outstanding and subsequent request(s) for the same data line are blocked until the pending request has been serviced. In this implementation, blocker 485d, associated with DA8, blocks new requests for the data line by temporarily storing the requests in a local first-in-first-out (FIFO) queue or by flow controlling new requests. SA2 responds to the RTS_req packet by sending a data packet to RA1 and remains the "owner" of the data line, as shown in Figure 5C. Finally, as shown in Figure 5D, upon receipt of the data packet from SA2, RA1 sends a RTS-completion (RTS_compt) packet to blocker 485d. Blocker 485d causes DA8 to leave the B state. DA8 is now able to service any queued request for the data line of interest. In example II illustrated by Figures 6A-6B and 6C-6D, subsystem 410 needs write permission, i.e., an owned (O) copy, from a data line whose home address space is in subsystem 480 but is owned by subsystem 420. In addition, a third subsystem and fourth subsystem (both not shown in Figure 4A) have S copies of the data line. First, as shown in Figure 6A, RA1 sends a read-to-own request (RTO_req) packet to DA8 via blocker 485d. Next, as shown in Figure 6B, blocker 485d causes DA8 to enter the B state and freezes new transactions to the cache line. DA8 marks its home directory to reflect the requester RA1's new status as the owner of the data line of interest, i.e., RA1's new status is owned (O). DA8 retrieves the number of sharers (#_sharers), two sharers in this example, of this data line from its directory, appends the #_sharers to the RTO_req packet and forwards the RTO_req packet to SA2. DA8 is also responsible for sending invalidate request (Invld_req) packets to SA3 and SA4 of the third and fourth subsystems, respectively. The Invld_req packets also include the identity of requesting RA1. Figure 6C shows SA2 responding to arrival of the RTO_req packet by invalidating its copy of the data line, and sending a data packet to RA1 together with the total number of sharers. Upon receipt of the respective Invld_req packets from DA8, both SA3 and SA4 send invalid acknowledgment (Invld_ack) packets to RA1 and also invalidate their respective S copies. Finally, as shown in Figure 6D, after the arrival of the data packet from SA2 and after the arrival of the correct number of Invld_ack packets, i.e., the #_sharers, from SA3 and SA4, RA1 sends a RTO-completion (RTO_compt) packet to blocker 485d. The RTO_compt packet completes a "three-hop dirty reply" sequence. DA8 responds to the RTO_compt packet by leaving the B state and by releasing the data line of interest. Note that the protocol illustrated by example II can be simplified by sending #_sharers together with the Invld_req packet(s). Accordingly, whenever no data is required by RA1 of the requesting subsystem, the SA2 does not need to send a packet to RA1 for the sole purpose of sending the #_shares to RA1. In other words, RA1 can obtain the #_sharers from either SA3 or DA4, thereby reducing the network traffic by one packet. An alternative example III is illustrated by Figures 6A-6B and 6E-6G. Starting with Figures 6A-6B as in example II, instead of sending Invld_ack packets to the requesting RA1 as shown in Figure 6C of example II, sharers SA3, SA4 send the Invld_ack packets to the blocker of the home DA, i.e., blocker 485d of home subsystem 480, as shown in Figure 6E. Consequently, home DA8 is responsible for counting the correct number of Invld_ack packets and also responsible for receiving the RTO-compt packet from RA1 before leaving the B state. In this example, DA8 no longer needs to send #_sharers to other nodes since SA3 and SA4 send Invld_ack packets to blocker 485d instead of RA1. Next, as shown in Figure 6F, RA1 sends the RTO_compt packet to blocker 485d after receiving the data packet from S A2. Blocker 485d is responsible for counting the Invld_acks packets and recognizing the arrival of the RTO_compt packet. Finally, as shown in Figure 6G, Blocker 285d sends a Completion_acknowledge (Compt_ack) packet to RA1 upon receipt of all expected Invld_ack(s), in this example, from both SA3 and SA4. The Compt_ack packet completes the "four-hop dirty reply" sequence and DA8 can now leave the B state and release the data line of interest. Figures 7A-7C illustrate example IV in which a remote owner writes back to the host directory, e.g., subsystem 410 is the "owner" and sends a write-back request (WB_req) to subsystem 420. First, as shown in Figure 7A, RA1 of global interface 415 of subsystem 410 sends a WB_req packet to DA8 of global interface 485 of subsystem 480. DA8 responds by entering the B state and freezing new transactions to the cache line. Next, as shown in Figure 7B, DA8 marks its home directory to reflect the requester RA1's new status as invalid, and sends a write-back-permission (WB_permit) back to RA1. Finally, RA1 sends the data to DA8 via a write-back-complete (WB_compt) packet which includes data to be replaced, as shown in Figure 7C. Upon receiving the WB_compt packet at DA8, the write-back transaction is complete and blocker 485d releases the data line of interest by leaving the B state. Note that RA1 only leaves the "O" state after RA1 has received the WB_permit packet. Waiting for the WB_permit packet from blocker 485d eliminates the corner case in which a subsequent read request to RA1 fails because RA1 replaced the cache line (using a write back) before the subsequent read request reaches RA1. Various optimizations of the above described cache coherent mechanism are possible. For example, instead of a single FIFO queue for storing all outstanding read and write requests, RTO requests are stored in a separate queue and given priority for processing so that RTO requests are serviced first. Prioritizing write-back requests improves processor performance because typically processors of subsystems protect their respective critical code sections using a shared "key". A processor locks the key before executing its critical code section and releases the key upon completion. Thereafter, a second processor can lock the key to execute its critical code section. Hence, by giving RTO requests priority, the key can be released rapidly, allowing other processors to quickly gain access to the key. In another embodiment, selective blocking is provided. For example, blocking is selectively activated when an RTS request or a write-back request is outstanding, but is deactivated when a RTO request is outstanding. Subsystem 410, 420 ... 480 of computer system 400 can be arranged in many configurations. For example, system 400 may be configured as a wide area network (WAN), such as the internet, a local area network (LAN) or a tightly coupled multiprocessor system. Other modifications and additions are possible. For example, instead of blocking all read and write requests arriving at the DA, RTO requests whenever a request is outstanding, read-to-share requests are blocked only if there is a read-to-own or a write-back request outstanding. In addition, each subsystem may be equipped with additional circuitry to perform "local data forwarding" so that processors within a subsystem can provide data to each other without accessing the host directory of another subsystem.

Claims: A directory-based cache accessing method for a multiprocessing computer system (400) having a plurality of subsystems (410, 420, 480) coupled to each other via a global interconnect (490), each of said subsystems including a main memory (414, 424, 484), a cache (413, 423, 483) and global interface (415, 425, 485) and wherein each of said caches includes a plurality of cache lines, the method comprising: sending a first request from a first subsystem for a cache line of a second subsystem via said global interconnect, said second subsystem being the home main memory subsystem of said cache line and including a directory; receiving said first request at said second subsystem; said method being characterised by : blocking at said second subsystem, with a blocking agent associated with a directory agent responsible for accessing and updating said directory, all new requests for said cache line of said second subsystem sent via said global interface and received after said receiving said first request; receiving at said subsystem a completion acknowledgement from said first subsystem indicating that said first request has been serviced, said first subsystem sending said completion acknowledgement responsive to receiving data corresponding to said first request and an invalidate acknowledgement from a third subsystem; terminating said blocking by said blocking agent responsive to receiving said completion acknowledgement; queuing said all requests received after said receiving said first request until said first request has been serviced; and processing a second request for said cache line of said second subsystem. The method of claim 1, wherein said request is a read request. The method of claim 2, wherein said read request is a read-to-share request. The method of claim 2, wherein said read request is a read-to-own request. The method of any of claims 2 to 4, further comprising the step of: sending data from said second subsystem to said first subsystem in response to said read request. The method of claim 5, further comprising the steps of: sending an invalidate request from said second subsystem to a third said subsystem in response to said read request; and sending an invalidate acknowledgement from said third subsystem to said first subsystem in response to said invalidate request. The method of claim 2, further comprising the steps of: forwarding said read request from said second subsystem to a third said subsystem; and sending data from said third subsystem to said first subsystem. The method of claim 7, further comprising the steps of: sending an invalidate request from said second subsystem to a fourth said subsystem in response to said read request; and sending an invalidate acknowledgement from said fourth subsystem to said first subsystem in response to said invalidate request. The method of claim 8, wherein said blocking step is terminated when said second subsystem receives a completion acknowledgement from said first subsystem indicating that said data and said invalidate acknowledgement have arrived at said first subsystem. The method of any of claims 2 to 9, wherein said step of receiving said read request includes the step of updating said directory. The method of claim 1, wherein said request is a write-back request. The method of claim 11, further comprising the steps of: sending a write-back-permission from said second subsystem to said first subsystem in response to said write-back request. The method of claim 12, wherein said blocking step is terminated when said second subsystem receiving a completion acknowledgement with data from said first subsystem indicating that said write-back-permission has arrived at said first subsystem. The method of claim 13, wherein said second subsystem includes a directory and said step of receiving said write-back request includes the step of updating said directory. The method of any proceeding claim, wherein said blocking step includes the step of queuing said new requests. The method of claim 1, wherein said second requests for said cache line of said second said subsystem includes a read request and a write-back request and said queuing includes a queuing of said read request in a read queue and said write-back request in a write queue. The method of claim 1, wherein said all new requests include said second request for said cache line of said second subsystem sent via said global interface. The method of claim 17, wherein said second request for said cache line of said second subsystem sent via said global interface originates from said first subsystem. The method of claim 1, wherein said second request for said cache line of said second subsystem sent via said global interface originates from a third subsystem. The method of claim 1, wherein said all new requests include a third request for said cache line of said second subsystem sent via said global interface and originated from said first subsystem. The method of claim 20, wherein said all new requests include said third request for said cache line of said second subsystem sent via said global interface and originated from a third subsystem. The method of claim 20, wherein said third request for said cache line of said second subsystem sent via said global interface includes a read request and a write-back request. A directory-based multiprocessing computer system comprising: a first subsystem (410) including a first main memory (414) coupled to a first local interconnect (419), a first cache (413) coupled to said first local interconnect, a first processor (411) coupled to said first cache, and a first global interface (415) coupled to said first local interconnect; a second subsystem (420) including a second main memory (424) coupled to a second local interconnect (429), a second cache (413) coupled to said local interconnect, a second processor (421) coupled to said second cache, and a second global interface (425) coupled to said second local interconnect; a global interconnect (490) coupled to said first global interface of said first subsystem and said second global interface of said second subsystem; wherein said global interconnect provides a communication path between said first subsystem and said second subsystem and wherein said first global interface is configured to pass a first request from said second subsystem for a cache line of said first cache of said first subsystem sent via said global interconnect, said first subsystem being the home main memory subsystem of said cache line, said multiprocessing computer system being characterised by : said first global interface being further configured to block, with a blocking agent associated with a directory agent responsible for accessing and updating said directory, all new requests for said cache line of said first cache sent via said global interconnect and received after said first request, queuing said all requests received after said receiving of said first request until said first request for said cache line has been serviced, and wherein said second global interface is configured to transmit a completion acknowledgement to said first global interface upon said global interconnect, said second global interface transmitting said completion acknowledgement responsive to receiving data corresponding to said first request and an invalidate acknowledgement from a third subsystem, and wherein said completion acknowledgement indicates that said first request has been serviced, and wherein said first global interface is configured to terminate blocking by said blocking agent said all new requests upon receiving said completion acknowledgement, and wherein said all new requests are received from said global interconnect. The multiprocessing computer system of claim 23, wherein said first global interface further includes said directory configured to store a status of said cache line. The multiprocessing computer system of claim 23, wherein said first global interface either includes a blocker configured to block a second request for a cache line of said first cache of said first subsystem sent via said global interconnect and received after said first request. The multiprocessing computer system of claim 25, wherein said blocker is further configured to block access to said directory configured to store the status of said cache line. The multiprocessing computer system of claim 26, wherein said blocker includes a queue configured to store said second request for said cache line of said first subsystem sent via said global interconnect and received after said first request. The multiprocessing computer system of claim 27, wherein said queue includes a read queue and a write queue. The multiprocessing computer system of claim 27, wherein said queue is further configured to store said third request for said cache line of said first subsystem sent via said global interconnect and received after said first request. The multiprocessing computer system of claim 29, wherein said store of said third request for said cache line of said first subsystem sent via said global interconnect continues until said first request has been serviced. The multiprocessing computer system of claim 23, wherein said blocker is further configured to block a third request for said cache line sent via said global interconnect and received after said first request.

Inventors: HAGERSTEN ERIK, LOEWENSTEIN PAUL N, HAGERSTEN, ERIK, LOEWENSTEIN, PAUL N.
Applicants: SUN MICROSYSTEMS INC, SUN MICROSYSTEMS, INC.
Application year: 1997
Main classification: G06F12
IPC codes: G06F12