A method for maintaining cache coherence in a multiprocessor computer system

Abstract: A method is provided for maintaining cache coherence in a multiprocessor (12, 14, 16) computer system (10) having a potential for duplication of data in a plurality of storage locations, where there is cache (18, 20, 22) associated with each processor (12, 14, 16) by storing a processor address and a hint for each cache. More specifically, where the multiprocessor computer system (10) employs doubly-linked lists for identifying a previous master of the sharable data and a next master of the sharable data, the method includes the steps of 1) passing (42) the current address (44) of the current master (20) from the current master (20) to the previous master (28); 2) passing (42) the current address (46) of the current master (20) from the current master (20) to the next master (18); 3) passing (42) the current index (46) of the current master (20) from the current master (20) to the previous master (28) (the current index being a collection of all information needed by the current master (20) to find a coherence block); and 4) passing (42) the current index (46) of the current master (20) from the current master (20) to the next master (18).

Description: BACKGROUND OF THE INVENTION This invention relates to computers and more specifically to intercomputer interfaces wherein caching schemes are employed. In particular the invention relates to cache coherence protocols. Cache coherence protocols are unidirectional transfer protocols used to achieve the performance advantages of fast local cache memories while maintaining a flat shared-memory model for a plurality of computer processors operating simultaneously. In a cache coherence protocol, system processes must be controlled such that multiple valid copies of the same data never become different, that is, never become incoherent, without communicating the change directly or indirectly to other processors which have a need to access the data. The time required to transfer data on high-speed computer interfaces, such as multiple-bit buses, is limited by distance, propagation delays, flux (bandwidth), noise and signal distortion. Both asynchronous and synchronous interfaces have distinctive limitations. The speeds of synchronous buses are limited for example by the propagation delay associated with the handshaking required for each self-contained unit of data. The speeds of synchronous buses are limited by time differences between clock signals and data signals originating at different sources. A traditional bus becomes a bottleneck in multiprocessor system when many processors need more cumulative bandwidth than is available on the bus. One approach addressing the fundamental limitations on data flow is the use of packetized unidirectional signalling schemes and distributed cache protocols. These schemes help to reduce bus traffic while providing the high speed processors rapid access to much of the memory space. There is nevertheless an overhead penalty in terms of complexity associated with the controlling protocols. In some cache coherency schemes there is support for virtual-physical caches, such as those commonly employed in Hewlett-Packard Precision Architecture (HPPA) machines (HP/9000 Series 800 and HP/3000 Series 900). However, such support frequently requires that all caches be homogeneous, that is, they must be of the same depth and have the same indexing mechanism. Certain limited non-homogeneous caching schemes can be accommodated by prior planning. The obvious way to handle generalized nonhomogeneous caches is to pass the entire virtual address as a "cache hint" or "virtual hint." Such a solution is costly in terms of time and bandwidth utilization. What is needed is a method for passing a cache hint while avoiding having to pass the entire virtual address wherein each master on the bus has no need to know how other resources on a bus index their caches. Cache consistency control is known. A recent patent, U.S. Patent No. 4,713,755, issued December 15, 1987, to Worely et al., describes one approach for maintaining memory integrity in a system having hierarchical memory by use of explicit software control of caches. The technique therein employs status flags for each block of stored information indicative of valid data and contaminated data. The status flags are employed by the operating system of an individual processor to initiate corrective action to cure the contaminated data. An IEEE study group project has been proposed by representatives of the computer industry to develop a scalable coherent interface standard for high performance multiprocessor environments. (Gustavson et al., "The Scalable Coherent Interface Project (SuperBus)," Rev. 13, No. 1, August 22, 1988, (IEEE direct circulation, David Gustavson, SLAC Bin 88, P.O. Box 4349, Stanford, CA 94309). The approach being proposed is a directory-based coherence mechanism whereby cache controllers and memory cooperate. One of the features of the proposed scheme is the use of a doubly-linked list scheme to support a directory-based cache coherency mechanism. If virtually-indexed caches are employed in the above scheme, it is necessary to have a mechanism to know where the index is to find the tag. However, with such a tag, a check can be made of the bus to determine if there is a coherency hit, that is, a potential conflict with another requestor of data. In the environment of interest, the bus shared by the processors carries the physical address of the tag and sometimes "virtual hints". A virtual hint is some information about the virtual address which can help in finding the tag. Virtual hints have the limitation that they tend to be processor specific. As the present invention is based on use of a doubly-linked list directory cache coherency scheme, it is believed helpful to provide here a brief description of the operation of such a scheme. Directory-based cache coherency techniques involve the use of tagged memory, that is, the use of memory tags which indicate which masters contain copies of the data of interest. A master may be any contiguous memory unit, processor cache or bus converter having a need for data. The tags typically are storable values containing information about the status of data. Such information may include whether the data is owned by a remote master as shared (nonexclusive) or private (exclusive) data. A mechanism is also needed to indicate which masters have a copy of the data of interest. One technique, based on the singly-linked list, is to store the master's address of the first such master to have a copy. (The master's address is an address to which a master will respond.) The singly-linked list is distributed among all masters so that each master has one link per valid cache coherence block in the master's cache. A doubly-linked list is formed by adding another link or master's address which aids in the removal of entries from the list. The list is two directional, and the two links are called the nextmaster link and the previousmaster link. The object of the use of the linked list is to prevent corruption of data stored in duplicated contiguous units of sharable information called coherence blocks. A coherence block is that unit of information specified as having to remain internally coherent. It is not necessarily the unit of transferable information. In a first operation, called a sharable request, a master, such as a cache, requests a sharable copy of a portion of a coherence block, the main memory first checks its tags to determine if any master has a private copy of the requested coherence block. If yes, then the memory attends to removing the private copy before proceeding. If either no copy exists or only a shared copy exists, the memory saves the master's address of the requesting master as the nextmaster link and returns its old nextmaster link to the requesting master so that it can be stored as the nextmaster link there in its directory. The requesting master then also sets its nextmaster link to the value indicating memory. Either a special null master address or the memory's master address can be used to mark each end of the list. A list is thus built whereby all copies of a coherency block can be traced by "patching" of addresses forward through the next master links and by "patching" of addresses backward through the previous master links. In another operation, called a private request, a master requests a private copy of a coherence block. The process is the same as a sharable request, except that the memory must remove any copies than outstanding prior to proceeding. In another operation, called a shared copy removal, a master removes a shared copy of a coherence block which causes a transaction to be issued to inform the previous master of a new next master link. Thereafter, the next master must also be informed and acknowledgement of completion of both operations must also be received before the master can really forget about the coherence block. In another operation, called a coherence block forget, the memory instructs all caches to forget about a coherence block by issuing a transaction which walks down the next master links of each cache until the end of the list is found, which points back to the memory. The memory is thereby informed that no cache has retained a copy of the coherency block. What remains to be solved is how to inform all next and previous masters in such a manner that the current master can actually forget about a line. The invention can be used for maintaining cache coherence in a multiprocessor computer system having a potential for duplication of data in a plurality of storage locations, where there is cache associates with each processor, by storing a processor address and a hint for each cache. The features of the invention are defined by Claim 1. The method of at least the preferred embodiment of the invention allows the general mixing of processors on a complex network-like-bus while still maintaining cache coherence protocol from issuing a broadcast transaction, which makes implementation of the network-like bus much easier than heretofore. Within the general technique, it is allowable to even mix cache indexing schemes within the cache coherency protocol. While the enhancements to the cache coherency protocol according to the invention are eliminated by the number of available special index bits, the invention is not limited in the manner of use of the indexing bits. Thus, schemes may be mixed. The invention will be better understood upon reference to the following detailed description of a preferred embodiment of the invention in connection with the following drawings. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram of a portion of a multiprocessor computer system in which a software operating system may employ the method according to the invention. Figure 2 is a schematic diagram illustrating a first state of a common memory, a first cache and a second cache. Figure 3 is a schematic diagram illustrating a second state of the common memory, the first cache and the second cache. Figure 4 is a schematic diagram illustrating a third state of the common memory, the first cache and the second cache. Figure 5 is a schematic diagram illustrating a fourth state of the common memory, the first cache and the second cache. Figure 6 is a schematic diagram illustrating a fifth state of the common memory, the first cache and the second cache. Figure 7 is a schematic diagram illustrating a sixth state of the common memory, the first cache and the second cache. Figure 8 is a schematic diagram illustrating a seventh state of the common memory, the first cache and the second cache. Figure 9 is a schematic diagram illustrating a eighth state of the common memory, the first cache and the second cache. Figure 10 is a schematic diagram illustrating a ninth state of the common memory, the first cache and the second cache. Figure 11 is a schematic diagram illustrating a tenth state of the common memory, the first cache and the second cache. Figure 12 is a schematic diagram illustrating an eleventh state of the common memory, the first cache and the second cache. Figure 13 is a schematic diagram illustrating a twelfth state of the common memory, the first cache and the second cache. Figure 14 is a schematic diagram illustrating a thirteenth state of the common memory, the first cache and the second cache. Figure 15 is a schematic diagram illustrating a fourteenth state of the common memory, the first cache and the second cache. Figure 16 is a schematic diagram illustrating an fifteenth state of the common memory, the first cache and the second cache. Figure 17 is a schematic diagram illustrating a sixteenth state of the common memory, the first cache and the second cache. DESCRIPTION OF SPECIFIC EMBODIMENTS Referring to Figure 1, there is shown a segment of a bus-based multiprocessor system 10 comprising a plurality of processors 12, 14, 16 and a plurality of high-speed cache memory systems, herein cache 18, cache 20, and cache 22, respectively, a memory bus 24, a main memory system 26 and an input/output system 30. Each processor 12, 14, 16 is coupled through and controls a respective cache 18, 20, 22. The caches 18, 20, 22 are thereby coupled via the memory bus 24 to the main memory system 28. As is known in the art, each of the individual systems, including the memory and each of the caches, is provided with independent management and control functionality. For the purposes of illustration of this invention, each of the caches 18, 20, 22 (with the associated processor) and the main memory system 28 may serve as masters in a doubly-linked list cache coherency scheme. As used generally, a master is a subsystem unit asserting control over information. Because the master/slave relationship changes quickly within the context of a single transaction in the present invention, the traditional position of master and slave is not easily tracked. There are in fact many masters. Hence, reference will generally be made to many masters and to specific elements as masters, and the master/slave relationship will be understood by context. The indexing scheme of each cache and the memory in a doubly-linked protocol scheme in accordance with the invention is master specific. In accordance with the invention, only master-specific information in the form of an index value, or simply and "index," along with a master's address is passed, and only the master needing information which is specific to it uses the index information, even though it may be stored physically at another master. A master always knows what is required to process its own information and what information is needed by others to process its own information. Thus, any master can supply to the other masters communicating with it this information. The other masters may store information which is used by the request-receiving process which is never used locally. Figure 2 illustrates the state and linkage of portions of links of three different potential masters, namely, main memory system 28, first cache 18 and second cache 20. The principles illustrated apply to any number of masters. In the example shown, the state of the masters is shown as it would be immediately after initialization when the data of any and all coherence blocks is stored in memory and no valid data is stored in any cache. Specifically, the directory of each coherence block of data includes a tag slot, such as element 32 of the memory 28, a "next" address and index slot 34, such as element 34 of the memory 28, and a "previous" address and index slot 36, such as element 36 of the memory 28. In addition, pointers are shown, such as a first pointer 38 and a second pointer 40. The tag slot 32 may hold the values of either "valid" or "invalid," indicating whether the data in the coherence block to which it refers is safe to use. The "next," or nextmaster, slot 34 may include as values called a link the address and index of any master including itself. The "previous," or previous master slot 36, may also include as values called a link the address and index of any master including itself. The first pointer 38 and the second pointer 40 each point from one slot to another according to the link. In Figure 2, two pointers 38 and 40 point back to themselves. A slot containing a link and illustrated by its related pointer to a next slot may be physically represented according to the invention as an address and an index appended to the address stored in the same slot. Finally, as used herein, transactions, by which data and instructions are passed between masters, are represented as arrows from a first label (e.g., "memory") to a second label (e.g., "cache A"), as will be apparent hereinafter. The memory 28 always has a valid copy of the data when data is accessible. (Hence the tag in memory is permanently set to the value "valid.") The caches 18 and 20 have valid data only when the tags have been set and otherwise it is assumed that all caches have invalid data. To illustrate the process of requesting and tagging valid data, reference is made to the Figures beginning with Figure 3. A request for data transaction 42 is instituted by Cache B 20 acting as a master whereby Cache B requests sharable data from the memory 28. Such a request typically contains a tag as to transaction type and the physical address of the requested coherence block. However, according to the invention, the request also contains Cache B's master address 44 and the index 46 which Cache B needs in order to index into its cache, as represented by a data block 43 as might be found in a temporary buffer associated with data received off the bus 24 (Figure 1). This two-element block of information is stored in the corresponding slot. The links merely express graphically the connection defined by the contents of the respective slots. Referring to Figure 4, memory 28 thereupon issues a second transaction 48 to Cache B 20 whereby the data requested by Cache B 20 is passed to corresponding locations in Cache B along with the nextmaster link for Cache B. The memory 28 will recognize that its own list was previously empty. Therefore, memory 28 places Cache B's link address and index in the nextmaster slot 34 and the previous master slot 36. However, the pointers 38 and 40 now point to nextmaster slot 50 and previousmaster slot 52 of Cache B 20, and the memory link address and index are loaded into to those slots (Figure 5). Cache B 20 recognizes that the nextmaster slot 50 and the previousmaster slot 52 now contain links which point to memory 28, as indicated by pointers 54 and 56. Cache B 20 then recognizes that its request has been completed. The final state, with the tag 51 of Cache B 20 set to the value valid, is shown in Figure 5. At the conclusion of the transactions in each instance, according to the invention, the link indexes of the previousmaster and the nextmaster as well as the link addresses are stored in the respective nextmaster slot and the previousmaster slot of the requesting master. Figure 6 illustrates a first transaction in a further transaction set according to the invention which might follow the previous sequence of transactions. Cache A issues a request 58 to memory 28 for a sharable copy of the same coherency block previously requested by Cache B 20 (Figure 3). In Figure 7, memory 28 sends back the requested cache data (path 60) and forwards its nextmaster link index and address stored in nextmaster slot 34 to next master slot 62 as represented by link 64. The link value in slot 34 is thus replaced by the value Cache A. In Figure 8, it is seen that Cache A 18 stores memory's old nextmaster link value of Cache B in nextmaster slot 62, thereby linking nextmaster slot 62 to nextmaster slot 50, as indicated by pointer 49. Cache A 18 also sends a transaction 66 to Cache B 20 to advise it that there is a new previousmaster. Cache A 18 has loaded its previousmaster slot 68 with the link index and address of the memory 28, as indicated by link 70. Cache A 20 knows that the previousmaster link value is memory because the transaction 60 was issued by the memory 28. In the next sequence, shown in Figure 9, Cache B 20 issues an acknowledgment 72 to Cache A 18 and updates its previousmaster link value stored in previousmaster slot 52 as indicated by link 74. Cache B 20 knows the previousmaster slot value is Cache A because Cache A 18 initiated the transaction. In Figure 10, the final state is illustrated. All links are now completed, and the value of the tag 76 of Cache A 18 has been set to valid. Cache A 18 and its associated processor can thereupon employ all the content of the coherence block without risk of use of invalid data. Figure 11 begins a sequence illustrating removal of a coherence block from a cache, in this example, Cache A. Cache A institutes a transaction 80 to its previous master, namely, memory 28, as indicated by its previousmaster value in slot 68. Cache A thereby passes the index and address of a new nextmaster link, which is the value Cache B stored in its nextmaster slot 62. Cache A's previousmaster slot 68 is used to get its previous master's index and address, which value is also passed in the transaction 80 to memory. In this case, the previousmaster slot value is memory. In Figure 12, Cache A's previous master (memory 28) sends a transaction 82 to its new next master (Cache B 20, as indicated by the value in slot 34 with link 84), in order to fix Cache B's previousmaster link. In other words, the transaction 82 is to fix Cache A's nextmaster's previousmaster link. In Figure 13, Cache B 20 issues a transaction 86 to Cache A 18 in order to inform Cache A 18 that the operation is complete. The previousmaster slot 52 has been set to the value memory as indicated by pointer 86. Bad pointers 38, 49 and 70, 86 are corrected by removal of Cache A 18, indicated by setting the value of its tag 76 to invalid. Figure 14 shows this final state after completion of the transaction sequence. A frequent type of transaction is a flush or purge of a coherence block initiated by a cache which is not in the list. Referring to Figure 15, Cache A 18 issues a transaction 88 to memory 28 requesting such a purge from all caches. Memory 28 receives the transaction 88 and notes from its link list that other copies of the coherence block exists. Thereupon, memory 28 issues a transaction 90 to start the flush/purge operation (Figure 16). The transaction 90 is directed along its nextmaster link indicated by pointer 84 (Figure 15) to Cache B and removes its own nextmaster link value indicated by pointer 84 and previousmaster link value indicated by pointer 40. Cache B 20, which is the subject next master and previous master, receives the transaction 90, forwards the purge request to memory 28 via transaction 92 based on its nextmaster link, after it removes its own previousmaster link as indicated by pointer 86 and nextmaster link as indicated by pointer 54 (Figure 16). Cache B 20 sets its tag to the value invalid. Once memory 28 has received the forwarded flush/purge request, it can mark the original request complete. The above examples are merely a sampling of the type of transactions which can be carried out with reliability in a multiprocessor environment employing doubly-linked list cache coherency protocols modified in accordance with the invention. Another example is a request for a private coherence block, namely, a cache coherence block which is not to be shared with other caches. The request is similar to that for a shared coherence block with the following exception: Memory 28 must issue and complete a flush operation prior to returning data to the requesting master. This invention, particularly the indexing and the passing of indexes, allows general mixing of processors, each employing its own indexing scheme, on a complex network-like bus while maintaining cache coherence across all processors. This invention becomes acutely important in large multiprocessor bus networks. It is unnecessary to issue broadcast transactions, which are difficult to recover from if cache coherency is a concern. The size of the bus network is limited only by the number of index bits. The nature of the use of the index is not affected by the cache in which it is stored. The invention has now been explained with reference to specific embodiments. Other embodiments will be apparent to those of ordinary skill in the art. For example, while the method disclosed is described in connection with only a single central memory, a system could comprise multiple memories having a valid copy of elements of the subject data. It is therefore not intended that the invention be limited, except as indicated by the appended claims.

Claims: A method of maintaining cache coherence in a multiprocessor computer system (10) having a potential for duplication of data in a plurality of units identifiable as coherence blocks, the directory of each coherence block of data including a tag location identifying the validity of the data, a nextmaster storage location for storing a next address and index, and a previousmaster location for storing a previous address and index, the multiprocessor computer system (10) employing doubly-linked lists for identifying previous and next masters, the method being characterised by employing a master specific indexing scheme so that: when a first master (28) receives a request (42) for a data transaction from a second master (20) for a copy of a coherence block which resides solely in the first master (28), the request (42) including the address (44) of the second master (20) and a second index (46) which only the second master (20) uses for indexing purposes, the second master address (44) and the second index (46) is stored by the first master (28) in a nextmaster storage location (34), and in a previous master storage location (36), the first master (28) subsequently sends the requested data to the second master (20), together with the first master address and a first index which only the first master uses for indexing purposes; and the first master address and the first index is stored by the second master (20) in a previousmaster storage location (52) and in a next master storage location (50), whereafter the tag location of the second master is set to the value valid, and when a third master (18) subsequently sends a request (58) for the same coherence block, the latter request (58) including the address of the third master (18) and a third index, which only the third master (18) uses for indexing purposes, the first master (28) stores the address of the third master (18) and the third index in its nextmaster storage location (34), the third master (18) then stores, in its nextmaster storage location (62), the address of the second master (20) and the second index (46), and the third master (18) then stores, in its previousmaster storage location (68), the address of the first master (28) and the first index; the previousmaster storage location (52) of the second master (20) being updated to point to the third master (18), whereafter the tag location of the third master is set to the value valid. A method according to Claim 1, wherein each request (42,58) includes a tag designating the type of request and the physical address of the requested coherence block, said tag denoting a transaction type, said request for a coherence block being one type of transaction. A method according Claim 1 or 2 wherein said masters (18,20,28) include one central memory system (28) containing a valid copy of all data, and at least two caches (18,20) having cache memories associated with respective processors (12,14). The method according to any preceding Claim further including the step of tagging each master (18,20,28) as containing valid data in a coherence block after updating the respective master with the appropriate indices and addresses.

Inventors: LA FETRA ROSS V, LA FETRA, ROSS V.
Applicants: HEWLETT PACKARD CO, HEWLETT-PACKARD COMPANY
Application year: 1990
Main classification: G06F12
IPC codes: G06F12