Patent Publication Number: US-7584331-B2

Title: Data processing system and method for selectively updating an invalid coherency state in response to snooping a castout

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
   The present application is related to U.S. patent application Ser. No. 11/055,305 now U.S. Pat. No. 7,389,388, which is assigned to the assignee of the present invention and incorporated herein by reference in its entirety. 
   BACKGROUND OF THE INVENTION 
   1. Technical Field 
   The present invention relates in general to data processing and, in particular, to data processing in a cache coherent data processing system. 
   2. Description of the Related Art 
   A conventional symmetric multiprocessor (SMP) computer system, such as a server computer system, includes multiple processing units all coupled to a system interconnect, which typically comprises one or more address, data and control buses. Coupled to the system interconnect is a system memory, which represents the lowest level of volatile memory in the multiprocessor computer system and which generally is accessible for read and write access by all processing units. In order to reduce access latency to instructions and data residing in the system memory, each processing unit is typically further supported by a respective multi-level cache hierarchy, the lower level(s) of which may be shared by one or more processor cores. 
   Because multiple processor cores may request write access to a same cache line of data and because modified cache lines are not immediately synchronized with system memory, the cache hierarchies of multiprocessor computer systems typically implement a cache coherency protocol to ensure at least a minimum level of coherence among the various processor core&#39;s “views” of the contents of system memory. In particular, cache coherency requires, at a minimum, that after a processing unit accesses a copy of a memory block and subsequently accesses an updated copy of the memory block, the processing unit cannot again access the old copy of the memory block. 
   A cache coherency protocol typically defines a set of cache states stored in association with the cache lines of each cache hierarchy, as well as a set of coherency messages utilized to communicate the cache state information between cache hierarchies. In a typical implementation, the cache state information takes the form of the well-known MESI (Modified, Exclusive, Shared, Invalid) protocol or a variant thereof, and the coherency messages indicate a protocol-defined coherency state transition in the cache hierarchy of the requester and/or the recipients of a memory access request. 
   In some cache coherency protocols, one or more coherency states are either not updated or only imprecisely updated in response to subsequent operations. Consequently, these coherency states may become “stale” over time in that they no longer accurately reflect a system-wide coherency state of the associated memory blocks. The present invention recognizes that the existence of “stale” coherency states can lead to system operations being performed that would otherwise not be required if the “stale” coherency states were updated and/or removed. 
   SUMMARY OF THE INVENTION 
   In view of the foregoing and other shortcomings in the art, the present invention provides an improved cache coherent data processing system and method of data processing in a cache coherent data processing system. 
   In one embodiment, in an entry of a first cache memory within a first coherency domain of a data processing system including at least first and second coherency domains, a coherency state field is set to a first state that indicates that an associated address tag is valid, an associated storage location does not contain valid data, and a memory block identified by the address tag is likely cached outside the first coherency domain. In response to snooping a castout operation, the first cache memory determines if the castout operation hits in the entry and, if so, updates the coherency state field from the first state to a second state indicating that the associated address tag is invalid. 
   All objects, features, and advantages of the present invention will become apparent in the following detailed written description. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The novel features believed characteristic of the invention are set forth in the appended claims. However, the invention, as well as a preferred mode of use, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein: 
       FIG. 1  is a high level block diagram of an exemplary data processing system in accordance with the present invention; 
       FIG. 2  is a more detailed block diagram of a processing unit in accordance with the present invention; 
       FIG. 3  is a more detailed block diagram of the L2 cache array and directory depicted in  FIG. 2 ; 
       FIG. 4  is a time-space diagram of an exemplary transaction on the system interconnect of the data processing system of  FIG. 1 ; 
       FIG. 5  illustrates a domain indicator in accordance with a preferred embodiment of the present invention; 
       FIG. 6  is a high level logical flowchart of an exemplary method of servicing a read operation by a processor core in a data processing system in accordance with the present invention; 
       FIGS. 7A-7B  together form a high level logical flowchart of an exemplary method of servicing a processor update operation in a data processing system in accordance with the present invention; 
       FIG. 8A-8B  together form a high level logical flowchart of an exemplary method of servicing a processor write operation in a data processing system in accordance with the present invention; 
       FIG. 9  is a high level logical flowchart of an exemplary cache castout operation for a data processing system in accordance with the present invention; 
       FIG. 10  is a high level logical flowchart of an exemplary method of performing a local bus read operation in a data processing system in accordance with the present invention; 
       FIGS. 11A-11B  together form a high level logical flowchart of an exemplary method of performing a global bus read operation in a data processing system in accordance with the present invention; 
       FIG. 12  is a high level logical flowchart of an exemplary method of performing a local bus RWITM operation in a data processing system in accordance with the present invention; 
       FIGS. 13A-13B  together form a high level logical flowchart of an exemplary method of performing a global bus RWITM operation in a data processing system in accordance with the present invention; 
       FIG. 14  is a high level logical flowchart of an exemplary method of performing a local bus DClaim operation in a data processing system in accordance with the present invention; 
       FIG. 15  is a high level logical flowchart of an exemplary method of performing a global bus DClaim operation in a data processing system in accordance with the present invention; 
       FIG. 16  is a high level logical flowchart of an exemplary method of performing a local bus kill operation in a data processing system in accordance with the present invention; 
       FIG. 17  is a high level logical flowchart of an exemplary method of performing a global bus kill operation in a data processing system in accordance with the present invention; 
       FIG. 18  is a high level logical flowchart of an exemplary method of performing a local bus DCBZ operation in a data processing system in accordance with the present invention; 
       FIG. 19  is a high level logical flowchart of an exemplary method of performing a global bus DCBZ operation in a data processing system in accordance with the present invention; 
       FIG. 20A  is a high level logical flowchart of an exemplary method of performing a local bus castout operation in a data processing system in accordance with the present invention; 
       FIG. 20B  is a high level logical flowchart of an exemplary method of processing an Ig castout operation at a cache snooper in accordance with the present invention; and 
       FIG. 21  is a high level logical flowchart of an exemplary method of performing a global bus castout operation in a data processing system in accordance with the present invention; 
   

   DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENT 
   I. Exemplary Data Processing System 
   With reference now to the figures and, in particular, with reference to  FIG. 1 , there is illustrated a high level block diagram of an exemplary embodiment of a cache coherent symmetric multiprocessor (SMP) data processing system in accordance with the present invention. As shown, data processing system  100  includes multiple processing nodes  102   a ,  102   b  for processing data and instructions. Processing nodes  102   a ,  102   b  are coupled to a system interconnect  110  for conveying address, data and control information. System interconnect  110  may be implemented, for example, as a bused interconnect, a switched interconnect or a hybrid interconnect. 
   In the depicted embodiment, each processing node  102  is realized as a multi-chip module (MCM) containing four processing units  104   a - 104   d , each preferably realized as a respective integrated circuit. The processing units  104   a - 104   d  within each processing node  102  are coupled for communication by a local interconnect  114 , which; like system interconnect  110 , may be implemented with one or more buses and/or switches. 
   The devices coupled to each local interconnect  114  include not only processing units  104 , but also one or more system memories  108   a - 108   d . Data and instructions residing in system memories  108  can generally be accessed and modified by a processor core in any processing unit  104  in any processing node  102  of data processing system  100 . In alternative embodiments of the invention, one or more system memories  108  can be coupled to system interconnect  110  rather than a local interconnect  114 . 
   Those skilled in the art will appreciate that SMP data processing system  100  can include many additional unillustrated components, such as interconnect bridges, non-volatile storage, ports for connection to networks or attached devices, etc. Because such additional components are not necessary for an understanding of the present invention, they are not illustrated in  FIG. 1  or discussed further herein. It should also be understood, however, that the enhancements provided by the present invention are applicable to cache coherent data processing systems of diverse architectures and are in no way limited to the generalized data processing system architecture illustrated in  FIG. 1 . 
   Referring now to  FIG. 2 , there is depicted a more detailed block diagram of an exemplary processing unit  104  in accordance with the present invention. In the depicted embodiment, each processing unit  104  includes two processor cores  200   a ,  200   b  for independently processing instructions and data. Each processor core  200  includes at least an instruction sequencing unit (ISU)  208  for fetching and ordering instructions for execution and one or more execution units  224  for executing instructions. The instructions executed by execution units  224  include instructions that request access to a memory block or cause the generation of a request for access to a memory block. 
   The operation of each processor core  200  is supported by a multi-level volatile memory hierarchy having at its lowest level shared system memories  108   a - 108   d , and at its upper levels one or more levels of cache memory. In the depicted embodiment, each processing unit  104  includes an integrated memory controller (IMC)  206  that controls read and write access to a respective one of the system memories  108   a - 108   d  within its processing node  102  in response to requests received from processor cores  200   a - 200   b  and operations snooped by a snooper (S)  222  on the local interconnect  114 . 
   In the illustrative embodiment, the cache memory hierarchy of processing unit  104  includes a store-through level one (L1) cache  226  within each processor core  200  and a level two (L2) cache  230  shared by all processor cores  200   a ,  200   b  of the processing unit  104 . L2 cache  230  includes an L2 array and directory  234 , a master  232  and a snooper  236 . Master  232  initiates transactions on local interconnect  114  and system interconnect  110  and accesses L2 array and directory  234  in response to memory access (and other) requests received from the associated processor cores  200   a - 200   b . Snooper  236  snoops operations on local interconnect  114 , provides appropriate responses, and performs any accesses to L2 array and directory  234  required by the operations. 
   Although the illustrated cache hierarchy includes only two levels of cache, those skilled in the art will appreciate that alternative embodiments may include additional levels (L3, L4, etc.) of on-chip or off-chip in-line or lookaside cache, which may be fully inclusive, partially inclusive, or non-inclusive of the contents the upper levels of cache. 
   Each processing unit  104  further includes an instance of response logic  210 , which as discussed further below, implements a portion of the distributed coherency signaling mechanism that maintains cache coherency within data processing system  100 . In addition, each processing unit  104  includes an instance of forwarding logic  212  for selectively forwarding communications between its local interconnect  114  and system interconnect  110 . Finally, each processing unit  104  includes an integrated I/O (input/output) controller  214  supporting the attachment of one or more I/O devices, such as I/O device  216 . As described further below, I/O controller  214  may issue operations on local interconnect  114  and/or system interconnect  110  in response to requests by I/O device  216 . 
   With reference now to  FIG. 3 , there is illustrated a more detailed block diagram of an exemplary embodiment of L2 array and directory  234 . As illustrated, L2 array and directory  234  includes a set associative L2 cache array  300  and an L2 cache directory  302  of the contents of L2 cache array  300 . As in conventional set associative caches, memory locations in system memories  108  are mapped to particular congruence classes within cache arrays  300  utilizing predetermined index bits within the system memory (real) addresses. The particular cache lines stored within cache array  300  are recorded in cache directory  302 , which contains one directory entry for each cache line in cache array  300 . As understood by those skilled in the art, each directory entry in cache directory  302  comprises at least a tag field  304 , which specifies the particular cache line stored in cache array  300  utilizing a tag portion of the corresponding real address, a state field  306 , which indicates the coherency state of the cache line, and a LRU (Least Recently Used) field  308  indicating a replacement order for the cache line with respect to other cache lines in the same congruence class. 
   II. Exemplary Operation 
   Referring now to  FIG. 4 , there is depicted a time-space diagram of an exemplary operation on a local or system interconnect  110 ,  114  of data processing system  100  of  FIG. 1 . The operation begins when a master  232  of an L2 cache  230  (or another master, such as an I/O controller  214 ) issues a request  402  on a local interconnect  114  and/or system interconnect  110 . Request  402  preferably includes a transaction type indicating a type of desired access and a resource identifier (e.g., real address) indicating a resource to be accessed by the request. Common types of requests preferably include those set forth below in Table I. 
   
     
       
         
             
             
           
             
               TABLE I 
             
             
                 
             
             
               Request 
               Description 
             
             
                 
             
           
          
             
               READ 
               Requests a copy of the image of a memory block 
             
             
                 
               for query purposes 
             
             
               RWITM (Read-With- 
               Requests a unique copy of the image of a memory 
             
             
               Intent-To-Modify) 
               block with the intent to update (modify) it and 
             
             
                 
               requires destruction of other copies, if any 
             
             
               DCLAIM (Data 
               Requests authority to promote an existing query- 
             
             
               Claim) 
               only copy of memory block to a unique copy with 
             
             
                 
               the intent to update (modify) it and requires 
             
             
                 
               destruction of other copies, if any 
             
             
               DCBZ (Data Cache 
               Requests authority to create a new unique copy of 
             
             
               Block Zero) 
               a memory block without regard to its present state 
             
             
                 
               and subsequently modify its contents; requires 
             
             
                 
               destruction of other copies, if any 
             
             
               CASTOUT 
               Copies the image of a memory block from a 
             
             
                 
               higher level of memory to a lower level of 
             
             
                 
               memory in preparation for the destruction of the 
             
             
                 
               higher level copy 
             
             
               WRITE 
               Requests authority to create a new unique copy of 
             
             
                 
               a memory block without regard to its present state 
             
             
                 
               and immediately copy the image of the memory 
             
             
                 
               block from a higher level memory to a lower level 
             
             
                 
               memory in preparation for the destruction of the 
             
             
                 
               higher level copy 
             
             
               PARTIAL WRITE 
               Requests authority to create a new unique copy of 
             
             
                 
               a partial memory block without regard to its 
             
             
                 
               present state and immediately copy the image 
             
             
                 
               of the partial memory block from a higher level 
             
             
                 
               memory to a lower level memory in preparation 
             
             
                 
               for the destruction of the higher level copy 
             
             
                 
             
          
         
       
     
   
   Request  402  is received by the snooper  236  of L2 caches  230 , as well as the snoopers  222  of memory controllers  206  ( FIG. 1 ). In general, with some exceptions, the snooper  236  in the same L2 cache  230  as the master  232  of request  402  does not snoop request  402  (i.e., there is generally no self-snooping) because a request  402  is transmitted on local interconnect  114  and/or system interconnect  110  only if the request  402  cannot be serviced internally by a processing unit  104 . Each snooper  222 ,  236  that receives request  402  provides a respective partial response  406  representing the response of at least that snooper to request  402 . A snooper  222  within a memory controller  206  determines the partial response  406  to provide based, for example, whether the snooper  222  is responsible for the request address and whether it has resources available to service the request. A snooper  236  of an L2 cache  230  may determine its partial response  406  based on, for example, the availability of its L2 cache directory  302 , the availability of a snoop logic instance within snooper  236  to handle the request, and the coherency state associated with the request address in L2 cache directory  302 . 
   The partial responses of snoopers  222  and  236  are logically combined either in stages or all at once by one or more instances of response logic  210  to determine a system-wide combined response (CR)  410  to request  402 . Subject to the scope restrictions discussed below, response logic  210  provides combined response  410  to master  232  and snoopers  222 ,  236  via its local interconnect  114  and/or system interconnect  110  to indicate the system-wide response (e.g., success, failure, retry, etc.) to request  402 . If CR  410  indicates success of request  402 , CR  410  may indicate, for example, a data source for a requested memory block, a cache state in which the requested memory block is to be cached by master  232 , and whether “cleanup” operations invalidating the requested memory block in one or more L2 caches  230  are required. 
   In response to receipt of combined response  410 , one or more of master  232  and snoopers  222 ,  236  typically perform one or more operations in order to service request  402 . These operations may include supplying data to master  232 , invalidating or otherwise updating the coherency state of data cached in one or more L2 caches  230 , performing castout operations, writing back data to a system memory  108 , etc. As discussed further below, if required by request  402 , a requested or target memory block may be transmitted to or from master  232  before or after the generation of combined response  410  by response logic  210 . 
   In the following description, partial response of a snooper  222 ,  236  to a request and the operations performed the snooper in response to the request and/or its combined response will be described with reference to whether that snooper is a Highest Point of Coherency (HPC), a Lowest Point of Coherency (LPC), or neither with respect to the request address specified by the request. An LPC is defined herein as a memory device or I/O device that serves as the repository for a memory block. In the absence of a HPC for the memory block, the LPC holds the true image of the memory block and has authority to grant or deny requests to generate an additional cached copy of the memory block. For a typical request in the data processing system embodiment of  FIGS. 1 and 2 , the LPC will be the memory controller  206  for the system memory  108  holding the referenced memory block. An HPC is defined herein as a uniquely identified device that caches a true image of the memory block (which may or may not be consistent with the corresponding memory block at the LPC) and has the authority to grant or deny a request to modify the memory block. Descriptively, the HPC may also provide a copy of the memory block to a requestor in response to an operation that does not modify the memory block. Thus, for a typical request in the data processing system embodiment of  FIGS. 1 and 2 , the HPC, if any, will be an L2 cache  230 . Although other indicators may be utilized to designate an HPC for a memory block, a preferred embodiment of the present invention designates the HPC, if any, for a memory block utilizing selected cache coherency state(s) within the L2 cache directory  302  of an L2 cache  230 , as described further below with reference to Table II. 
   Still referring to  FIG. 4 , the HPC, if any, for a memory block referenced in a request  402 , or in the absence of an HPC, the LPC of the memory block, preferably has the responsibility of protecting the transfer of ownership of a memory block in response to a request  402  during a protection window  404   a . In the exemplary scenario shown in  FIG. 4 , the snooper  236  that is the HPC for the memory block specified by the request address of request  402  protects the transfer of ownership of the requested memory block to master  232  during a protection window  404   a  that extends from the time that snooper  236  determines its partial response  406  until snooper  236  receives combined response  410 . During protection window  404   a , snooper  236  protects the transfer of ownership by providing partial responses  406  to other requests specifying the same request address that prevent other masters from obtaining ownership until ownership has been successfully transferred to master  232 . Master  232  likewise initiates a protection window  404   b  to protect its ownership of the memory block requested in request  402  following receipt of combined response  410 . 
   Because snoopers  222 ,  236  all have limited resources for handling the CPU and I/O requests described above, several different levels of partial responses and corresponding CRs are possible. For example, if a snooper  222  within a memory controller  206  that is responsible for a requested memory block has queue available to handle a request, the snooper  222  may respond with a partial response indicating that it is able to serve as the LPC for the request. If, on the other hand, the snooper  222  has no queue available to handle the request, the snooper  222  may respond with a partial response indicating that is the LPC for the memory block, but is unable to currently service the request. 
   Similarly, a snooper  236  in an L2 cache  230  may require an available instance of snoop logic and access to L2 cache directory  302  in order to handle a request. Absence of access to either (or both) of these resources results in a partial response (and corresponding CR) signaling an inability to service the request due to absence of a required resource. 
   Hereafter, a snooper  222 ,  236  providing a partial response indicating that the snooper has available all internal resources required to service a request, if required, is said to “affirm” the request. For snoopers  236 , partial responses affirming a snooped operation preferably indicate the cache state of the requested or target memory block at that snooper  236 . A snooper  236  providing a partial response indicating that the snooper  236  does not have available all internal resources required to service the request may be said to be “possibly hidden.” Such a snooper  236  is “possibly hidden” because the snooper  236 , due to lack of an available instance of snoop logic or access to L2 cache directory  302 , cannot “affirm” the request in sense defined above and has, from the perspective of other masters  232  and snoopers  222 ,  236 , an unknown coherency state. 
   III. Data Delivery Domains 
   Conventional broadcast-based data processing systems handle both cache coherency and data delivery through broadcast communication, which in conventional systems is transmitted on a system interconnect to at least all memory controllers and cache hierarchies in the system. As compared with systems of alternative architectures and like scale, broadcast-based systems tend to offer decreased access latency and better data handling and coherency management of shared memory blocks. 
   As broadcast-based system scale in size, traffic volume on the system interconnect is multiplied, meaning that system cost rises sharply with system scale as more bandwidth is required for communication over the system interconnect. That is, a system with m processor cores, each having an average traffic volume of n transactions, has a traffic volume of m×n, meaning that traffic volume in broadcast-based systems scales multiplicatively not additively. Beyond the requirement for substantially greater interconnect bandwidth, an increase in system size has the secondary effect of increasing some access latencies. For example, the access latency of read data is limited, in the worst case, by the combined response latency of the furthest away lower level cache holding the requested memory block in a shared coherency state from which the requested data can be sourced. 
   In order to reduce system interconnect bandwidth requirements and access latencies while still retaining the advantages of a broadcast-based system, the present invention reduces data access latency by decreasing the average distance between a requesting L2 cache  230  and an data source. One technique for do so is to reducing the average distance between a requesting L2 cache  230  and a data source is to permit multiple L2 caches  230  distributed throughout data processing system  100  to hold copies of the same memory block in a “special” shared coherency state that permits these caches to supply the memory block to requesting L2 caches  230  using cache-to-cache intervention. 
   In order to implement multiple concurrent and distributed sources for shared memory blocks in an SMP data processing system, such as data processing system  100 , two issues must be addressed. First, some rule governing the creation of copies of memory blocks in the “special” shared coherency state alluded to above must be implemented. Second, there must be a rule governing which snooping L2 cache  230 , if any, provides a shared memory block to a requesting L2 cache  230 , for example, in response to a bus read operation or bus RWITM operation. 
   According to the present invention, both of these issues are addressed through the implementation of data sourcing domains. In particular, each domain within a SMP data processing system, where a domain is defined to include one or more lower level (e.g., L2) caches that participate in responding to data requests, is permitted to include only one cache hierarchy that holds a particular memory block in the “special” shared coherency state at a time. That cache hierarchy, if present when a bus read-type (e.g., read or RWITM) operation is initiated by a requesting lower level cache in the same domain, is responsible for sourcing the requested memory block to the requesting lower level cache. Although many different domain sizes may be defined, in data processing system  100  of  FIG. 1 , it is convenient if each processing node  102  (i.e., MCM) is considered a data sourcing domain. One example of such a “special” shared state (i.e., Sr) is described below with reference to Table II. 
   IV. Coherency Domains 
   While the implementation of data delivery domains as described above improves data access latency, this enhancement does not address the m×n multiplication of traffic volume as system scale increases. In order to reduce traffic volume while still maintaining a broadcast-based coherency mechanism, preferred embodiments of the present invention additionally implement coherency domains, which like the data delivery domains hereinbefore described, can conveniently (but are not required to be) implemented with each processing node  102  forming a separate coherency domain. Data delivery domains and coherency domains can be, but are not required to be coextensive, and for the purposes of explaining exemplary operation of data processing system  100  will hereafter be assumed to have boundaries defined by processing nodes  102 . 
   The implementation of coherency domains reduces system traffic by limiting inter-domain broadcast communication over system interconnect  110  in cases in which requests can be serviced with participation by fewer than all coherency domains. For example, if processing unit  104   a  of processing node  102   a  has a bus read operation to issue, then processing unit  104   a  may elect to first broadcast the bus read operation to all participants within its own coherency domain (e.g., processing node  102   a ), but not to participants in other coherency domains (e.g., processing node  102   b ). A broadcast operation transmitted to only those participants within the same coherency domain as the master of the operation is defined herein as a “local operation”. If the local bus read operation can be serviced within the coherency domain of processing unit  104   a , then no further broadcast of the bus read operation is performed. If, however, the partial responses and combined response to the local bus read operation indicate that the bus read operation cannot be serviced solely within the coherency domain of processing node  102   a , the scope of the broadcast may then be extended to include, in addition to the local coherency domain, one or more additional coherency domains. 
   In a basic implementation, two broadcast scopes are employed: a “local” scope including only the local coherency domain and a “global” scope including all of the other coherency domains in the SMP data processing system. Thus, an operation that is transmitted to all coherency domains in an SMP data processing system is defined herein as a “global operation”. Importantly, regardless of whether local operations or operations of more expansive scope (e.g., global operations) are employed to service operations, cache coherency is maintained across all coherency domains in the SMP data processing system. 
   In a preferred embodiment, the scope of an operation is indicated in a bus operation by a local/global indicator (signal), which in one embodiment may comprise a 1-bit flag. Forwarding logic  212  within processing units  104  preferably determines whether or not to forward an operation received via local interconnect  114  onto system interconnect  110  based upon the setting of the local/global indicator (signal) in the operation. 
   V. Domain Indicators 
   In order to limit the issuance of unneeded local operations and thereby reduce operational latency and conserve additional bandwidth on local interconnects, the present invention preferably implements a domain indicator per memory block that indicates whether or not a copy of the associated memory block is cached outside of the local coherency domain. For example,  FIG. 5  depicts a first exemplary implementation of a domain indicator in accordance with the present invention. As shown in  FIG. 5 , a system memory  108 , which may be implemented in dynamic random access memory (DRAM), stores a plurality of memory blocks  500 . System memory  108  stores in association with each memory block  500  an associated error correcting code (ECC)  502  utilized to correct errors, if any, in memory block  500  and a domain indicator  504 . Although in some embodiments of the present invention, domain indicator  504  may identify a particular coherency domain (i.e., specify a coherency domain or node ID), it is hereafter assumed that domain indicator  504  is a 1-bit indicator that is set (e.g., to ‘1’ to indicate “local”) if the associated memory block  500  is cached, if at all, only within the same coherency domain as the memory controller  206  serving as the LPC for the memory block  500 . Domain indicator  504  is reset (e.g., to ‘0’ to indicate “global”) otherwise. The setting of domain indicators  504  to indicate “local” may be implemented imprecisely in that a false setting of “global” will not induce any coherency errors, but may cause unneeded global broadcasts of operations. 
   Importantly, memory controllers  206  (and L2 caches  230 ) that source a memory block in response to an operation preferably transmit the associated domain indicator  504  in conjunction with the requested memory block. 
   VI. Exemplary Coherency Protocol 
   The present invention preferably implements a cache coherency protocol designed to leverage the implementation of data delivery and coherency domains as described above. In a preferred embodiment, the cache coherency states within the protocol, in addition to providing (1) an indication of whether a cache is the HPC for a memory block, also indicate (2) whether the cached copy is unique (i.e., is the only cached copy system-wide) among caches at that memory hierarchy level, (3) whether and when the cache can provide a copy of the memory block to a master of a request for the memory block, (4) whether the cached image of the memory block is consistent with the corresponding memory block at the LPC (system memory), and (5) whether another cache in a remote coherency domain (possibly) holds a cache entry having a matching address. These five attributes can be expressed, for example, in an exemplary variant of the well-known MESI (Modified, Exclusive, Shared, Invalid) protocol summarized below in Table II. 
   
     
       
         
             
             
             
             
             
             
             
           
             
               TABLE II 
             
             
                 
             
             
               Cache 
                 
                 
                 
               Consistent 
               Cached outside 
               Legal concurrent 
             
             
               state 
               HPC? 
               Unique? 
               Data source? 
               with LPC? 
               local domain? 
               states 
             
             
                 
             
           
          
             
               M 
               yes 
               yes 
               yes, before 
               no 
               no 
               I, Ig, In (&amp; LPC) 
             
             
                 
                 
                 
               CR 
             
             
               Me 
               yes 
               yes 
               yes, before 
               yes 
               no 
               I, Ig, In (&amp; LPC) 
             
             
                 
                 
                 
               CR 
             
             
               T 
               yes 
               unknown 
               yes, after CR 
               no 
               unknown 
               Sr, S, I, Ig, In (&amp; 
             
             
                 
                 
                 
               if none 
                 
                 
               LPC) 
             
             
                 
                 
                 
               provided 
             
             
                 
                 
                 
               before CR 
             
             
               Tn 
               yes 
               unknown 
               yes, after CR 
               no 
               no 
               Sr, S, I, Ig, In (&amp; 
             
             
                 
                 
                 
               if none 
                 
                 
               LPC) 
             
             
                 
                 
                 
               provided 
             
             
                 
                 
                 
               before CR 
             
             
               Te 
               yes 
               unknown 
               yes, after CR 
               yes 
               unknown 
               Sr, S, I, Ig, In (&amp; 
             
             
                 
                 
                 
               if none 
                 
                 
               LPC) 
             
             
                 
                 
                 
               provided 
             
             
                 
                 
                 
               before CR 
             
             
               Ten 
               yes 
               unknown 
               yes, after CR 
               yes 
               no 
               Sr, S, I, Ig, In (&amp; 
             
             
                 
                 
                 
               if none 
                 
                 
               LPC) 
             
             
                 
                 
                 
               provided 
             
             
                 
                 
                 
               before CR 
             
             
               Sr 
               no 
               unknown 
               yes, before 
               unknown 
               unknown 
               T, Tn, Te, Ten, 
             
             
                 
                 
                 
               CR 
                 
                 
               S, I, Ig, In (&amp; 
             
             
                 
                 
                 
                 
                 
                 
               LPC) 
             
             
               S 
               no 
               unknown 
               no 
               unknown 
               unknown 
               T, Tn, Te, Ten, 
             
             
                 
                 
                 
                 
                 
                 
               Sr, S, I, Ig, In (&amp; 
             
             
                 
                 
                 
                 
                 
                 
               LPC) 
             
             
               I 
               no 
               n/a 
               no 
               n/a 
               unknown 
               M, Me, T, Tn, 
             
             
                 
                 
                 
                 
                 
                 
               Te, Ten, Sr, S, I, 
             
             
                 
                 
                 
                 
                 
                 
               Ig, In (&amp; LPC) 
             
             
               Ig 
               no 
               n/a 
               no 
               n/a 
               Assumed so, in 
               M, Me, T, Tn, 
             
             
                 
                 
                 
                 
                 
               absence of other 
               Te, Ten, Sr, S, I, 
             
             
                 
                 
                 
                 
                 
               information 
               Ig, In (&amp; LPC) 
             
             
               In 
               no 
               n/a 
               no 
               n/a 
               Assumed not, in 
               M, Me, T, Tn, 
             
             
                 
                 
                 
                 
                 
               absence of other 
               Te, Ten, Sr, S, I, 
             
             
                 
                 
                 
                 
                 
               information 
               Ig, In (&amp; LPC) 
             
             
                 
             
          
         
       
     
   
   A. Ig State 
   In order to avoid having to access the LPC to determine whether or not the memory block is known to be cached, if at all, only locally, the Ig (Invalid global) coherency state is utilized to maintain a domain indication in cases in which no copy of a memory block remains cached in a coherency domain. The Ig state is defined herein as a cache coherency state indicating (1) the associated memory block in the cache array is invalid, (2) the address tag in the cache directory is valid, and (3) a copy of the memory block identified by the address tag may possibly be cached in another coherency domain. The Ig indication is preferably imprecise, meaning that it may be incorrect without a violation of coherency. 
   The Ig state is formed in a lower level cache in response to that cache providing a requested memory block to a requester in another coherency domain in response to an exclusive access request (e.g., a bus RWITM operation). In some embodiments of the present invention, it may be preferable to form the Ig state only in the coherency domain containing the LPC for the memory block. In such embodiments, some mechanism (e.g., a partial response by the LPC and subsequent combined response) must be implemented to indicate to the cache sourcing the requested memory block that the LPC is within its local coherency domain. In other embodiments that do not support the communication of an indication that the LPC is local, an Ig state may be formed any time that a cache sources a memory block to a remote coherency domain in response to an exclusive access request. 
   Because cache directory entries including an Ig state carry potentially useful information, it is desirable in at least some implementations to preferentially retain entries in the Ig state over entries in the I state (e.g., by modifying the Least Recently Used (LRU) algorithm utilized to select a victim cache entry for replacement). As Ig directory entries are retained in cache, it is possible for some Ig entries to become “stale” over time in that a cache whose exclusive access request caused the formation of the Ig state may deallocate or writeback its copy of the memory block without notification to the cache holding the address tag of the memory block in the Ig state. In such cases, the “stale” Ig state, which incorrectly indicates that a global operation should be issued instead of a local operation, will not cause any coherency errors, but will merely cause some operations, which could otherwise be serviced utilizing a local operation, to be issued as global operations. Occurrences of such inefficiencies will be limited in duration by the eventual replacement of the “stale” Ig cache entries and by snooping castouts of Ig entries, as described further below. 
   Several rules govern the selection and replacement of Ig cache entries. First, if a cache selects an Ig entry as the victim for replacement, a castout of the Ig entry is performed (unlike the case when an I entry is selected). Second, if a request that causes a memory block to be loaded into a cache hits on an Ig cache entry in that same cache, the cache treats the Ig hit as a cache miss and performs a castout operation with the Ig entry as the selected victim. The cache thus avoids avoid placing two copies of the same address tag in the cache directory. Third, the castout of the Ig state is preferably performed as a local operation, or if performed as a global operation, ignored by memory controllers of non-local coherency domains. If an Ig entry is permitted to form in a cache that is not within the same coherency domain as the LPC for the memory block, no update to the domain indicator in the LPC is required. Fourth, the castout of the Ig state is preferably performed as a dataless address-only operation in which the domain indicator is written back to the LPC (if local to the cache performing the castout). 
   Implementation of an Ig state in accordance with the present invention improves communication efficiency by maintaining a cached domain indicator for a memory block in a coherency domain even when no valid copy of the memory block remains cached in the coherency domain. As a consequence, an HPC for a memory block can service an exclusive access request (e.g., bus RWITM operation) from a remote coherency domain without retrying the request and performing a push of the requested memory block to the LPC. 
   B. In State 
   The In state is defined herein as a cache coherency state indicating (1) the associated memory block in the cache array is invalid, (2) the address tag in the cache directory is valid, and (3) a copy of the memory block identified by the address tag is likely cached, if at all, only by one or more other cache hierarchies within the local coherency domain. The In indication is preferably imprecise, meaning that it may be incorrect without a violation of coherency. The In state is formed in a lower level cache in response to that cache providing a requested memory block to a requestor in the same coherency domain in response to an exclusive access request (e.g., a bus RWITM operation). 
   Because cache directory entries including an In state carry potentially useful information, it is desirable in at least some implementations to preferentially retain entries in the In state over entries in the I state (e.g., by modifying the Least Recently Used (LRU) algorithm utilized to select a victim cache entry for replacement). As In directory entries are retained in cache, it is possible for some In entries to become “stale” over time in that a cache whose exclusive access request caused the formation of the In state may itself supply a shared copy of the memory block to a remote coherency domain without notification to the cache holding the address tag of the memory block in the In state. In such cases, the “stale” In state, which incorrectly indicates that a local operation should be issued instead of a global operation, will not cause any coherency errors, but will merely cause some operations to be erroneously first issued as local operations, rather than as global operations. Occurrences of such inefficiencies will be limited in duration by the eventual replacement of the “stale” In cache entries. In a preferred embodiment, cache entries in the In coherency state are not subject to castout, but are instead simply replaced. Thus, unlike Ig cache entries, In cache entries are not utilized to update domain indicators  504  in system memories  108 . 
   Implementation of an In state in accordance with the present invention improves communication efficiency by maintaining a cached domain indicator for a memory block that may be consulted by a master in order to select a local scope for one of its operations. As a consequence, bandwidth on system interconnect  110  and local interconnects  114  in other coherency domains is conserved. 
   C. Sr State 
   In the operations described below, it is useful to be able to determine whether or not a lower level cache holding a shared requested memory block in the Sr coherency state is located within the same domain as the requesting master. In one embodiment, the presence of a “local” Sr snooper within the same domain as the requesting master can be indicated by the response behavior of a snooper at a lower level cache holding a requested memory block in the Sr coherency state. For example, assuming that each bus operation includes a range indicator indicating whether the bus operation has crossed a domain boundary (e.g., an explicit domain identifier of the master or a single local/not local range bit), a lower level cache holding a shared memory block in the Sr coherency state can provide a partial response affirming the request in the Sr state only for requests by masters within the same data sourcing domain and provide partial responses indicating the S state for all other requests. In such embodiments the response behavior can be summarized as shown in Table III, where prime (′) notation is utilized to designate partial responses that may differ from the actual cache state of the memory block. 
                               TABLE III                       Partial response   Partial response           Cache state   (adequate   (adequate       Domain of master of   in   resources   resources       read-type request   directory    available)   unavailable)                  “local” (i.e., within   Sr   Sr′ affirm   Sr′ possibly       same domain)           hidden       “remote” (i.e., not   Sr   S′ affirm   S′ possibly hidden       within same domain)       “local” (i.e., within   S   S′ affirm   S′ possibly hidden       same domain)       “remote” (i.e., not   S   S′ affirm   S′ possibly hidden       within same domain)                    
Assuming the response behavior set forth above in Table III, the average data latency for shared data can be significantly decreased by increasing the number of shared copies of memory blocks distributed within an SMP data processing system that may serve as data sources.
 
   VII. Exemplary Operations 
   With reference now generally to  FIGS. 6-27 , several high level logical flowcharts depicting the logical steps involved in servicing requests of processor cores  200 , L2 caches  230  and I/O controllers  214  are given. In particular,  FIGS. 6-12  depict the various processes within masters of the requests, and  FIGS. 13-27  illustrate operations involved with communicating and servicing the requests via local and system interconnects  114 ,  110 . Even though interconnects  110 ,  114  are not necessarily bused interconnects, such operations are termed “bus operations” (e.g., bus read operation, bus write operation, etc.) herein to distinguish them from cache or CPU (processor) operations. As logical flowcharts, it should be understood that these figures are not intended to convey a strict chronology of operations and that many of the illustrated operations may be performed concurrently or in a different order than that shown. 
   A. CPU and Cache Operations 
   With reference first to  FIG. 6 , there is depicted a high level logical flowchart of an exemplary method of servicing a processor read operation in a data processing system in accordance with the present invention. As shown, the process begins at block  600 , which represents a master  232  in an L2 cache  230  receiving a read request from an associated processor core  200 . In response to receipt of the read request, master  232  determines at block  602  whether or not the requested memory block is held in L2 cache directory  302  in any of the M, Me, Tx (e.g., T, Tn, Te or Ten), Sr or S states. If so, master  232  accesses L2 cache array  300  to obtain the requested memory block and supplies the requested memory block to the requesting processor core  200 , as shown at block  624 . The process thereafter terminates at block  626 . 
   Returning to block  602 , if the requested memory block is not held in L2 directory  302  in any of the M, Me, Tx, S, or Sr states, a determination is also made at block  604  whether or not a castout of an existing cache line is required to accommodate the requested memory block in L2 cache  230 . In one embodiment, a castout operation is required at block  604  and at similar blocks in succeeding figures if the memory block selected as a victim for eviction from the L2 cache  230  of the requesting processor is marked in L2 directory  302  as being in any of the M, T, Te, Tn or Ig coherency states. In response to a determination at block  604  that a castout is required, a cache castout operation is performed, as indicated at block  606 . Concurrently, the master  232  determines at block  610  whether or not to issue a bus read operation as a local operation or a global operation. 
   In a first embodiment in which each bus operation is initially issued as a local operation and issued as a local operation only once, the determination depicted at block  610  (and like determinations in succeeding figures) can simply represent a determination by the master of whether or not the bus read operation has previously been issued as a local bus read operation. In a second alternative embodiment in which local bus operations can be retried, the determination depicted at block  610  can represent a determination by the master of whether or not the bus read operation has previously been issued more than a threshold number of times. In a third alternative embodiment, the determination made at block  610  can be based upon a prediction by the master of whether or not a local operation is likely to be successful (e.g., is the HPC or is likely to find the HPC in the local coherency domain). Exemplary implementations of this third alternative embodiment are described in greater detail below with reference to  FIGS. 28-29 . 
   In response to a determination at block  610  to issue a global bus read operation rather than a local bus read operation, the process proceeds from block  610  to block  620 , which is described below. If, on the other hand, a determination is made at block  610  to issue a local bus read operation, master  232  initiates a local bus read operation on its local interconnect  114 , as illustrated at block  612  and described below with reference to  FIG. 13 . The local bus read operation is broadcast only within the local coherency domain (e.g., processing node  102 ) containing master  232 . If master  232  receives a CR indicating “success” (block  614 ), master  232  receives the requested memory block and returns the requested memory block (or at least a portion thereof) to the requesting processor core  200 , as shown at block  624 . Thereafter, the process ends at block  626 . 
   Returning to block  614 , if the CR for the local bus read operation does not indicate “success”, master  232  makes a determination at block  616  whether or the CR definitively indicates that the bus read operation cannot be serviced within the local coherency domain and should therefore be reissued as a global bus read operation. If so (e.g., if an L2 cache  230  in another coherency domain holds the requested memory block in the M state or Me state), the process passes to block  620 , which is described below. If, on the other hand, the CR does not definitively indicate that the bus read operation cannot be serviced within the local coherency domain, the process returns from block  616  to block  610 , which illustrates master  232  again determining whether or not to issue a local bus read operation. In this case, master  232  may employ in the determination any additional information provided by the CR. Following block  610 , the process passes to either block  612 , which is described above, or to block  620 . 
   Block  620  depicts master  230  issuing a global bus read operation as described below with reference to  FIG. 14 . If the CR of the global bus read operation does not indicate “success” at block  622 , master  232  repeats the global bus read operation at block  620  until a CR indicating “success” is received. If the CR of the global bus read operation indicates “success”, the master  232  receives the requested memory block and returns the requested memory block (or at least a portion thereof) to the requesting processor core  200  at block  624 . The process thereafter terminates at block  626 . 
   Thus, assuming affinity between processes and their data within the same coherency domain, operations, such as the CPU read operation depicted in  FIG. 6 , can frequently be serviced utilizing broadcast communication limited in scope to the coherency domain of the requesting master. The combination of data delivery domains as hereinbefore described and coherency domains thus improves not only data access latency, but also reduces traffic on the system interconnect (and other local interconnects) by limiting the scope of broadcast communication. 
   Referring now to  FIG. 7A-7B , there is illustrated a high level logical flowchart of an exemplary method of servicing a processor update operation in a data processing system in accordance with the present invention. As depicted, the process begins at block  700  in response to receipt by an L2 cache  230  of an update request by an associated one of the processor cores  200  within the same processing unit  104 . In response to the receipt of the update request, master  232  of the L2 cache  230  accesses L2 cache directory  302  to determine if the memory block referenced by the request address specified by the update request is cached within L2 cache  230  in M state, as shown at block  702 . If so, the master  232  updates the memory block in L2 cache  232  within the new data supplied by the processor core  200 , as illustrated at block  704 . Thereafter, the update process ends at block  706 . 
   As shown at blocks  710 - 712 , if L2 cache directory  302  instead indicates that L2 cache  230  holds the specified memory block in the Me state, master  232  updates the state field  306  for the requested memory block to M state in addition to updating the memory block as shown at block  704 . Thereafter, the process terminates at block  706 . 
   Following page connector A to  FIG. 7B , if L2 cache directory  302  indicates that L2 cache  230  holds the requested memory block in either of the T or Te states (block  720 ), meaning that the L2 cache  230  is the HPC for the requested memory block and the requested memory block may possibly be held in one or more other L2 caches  230 , master  232  must gain exclusive access to the requested memory block in order to perform the requested update to the memory block. The process by which master  232  gains exclusive access to the requested memory block is shown at block  722  and following blocks. 
   According to this process, master  232  updates the state of the requested memory block in the associated state field  306  of L2 cache directory  302  to the M state, as depicted at block  722 . This upgrade is cache state is permissible without first informing other L2 caches  230  because, as the HPC, the L2 cache  230  has the authority to award itself exclusive access to the requested memory block. As illustrated at block  724 , the snooper  236  of the L2 cache  230  provides “downgrade” partial responses to competing DClaim operations snooped on its local interconnect  114 , if any, by which other masters are seeking ownership of the requested memory block. These partial responses indicate that the other requestors must reissue any such competing operations as bus RWITM operations. In addition, as depicted at block  726 , master  232  issues a global bus kill operation on system interconnect  110  to invalidate any other cached copies of the memory block, as described below with reference to  FIG. 20 . 
   Master  232  next determines at blocks  790  and  728  whether or not the CR for the bus kill operation indicates that the bus kill operation successfully invalidated all other cached copies of the requested memory block or whether additional local or global “cleanup” (i.e., invalidation of other cached copies) is required. If the CR indicates that additional cleanup is not required, the process proceeds through page connector C to block  704  of  FIG. 7A , which has been described. If the CR indicates that additional cleanup is required, master  232  additionally determines whether the CR indicates that the other cached copy or copies of the requested memory block reside entirely within its local coherency domain or whether at least one copy of the requested memory block is cached outside the local coherency domain of master  232  (blocks  790  and  728 ). If the CR indicates that each remaining cached copy of the requested memory block resides in the local coherency domain of master  232 , the snooper  236  of the requesting L2 cache  230  continues to downgrade active bus DClaim operations (block  786 ), and the master  232  of the requesting L2 cache  230  continues to issue local bus kill operation (block  788 ) limited in scope to the local coherency domain of master  232  until all other cached copies of the memory block are invalidated. If the CR indicates that at least one remaining cached copy of the requested memory block resides in a remote coherency domain, the process returns to block  724 , which has been described. 
   With reference now to block  780 , if the access to the L2 cache directory  302  indicates that the requested memory block is held in one of the Tn or Ten states, then master  232  knows that the requesting L2 cache  230  is the HPC for the requested memory block and that any other cached copy of the requested memory block is held by a cache in its local coherency domain. Accordingly, master  232  updates the state of the requested memory block in the associated state field  306  of L2 cache directory  302  to the M state, as depicted at block  784 . In addition, the snooper  236  of the requesting L2 cache  230  provides “downgrade” partial responses to any competing DClaim operations snooped on its local interconnect  114  (block  786 ), and the master  232  of the requesting L2 cache  230  continues to issue local bus kill operation (block  788 ) limited in scope to the local coherency domain of master  232  until any other cached copies of the memory block are invalidated. If the master  232  determines by reference to the CR for a local bus kill operation that no further local cleanup is required (block  790 ), the process passes through block  728  and page connector C to block  704 , which has been described. 
   Referring now to block  730  of  FIG. 7A , if the access to L2 cache directory  302  indicates that the requested memory block is held in the Sr or S states, the requesting L2 cache  230  is not the HPC for the requested memory block, and master  232  must gain ownership of the requested memory block from the HPC, if any, or in the absence of an HPC, the LPC, prior to updating the memory block. 
   Accordingly, master  232  first determines at block  731  whether to issue a bus DClaim operation as a local or global operation. If master  232  makes a determination to issue a global bus DClaim operation, the process proceeds to block  740 , which is described below. In response to a determination at block  731  to issue a bus DClaim operation as a local operation, master  232  issues a local bus DClaim operation at block  732 , as described below in greater detail with reference to  FIG. 17 . Master  232  then awaits receipt of the CR of the local bus DClaim operation, which is represented by the collection of decision blocks  734 ,  736  and  738 . If the CR indicates “retry” (block  734 ), the process returns to block  731 , which has been described. If the CR alternatively indicates definitively that the bus DClaim operation cannot be serviced with the local coherency domain (block  736 ), the process proceeds to block  740 , which is described below. If the CR alternatively indicates “downgrade”, meaning that another requestor has obtained ownership of the requested memory block via a bus DClaim operation, the process passes to block  748 , which is described below. If the CR alternatively indicates that master  232  has been awarded ownership of the requested memory block by the HPC based upon the local bus DClaim operation, the process passes through page connector D to block  790  of  FIG. 7B  and following blocks, which have been described. 
   Block  740  depicts master  232  issuing a global bus DClaim operation, as described below with respect to  FIG. 18 . Master  232  next determines at blocks  742 - 744  whether or not the CR for the global bus DClaim operation indicates that it succeeded, should be retried, or was “downgraded” to a RWITM operation. If the CR indicates that the bus DClaim operation should be retried (block  742 ), master  232  reissues a global bus DClaim operation at block  740  and continues to do so until a CR other than “retry” is received. If the CR is received indicating that the global bus DClaim operation has been downgraded in response to another requestor successfully issuing a bus DClaim operation targeting the requested memory block, the process proceeds to block  746 , which is described below. If the CR alternatively indicates that master  232  has been awarded ownership of the requested memory block by the HPC based upon the global bus DClaim operation, the process passes through page connector D to block  790  of  FIG. 7B  and following blocks, which have been described. 
   Block  746  depicts master  232  of the requesting L2 cache  230  determining whether or not to issue a bus RWITM operation as a local or global operation. If master  232  elects to issue a global RWITM operation, the process passes to block  754 , which is described below. If, however, master  232  elects to issue a local bus RWITM operation, the process proceeds to block  748 , which illustrates master  232  issuing a local bus RWITM operation and awaiting the associated CR. As indicated at block  750 , if the CR indicates “retry”, the process returns to block  746 , which represents master  232  again determining whether to issue a local or global RWITM operation utilizing the additional information, if any, provided in the retry CR. If the CR to the local bus RWTIM operation issued at block  748  does not indicate “retry” (block  750 ) but instead indicates that the bus RWITM operation was successful in obtaining ownership of the requested memory block (block  752 ), the process passes through page connector D to block  790  of  FIG. 7B , which has been described. If master  232  determines at block  752  that the CR to the local bus RWITM operation indicates that the operation cannot be serviced within the local coherency domain, the process passes to block  754  and following blocks. 
   Blocks  754  and  756  depict master  232  iteratively issuing a global bus RWITM operation for the requested memory block, as described below with reference to  FIGS. 16A-16B , until a CR other than “retry” is received. In response to master  232  receiving a non-retry CR indicating that it succeeded in obtaining ownership of the requested memory block (block  756 ), the process passes through page connector D to block  790  and following blocks, which have been described. 
   With reference now to block  760 , if a negative determination has been made at blocks  702 ,  710 ,  720 ,  5502  and  730 , L2 cache  230  does not hold a valid copy of the requested memory block. Accordingly, as indicated at blocks  760  and  770 , L2 cache  230  performs a cache castout operation if needed to allocate a cache line for the requested memory block. Thereafter, the process passes to block  746  and following blocks as described above. 
   With reference now to  FIGS. 8A-8B , there is depicted a high level logical flowchart of an exemplary method of servicing a processor write operation in a data processing system in accordance with the present invention. The process begins at block  800  in response to receipt by an L2 cache  230  of a write request by an associated one of the processor cores  200  within the same processing unit  104 . In response to the receipt of the write request, master  232  of the L2 cache  230  accesses L2 cache directory  302  to determine if the memory block referenced by the request address specified by the update request is cached within L2 cache  230  in M state, as shown at block  802 . If so, the master  232  writes the data supplied by the processor core  200  into L2 cache array  300 , as illustrated at block  804 . Thereafter, the process ends at block  806 . 
   As shown at blocks  810 - 812 , if L2 cache directory  302  instead indicates that L2 cache  230  holds the specified memory block in the Me state, master  232  updates the state field  306  for the requested memory block to M state in addition to writing the memory block as shown at block  804 . Thereafter, the process terminates at block  806 . 
   Passing through page connector E to block  820  of  FIG. 8B , if L2 cache directory  302  indicates that L2 cache  230  holds the requested memory block in either of the T or Te states, meaning that the L2 cache  230  is the HPC for the requested memory block and the requested memory block may possibly be held in one or more other L2 caches  230 , master  232  must gain exclusive access to the requested memory block in order to perform the requested write to the memory block. The process by which master  232  gains exclusive access to the requested memory block is shown at block  822  and following blocks. 
   According to this process, master  232  updates the state of the requested memory block in the associated state field  306  of L2 cache directory  302  to the M state, as depicted at block  822 . As illustrated at block  824 , the snooper  236  of the requesting L2 cache  230  provides “downgrade” partial responses to competing DClaim operations snooped on its local interconnect  114  to attempt to force other requesters for the memory block to reissue any such competing requests as RWITM requests. In addition, as depicted at block  826 , master  232  issues a global bus kill operation to invalidate any other cached copies of the memory block, as described in detail below with reference to  FIG. 17 . 
   Master  232  next determines at blocks  890  and  828  whether or not the CR for the bus kill operation indicates that the bus kill operation successfully invalidated all other cached copies of the requested memory block or whether additional local or global “cleanup” (i.e., invalidation of other cached copies) is required. If the CR indicates that additional cleanup is not required, the process proceeds through page connector H to block  804  of  FIG. 8A , which has been described. If the CR indicates that additional cleanup is required, master  232  additionally determines whether the CR indicates that the other cached copy or copies of the requested memory block reside entirely within its local coherency domain (block  890 ) or whether at least one copy of the requested memory block is cached outside the local coherency domain of master  232  (block  828 ). If the CR indicates that each remaining cached copy of the requested memory block resides in the local coherency domain of master  232 , the snooper  236  of the requesting L2 cache  230  continues to downgrade active bus DClaim operations (block  876 ), and the master  232  of the requesting L2 cache  230  continues to issue local bus kill operation (block  878 ) limited in scope to the local coherency domain of master  232  until all other cached copies of the memory block are invalidated. If the CR indicates that at least one remaining cached copy of the requested memory block resides in a remote coherency domain, the process returns to block  824 , which has been described. 
   With reference now to block  870 , if the access to the L2 cache directory  302  indicates that the requested memory block is held in one of the Tn or Ten states, then master  232  knows that the requesting L2 cache  230  is the HPC for the requested memory block and that any other cached copy of the requested memory block is held by another cache in its local coherency domain. Accordingly, master  232  updates the state of the requested memory block in the associated state field  306  of L2 cache directory  302  to the M state, as depicted at block  874 . In addition, the snooper  236  of the requesting L2 cache  230  provides “downgrade” partial responses to any competing DClaim operations snooped on its local interconnect  114  (block  876 ), and the master  232  of the requesting L2 cache  230  continues to issue local bus kill operation (block  878 ) limited in scope to the local coherency domain of master  232  until any other cached copies of the memory block are invalidated. If the master  232  determines by reference to the CR for a local bus kill operation that no further local cleanup is required (block  890 ), the process passes through block  828  and page connector H to block  804 , which has been described. 
   Referring now to block  830  of  FIG. 8A , if the access to L2 cache directory  302  indicates that the requested memory block is held in the Sr or S states, the requesting L2 cache  230  is not the HPC for the requested memory block, and master  232  must gain ownership of the requested memory block from the HPC, if any, or in the absence of an HPC, the LPC, prior to writing the memory block. Accordingly, master  232  first determines at block  850  whether to issue a bus DBCZ operation as a local or global operation. 
   If master  232  elects to issue a global bus DCBZ operation, the process passes to block  860 , which is described below. If, however, master  232  elects to issue a local bus DCBZ operation, the process proceeds to block  852 , which illustrates master  232  issuing a local bus DCBZ operation, as described below with reference to  FIG. 21 , and then awaiting the associated CR. As indicated at block  854 , if the CR for the local bus DCBZ operation indicates “retry”, the process returns to block  850 , which represents master  232  again determining whether to issue a local or global bus DCBZ operation utilizing the additional information, if any, provided in the retry CR. If the CR to the local bus DCBZ operation issued at block  852  does not indicate “retry” (block  854 ) but instead indicates that the bus RWITM operation was successful in obtaining ownership of the requested memory block (block  856 ), the process passes through page connector G to block  890  of  FIG. 8B , which has been described. If master  232  determines at block  856  that the CR to the local bus DCBZ operation indicates that the operation cannot be serviced within the local coherency domain, the process passes to block  860  and following blocks. 
   Block  860  illustrates master  232  issuing a global bus DCBZ operation, as described below with respect to  FIG. 22 . As shown at block  862 , master  232  reissues the global bus DCBZ operation at block  860  until a CR other than “retry” is received. Following receipt of a CR to the global bus DCBZ operation other than “retry” at block  862 , the process passes through page connector G to block  890  of  FIG. 8B  and following blocks, which have been described. 
   With reference now to block  840 , if a negative determination has been made at blocks  802 ,  810 ,  820 ,  870  and  830 , L2 cache  230  does not hold a valid copy of the requested memory block. Accordingly, as indicated at block  840  and  842 , L2 cache  230  performs a cache castout operation if needed to allocate a cache line for the requested memory block. Thereafter, the process passes to block  850  and following blocks, which have been described. 
   Referring now to  FIG. 9 , a high level logical flowchart of an exemplary cache castout operation for a data processing system in accordance with the present invention. The illustrated process begins at block  1200  when an L2 cache  230  determines that a castout of a cache line is needed, for example, at block  606  of  FIG. 6 , block  770  of  FIG. 7A , or block  842  of  FIG. 8A . To perform the castout operation, the L2 cache  230  first determines at block  1202  whether or not the victim entry selected for replacement from the target congruence class is in the Ig state. If so, an address-only local bus castout operation is issued at block  1212  and, if necessary, retried (as indicated by block  1214 ) in order to update the corresponding domain indicator in the LPC system memory  108 . As noted above, the castout of the Ig entry is preferably performed only as a local operation, meaning that if the LPC system memory  108  is not within the local coherency domain, the CR does not indicate “retry local” at block  1214 . Thereafter, the cache castout operation ends at block  1224 . 
   Returning to block  1202 , if the victim entry selected for replacement is not in the Ig state, the L2 cache  230  determines at block  1204  whether to issue a global or local bus castout operation for the selected memory block. If L2 cache  230  elects to issue a global bus castout operation, the process passes to block  1220 , which is described below. If, however, L2 cache  230  elects to issue a local bus castout operation, the process proceeds to block  1206 , which illustrates the L2 cache  230  issuing a local bus castout operation, as described above with reference to  FIG. 20 , and then awaiting the associated CR. As indicated at block  1208 , if the CR indicates “retry local”, meaning that the local bus write operation can definitely be serviced within the local coherency domain if retried, L2 cache  230  reissues the local bus castout operation at block  1206 . Alternatively, if L2 cache  230  receives a CR indicating definitively that the bus write operation cannot be serviced within the local coherency domain (block  1210 ), the process proceeds to block  1220 , which is described below. Finally, if L2 cache  230  receives a CR indicating that the castout of the selected memory block succeeded, the process simply ends at block  1224 . 
   Block  1220  depicts L2 cache  230  issuing a global bus castout operation on system interconnect  110  via local interconnect  114 , as described below with reference to  FIG. 21 . As indicated at block  1222 , the L2 cache  230  reissues the global bus castout operation until a CR other than “retry” is received. Thereafter, the process ends at block  1224 . 
   B. Interconnect Operations 
   Referring now to  FIGS. 10-21 , exemplary local and global bus operations in an illustrative data processing system  100  will now be described. Referring first to  FIG. 10 , there is depicted a high level logical flowchart of an exemplary method of performing a local bus read operation in a data processing system in accordance with the present invention. The process begins at block  1300 , for example, at block  612  of  FIG. 6 , with an L2 cache  230  issuing a local bus read operation on its local interconnect  114 . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  in response to snooping the local bus read operation are represented in  FIG. 10  by the outcomes of decision blocks  1302 ,  1310 ,  1312 ,  1314 ,  1320 ,  1330 ,  1332 ,  1340 ,  1344 ,  1346  and  1348 . These partial responses in turn determine the CR for the local bus read operation. 
   As shown at block  1302 , if a snooper  236  of an L2 cache  230  affirms the local bus read operation with a partial response indicating that the L2 cache  230  holds the requested memory block in either the M or Me state, the process proceeds from block  1302  to block  1304 . Block  1304  indicates the operations of the requesting L2 cache  230  and the affirming L2 cache  230  in response to the local bus read operation. In particular, the snooper  236  in the affirming L2 cache  230  updates the cache state of the requested memory block from M to Tn or from Me to Ten. In addition, the snooper  236  in the affirming L2 cache  230  may initiate transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (i.e., provides “early” data). Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache array  300  in the Sr state. The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1308 . 
   If, on the other hand, a snooper  236  of an L2 cache  230  affirms the local bus read operation with a partial response indicating that the L2 cache  230  holds the requested memory block in the Tx state (block  1310 ) and an Sr′ snooper  236  also affirms the bus read operation (block  1312 ), the process passes to block  1318 . Block  1318  represents the Sr′ snooper  236  updating the cache state of the requested memory block to S and initiating transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (i.e., provides “early” data). The Tx snooper  236  remains unchanged. Upon receipt of the requested memory block, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache array  300  in the Sr state. The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1308 . 
   If the complex of partial responses includes a Tx snooper  236  affirming the local bus read operation (block  1310 ), no Sr′ snooper  236  affirming the bus read operation (block  1312 ), and a snooper  236  providing an partial response (e.g., a type of retry) indicating that an Sr′ snooper  236  may be possibly hidden in the local data delivery domain (block  1314 ), the process passes to block  1316 . Block  1316  represents the Tx snooper  236  that affirmed the bus read operation initiating transmission of the requested memory block to the requesting L2 cache  230  after receipt of the CR (i.e., provides “late” data) and retaining the requested memory block in the Tx state. Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache directory  300  in the S state (since an Sr′ snooper  236  may be hidden and only one Sr′ snooper  236  is permitted in each data delivery domain for the requested memory block). The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1308 . 
   If the complex of partial responses includes a T or Te snooper  236  affirming the local bus read operation (block  1310 ), no Sr′ snooper  236  affirming the bus read operation (block  1312 ), and no snooper  236  providing a partial response that may possibly hide a Sr′ snooper  236  (block  1314 ), the process passes to block  1306 . Block  1306  represents the T or Te snooper  236  that affirmed the bus read operation initiating transmission of the requested memory block to the requesting L2 cache  230  after receipt of the CR (i.e., provides “late” data) and retaining the requested memory block in the T or Te state. Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache array  300  in the Sr state (since no other Sr′ snooper  236  exists for the requested memory block in the local data delivery domain). The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1308 . 
   Referring now to block  1320 , if no M, Me, or Tx snooper  236  affirms the local bus read operation, but an Sr′ snooper  236  affirms the local bus read operation, the local bus read operation is serviced in accordance with block  1322 . In particular, the Sr′ snooper  236  affirming the bus read operation initiates transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of CR and updates the state of the requested memory block in its L2 cache directory  302  to the S state. The master  232  in the requesting L2 cache  230  places the requested memory block in its L2 cache array  300  in the Sr state. The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1308 . 
   With reference now to block  1324 , if no M, Me, Tx or Sr′ snooper  236  affirms the local bus read operation, but an L2 cache  230  provides a partial response affirming the local bus read operation indicating that the L2 cache  230  holds the address tag of the requested memory block in the Ig state. If no M, Me, Tx or Sr′ snooper  236  is possibly hidden by an incomplete partial response (block  1332 ), distributed response logic  210  provides a “go global” CR, as depicted at block  3164 . If, on the other hand, an Ig snooper  236  affirms the local bus read operation and the complex of partial responses indicates an M, Me, Tx or Sr′ snooper  236  is possibly hidden, response logic  210  generates a “retry” CR, as depicted at block  1342 . 
   Turning now to block  1330 , if no M, Me, Tx, Sr′ or Ig snooper  236  affirms the local bus read operation, and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, the process passes to block  1332 , which has been described. If, however, no M, Me, Tx, Sr′ or Ig snooper  236  affirms the local bus read operation, and further, if a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, the process proceeds to block  1340 . 
   Referring now to block  1340 , if a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the local bus read operation, response logic  210  generates a CR indicating “retry”, as depicted at block  1342 . If, however, a snooper  222  affirms the local bus read operation, the process proceeds to block  1344 . As indicated by decision block  1344 , response logic  210  also generates a “retry” CR at block  1342  if a memory controller snooper  222  affirms the bus read operation and an L2 cache snooper  236  provides a partial response indicating that it may hold the requested memory block in one of the M, Me, Tx or Ig states but cannot affirm the local bus read operation. In each of these cases, response logic  210  generates a “retry” CR because the bus read operation, if reissued as a local operation, may be able to be serviced without resorting to a global broadcast. 
   With reference now to block  1346 , if no M, Me, Tx or Ig snooper  236  affirms the local bus read operation, no M, Me, Tx or Ig snooper  236  is possibly hidden, and a memory controller snooper  222  affirms the local bus read operation, the snooper  222  affirming the local bus read operation provides the requested memory block and the associated domain indicator  504  to the requesting L2 cache  230  in response to the CR, as depicted at each of blocks  1350 ,  1352  and  1354 . As shown at blocks  1350 ,  1352  and  1354 , the master  232  of the requesting L2 cache  230  handles the requested memory block in accordance with the CR and the state of the domain indicator  504 . In particular, if master  232  determines at block  1360  that the domain indicator  3004  is reset to “global”, meaning that a modified copy of the requested memory block may be cached outside the local domain, master  232  of the requesting L2 cache  230  discards the requested memory block, remaining in the I state with respect to the requested memory block. In addition, in light of the “global” domain indicator  504 , master  232  interprets the CR as indicating “go global” (block  1364 ), meaning that master  232  will reissue the bus read operation as a global bus read operation. 
   If, on the other hand, the domain indicator  504  is set to indicate “local” (block  1360 ), the master  232  of the requesting cache  230  interprets the CR as indicating “success” (block  1308 ) and places both the requested memory block and domain indicator  504  within its L2 cache array  300 . The master  232  also sets the state field  306  associated with the requested memory block to a state indicated by the CR. In particular, if the partial responses and hence the CR indicate that a Sr′ snooper  236  may be hidden (block  1346 ), the requesting L2 cache  230  holds the requested memory block in the S state (block  1350 ) because only one Sr copy of the memory block is permitted in any domain. Alternatively, if the partial responses and CR indicate that no Sr′ snooper  236  may be hidden, but an S′ snooper  236  may be hidden, the requesting L2 cache  236  holds the requested memory block in the Sr state (block  1352 ). Finally, if neither a Sr′ or S′ snooper  236  may be possibly hidden (block  1348 ), the requesting L2 cache  230  holds the requested memory block in the Me state (block  1354 ) because the requesting L2 cache  230  is guaranteed to be the only cache system-wide holding the requested memory block. 
   With reference now to  FIGS. 11A-11B , there is depicted a high level logical flowchart of an exemplary method of performing a global bus read operation in a data processing system implementing Tn and Ten coherency states in accordance with the present invention. The process begins at block  1400 , for example, at block  620  of  FIG. 6 , with an L2 cache  230  issuing a global bus read operation on its local interconnect  114 . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  in response to snooping the global bus read operation are represented in  FIG. 11A  by the outcomes of decision blocks  1402 ,  1410 ,  1412 ,  1414 ,  1420 ,  1430 ,  1440 ,  1442 ,  1444 , and  1446 . These partial responses in turn determine the CR for the global bus read operation. 
   As shown at block  1402 , if a snooper  236  of an L2 cache  230  affirms the global bus read operation with a partial response indicating that the L2 cache  230  holds the requested memory block in either the M or Me state, the process proceeds from block  1402  through page connector J to block  1480  of  FIG. 11B . Block  1480  represents the fact that the M or Me snooper  236  updates its cache state differently depending upon whether the M or Me snooper  236  is local (i.e., within the same coherency domain) as the requesting L2 cache  230  as indicated by the scope indicator in the global bus read operation. In either case, the snooper  236  in the affirming L2 cache  230  may initiate transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (i.e., provides “early” data), and upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in its L2 cache array  300  in the Sr state (blocks  1481  and  1482 ). However, the snooper  236  in the affirming L2 cache  230  updates the state of the requested memory block from M to T or from Me to Te if the snooper  236  is not local to the requesting L2 cache  230  (block  1481 ) and updates the state of the requesting memory block from M to Tn or from Me to Ten if the snooper  236  is local (block  1482 ). The process then returns to  FIG. 11A  through page connector N and ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1408 . 
   If a snooper  236  of an L2 cache  230  affirms the global bus read operation with a partial response indicating that the L2 cache  230  holds the requested memory block in any the T, Tn, Te or Ten states (generically designated in block  1410  as Tx) and an Sr′ snooper  236  also affirms the bus read operation (block  1412 ), the process passes through page connector M to block  1492 . Block  1492  indicates that the affirming Tx snooper  236  updates the state of the requested memory block differently depending upon whether the scope indicator of the global bus read operation indicated that the snooper  236  is within the coherency domain of the requesting L2 cache  230 . In either case, the Sr′ snooper  236  updates the state of the requested memory block to S and initiates transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (blocks  1494  and  1495 ). Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache array  300  in the Sr state (blocks  1494  and  1495 ). In addition, the Tx snooper  236  updates the state of the requested memory block, if necessary, from Tn to T or from Ten to Te if the snooper  236  is not local to the requesting L2 cache  230  (block  1494 ), but leaves the state of the requested memory block unchanged if the Tx snooper  236  is local to the requesting L2 cache (block  1495 ). The process then returns to  FIG. 11A  through page connector N and ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1408 . 
   If the complex of partial responses includes a Tx snooper  236  affirming the global bus read operation (block  1410 ), no Sr′ snooper  236  affirming the bus read operation (block  1412 ), and a snooper  236  providing an partial response (e.g., a type of retry) indicating that an Sr′ snooper  236  may exist in the local data delivery domain but did not affirm the global bus read operation, the process passes through page connector L to block  1488  of  FIG. 11B . Block  1488  indicates that the affirming Tx snooper  236  updates the state of the requested memory block differently depending upon whether the scope indicator of the global bus read operation indicated that the snooper  236  is within the coherency domain of the requesting L2 cache  230 . In either case, the Tx snooper  236  that affirmed the global bus read operation initiates transmission of the requested memory block to the requesting L2 cache  230  after receipt of the CR (blocks  1489  and  1490 ). Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache directory  300  in the S state (since an Sr′ snooper  236  may be hidden within the local domain the requesting cache  236  and only one Sr′ snooper  236  is permitted in each domain for the requested memory block). In addition, the Tx snooper  236  updates the state of the requested memory block, if necessary, from Tn to T or from Ten to Te if the snooper  236  is not local to the requesting L2 cache  230  (block  1489 ), but leaves the state of the requested memory block unchanged if the Tx snooper  236  is local to the requesting L2 cache (block  1490 ). The process then returns to  FIG. 11A  through page connector N and ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1408 . 
   If the complex of partial responses includes a Tx snooper  236  affirming the global bus read operation, no Sr′ snooper  236  affirming the bus read operation, and no snooper  236  providing a partial response that may hide a Sr′ snooper  236 , the process passes through page connector K to block  1484  of  FIG. 11B . Block  1484  indicates that the affirming Tx snooper  236  updates the state of the requested memory block differently depending upon whether the scope indicator of the global bus read operation indicated that the snooper  236  is within the coherency domain of the requesting L2 cache  230 . In either case, the Tx snooper  236  that affirmed the global bus read operation initiates transmission of the requested memory block to the requesting L2 cache  230  after receipt of the CR (i.e., provides “late” data), the master  232  in the requesting L2 cache  230  places the requested memory block in its L2 cache array  300  in the Sr state (since no other Sr′ snooper  236  exists for the requested memory block in the local domain). In addition, the Tx snooper  236  updates the state of the requested memory block, if necessary, from Tn to T or from Ten to Te if the snooper  236  is not local to the requesting L2 cache  230  (block  1485 ), but leaves the state of the requested memory block unchanged if the Tx snooper  236  is local to the requesting L2 cache (block  1486 ). The process then returns to FIG.  11 A through page connector N and ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1408 . 
   Referring now to block  1420 , if no M, Me, or Tx snooper  236  affirms the global bus read operation, but an Sr′ snooper  236  affirms the global bus read operation, the global bus read operation is serviced in accordance with block  1422 . In particular, the Sr′ snooper  236  that affirmed the global bus read operation initiates transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of CR and updates the state of the requested memory block in its L2 cache directory  302  to the S state. The master  232  in the requesting L2 cache  230  places the requested memory block in L2 cache array  300  in the Sr state. The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1408 . 
   Turning now to block  1430 , if no M, Me, Tx or Sr′ snooper  236  affirms the global bus read operation, and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, an error occurs that halts processing as shown at block  1432  because every memory block is required to have an LPC. 
   Referring now to block  1440 , if a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the global bus read operation, response logic  210  generates a CR indicating “retry”, as depicted at block  1450 . As indicated by decision block  1442 , response logic  210  similarly generates a “retry” CR at block  1450  if a memory controller snooper  222  affirms the global bus read operation and an L2 cache snooper  236  provides a partial response indicating that it may hold the requested memory block in one of the M, Me, or Tx states but cannot affirm the global bus read operation. In each of these cases, response logic  210  generates a “retry” CR to cause the operation to be reissued because one of the possibly hidden snoopers  236  may be required to source the requested memory block to the requesting L2 cache  230 . 
   With reference now to block  1444 , if no M, Me, Tx or Sr′ snooper  236  affirms the bus read operation, no M, Me, or Tx snooper  236  is possibly hidden, and a memory controller snooper  222  affirms the global bus read operation, the snooper  222  affirming the global bus read operation provides the requested memory block and the associated domain indicator  504  to the requesting L2 cache  230  in response to the CR, as depicted at each of blocks  1452  and  1454 . As shown at blocks  1444 ,  1446 ,  1452 ,  1454  and  1456 , the master  232  of the requesting L2 cache  230  handles the requested memory block in accordance with the partial responses compiled into the “success” CR represented at block  1408 . In particular, if the CR indicates that no Sr′ or S′ snooper  236  is possibly hidden, the requesting L2 cache  230  holds the requested memory block in the Me state (block  1456 ); the requesting L2 cache  230  holds the requested memory block in the Sr state if no Sr′ snooper  236  is possibly hidden and a S′ snooper  236  is possibly hidden; and the requesting L2 cache  230  holds the requested memory block in the S state if an Sr′ snooper  236  is possibly hidden. 
   In response to the CR, the memory controller snooper  222  that is the LPC for the requested memory block then determines whether to update the domain indicator for the requested memory block, as illustrated at blocks  1460 ,  1462 ,  1470 ,  1472  and  1474 . If the CR indicates that the new cache state for the requested memory block is Me, the LPC snooper  222  determines whether it is within the same domain as the requesting L2 cache  230  (block  1460 ), for example, by reference to the scope indicator in the global bus read operation, and whether the domain indicator  504  indicates local or global (blocks  1460  and  1472 ). If the LPC is within the same domain as the requesting L2 cache  230  (block  1460 ), the LPC snooper  222  sets the domain indicator  504  to “local” if it is reset to “global” (block  1462  and  1464 ). If the LPC is not within the same domain as the requesting L2 cache  230  (block  1460 ), the LPC snooper  222  resets the domain indicator  504  to “global” if it is set to “local” (block  1472  and  1474 ). 
   If the CR indicates that the new cache state for the requested memory block is S or Sr, the LPC snooper  222  similarly determines whether it is within the same domain as the requesting L2 cache  230  (block  1470 ) and whether the domain indicator  504  indicates local or global (block  1472 ). If the LPC is within the same domain as the requesting L2 cache  230  (block  1470 ), no update to the domain indicator  504  is required. If, however, the LPC is not within the same domain as the requesting L2 cache  230  (block  1470 ), the LPC snooper  222  resets the domain indicator  504  to “global” if it is set to “local” (block  1472  and  1474 ). Thus, LPC snooper  222  updates the domain indicator  504 , if required, in response to receipt of the CR. 
   Referring now to  FIG. 12 , there is depicted a high level logical flowchart of an exemplary method of performing a local bus RWITM operation in a data processing system in accordance with the present invention. The process begins at block  1500 , for example, with a master  232  of an L2 cache  230  issuing a local bus RWITM operation its local interconnect  114  at block  748  of  FIG. 7A . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  are represented in  FIG. 12  by the outcomes of decision blocks  1502 ,  1510 ,  1512 ,  1520 ,  1524 ,  1530 ,  1534 ,  1540  and  1544 . These partial responses in turn determine the CR for the local bus RWITM operation. 
   If a snooper  236  affirms the local bus RWITM operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in either the M or Me state as shown at block  1502 , the process proceeds from block  1502  to block  1504 . Block  1504  indicates the operations of the requesting L2 cache  230  and the affirming L2 cache  230  in response to the local bus RWITM operation. In particular, the snooper  236  in the affirming L2 cache  230  updates the cache state of the requested memory block from the M or Me state to the In state and may initiate transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (i.e., provides “early” data). Upon receipt, the master  232  in the requesting L2 cache  230  places the requested memory block in its L2 cache array  300  in the M state. The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  1506 . 
   Referring to block  1510 , if a snooper  236  affirms the local bus RWITM operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in any of the T, Tn, Te or Ten states (generically designated as Tx in  FIG. 12 ) and no Sr′ snooper  236  affirms the local bus RWITM operation (block  1512 ), the process passes to block  1514 . Block  1514  represents the Tx snooper  236  that affirmed the local bus RWITM operation initiating transmission of the requested memory block to the requesting L2 cache  230  in response to receipt of the CR from response logic  210 . In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. All valid affirming snoopers  236  update their respective cache states for the requested memory block to In. 
   If the complex of partial responses includes a Tx snooper  236  and an Sr′ snooper  236  both affirming the local bus RWITM operation (blocks  1510  and  1512 ), the process passes to block  1516 . Block  1516  represents the Sr′ snooper  236  that affirmed the local bus RWITM operation initiating transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR provided by response logic  210 . In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. All valid affirming snoopers  236  update their respective cache states for the requested memory block to In. 
   As shown at block  1517 , in either of the cases represented by blocks  1514  and  1516 , response logic  210  generates a CR dependent upon whether the Tx affirming snooper  236  held the requested memory block in one of the T/Te states or the Tn/Ten states. If the Tx snooper  236  was T or Te, response logic  210  generates a CR indicating “cleanup”, as shown at block  1518 . If, however, the Tx snooper  236  was Tn or Ten, response logic  210  advantageously restricts the scope of the cleanup operations to the local domain by generating a CR indicating “local cleanup”, as shown at block  1556 . The limited scope of cleanup operations is permitted because the existence of a Tn or Ten coherency state guarantees that no remote cache holds the requested memory block, meaning that coherency can be maintained without a wider broadcast of the local bus RWITM operation or attendant bus kill operations. 
   The local bus RWITM operation cannot be serviced by a L2 cache snooper  236  without retry if no M, Me, or Tx snooper  236  (i.e., HPC) affirms the local bus RWITM operation to signify that it can mediate the data transfer. Accordingly, if an Sr′ snooper  236  affirms the local bus RWITM operation and supplies early data to the requesting L2 cache  230  as shown at block  1520 , the master  232  of the requesting L2 cache  230  discards the data provided by the Sr′ snooper  236 , as depicted at block  1522 . 
   Block  1524  represents the differences in handling the local bus RWITM operation depending upon whether a snooper  236  of an L2 cache  230  provides a partial response affirming the local bus RWITM operation and indicating that the L2 cache  230  holds the address tag of the requested memory block in the Ig state. If so, any valid affirming snooper  236  (i.e., not Ig snoopers  236 ) invalidates the relevant cache entry (block  1532 ). If no M, Me, or Tx snooper  236  is possibly hidden by an incomplete partial response (block  1534 ), distributed response logic  210  provides a “go global” CR, as depicted at block  1536 . If, on the other hand, an Ig snooper  236  affirms the local bus RWITM operation and the complex of partial responses indicates an M, Me, or Tx snooper  236  is possibly hidden, response logic  210  generates a “retry” CR, as depicted at block  1538 . Thus, the affirmance of the local bus RWITM operation by an Ig snooper  236  will cause the operation to be reissued as a global operation if no HPC is possibly hidden in the local coherency domain. 
   If an Ig snooper  236  does not affirm the local bus RWITM operation at block  1524 , the local bus RWITM operation is handled in accordance with block  1530  and following blocks. In particular, if no memory controller snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block (block  1530 ), each valid affirming snooper  236  updates the requested memory block in its respective L2 cache directory  302  to the In coherency state (block  1532 ). The CR generated by response logic  210  depends upon whether any partial responses indicate that an M, Me, or Tx snooper  236  may be hidden (block  1534 ). That is, if no M, Me, or Tx snooper  236  may be hidden, response logic  210  generates a “go global” CR at block  1536  to inform the master  232  that the local bus RWITM operation must be reissued as a global RWITM operation. On the other hand, if an M, Me, or Tx snooper  236  (i.e., an HPC) for the requested memory block may be hidden, response logic  210  generates a CR indicating “retry”, as depicted at block  1538 , because the operation may be serviced locally if retried. 
   Similarly, valid affirming snoopers  236  update their respective copies of the requested memory block to the In coherency state (block  1542 ), and response logic  210  provides a “retry” CR for the local bus RWITM operation (block  1538 ) if no M, Me, or Tx snooper  236  affirms the local bus RWITM operation and a snooper  222  provides a partial response indicating that it is the LPC but does not affirm the local bus RWITM operation. A “retry” CR is also generated at block  1538 , and snoopers  236  invalidate their respective copies of the requested memory block (block  1542 ) if no M, Me, or Tx snooper  236  affirmed the local bus RWTIM operation (blocks  1502 ,  1510 ), a snooper  222  affirmed the local bus RWITM operation (block  1540 ), and an M, Me, Tx or Ig snooper  236  may be possibly hidden (block  1544 ). 
   As shown at block  1546 , if no M, Me, or Tx snooper  236  affirms the local bus RWITM operation or is possibly hidden and the LPC snooper  222  affirms the local bus RWITM operation, each valid affirming snooper  236  updates its respective copy of the requested memory block to the In coherency state. In addition, the LPC snooper  222  provides the requested memory block and associated domain indicator  504  to the requesting L2 cache  230  in response to receipt of the CR from response logic  210 . The master  232  of the requesting L2 cache  230  handles the data in accordance with the domain indicator  504 . In particular, if the domain indicator  504  is reset to “global”, meaning that a remote cached copy may exist that renders stale the data received from the LPC snooper  222 , master  232  discards the data received from the LPC snooper  222 , maintains an invalid coherency state with respect to the requested memory block (block  1552 ), and interprets the CR provided by response logic  210  as “go global” (block  1536 ). If, on the other hand, the domain indicator  504  is set to “local”, meaning that no remote cached copy of the requested memory block renders the data received from the LPC snooper  222  potentially stale, the master  232  places the requested memory block and domain indicator  504  in its L2 cache array  300  and sets the associated state field  306  to M (block  1546 ). If the partial responses and hence the CR indicate an S′ or Sr′ snooper  236  is possibly hidden (block  1554 ), the CR indicates local “cleanup” (block  1556 ), meaning that the requesting L2 cache  230  must invalidate the other valid locally cached copies of the requested memory block, if any, through one or more local bus kill operations. If no such S′ or Sr′ snoopers  236  are possibly hidden by incomplete partial responses, the CR indicates “success”, as depicted at block  1506 . 
   It will be further appreciated that in some embodiments, the master of the local bus RWITM operation may speculatively perform a local cleanup as shown at block  1556  prior to receipt of the domain indicator  3004  from the LPC (block  1550 ). In this manner, the latency associated with data delivery from the LPC can be masked by the one or more local bus kill operations involved in the local cleanup operations. 
   With reference now to  FIGS. 13A-13B , there is illustrated a high level logical flowchart of an exemplary method of performing a global bus RWITM operation in a data processing system in accordance with the present invention. As shown, the process begins at block  1600  in response to the master  232  of a requesting L2 cache  230  issuing a global bus RWITM operation, for example, at block  754  of  FIG. 7A . If a snooper  236  affirms the global bus RWITM operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in the M or Me state as shown at block  1602 , the M or Me snooper  236  provides early data to the requesting master  232 , which holds the requested memory block in the M state (block  1604  or block  1606 ). Response logic  210  generates a CR indicating “success”, as shown at block  1607 . In addition, the M or Me snooper  236  updates its cache state to either In or Ig depending upon whether or not it is local to (i.e., in the same coherency domain as) the requesting master  232  (block  1603 ). If the M or Me snooper  236  determines it belongs to the same coherency domain as the requesting master  232 , for example, by reference to the scope indicator in the bus operation, the M or Me snooper  236  updates its cache state for the requested memory block to In (block  1606 ). On the other hand, if the M or Me snooper  236  determines it does not belong to the same coherency domain as the requesting master  232 , the M or Me snooper  236  updates its cache state for the requested memory block to Ig in order to maintain a cached domain indicator for the requested memory block in its coherency domain (block  1604 ). Consequently, no retry-push is required in response to the global bus RWITM operation in order to update the domain indicator  504  in the LPC system memory  108 . 
   Turning now to block  1610 , if a snooper  236  affirms the global bus RWITM operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in either the Tn or Ten state, the process passes to block  1612 , which represents the Tn or Ten snooper  236  determining whether or not it is local to the requesting master  232 . If so, the global bus RWITM operation is handled in accordance with blocks  1614  and following blocks, which are described below. If, however, the Tn or Ten snooper  236  affirming the global bus RWITM operation determines that it is not local to the requesting master  232 , the global bus RWITM operation is serviced in accordance, with either block  1618  or block  1620 , depending upon whether or not an Sr′ snooper  236  also affirmed the global bus RWITM operation. 
   As shown at blocks  1618 , if an Sr′ snooper  236  affirmed the global bus RWITM operation, the Sr′ snooper  236  provides early data to the requesting master  232 , and the Tn or Ten snooper  236  that affirmed the global bus RWITM operation updates its cache state for the entry containing the requested memory block to Ig. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, any valid affirming snooper  236  (i.e., not an Ig snooper  236 ) other than the Tn or Ten snooper  236  updates its respective cache state for the requested memory block to I. Alternatively, as depicted at block  1620 , if an Sr′ snooper  236  does not affirm the global bus RWITM operation, the Tn or Ten snooper  236  provides late data in response to receipt of the CR. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, the Tn or Ten snooper  236  updates its cache state to Ig, and any other valid affirming snooper  236  (i.e., not an Ig snooper  236 ) updates its respective cache state for the requested memory block to I. Thus, if a remote Tn or Ten snooper  236  affirms the global bus RWITM operation, the affirming Tn or Ten snooper  236  enters the Ig state in order to maintain a cached domain indicator for the requested memory block in its coherency domain. Consequently, no retry-push is required in response to the global bus RWITM operation in order to update the domain indicator  504  in the LPC system memory  108 . 
   In either of the cases represented by blocks  1618  and  1620 , response logic  210  generates a CR dependent upon whether an S′ or Sr′ snooper  236  is possibly hidden and thus unable to invalidate its copy of the requested memory block in response to snooping the global bus RWITM operation. If response logic  210  makes a determination at block  1626  based upon the partial responses to the global bus RWITM operation that an S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “cleanup”, as shown at block  1628 . Alternatively, if response logic  210  determines that no S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “success”, as depicted at block  1607 . 
   Returning to block  1612 , if a Tn or Ten snooper  236  that is local to the requesting master  232  affirms the global bus RWITM operation, the global bus RWITM operation is serviced in accordance with either block  1624  or block  1622 , depending upon whether or not an Sr′ snooper  236  also affirmed the global bus RWITM operation. 
   As shown at block  1624 , if an Sr′ snooper  236  affirmed the global bus RWITM operation, the Sr′ snooper  236  provides early data to the requesting master  232 , and each valid snooper  236  that affirmed the global bus RWITM operation updates its respective cache state for the entry containing the requested memory block to In. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. Alternatively, as depicted at block  1622 , if an Sr′ snooper  236  does not affirm the global bus RWITM operation, the Tn or Ten snooper  236  provides late data in response to receipt of the CR. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, each valid affirming snooper  236  updates its respective cache state for the requested memory block to In. 
   In either of the cases represented by blocks  1624  and  1622 , response logic  210  generates a CR dependent upon whether an S′ or Sr′ snooper  236  is possibly hidden and thus unable to invalidate its copy of the requested memory block in response to snooping the global bus RWITM operation. If response logic  210  makes a determination at block  1625  based upon the partial responses to the global bus RWITM operation that an S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “local cleanup”, as shown at block  1632 . Thus, the scope of the bus kill operations required to ensure coherency are advantageously limited to the local coherency domain containing the requesting L2 cache  230  and the (former) Tn or Ten snooper  236 . Alternatively, if response logic  210  determines that no S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “success”, as depicted at block  1607 . 
   Following page connector  0  to block  1630  of  FIG. 13B , if a T or Te snooper  236  affirms the global bus RWITM operation, the process passes to block  1632 , which represents the T or Te snooper  236  determining whether or not it is local to the requesting master  232 . If so, the global bus RWITM operation is handled in accordance with blocks  1638  and following blocks, which are described in detail below. If, however, the T or Te snooper  236  affirming the global bus RWITM operation determines that it is not local to the requesting master  232 , the global bus RWITM operation is serviced in accordance with either block  1636  or block  1635 , depending upon whether or not an Sr′ snooper  236  affirmed the global bus RWITM operation. 
   As shown at blocks  1635 , if an Sr′ snooper  236  affirmed the global bus RWITM operation, the Sr′ snooper  236  provides early data to the requesting master  232 , and the T or Te snooper  236  that affirmed the global bus RWITM operation updates its cache state for the entry containing the requested memory block to Ig. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, any valid affirming snooper  236  other than the T or Te snooper  236  updates its respective cache state for the requested memory block to I. Alternatively, as depicted at block  1636 , if an Sr′ snooper  236  does not affirm the global bus RWITM operation, the T or Te snooper  236  provides late data in response to receipt of a CR. In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, the T or Te snooper  236  updates its cache state to Ig, and any other valid affirming snooper  236  updates its respective cache state for the requested memory block to I. Thus, if a remote T or Te snooper  236  affirms the global bus RWITM operation, the affirming T or Te snooper  236  enters the Ig state in order to maintain a cached domain indicator for the requested memory block in its coherency domain. Consequently, no retry-push is required in response to the global bus RWITM operation in order to update the domain indicator  504  in the LPC system memory  108 . 
   In either of the cases represented by block  1635  or block  1636 , response logic  210  generates a CR dependent upon whether an S′ or Sr′ snooper  236  is possibly hidden and thus unable to invalidate its copy of the requested memory block in response to snooping the global bus RWITM operation. If response logic  210  makes a determination at block  1644  based upon the partial responses to the bus RWITM operation that an S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “cleanup”, as shown at block  1626 . Alternatively, if response logic  210  determines that no S′ or Sr′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “success”, as depicted at block  1607 . 
   Returning to blocks  1632  and  1638 , if the T or Te snooper  236  determines at block  3412  that it is local the requesting master  232 , the global bus RWITM operation is serviced in accordance with either block  1640  or block  1642 , depending upon whether an Sr′ snooper  236  also affirmed the global bus RWITM operation. That is, as shown at block  1640 , if no Sr′ snooper  236  affirms the global bus RWITM operation (block  1638 ), the T or Te snooper  236  that affirmed the global bus RWITM operation initiates transmission of the requested memory block to the requesting L2 cache  230  in response to receipt of the CR (i.e., provides “late” data). In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, each valid affirming snooper  236  updates its respective coherency state for the requested memory block to In. Alternatively, as depicted at block  1642 , if an Sr′ snooper  236  affirms the global bus RWITM operation (block  1638 ), the Sr′ snooper  236  initiates transmission of the requested memory block to the requesting L2 cache  230  prior to receipt of the CR (i.e., provides “early” data). In response to receipt of the requested memory block, the requesting L2 cache  230  holds the requested memory block in the M state. In addition, each valid affirming snooper  236  within the same coherency domain as the requesting master  232  updates its respective coherency state for the requested memory block to In. Following either block  1640  or block  1642 , the process passes to block  1644 , which has been described. 
   Referring now to block  1650 , if no M, Me, or Tx snooper  236  affirms the global bus RWITM operation, and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, an error occurs causing processing to halt, as depicted at block  1652 . If, on the other hand, no M, Me, or Tx snooper  236  affirms the bus RWITM operation and a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the bus RWITM operation (block  1660 ), each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block, either to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  or to the I coherency state otherwise. (block  1672 ). Response logic  210  also generates a CR indicating “retry”, as depicted at block  1674 . In addition, data provided by an Sr′ snooper  236  affirming the global bus RWITM operation, if any, is discarded by the master  232  (blocks  1668  and  1670 ). 
   As indicated by decision block  1662 , affirming snoopers  236  similarly update the coherency states of their respective copies of the requested memory block at block  1672  and response logic  210  generates a “retry” CR at block  1674  if a memory controller snooper  222  affirms the global bus RWITM operation (block  1660 ) and an L2 cache snooper  236  provides a partial response indicating that it may hold the requested memory block in one of the M, Me, or Tx states but cannot affirm the global bus RWITM operation (block  1662 ). 
   With reference now to block  1664 , if no M, Me, or Tx snooper  236  affirms the global bus RWITM operation or is possibly hidden, a snooper  222  affirms the global bus RWITM operation, and a Sr′ snooper  236  affirms the global bus RWITM operation, the global bus RWITM operation is serviced in accordance with block  1642  and following blocks, which are described above. Assuming these same conditions except for the absence of an Sr′ snooper  236  affirming the global bus RWITM operation, the global bus RWITM operation is serviced in accordance with block  1666 . In particular, in response to the CR, the LPC snooper  222  provides the requested memory block to the requesting L2 cache  230 , which obtains the requested memory block in the M state. In addition, each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block, either to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  or to the I coherency state otherwise. 
   Following block  1666 , the process passes to blocks  1680 - 1686 , which collectively represent the LPC snooper  222  determining whether or not to update the domain indicator  504  for the requested memory block based upon whether the LPC snooper  222  is local to the requesting master  232  (block  1680 ) and the present state of the domain indicator (blocks  1682  and  1684 ). If the LPC snooper  222  is local to the requesting L2 cache  230  and the domain indicator  504  in system memory  108  is set to indicate “local”, no update is required, and the process passes through page connector P to block  1625  of  FIG. 13A , which has been described. On the other hand, LPC snooper  222  changes the state of the domain indicator  504  at block  1686  if LPC snooper  222  is local to the requesting master  232  and domain indicator  504  is reset to indicate “global” or if LPC snooper  222  is not local to the requesting master  232  and domain indicator  504  is reset to indicate “local”. 
   If the partial responses indicate an S′ or Sr′ snooper  236  is possibly hidden (block  1644 ), the requesting L2 cache  230  receives a “cleanup” CR at block  1628 , indicating that it must invalidate any other valid cached copies of the requested memory block. If no S′ or Sr′ snoopers  236  are possibly hidden by incomplete partial responses, response logic  210  generates a “success” CR, as depicted at block  1607 . 
   With reference now to  FIG. 14 , there is illustrated a high level logical flowchart of an exemplary method of performing a local bus DClaim operation in a data processing system in accordance with the present invention. As shown, the process begins at block  1700 , for example, with a master  232  issuing a local bus DClaim operation on a local interconnect  114  at block  732  of  FIG. 7A . The various partial responses that snoopers  236  may provide to distributed response logic  210  in response to the local bus DClaim operation are represented in  FIG. 14  by the outcomes of decision blocks  1702 ,  1710 ,  1720 ,  1740 , and  1744 . These partial responses in turn determine what CR response logic  210  generates for the local bus DClaim operation. 
   As shown at block  1702 , if any snooper  236  issues a partial response downgrading the local bus DClaim operation to a bus RWITM operation as illustrated, for example, at blocks  748  and  754  of  FIG. 7A , each other affirming snooper  236  holding the requested memory block in a valid state updates the coherency state of its respective copy of the requested memory block to the In state, as shown at block  1703 . In response to the local bus DClaim operation and the partial responses, distributed response logic  210  generates a CR indicating “downgrade”, as shown at block  1704 . In response to this CR, the master  232  of the local bus DClaim operation must next attempt to gain ownership of the requested memory block utilizing a local bus RWITM operation, as depicted at block  748  of  FIG. 7A . 
   If a snooper  236  affirms the local bus DClaim operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in either the T or Te state as shown at block  1710 , the process passes to block  1712 . Because no data transfer is required in response to a bus DClaim operation, block  1712  indicates that the master  232  in the requesting L2 cache  230  updates the cache state of the requested memory block in L2 cache directory  302  to the M state. In addition, each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block to the In coherency state. As shown at block  1718 , distributed response logic  210  generates a CR indicating “cleanup”, meaning that the requesting L2 cache  230  must issue one or more bus kill operations to invalidate copies of the requested memory block, if any, held outside of the local coherency domain. 
   As illustrated at block  1740 , if a Tn or Ten snooper  236  affirms the local bus DClaim operation, the process passes to block  1742 . Because no data transfer is required in response to a bus DClaim operation, block  1742  indicates that the master  232  in the requesting L2 cache  230  updates the cache state of the requested memory block in L2 cache directory  302  to the M state. Each valid affirming snooper  236 , if any, updates the coherency state for the requested memory block to In. As shown at block  1744 , distributed response logic  210  generates a CR that is dependent upon whether the partial responses received by response logic  210  indicate that an Sr′ or S′ snooper  236  may be possibly hidden. If not, distributed response logic  210  generates a response indicating “success”, as shown at block  1746 , because the presence of the Tn or Ten coherency state guarantees that no L2 cache  230  outside of the local coherency domain holds a copy of the requested memory block. If the partial responses indicate that an Sr′ or S′ snooper  236  may be possibly hidden, response logic  210  generates a CR indicating “local cleanup”, as shown at block  1748 . Only local cleanup operations are required because the Tn or Ten coherency state again guarantees that no L2 cache  230  outside of the local coherency domain holds a valid copy of the requested memory block. 
   Turning now to block  1720 , if no snooper downgrades the local bus DClaim operation (block  1702 ), no Tx snooper  236  affirms the local bus DClaim operation (blocks  1710  and  1740 ), and further, and a snooper  236  provides a partial response indicating that it may hold the requested memory block in a Tx state but cannot affirm the local bus DClaim operation, each valid affirming snoopers  236  updates its respective coherency state for the requested memory block to the In state (block  1721 ). In addition, response logic  210  generates a CR indicating “retry”, as depicted at block  1722 . In response to the “retry” CR, the requesting master  232  may reissue the bus DClaim operation as either a local or global operation, as explained above with reference to block  736  of  FIG. 7A . If, however, no snooper downgrades the local bus DClaim operation (block  1702 ), no Tx snooper  236  affirms the bus DClaim operation or is possibly hidden (blocks  1702 ,  1710 ,  1740 , and  1720 ), response logic  210  provides a “go global” CR, as shown at block  1732 , and each affirming snooper  236 , if any, having a valid copy of the requested memory block updates the coherency state of its respective copy of the requested memory block to In, as shown at block  1730 . In response to the “go global” CR, the master  232  reissues the bus DClaim operation as a global operation, as depicted at block  740  of  FIG. 7A . 
   Referring now to  FIG. 15 , there is depicted a high level logical flowchart of an exemplary method of performing a global bus DClaim operation in a data processing system in accordance with the present invention. The process begins at block  1800 , for example, with a master  232  of an L2 cache  230  issuing a global bus DClaim operation on system interconnect  110  at block  740  of  FIG. 7A . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  in response to the global bus DClaim operation are represented in  FIG. 15  by the outcomes of decision blocks  1802 ,  1810 ,  1818 ,  1830 ,  1840 ,  1842  and  1819  These partial responses in turn determine what CR response logic  210  generates for the global bus DClaim operation. 
   As shown at block  1802 , if any snooper  236  issues a partial response downgrading the global bus DClaim operation to a bus RWITM operation, each valid affirming snooper  236  other than the downgrading snooper  236  updates the coherency state of its copy of the requested memory block, as shown at block  1803 . That is, each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  and to the I coherency state otherwise. In addition, distributed response logic  210  generates a CR indicating “downgrade”, as shown at block  1804 . In response to this CR, the master  232  of the global bus DClaim operation must next attempt to gain ownership of the requested memory block utilizing a bus RWITM operation, as depicted at blocks  748  and  754  of  FIG. 7A . 
   If a Tx (e.g., T, Te, Tn, or Ten) snooper  236  affirms the global bus DClaim operation as shown at block  1810 , the process passes to block  1812 . Block  1812  depicts the Tx snooper  236  determining whether it is local to the requesting master  232 . If not, the Tx snooper  236  updates the state of its relevant entry to Ig to maintain a cached domain indicator for the requested memory block as shown at block  1814 . In addition, the requesting master  232  updates the coherency state of its copy of the requested memory block to M, and each valid affirming snooper  236  other than the Tx snooper  236  updates its coherency state for the requested memory block to I (block  1814 ). 
   Returning to block  1812 , if the Tx snooper  236  determines that it is local to the requesting master  232 , the global bus DClaim operation is handled in accordance with block  1816 . In particular, the master  232  in the requesting L2 cache  230  updates the state of its copy of the requested memory block to the M state. In addition, each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  and to the I coherency state otherwise. 
   As shown at blocks  1818  and  1822 , if the partial responses indicate that no S′ or Sr′ snooper  236  is possibly hidden, the process ends with distributed response logic  210  generating a CR indicating “success” (block  1822 ). If, on the other hand, a determination is made at block  1818  that at least one partial response indicating the presence of a possibly hidden S′ or Sr′ snooper  236  was given in response to the global bus DClaim operation, some type of cleanup operation will be required. If the affirming Tx snooper  236  is within the same coherency domain as the requesting master  232  and, prior to the operation, was in one of the Te and Ten states, distributed response logic  210  generates a CR indicating “local cleanup” (block  1824 ), meaning that the requesting L2 cache  230  must issue one or more local bus kill operations to invalidate the requested memory block in any such hidden S′ or Sr′ snooper  236 . If the affirming Tx snooper  236  is not within the same coherency domain as the requesting master  232  or the affirming Tx snooper  236  was, prior to the operation, in one of the T or Te coherency states, global cleanup is required, and response logic  210  generates a CR indicating “cleanup” (block  1820 ). Thus, the presence of a Tn or Ten coherency state can again be utilized to limit the scope of bus kill operations. 
   Turning now to block  1830 , if no Tx snooper  236  affirms the global bus DClaim operation, and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, an error occurs causing processing to halt, as depicted at block  1832 . If, on the other hand, no Tx snooper  236  affirms the global bus DClaim operation and a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the global bus DClaim operation (block  1840 ), each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  and to the I coherency state otherwise (block  1843 ). In addition, response logic  210  generates a CR indicating “retry”, as depicted at block  1844 . As indicated by decision block  1842 , each valid affirming snooper also updates the coherency state of its respective copy of the requested memory block at block  1843 , and response logic  210  similarly generates a “retry” CR at block  1844  if a memory controller snooper  222  affirms the bus DClaim operation (block  1840 ) and an Tx snooper  236  may be possibly hidden (block  1842 ). 
   As depicted at block  1842 , if no Tx snooper  236  affirms the global bus DClaim operation or is possibly hidden and a snooper  222  affirms the global bus DClaim operation, the global bus DClaim operation is serviced in accordance with block  1816 , which is described above. 
   With reference now to  FIG. 16 , there is illustrated a high level logical flowchart of an exemplary method of performing a local bus kill operation in a data processing system in accordance with the present invention. The limitation of scope of the local bus kill operation to one coherency domain is enabled by the additional information provided by the Tn and Ten coherency states, namely, that no shared copy of the memory block resides outside of the coherency domain. 
   As depicted, the process begins at block  1900 , for example, with the master  232  of an L2 cache  230  issuing a local bus kill operation on its local interconnect  114 , for example, at block  788  of  FIG. 7B  or block  878  of  FIG. 8B . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  in response to the local bus kill operation are represented in  FIG. 16  by the outcomes of decision blocks  1902  and  1906 . These partial responses in turn determine what CR response logic  210  generates for the local bus kill operation. 
   In particular, as depicted at blocks  1902  and  1904 , any snooper  236  affirming the bus kill operation in any of the M, Me, Tx, Sr′ or S′ states updates the coherency state of its copy of the requested memory block to In without any transmission of data in response to receipt of the CR. An affirming Ig or In snooper  236 , if any, remains unchanged. As further shown at blocks  1906 ,  1908  and  1910 , response logic  210  generates a CR indicating “local cleanup” if any snooper  236  provides a partial response not affirming the local bus kill operation and otherwise generates a CR indicating “success”. 
   With reference now to  FIG. 17 , there is illustrated a high level logical flowchart of an exemplary method of performing a global bus kill operation in accordance with the present invention. As depicted, the process begins at block  2000 , for example, with the master  232  of an L2 cache  230  issuing a bus kill operation on system interconnect  110 , for example, at block  626  of  FIG. 6  or block  726  of  FIG. 7 . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  in response to the global bus kill operation are represented in  FIG. 17  by the outcomes of decision blocks  2002  and  2006 . These partial responses in turn determine what CR response logic  210  generates for the bus kill operation. 
   In particular, as depicted at blocks  2002  and  2004 , any snooper  236  affirming the bus kill operation in any of the M, Me, Tx, Sr′ or S′ states updates its copy of the requested memory block without any transmission of data in response to receipt of the CR. In particular, each valid affirming snooper  236 , if any, updates the coherency state of its respective copy of the requested memory block to the In coherency state if the affirming snooper  236  is within the same coherency domain as the master  232  and to the I coherency state otherwise. An affirming Ig or In snooper  236 , if any, remains unchanged. As further shown at blocks  2006 ,  2008  and  2010 , response logic  210  generates a CR indicating “cleanup” if any snooper  236  provided a partial response not affirming the bus kill operation and otherwise generates a CR indicating “success”. 
   With reference now to  FIG. 18 , there is depicted a high level logical flowchart of an exemplary method of performing a local bus DCBZ operation in a data processing system in accordance with the present invention. The process begins at block  2100 , for example, with the issuance of a local bus DCBZ operation on a local interconnect  114  at block  852  of  FIG. 8A . The various partial responses that snoopers  236  may provide to distributed response logic  210  are represented in  FIG. 18  by the outcomes of decision blocks  2102 ,  2103 ,  2107 ,  2110 , and  2120 . These partial responses in turn determine the CR for the local bus DCBZ operation. 
   If a snooper  236  affirms the local bus DCBZ operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in either the M or Me state as shown at block  2102 , the process proceeds to block  2104 . Block  2104  indicates the operations of the requesting L2 cache  230  and affirming L2 cache  230  in response to the request. In particular, the master  232  in the requesting L2 cache  230  updates the cache state of the requested memory block to the M state (no data is transferred), and the snooper  236  in the affirming L2 cache  230  updates the cache state of the requested memory block to the In state. The process then ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  2106 . 
   As depicted at blocks  2103  and  2105 , if a Tn or Ten snooper  236  affirms the local bus DCBZ operation, the Tn or Ten snooper  236  and any other valid affirming snooper  236  updates the coherency state of its copy of the requested memory block to In, and the requesting L2 cache  230  updates its cache state for the requested memory block to the M state. If response logic  210  received a partial response indicating that an Sr′ or S′ snooper  236  is possibly hidden (block  2107 ), response logic  210  generates a CR indicating “local cleanup”, as illustrated at block  2109 . Thus, the existence of the Tn or Ten state enables the scope of cleanup operations to be restricted to the local coherency domain. If response logic  210  determines at block  2107  that no Sr′ or S′ snooper  236  is possibly hidden, response logic  210  generates a CR indicating “success”, as shown at block  2106 . 
   Referring now to block  2110 , if a T or Te snooper  236  affirms the local bus DCBZ operation, the process passes to block  2112 . Block  2112  represents the T or Te snooper  236  and any other valid affirming snooper  236  updating the coherency state of its respective copy of the requested memory block to In and the master  232  in the requesting L2 cache  230  updating the cache state of the requested memory block to the M state. As further illustrated at block  2116 , distributed response logic  210  generates a CR indicating “cleanup” in order to ensure the invalidation of copies of the requested memory block, if any, held in L2 caches  230  outside of the local coherency domain. 
   Turning now to block  2120 , if no M, Me, or Tx snooper  236  affirms the local bus DCBZ operation (blocks  2102  and  2110 ), and further, a snooper  236  provides a partial response indicating that it may hold the requested memory block in the M, Me, or Tx state but cannot affirm the local bus DCBZ operation, each valid affirming snooper  236  updates the coherency state of its respective copy of the requested memory block to In (block  2121 ), and response logic  210  generates a CR indicating “retry”, as depicted at block  2122 . In response to the “retry” CR, the requesting master  232  may reissue the bus DCBZ operation as either a local or global operation, as explained above with reference to block  2050  of  FIG. 17 . If, however, no M, Me, or Tx snooper  236  affirms the bus DClaim operation or is possibly hidden (blocks  2102 ,  2110 ,  2120 ), response logic  210  provides a “go global” CR, as shown at block  2132 , and each affirming snooper  236 , if any, having a valid copy of the requested memory block updates its coherency state to In, as shown at block  2130 . In response to the “go global” CR, the master  232  reissues the bus DCBZ operation as a global operation, as depicted at block  860  of  FIG. 8A . 
   Referring now to  FIG. 19 , there is depicted a high level logical flowchart of an exemplary method of performing a global bus DCBZ operation in a data processing system in accordance with the present invention. The process begins at block  2200 , for example, with the master  232  of an L2 cache  230  issuing a global bus DCBZ operation on system interconnect  110  at block  860  of  FIG. 8A . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  are represented in  FIG. 19  by the outcomes of decision blocks  2202 ,  2210 ,  2212 ,  2230 ,  2238 ,  2239  and  2250 . These partial responses in turn determine the CR for the global bus DCBZ operation. 
   As indicated at blocks  2202 - 2204 , if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, an error halting processing occurs, since the no LPC was found. If a snooper  222  indicates that it is the LPC for the requested memory block, but does not affirm the global DCBZ operation, each affirming snooper  236  updates the coherency state of its respective copy of the requested memory block to the I state if it is local to the requesting master  232  and to the I state otherwise (block  2251 ). In addition, response logic  210  generates a CR indicating “retry”, as depicted at block  2252 . A “retry” CR is similarly generated by response logic  210  at block  2252  and each valid affirming snooper  236  updates the coherency state of its respective copy of the requested memory block at block  2251  if a snooper  222  affirms the global bus DCBZ operation, no M, Me, or Tx snooper  236  affirms the global bus DCBZ operation, and an M, Me, or Tx snooper  236  is possibly hidden. 
   If a snooper  236  affirms the global bus DCBZ operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in either the M or Me state as shown at block  2212 , the process proceeds to block  2214 . Block  2214  indicates the operations of the requesting L2 cache  230  and the affirming L2 cache  230  in response to the global bus DCBZ operation. In particular, the master  232  in the requesting L2 cache  230  updates the cache state of the requested memory block to the M state (no data is transferred), and the snooper  236  in the affirming L2 cache  230  updates the cache state of the requested memory block to the In state if it is local to the requesting master  232  and to the I state otherwise. As further shown at block  2216  and  2218 , the LPC snooper  222  also resets the domain indicator  504  associated with the requested memory block to “global” if the LPC snooper  222  is not within the same coherency domain as the requesting master  232 . The process ends with distributed response logic  210  generating a CR indicating “success”, as depicted at block  2220 . 
   If a Tx snooper  236  affirms the global bus DCBZ operation as shown at block  2230 , the process passes to block  2232 . Block  2232  represents the Tx snooper  236  and any other valid affirming snooper  236  updating the coherency state of its copy of the requested memory block to the In state if it is local to the requesting master  232  and to the I state otherwise. In addition, the master  232  in the requesting L2 cache  230  updates the coherency state of its copy of the requested memory block to the M state. As further shown at block  2234  and  2236 , the LPC snooper  222  also resets the domain indicator  504  associated with the requested memory block to “global” if the LPC snooper  222  is not within the same coherency domain as the requesting master  232 . 
   If response logic  210  determines at block  2238  that the partial responses indicate that no S′ or Sr′ snooper  236  is possibly hidden, distributed response logic  210  provides a CR indicating “success” as shown at block  2220 . If, on the other hand, at least one partial response indicating the presence of a possibly hidden S′ or Sr′ snooper  236  was given in response to the global bus DCBZ operation, cleanup operations are required. Accordingly, as shown at blocks  2239 ,  2242  and  2240 , distributed response logic  210  generates a CR indicating “local cleanup” if the LPC snooper  222  is local to the requesting master  232  and the affirming snooper  236  held the requested memory block in one of the Tn or Ten coherency states, and otherwise generates a CR indicating global “cleanup”. 
   As indicated by decision block  2250 , if a memory controller snooper  222  affirms the global bus DCBZ operation (block  2210 ) and no M, Me, or Tx snooper  236  affirms the global bus DCBZ operation or is possibly hidden (blocks  2212 ,  2230  and  2250 ), the global bus DCBZ operation is serviced as described above with reference to block  2232  and following blocks. 
   With reference now to  FIG. 20A , there is illustrated a high level logical flowchart of an exemplary method of performing a local bus castout operation in a data processing system in accordance with preferred embodiments of the present invention. The process begins at block  2300 , for example, with the issuance of a local bus castout operation on a local interconnect  114  at block  1206  or block  1212  of  FIG. 9 . The process then proceeds to block  2301 , which illustrates the process bifurcating depending upon whether the local bus castout operation is an Ig castout operation. If not, the process passes to block  2302 , which is described below. If, however, the local bus castout operation is an Ig castout operation, the process proceeds to block  2303 , which illustrates cache snoopers such as L2 cache snoopers  236  (and L3 cache snoopers, if present) processing the Ig castout operation, as described below in greater detail with reference to  FIG. 20B . The process then proceeds to block  2310 , which is described below. 
   Referring now to block  2302 , if a snooper  236  affirms the local bus castout operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in any of the M, Me, or Tx states, an error halting processing occurs, as indicated at block  2304 , because the memory block being castout can have only one HPC (i.e., the requesting L2 cache  230 ). 
   As depicted at block  2310 , if no M, Me or Tx snooper  236  affirms the local bus castout operation (block  2302 ), and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, response logic  210  provides a “go global” CR, as depicted at block  2312 , because the LPC is a required participant to receive the castout memory block. If, however, no M, Me, or Tx snooper  236  affirms the bus castout operation (block  2302 ) and a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the bus castout operation (blocks  2310  and  2320 ), response logic  210  generates a CR indicating “local retry”, as depicted at block  2330 , because the LPC is in the local coherency domain but must be available to receive the castout memory block. If a memory controller snooper  222  affirms the bus castout operation (block  2320 ) and no M, Me, or Tx snooper  236  affirms the bus castout operation (block  2302 ), the requesting L2 cache  230  invalidates the memory block within its cache directory  302  and transmits the memory block to the LPC (block  2324  or block  2328 ), unless the requesting L2 cache  230  is in the Ig state. In addition to updating the memory block, the LPC snooper  222  sets the associated domain indicator  504  to “local” if the memory block is in the M, Me, Tn or Ten state (blocks  2322  and  2324 ), and resets the associated domain indicator  504  to “global” if the memory block is in the T or Te state (blocks  2322  and  2328 ). The update of the domain indicator  504  to local is possible because a castout of a memory block in either of the M, Me, Tn or Ten states guarantees that no remotely cached copy of the memory block exists. In response to an affirmative determination at block  2320 , response logic  210  generates a CR indicating “success”, as illustrated at block  2326 . 
   Referring now to  FIG. 20B , there is a high level logical flowchart of an exemplary method of processing an Ig castout operation at an individual cache snooper in accordance with the present invention. The process begins at block  2338  in response to a cache snooper, for example, L2 cache snooper  236 , snooping a local bus castout operation. In response to receipt of the local bus castout operation, the snooper  236  determines at block  2340  if the local bus castout operation is a castout of an Ig cache entry. If not, processing of the local bus castout operation by the snooper  236  ends at block  2350 . If the local bus castout operation is a castout of an Ig cache entry, the process passes from block  2340  to block  2342 , which represents snooper  236  initiating a lookup of its L2 cache directory  302  to determine whether the Ig castout operation hits an Ig entry within its L2 cache directory  302 . If not, processing of the Ig castout operation by snooper  236  ends at block  2350 . 
   However, if the Ig castout operation hits an Ig entry within L2 cache directory  302 , snooper  236  determines at block  2344  whether an instance of snooper logic within snooper  236  is currently available to handle the Ig castout operation. If not, the process ends at block  2350 . However, if an instance of snooper logic within snooper  236  is available to handle the Ig castout operation, snooper  236  further determines at block  2346  whether the associated master  232  of the L2 cache  230  is currently in the process of casting out the Ig cache entry that generated a hit for the snooped Ig castout operation. If so, the processing of the Ig castout operation by snooper  236  ends at block  2350 . However, if the associated master  232  is not currently in the process of casting out the Ig cache entry that generated a hit for the snooped Ig castout operation, the process proceeds from block  2346  to block  2348 . Block  2348  illustrates the snooper  236  dispatching an instance of snooper logic within snooper  236  to update L2 cache directory  302 . In particular, the instance of snooper logic updates the coherency state recorded within the state field  306  of the cache entry that generated the snoop hit for the Ig castout operation from Ig to I. This coherency state update can be made without loss of communication efficiency (or coherency) because the Ig castout operation guarantees that the corresponding domain indicator  504  in system memory  108  will be updated to indicate that the associated memory block is likely cached outside the local coherency domain. Following block  2348 , processing of the Ig castout operation by snooper  236  ends at block  2350 . 
   As has been described, Ig entries are formed within an L2 cache directory  302 , in part, to maintain an imprecise indication that a particular memory block is cached outside of the local coherency domain. Ig directory entries can become “stale” over time in that a non-local L2 cache  230  whose exclusive access request caused the formation of the Ig coherency state may deallocate, writeback, or intervene and invalidate its copy of the memory block without notification to the L2 cache  230  holding the address tag of the memory block in the Ig state. In such cases, the “stale” Ig state, which incorrectly indicates that a global operation should be issued instead of a local operation to obtain a copy of the associated memory block, will not cause any coherency errors, but will merely cause some operations, which could otherwise be serviced utilizing a local operation, to be issued as global operations. The communication inefficiencies occasioned by these “stale” Ig coherency states and the general proliferation of Ig cache entries are reduced by updating Ig cache entries to the I coherency state in response to a snoop hit on an Ig castout operation. 
   Referring now to  FIG. 21 , there is depicted a high level logical flowchart of an exemplary method of performing a global bus castout operation in a data processing system in accordance with the present invention. The process begins at block  2400 , for example, with a master  232  of an L2 cache  230  issuing a global bus castout operation on system interconnect  110  at block  1220  of  FIG. 9 . The various partial responses that snoopers  222 ,  236  may provide to distributed response logic  210  are represented in  FIG. 21  by the outcomes of decision blocks  2402 ,  2410 ,  2420 , and  2427 . These partial responses in turn determine the CR for the global bus castout operation. 
   If a snooper  236  affirms the global bus castout operation with a partial response indicating that the L2 cache  230  containing the snooper  236  holds the requested memory block in any of the M, Me, or Tx states as shown at block  2402 , an error halting processing occurs, as indicated at block  2404 , because the memory block being castout can have only one HPC (i.e., the requesting L2 cache  230 ). 
   As depicted at block  2410 , if no M, Me, or Tx snooper  236  affirms the global bus castout operation, and further, if no snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block, an error occurs causing processing to halt, as depicted at block  2412 . If, however, no M, Me, or Tx snooper  236  affirms the bus castout operation and a snooper  222  provides a partial response indicating that it is responsible (i.e., the LPC) for the requested memory block but does not affirm the global bus castout operation (block  2420 ), response logic  210  generates a CR indicating “retry”, as depicted at block  2430 , because the LPC must be available to receive the castout memory block. 
   If a memory controller snooper  222  affirms the bus castout operation (block  2420 ) and no M, Me, or Tx snooper  236  affirms the global bus castout operation (block  2402 ), the requesting L2 cache  230  invalidates the memory block within its cache directory  302  and, except for Ig castouts, transmits the memory block to the LPC (block  2424  or block  2428 ). In addition to updating the target memory block, the LPC snooper  222  sets the associated domain indicator  504  to “local” if the memory block is in the M state (blocks  2422  and  2424 ), and resets the associated domain indicator  504  to “global” if the memory block is in the Ig state (blocks  2422  and  2428 ). As further shown at block  2427 , if the castout memory block is in one of the T, Tn or Te coherency states, the castout is handled in accordance with block  2428  if the partial responses and CR indicate that an S or Sr′ snooper  236  affirms the castout operation or is possibly hidden, and is otherwise handled in accordance with block  2024 . In response to an affirmative determination at block  2420 , response logic  210  generates a CR indicating “success”, as illustrated at block  2426 . 
   The update of the domain indicator  504  to “local” at block  2424  is possible because a castout of a memory block in the M state, or in the alternative, absence of an affirming or possibly hidden S′ or Sr′ snooper  236 , guarantees that no remotely cached copy of the memory block exists. 
   While the invention has been particularly shown as described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.