Abstract:
Methods and apparatus relating to processing out of order transactions for mirrored subsystems are described. In one embodiment, a device (that is mirroring data from another device) includes a cache to track out of order write operations prior to writing the data from the write operations to memory. A register may be used to track the state of the cache and cause acknowledgement of commitment of the data to memory once all cache entries, as recorded at a select point by the register, are emptied or otherwise invalidated. Other embodiments are also disclosed.

Description:
FIELD 
       [0001]    The present disclosure generally relates to the field of electronics. More particularly, an embodiment of the invention relates to processing out of order transactions for mirrored subsystems. 
       BACKGROUND 
       [0002]    Mirroring or backing up of data generally involves copying an original set of data from one location to one or more other locations. Generally, the original set of data may be logically split into smaller units prior to transfer to other locations. As computing systems become more complex, there may be more than one route via which each of the smaller units may be transmitted. As a result, different portions of the original set of data may arrive at the other locations out of order. To indicate successful completion of the mirroring operation, an acknowledgement may be sent to the location where the original set of data resides. However, the out of order nature of the transmissions may complicate the acknowledgement process. Also, as computing systems become more complex, the number of portions of the original set of data may significantly increase, resulting in more out of order transmissions for which an acknowledgement process needs to account. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0003]    The detailed description is provided with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items. 
           [0004]      FIGS. 1-2  and  4 - 5  illustrate block diagrams of embodiments of computing systems, which may be utilized to implement various embodiments discussed herein. 
           [0005]      FIG. 3  illustrates a flow diagram according to an embodiment. 
       
    
    
     DETAILED DESCRIPTION 
       [0006]    In the following description, numerous specific details are set forth in order to provide a thorough understanding of various embodiments. However, some embodiments may be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the particular embodiments. 
         [0007]    Some embodiments discussed herein are generally related to processing out of order transactions for mirrored subsystems. Mirrored subsystems may include battery backed DIMMs (Dual In-line Memory Modules). For example, in an embodiment, some storage systems may mirror data via a PCIe (Peripheral Component Interconnect (PCI) express) bus (e.g., in accordance with PCIe Specifications, Revision 2.0, 2006), available from the PCI Special Interest Group, Portland, Oreg., U.S.A.) into mirrored memory on one or more redundant Central Processing Unit (CPU) nodes. Such redundancy may prevent data loss even if one of the CPU nodes has a power failure. As will be discussed herein with reference to  FIGS. 1-5 , the redundancy may be achieved with a protocol that ensures successful data mirroring, without impacting system write performance while waiting for the acknowledgement. 
         [0008]    Various computing systems may be used to implement embodiments, discussed herein, such as the systems discussed with reference to  FIGS. 1-2  and  4 - 5 . More particularly,  FIG. 1  illustrates a block diagram of a computing system  100 , according to an embodiment of the invention. The system  100  may include one or more agents  102 - 1  through  102 -M (collectively referred to herein as “agents  102 ” or more generally “agent  102 ”). In an embodiment, one or more of the agents  102  may be any of components of a computing system, such as the computing systems discussed with reference to  FIGS. 4-5 . 
         [0009]    As illustrated in  FIG. 1 , the agents  102  may communicate via a network fabric  104 . In one embodiment, the network fabric  104  may include a computer network that allows various agents (such as computing devices) to communicate data. In an embodiment, the network fabric  104  may include one or more interconnects (or interconnection networks) that communicate via a serial (e.g., point-to-point) link and/or a shared communication network. For example, some embodiments may facilitate component debug or validation on links that allow communication with fully buffered dual in-line memory modules (FBD), e.g., where the FBD link is a serial link for coupling memory modules to a host controller device (such as a processor or memory hub). Debug information may be transmitted from the FBD channel host such that the debug information may be observed along the channel by channel traffic trace capture tools (such as one or more logic analyzers). 
         [0010]    In one embodiment, the system  100  may support a layered protocol scheme, which may include a physical layer, a link layer, a routing layer, a transport layer, and/or a protocol layer. The fabric  104  may further facilitate transmission of data (e.g., in form of packets) from one protocol (e.g., caching processor or caching aware memory controller) to another protocol for a point-to-point or shared network. Also, in some embodiments, the network fabric  104  may provide communication that adheres to one or more cache coherent protocols. 
         [0011]    Furthermore, as shown by the direction of arrows in  FIG. 1 , the agents  102  may transmit and/or receive data via the network fabric  104 . Hence, some agents may utilize a unidirectional link while others may utilize a bidirectional link for communication. For instance, one or more agents (such as agent  102 -M) may transmit data (e.g., via a unidirectional link  106 ), other agent(s) (such as agent  102 - 2 ) may receive data (e.g., via a unidirectional link  108 ), while some agent(s) (such as agent  102 - 1 ) may both transmit and receive data (e.g., via a bidirectional link  110 ). 
         [0012]      FIG. 2  illustrates a block diagram of an embodiment of a computing system  200 . As shown in  FIG. 2 , system  200  may include a Node A and a Node B. Each node may include one or more components (such as agents  102  discussed with reference to  FIG. 1 ) including, for example, a Gigabit Ethernet (GigE) controller or another type of a network controller for communication over a network, DIMM to store data (where the DIMM may or may not be provided on the same integrated circuit semiconductor device as one or more other components of each node), a CPU to process data (including one or more processor cores), one or more PCIe interfaces to communicate with other nodes, components, or network(s), interconnect to couple components of each node to each other (which may be a point-to-point or serial interconnect in some embodiments; for example, providing a serial link between the CPU and PCIe interface(s)), memory control unit (MCU)  201 -A/ 201 -B to perform memory-related operations (which may include a memory to store data (such as DDR3 memory shown in  FIG. 2 ), control logic (labeled as “Control” in  FIG. 2 ) to control operations (such as read or write) of the MCU such as operations on the DIMM (e.g., via DDR3), a write buffer to buffer write operations directed at the DIMM (e.g., via DDR3) associated with the memory, a cache  202 -A/ 202 -B to store information about input/output (I/O) write operations (e.g., such as write operations associated with mirroring of data), and a register  204 -A/ 204 -B), Peripheral Control Hub (PCH) to couple peripherals to the node(s), and Direct Media Interface (DMI) to provide an interface between the PCH and integrated portion of nodes ( 206 -A/ 206 -B). In one embodiment, each of the integrated portions  206 -A or  206 -B may be provided on a single Integrated Circuit (IC) semiconductor device. The cache  202 -A/ 202 -B and/or register  204 -A/ 204 -B may be stored in other locations than inside the MCU in some embodiments. Also, the cache  202 -A/ 202 -B may be kept coherent with other caches in system  200  such as discussed with reference to agents  102  of  FIG. 1  and/or caches within processors/CPUs (such as discussed with reference to  FIG. 4  or  5 ). Moreover, these caches may allow for ownership of multiple cachelines on PCIe write operations and a mechanism to write back the data once PCIe ordering rules have been met. 
         [0013]    As shown in  FIG. 2 , DIMM of Node A (or a portion of it) may be mirrored  210  to DIMM of Node B via PCIe Non-Transparent Bridges (NTBs) of nodes A and B. For example, one or more write operations may be performed from DIMM of Node A to DIMM of Node B, followed by a special transaction. The write operations are stored/cached in cache  202 -B prior to being written to DIMM of Node B. As will be further discussed with reference to  FIG. 3 , upon receipt of the special transaction from Node A at Node B, register  204 -B may be updated with a snapshot of state the cache  202 -B (e.g., each bit in the register  204 -B indicating whether a corresponding line or block in the cache  202 -B is invalid/empty or otherwise valid/owned). After all write operations prior to the special transaction are processed (e.g., committed or written to DIMM of Node B, where at least some of these operations are committed to DIMM of Node B out of order), Node B may send an acknowledgement  212  to Node A. Also, while not shown in  FIG. 2 , in some embodiments, DIMMs of Node A and/or Node B may be coupled to one or more uninterruptible power sources and/or battery packs such that data stored therein is protected even after a power failure. 
         [0014]    Furthermore, the special transaction may be a posted transaction with a posted response, which in general consumes fewer resources than a read transaction used in some solutions that is non-posted. Moreover, in some implementations that use a shared bus (e.g., front-side bus), a mechanism (such as a read operation) may ensure all previous write operations are committed to memory (i.e., flushing any write cache such as cache  204 -B). However, in coherent point-to-point systems (such as system  100  of  FIG. 1 ), the write operations may be performed out of order; hence, there is no guarantee that all previous write operations would be pushed to memory even after a read operation. In an embodiment, to provide the special transaction as a posted transaction, a one-bit register (or a designated location in any of the memory devices of system  200 ) may be updated (e.g., by software). In turn, a logic (such as control logic of MCU  201 -B or another logic in Node B (not shown)) may detect the updating (e.g., an edge, or change between a  0  or  1  in the register) and cause the register  204 -B to be updated with information about cache lines (or blocks) of the cache  202 -B. 
         [0015]      FIG. 3  illustrates a flow diagram of transactions between PCIe, Integrated I/O (HO), and memory, according to an embodiment. In one embodiment, PCIe of  FIG. 3  may be the same or similar to the PCIe coupling between PCIe NTB of Node A of  FIG. 2 . The IIO of  FIG. 3  may be the same or similar to the PCIe NTB of Node B and MCU  201 -B of  FIG. 2 , while “memory” may be the same or similar to the DIMM of Node B of  FIG. 2 . Also, dashed arrows in  FIG. 3  indicate operations that are guaranteed to complete in order which is required to guarantee PCI ordering rules. Also,  FIG. 3  is a simple example describing the mechanism for 2 writes, but embodiment of the invention may work with N number of writes followed by a Special Packet, followed by M number of subsequent writes, etc. 
         [0016]    As illustrated in  FIG. 3 , two PCIe write operations (Write  1  and Write  2 ) are sent to memory. In this example, the data being written to the DIMM will occur in different order than written on PCIe bus due to unordered data channels. Hence, WbIData (Writeback data, e.g., indicating downgrade to I (invalid) state (which means that the data is being written from the cache and this cache is now in the I-state)) packets may occur out of order and Acknowledge for Write  1  and Write  2  are to be sent after both write operations are committed to memory. 
         [0017]    Moreover, once the write operations are received at the IIO, InvWbMtoI. (request E (exclusive) state without data) packets are sent to memory and the cache  202 -B at Node B is updated with data from these write operations. In response to receipt of the special packet at IIO, a snapshot of state of the cache  202 -B is stored in the register  204 -B. The memory responds InvWbMtoI by sending Gnt_Cmp (grant E state ownership without data) packets. After receiving Gnt_Cmp&#39;s, the IIO issues WbMtol (downgrade from M (modified) to I, e.g., signal an in-flight WbIData message about to be sent) packets and WbIData packets). Upon receipt of Cmp (completed, e.g., indicating that WbIData has been received by the memory) packets at IIO, the register  204 -B is updated (e.g., indicating invalidity or empty status of a corresponding line or block in the cache  202 -B). Once register  204 -B indicates all entries of the cache  202 -B per the most recent snapshot) are empty/invalidated, the acknowledge packet is sent to the PCIe NTB of Node A. 
         [0018]    In an embodiment, any write operations occurring after the special packet may be ignored for the purposes of tracking until the acknowledgment is sent for any pending special packets. Alternatively, for multiple ports, the snapshot logic, cache, register, etc. may be replicated. 
         [0019]    Accordingly, in some embodiments, protocol to mirror data may provide for high memory bandwidth between the CPU nodes (e.g., to provide efficiency and/or speed through a sufficient number of write operations) and it may also provide for an acknowledgement when these write operations have been committed to the redundant memory. Moreover, in such systems, a large number of write operations are mirrored, followed by a special transaction (snapshot) that causes an acknowledgement to be returned once the previous write operations (issued prior to the snapshot) have been committed to the battery backed memory. After the special transaction is sent, the mirrored write operations may continue; however, these subsequent write operations are not considered protected until the next special transaction is sent and an acknowledgement is received. Accordingly, to provide for high bandwidth, no data stall may be introduced during the mirroring. 
         [0020]      FIG. 4  illustrates a block diagram of an embodiment of a computing system  400 . One or more of the agents  102  of  FIG. 1  may comprise one or more components of the computing system  400 . Also, in at least one embodiment, snapshot logic, cache, and register discussed with reference to  FIGS. 1-4  may be provided in the system  400 , e.g., within various components of system  400  such as Graphics Memory Control Hub (GMCH)  408 , I/O Control Hub (ICH)  420 , etc. The computing system  400  may include one or more central processing unit(s) (CPUs)  402  (which may be collectively referred to herein as “processors  402 ” or more generically “processor  402 ”) coupled to an interconnection network (or bus)  404 . The processors  402  may be any type of processor such as a general purpose processor, a network processor (which may process data communicated over a computer network  405 ), etc. (including a reduced instruction set computer (RISC) processor or a complex instruction set computer (CISC)). Moreover, the processors  402  may have a single or multiple core design. The processors  402  with a multiple core design may integrate different types of processor cores on the same integrated circuit (IC) die. Also, the processors  402  with a multiple core design may be implemented as symmetrical or asymmetrical multiprocessors. 
         [0021]    The processor  402  may include one or more caches, which may be private and/or shared in various embodiments. Generally, a cache stores data corresponding to original data stored elsewhere or computed earlier. To reduce memory access latency, once data is stored in a cache, future use may be made by accessing a cached copy rather than refetching or recomputing the original data. The cache(s) may be any type of cache, such a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L3), a mid-level cache, a last level cache (LLC), etc. to store electronic data (e.g., including instructions) that is utilized by one or more components of the system  400 . Additionally, such cache(s) may be located in various locations (e.g., inside other components to the computing systems discussed herein, including systems of  FIG. 1-2  or  5 ). 
         [0022]    A chipset  406  may additionally be coupled to the interconnection network  404 . Further, the chipset  406  may include a graphics memory control hub (GMCH)  408 . The GMCH  408  may include a memory controller  410  that is coupled to a memory  412 . The memory  412  may store data, e.g., including sequences of instructions that are executed by the processor  402 , or any other device in communication with components of the computing system  400 . Also, in one embodiment of the invention, the memory  412  may include one or more volatile storage (or memory) devices such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), etc. Nonvolatile memory may also be utilized such as a hard disk. Additional devices may be coupled to the interconnection network  404 , such as multiple processors and/or multiple system memories. 
         [0023]    The GMCH  408  may further include a graphics interface  414  coupled to a display device  416  (e.g., via a graphics accelerator in an embodiment). In one embodiment, the graphics interface  414  may be coupled to the display device  416  via an accelerated graphics port (AGP). In an embodiment of the invention, the display device  416  (such as a flat panel display) may be coupled to the graphics interface  414  through, for example, a signal converter that translates a digital representation of an image stored in a storage device such as video memory or system memory (e.g., memory  412 ) into display signals that are interpreted and displayed by the display  416 . 
         [0024]    As shown in  FIG. 4 , a hub interface  418  may couple the GMCH  408  to an input/output control hub (ICH)  420 . The ICH  420  may provide an interface to input/output (I/O) devices coupled to the computing system  400 . The ICH  420  may be coupled to a bus  422  through a peripheral bridge (or controller)  424 , such as a peripheral component interconnect (PCI) bridge that may be compliant with the PCIe specification, a universal serial bus (USB) controller, etc. The bridge  424  may provide a data path between the processor  402  and peripheral devices. Other types of topologies may be utilized. Also, multiple buses may be coupled to the ICH  420 , e.g., through multiple bridges or controllers. Further, the bus  422  may comprise other types and configurations of bus systems. Moreover, other peripherals coupled to the ICH  420  may include, in various embodiments of the invention, integrated drive electronics (IDE) or small computer system interface (SCSI) hard drive(s), USB port(s), a keyboard, a mouse, parallel port(s), serial port(s), floppy disk drive(s), digital output support (e.g., digital video interface (DVI)), etc. 
         [0025]    The bus  422  may be coupled to an audio device  426 , one or more disk drive(s)  428 , and a network adapter  430  (which may be a NIC in an embodiment). In one embodiment, the network adapter  430  or other devices coupled to the bus  422  may communicate with the chipset  406 . Also, various components (such as the network adapter  430 ) may be coupled to the GMCH  408  in some embodiments of the invention. In addition, the processor  402  and the GMCH  408  may be combined to form a single chip. In an embodiment, the memory controller  410  may be provided in one or more of the CPUs  402 . Further, in an embodiment, GMCH  408  and ICH  420  may be combined into a Peripheral Control Hub (PCH). 
         [0026]    Additionally, the computing system  400  may include volatile and/or nonvolatile memory (or storage). For example, nonvolatile memory may include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), a disk drive (e.g.,  428 ), a floppy disk, a compact disk ROM (CD-ROM), a digital versatile disk (DVD), flash memory, a magneto-optical disk, or other types of nonvolatile machine-readable media capable of storing electronic data (e.g., including instructions). 
         [0027]    The memory  412  may include one or more of the following in an embodiment: an operating system (O/S)  432 , application  434 , and/or device driver  436 . The memory  412  may also include regions dedicated to Memory Mapped I/O (MMIO) operations. Programs and/or data stored in the memory  412  may be swapped into the disk drive  428  as part of memory management operations. The application(s)  434  may execute (e.g., on the processor(s)  402 ) to communicate one or more packets with one or more computing devices coupled to the network  405 . In an embodiment, a packet may be a sequence of one or more symbols and/or values that may be encoded by one or more electrical signals transmitted from at least one sender to at least on receiver (e.g., over a network such as the network  405  or other channel such as those discussed with reference to  FIGS. 1-3 ). 
         [0028]    In an embodiment, the application  434  may utilize the O/S  432  to communicate with various components of the system  400 , e.g., through the device driver  436 . Hence, the device driver  436  may include network adapter  430  specific commands to provide a communication interface between the O/S  432  and the network adapter  430 , or other I/O devices coupled to the system  400 , e.g., via the chipset  406 . 
         [0029]    In an embodiment, the O/S  432  may include a network protocol stack. A protocol stack generally refers to a set of procedures or programs that may be executed to process packets sent over a network  405 , where the packets may conform to a specified protocol. For example, TCP/IP (Transport Control Protocol/Internet Protocol) packets may be processed using a TCP/IP stack. The device driver  436  may indicate the buffers in the memory  412  that are to be processed, e.g., via the protocol stack. 
         [0030]    The network  405  may include any type of computer network. The network adapter  430  may further include a direct memory access (DMA) engine, which writes packets to buffers (e.g., stored in the memory  412 ) assigned to available descriptors (e.g., stored in the memory  412 ) to transmit and/or receive data over the network  405 . Additionally, the network adapter  430  may include a network adapter controller, which may include logic (such as one or more programmable processors) to perform adapter related operations. In an embodiment, the adapter controller may be a MAC (media access control) component. The network adapter  430  may further include a memory, such as any type of volatile/nonvolatile memory (e.g., including one or more cache(s) and/or other memory types discussed with reference to memory  412 ). 
         [0031]      FIG. 5  illustrates a computing system  500  that is arranged in a point-to-point (PtP) configuration, according to an embodiment of the invention. In particular,  FIG. 5  shows a system where processors, memory, and input/output devices are interconnected by a number of point-to-point interfaces. The operations discussed with reference to  FIGS. 1-4  may be performed by one or more components of the system  500 . Also, one or more of the agents  102  of  FIG. 1  may comprise one or more components of the computing system  500 . 
         [0032]    As illustrated in  FIG. 5 , the system  500  may include several processors, of which only two, processors  502  and  504  are shown for clarity. The processors  502  and  504  may each include a local memory controller hub (GMCH)  506  and  508  to enable communication with memories  510  and  512 . The memories  510  and/or  512  may store various data such as those discussed with reference to the memory  412  of  FIG. 4 . As shown in  FIG. 5 , the processors  502  and  504  (or other components of system  500  such as chipset  520 , I/O devices  543 , etc.) may also include one or more cache(s) such as those discussed with reference to  FIGS. 1-4 . 
         [0033]    In an embodiment, the processors  502  and  504  may be one of the processors  402  discussed with reference to  FIG. 4 . The processors  502  and  504  may exchange data via a point-to-point (PtP) interface  514  using PtP interface circuits  516  and  518 , respectively. Also, the processors  502  and  504  may each exchange data with a chipset  520  via individual PtP interfaces  522  and  524  using point-to-point interface circuits  526 ,  528 ,  530 , and  532 . The chipset  520  may further exchange data with a high-performance graphics circuit  534  via a high-performance graphics interface  536 , e.g., using a PtP interface circuit  537 . 
         [0034]    In at least one embodiment, snapshot logic, cache, and register discussed with reference to  FIGS. 1-4  may be provided in the system  500 , e.g., within various components of system  500  such as the chipset  520 , I/O devices  543 , communication devices  546 , etc. Other embodiments of the invention, however, may exist in other circuits, logic units, or devices within the system  500  of  FIG. 5 . Furthermore, other embodiments of the invention may be distributed throughout several circuits, logic units, or devices illustrated in  FIG. 5 . 
         [0035]    The chipset  520  may communicate with the bus  540  using a PtP interface circuit  541 . The bus  540  may have one or more devices that communicate with it, such as a bus bridge  542  and I/O devices  543 . Via a bus  544 , the bus bridge  542  may communicate with other devices such as a keyboard/mouse  545 , communication devices  546  (such as modems, network interface devices, or other communication devices that may communicate with the computer network  405 ), audio I/O device, and/or a data storage device  548 . The data storage device  548  may store code  549  that may be executed by the processors  502  and/or  504 . 
         [0036]    In various embodiments of the invention, the operations discussed herein, e.g., with reference to  FIGS. 1-5 , may be implemented as hardware (e.g., circuitry), software, firmware, microcode, or combinations thereof, which may be provided as a computer program product, e.g., including a machine-readable or computer-readable medium having stored thereon instructions (or software procedures) used to program a computer to perform a process discussed herein. Also, the term “logic” may include, by way of example, software, hardware, or combinations of software and hardware. The machine-readable medium may include a storage device such as those discussed with respect to  FIGS. 1-5 . Additionally, such computer-readable media may be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) through data signals provided in a carrier wave or other propagation medium via a communication link (e.g., a bus, a modem, or a network connection). 
         [0037]    Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least an implementation. The appearances of the phrase “in one embodiment” in various places in the specification may or may not be all referring to the same embodiment. 
         [0038]    Also, in the description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. In some embodiments of the invention, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements may not be in direct contact with each other, but may still cooperate or interact with each other. 
         [0039]    Thus, although embodiments of the invention have been described in language specific to structural features and/or methodological acts, it is to be understood that claimed subject matter may not be limited to the specific features or acts described. Rather, the specific features and acts are disclosed as sample forms of implementing the claimed subject matter.