Write posting with global ordering in multi-path systems

An apparatus and method for permitting write posting with global ordering in a multipath system. The apparatus and method including a bus adapter having an input port to receive one or more operations from a processor. A queue controlled by the bus adapter buffers information from the one or more operations. A control circuit, coupled to the queue, generates an output signal that relates to the information from the one or more operations. The output signal is transmitted to the processor. An interconnect fabric, coupled to each bus adapter, transmits the one or more operations. A device, connected to the interconnect fabric, receives the transmitted operations, where the device sends a acknowledgment signal to the processor upon receiving the transmitted operation.

TECHNICAL FIELD

The invention relates to write posting in multi-processor systems. More particularly, the invention relates to methods and apparatus for write posting with global ordering in a multi-processor system having multiple paths from multiple processors to a single I/O device.

BACKGROUND ART

Computer systems include a means for CPUs to transmit data to input-output (I/O) units. The transit latency for such transactions is typically high, in the hundreds or thousands of CPUs cycles. In order to maximize the performance of sequences of such transactions, it has become commonplace to employ pipelining, also known as write-posting, in this particular context. With write-posting, a CPU may emit a successor write before the preceding write has progressed all the way to its destination.

Certain challenges to maintaining system ordering can emerge in the context of write-posting. For example, bus protocols frequently include a retry mechanism. If two successive posted writes are sent, and the first is retried, the writes may arrive at their destination in the opposite order in which they were transmitted.

Furthermore, in multiple CPU systems, it is generally necessary for writes from different CPUs sent to the same destination to be coordinated. Assuming some ordering relationship between CPUs established by some mechanism, such as a memory semaphore, I/O writes emitted by a CPU ordered first, arrive at the destination before those from a CPU ordered second.

Both of these problems are easily solved in single-bus systems, that is systems in which there is only one path from any CPU to each I/O device. The retry problem can be solved by retrying all writes to the same destination once the first is retried, coupled with a protocol requirement that only the oldest retry is reissued until it is accepted. The second situation cannot arise by definition; if there is only one path, arrival order must equal issue order.

Various means have been used to solve the above problem in multi-path systems. One is the use of a special sync operation. Here, I/O writes are posted as for a single-path system. But before a handoff permission for a different CPU to begin writing to a device is received, the previously writing CPU issues the sync . The sync acts as a probe to assure that all paths from the issuing CPU to all possible destinations have ben drained. When the probe indicates it is complete, the first CPU indicates to the second CPU, via a semaphore or interrupt, that it may proceed. An ordering fence, either implied or explicit, is used between the sync and the proceed indication. Performance and complexity are the drawbacks to this approach. Complexity increases with attempts to restrict the scope of the sync to the minimum necessary. Performance decreases, the more heavyweight and simple the sync operation is.

Another approach is to follow the last write from the first CPU with a read from that CPU to the same destination. An implicit or explicit fence can separate the return of that read with the indication for the next CPU to proceed. The disadvantage here is with software constraints and complexity. Legacy software written for single-path systems where this is not necessary can be difficult to modify. The transfer of control may be managed by a layer of software that is isolated from that doing the writes, thus, the necessity of the read or where to direct it is difficult to determine. An example of this is OS pre-emption and process migration.

Accordingly, a need exists for an apparatus to accommodate write posting with global ordering in multiple path systems.

SUMMARY OF INVENTION

An apparatus and method consistent with the present invention for permitting write posting with global ordering in a multipath system. The apparatus and method including a bus adapter having an input port to receive one or more operations from a processor. A queue controlled by the bus adapter to buffer information from the one or more operations. A control circuit, coupled to the queue, to generate an output signal that relates to the information from the one or more operations. The output signal is transmitted to the processor. An interconnect fabric, coupled to each bus adapter, to transmit the one or more operations. A device, connected to the interconnect fabric, to receive the transmitted operations where the device sends an acknowledgment signal to the processor upon receiving the transmitted operation.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

FIG. 1 is a diagram of a multi-processor computer system 100 having multiple paths from multiple processors to a single I/O device. In system 100 , a plurality of processors 105 , 110 writes to its bus adapters 115 , 120 via their processors bus 125 , 130 , respectively. Processor 105 writes to bus adapter 115 via bus 125 and processor 110 writes to bus adapter 120 via bus 130 . Two processors and bus adapters are shown, however, additional processors and bus adapters could be included in the system 100 .

The bus adapters 115 , 120 are interfaces which receive signals from the processors 105 , 110 . The processors 105 , 110 are connected to an interconnect fabric 135 which could be a mesh, ring, cross bar or other topology. The interconnect fabric 135 is not a simple bus, where only one processor can access it at a time and therefore assure ordering by virtue of that fact. The interconnect fabric 135 has various buffering stages within itself, where the bus adaptors are just the first stage of interconnect to the system's 100 interconnect fabric 135 . Also connected to the interconnect fabric 135 is an I/O device 140 . The I/O device 140 is connected to the interconnect fabric 135 via bus adapter 145 . The I/O device 140 gives the processors 105 , 110 access to other parts of the system 100 . Examples of I/O devices 140 are disc drives and graphic displays.

FIG. 2 is a timing diagram 150 of a guaranteed sequentiality signal. Initially, the processor 105 writes an address 155 of a memory the processor 105 is accessing. One address typically represents a packet of 32, 64 or 128 bites of data. Shortly after the first address 155 is sent, guaranteed sequentiality (GSEQ) signals 165 are generated by the bus adaptor 115 based on queue status within each bus adaptor. Subsequently, data 160 for the first write is sent. The GSEQ signals 165 may be sent before or after the data 160 is sent. Each bus adaptor 115 is driving the GSEQ signal 165 rapidly after an address emerges. The GSEQ signal 165 notifies the processor 105 it can send out successive addresses in order, rapidly and then data corresponding to the addresses shortly thereafter, in a pipeline fashion. The bus adapter 115 is guaranteeing sequentiality via the GSEQ signal 165 , that successive operations from the same processor 105 remain in that order. Sequentiality means that instructions from processor 105 are all arriving at the I/O device 140 in the order of issuance. The bus adapter 115 only generates the GSEQ signal 165 if it knows it will not issue a retry signal 170 .

In addition to the GSEQ signal 165 , the system 100 also can issue the retry signal 170 that is generated by each bus adapter based on the queue status within each bus adaptor. The retry indicates a queue full signal or similar resource constraint. Since the GSEQ signal 165 is a function of the number of entries available in the queue, the GSEQ signal 165 indicates that enough entries are still available. If the queue is full, then the retry signal 170 is sent to the processor 105 to resend the transaction, but if there are enough entries available in the queue, then the GSEQ signal 165 is asserted and guarantees acceptance of that transaction without retrying. The processors 105 , 110 can then send out the next address.

Assuming that the retry signal 170 was not sent, the data 160 for the transaction would be sent, and that might or might not be on the same set of wires. Frequently its efficient to have the data to be on a different set of wires then the address. Since these are write transactions, a processor is sending out both the address and the data. In some implementations, address and data are multiplexed under the same wires, and in other implementations they are separate. At some point, assuming there was no retry 170 , data 160 is sent out and some time later the I/O device 140 has received the data 160 , either the I/O device 140 or the I/O device's 140 bus adaptor 145 that is immediately up stream from the I/O device 140 , sends an acknowledgment 175 . The acknowledgment 175 comes back to the processors after the I/O device 140 has received the data 160 .

The system 100 maintains sequentiality by using the GSEQ signal 165 to send all the addresses 155 one after another and then the processor 105 waits until it gets the acknowledgment 175 back from the I/O device 140 after the processor 105 has sent the last address. This eliminates the need to add special read operations to follow the last write. If the processor 105 does not receive the GSEQ signal 165 , it will wait until the point in time the retry 170 may or may not be signaled. If the retry 170 is not signaled, then the processor 105 goes ahead and sends address 2 . If the retry 170 is signaled, then the processor 105 sends out address I again, and that time it might receive the GSEQ signal 165 . A sequentiality commit point is the point at which it is known the transaction will not be retried.

The acknowledgment 175 is being sent by either the I/O device 140 itself or the bus adaptor 145 for the I/O device 140 , such that the connection between the bus adaptor 145 and the I/O device 140 is a single path. The acknowledgment 175 , when it is received by the processor 105 is at an ordering commit point. The ordering commit point is the point at which a given write transaction cannot be passed by any other write transaction. Therefore, the processor 105 knows, when it received an acknowledgment 175 that the data 160 sent was successfully received by the I/O device 140 .

If the processor 105 issues an order dependant operation, such as a semaphore, the processor 105 will wait for any outstanding acknowledgments 175 , even though the processor 105 has sent off multiple writes rapidly in order. If the processor 105 executes a semaphore operation it will stall until all the acknowledgments 175 have come back. So if the operation requires another processor 115 it has to wait for the acknowledge 175 to ensure proper order. The system 100 is using ordering semantics attached to a semaphore to essentially lock out the bus. This is how the system 100 can safely hand off control of one path to a different processor and ensure that the operations don't get interlaced. This enables the system 100 to maintain ordering between the multiple processors.

FIG. 3 is a diagram of the bus adapter implementing the GSEQ signal. Guaranteed sequentiality is implemented by adding some additional logic gates to each bus adaptor 115 , 120 to enable each bus adapter to determine whether the GSEQ signal 165 should be sent. The address and data buses from the processor enter the bus adaptor. Each bus adaptor contains a queue 180 that the addresses and data enter. As an example, FIG. 3 illustrates an eight entry queue 180 . The queue 180 contains the address 185 and data 190 and it may also contain a valid bit 195 , although the valid bit 195 is not necessary for the particular implementation of the GSEQ signal 165 generation.

The queue 180 also contains an input pointer 200 which points to the next entry to be occupied with the incoming data. The queue 180 also has an output pointer 205 which points to the last occupied entry to go out to the interconnect fabric 135 . Alternatively, as an implementation choice, the input and output pointers 200 , 205 could have a pre-increment on the pointer as to whether it is actually occupied or not. No matter the implementation, both the input and output pointers 200 , 205 continually increment such that, the pointers wrap around. The pointer values enter a substracter 210 and the result 215 of that subtraction is the number of entries that are occupied in the queue 180 . In the present example, the white boxes represent the number of available entries in the queue 180 and the shaded boxes indicate occupied entries in the queue 180 .

In the example, the input pointer 200 is pointing to entry 2 and the output pointer 205 is pointing to entry 6 . The result 215 of 6 minus 2 is 4 entries are occupied within the queue 180 . That result 215 is compared by a comparator 220 to a fixed subset of the queue 180 , in this case 5 entries. Since the queue 180 has less than or equal to 5 entries occupied within it, the bus adaptor 115 drives the GSEQ signal 165 notifying the processor 105 it can accept more transactions. If the number of occupied entries is equal to the number of total entries available in the queue 180 , then the retry signal 170 would be sent to the processor 105 to indicate the queue 180 is filled up.

FIG. 4 is a flow chart illustrating a process of bus adapter 115 implementing the GSEQ signal 165 . The bus adapter 115 , as explained above with respect to FIGS. 2 and 3 , typically implements these functions using hardware modules. However, it may also alternatively implement these functions in software or firmware. In process 225 , the processor 105 sends a write transaction to its bus adaptor 115 (step 230 ). The bus adaptor 115 checks the queue 180 to determine if the GSEQ signal 165 should be sent to the processor 105 (step 235 ). If the GSEQ signal 165 cannot be sent to the processor 105 , the processor may receive a retry signal 170 (step 240 ). The processor 105 can then resend address 1 to the bus adaptor 115 . If the GSEQ signal 165 can be sent, the processor 105 determines if that was the last address it wants to send (step 245 ). If it is not the last address, the processor 105 determines the next address (step 250 ) and sends the next address to the bus adaptor 115 . If the last address was sent, then the processor waits for an acknowledge signal 175 from the I/O device 140 or from the I/O device's bus adaptor 145 (step 255 ). Upon receiving the acknowledge signal 175 , the processor 105 resets the semaphore bit to pass control to processor 110 (step 260 ). The determination of last address sent might not be made by the same software process or layer, as is sending the writes. For example, an operating system might interrupt and suspend the sending process, and reschedule that process on a different processor.

The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention as defined in the following claims, and their equivalents, in which all terms are to be understood in their broadest possible sense unless otherwise indicated.