Computer system with concurrent direct memory access

A computer system with concurrent direct memory access is provided including a computer node having a processor interface bus and a cut-through network interface controller installed on the processor interface bus. A data transfer is initiated through the cut-through network interface controller by starting a direct memory access to move data from a memory to a transmit buffer in the cut-through network interface controller and a network interface controller physical interface transmitting the data, to the computer node attached to a reliable network, before all of the data is in the transmit buffer.

CROSS-REFERENCE TO RELATED APPLICATION

The present application contains subject matter related to a concurrently filed application by Michael Schlansker and Erwin Oertli entitled “Virtual Network Interface System with Memory Management”. The related application is identified by application Ser. No. 11/553,976 and is assigned to Hewlett-Packard Development Company, LP.

TECHNICAL FIELD

The present invention relates generally to network interface controllers, and more particularly to a system for network interface transmission with enhanced direct memory access operation.

BACKGROUND ART

Computer networks are an increasingly important part of both private and business environments. Computing devices such as workstations, personal computers, server computers, storage devices, firewalls and other computing devices function as nodes of a network with at least one network element connecting the computing devices. The various nodes transmit and/or receive various kinds of information over the network. Computing devices and users are demanding higher communication speeds across networks as more and more information flows across the various networks. The introduction of new technologies will likely load down networks even more.

The state of the art in high speed data access on computer networks has in large part been driven by exponential growth in the Internet and e-commerce. Furthermore, as computers become more powerful, applications are always being developed which take advantage of any increase in computer performance. Often, these applications utilize networks, both local and global.

It is becoming increasingly important to keep pace with the increased demands for network services by the general public. This can be accomplished by removing the bottlenecks that inhibit data transfer across computer networks because the thirst for increased bandwidth is ever present. Internet users are becoming ubiquitous as home users and businesses tap into the resources of the information superhighway. Electronic mail, which is fast becoming the preferred method of communication in business as well as in the private sector, and new business models, such as the Virtual Office, rely on computer networks for their very existence. In essence, the demand for computer networking connectivity and bandwidth is large, and growing larger all the time.

In an effort to keep up with increasing network connectivity and bandwidth demands, makers of networking hardware and software, as well as the Information Services (IS) managers that operate computer networks are continually looking for ways to improve network connectivity and bandwidth, while reducing network traffic latency.

Increasingly, computer networks are being called upon to carry time-critical telecommunications and video data streams. Guaranteed bandwidth to residential communications ports that carry voice, video and data has increased from tens of kilobits/second to Megabits/second levels. Commercial communications bandwidth has increased to several Megabits/second guaranteed bandwidth per port. However, the infrastructure that enables Wide and Local Area Networks to operate is comprised of installed network gear that is running industry standard network protocols that are not well-suited for the performance demands of time-critical, latency-intolerant network traffic such as voice and video.

Thus, a need still remains for a computer system with concurrent direct memory access that can help to reduce the latency of the network. In view of the ever increasing demand for faster network response, it is increasingly critical that answers be found to these problems. In view of the ever-increasing commercial competitive pressures, along with growing consumer expectations and the diminishing opportunities for meaningful product differentiation in the marketplace, it is critical that answers be found for these problems. Additionally, the need to save costs, improve efficiencies and performance, and meet competitive pressures, adds an even greater urgency to the critical necessity for finding answers to these problems.

SUMMARY

The present invention provides a computer system with concurrent direct memory access which includes a computer node having a processor interface bus with a cut-through network interface controller carefully positioned on the processor interface bus of the computer node. A data transfer initiated through the cut-through network interface controller includes a direct memory access to move the data from a memory to a transmit buffer in the cut-through network interface controller and a network interface controller physical interface starts transmitting the data, to the computer node attached to a reliable network, before all of the data is in the transmit buffer.

DETAILED DESCRIPTION

In the following description, numerous specific details are given to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In order to avoid obscuring the present invention, some well-known circuits, system configurations, and process steps are not disclosed in detail. Likewise, the drawings showing embodiments of the system are semi-diagrammatic and not to scale and, particularly, some of the dimensions are for the clarity of presentation and are shown greatly exaggerated in the drawing FIGs. Where multiple embodiments are disclosed and described, having some features in common, for clarity and ease of illustration, description, and comprehension thereof, similar and like features one to another will ordinarily be described with like reference numerals.

Referring now toFIG. 1, therein is shown a block diagram of a computer system with concurrent direct memory access100with concurrent direct memory access, in an embodiment of the present invention. The block diagram of the computer system with concurrent direct memory access100depicts a computer node102having a network interface controller (NIC)104, such as an Ethernet controller, optical interface controller, or RF interface controller, supported by a composite virtual NIC106that is linked to an application108. The composite virtual NIC106comprises a virtual network interface controller (VNIC)110and a flow-control NIC112, such as a control circuit. The NIC104and the VNIC110are coupled to a memory114, such as a pinned memory within the system memory, for storing and sending messages. The pinned memory is any memory that is constantly dedicated to support a function, such as the NIC104. The pinned memory may not be off-loaded to disk storage or virtual memory. The NIC104is coupled to a reliable network118, which is further coupled to a plurality of other units of the computer node102.

The NIC104is the physical interface controller. It sends and receives the electronic, optical, or RF signals between the reliable network118and other units of the computer node102. Each of the applications108that utilize the services of the NIC104is assigned the composite virtual NIC106. The composite virtual NIC106comprises a block of the memory114and a driver program. The composite virtual NIC106is made up of the VNIC110, which manages the movement of messages and data between the NIC104and the application108, and the flow-control NIC112. The flow-control NIC112manages the utilization of the memory114in the VNIC110on the destination side of the reliable network118. Collectively, all of the computer nodes102attached to the reliable network118may be considered a computer cluster116or the computer cluster116contains the reliable network118.

The flow-control NIC112functions in circuitry that performs as the memory manager for the transfer of data for the application108to which it is linked. When the computer node102is initialized, the flow-control NIC112is assigned a unique identity for network communication. The unique identifier may consist of the MAC address of the NIC104and a port identifier associated with the VNIC110. The memory114available to the VNIC110is equally divided into segments called credits. Usually a credit represents a sufficient amount of the memory to transfer a packet of data. When the application108wishes to transfer a large amount of data, the flow-control NIC112assembles a message that requests an additional block of credits sufficient to handle a portion or the entire data transfer.

Referring now toFIG. 2, therein is shown a block diagram of a transmit path200of the computer system with concurrent direct memory access100, ofFIG. 1. The block diagram of the transmit path200depicts the reliable network118coupled to the NIC104and the NIC104is coupled to the memory114. The memory114is coupled to the NIC104by a processor interface bus202, such as a front side bus. A direct memory access (DMA)204is a circuit that receives the signals from the processor interface bus202. The DMA204copies message data form the memory114to a transmit buffer206. The transmit buffer206is a memory that acts as a temporary holding area for a NIC physical interface208. The NIC physical interface208is the circuitry that is responsible for generating the signals that interact with the reliable network118. A transfer processor210monitors the movement of the message date form the DMA204through the transmit buffer206to the NIC physical interface208.

The DMA204starts a message transmission by copying the data from the memory114to the to the transmit buffer206. When a sufficient amount of data is in the transmit buffer206, the NIC physical interface208starts to transfer the data to the reliable network118. The transfer is started prior to the arrival of all of the data that will be transferred. The DMA204can transfer the data at a much higher rate than the NIC physical interface208. The transfer speed advantage that the DMA204possesses allows the NIC physical interface208to start the transfer earlier than a prior art version of the NIC104, which would transfer all of the data into the transmit buffer206prior to the transfer to the reliable network118.

The pipelining of these two transfers, the DMA204into the transmit buffer206and the NIC physical interface208out of the transmit buffer, becomes a calculated risk. The overwhelming probability is in favor of the DMA204transferring all of the message data into the transmit buffer206before the NIC physical interface208can transfer it out. As long as the last byte of the message is in the transmit buffer206before the NIC physical interface208finishes, the transfer will be successful and a significant amount of latency has been cut-through. The NIC104that displays these characteristics may be called the “cut-through network interface controller”104.

The transfer processor210monitors the movement of the data through the transmit buffer206. If for some reason, such as the memory114refresh or contention on the processor interface bus202, the NIC physical interface208should overrun the transmit buffer206, the transfer processor210would signal the error. In response to the error flagged by the transfer processor210, the NIC physical interface208would end the message packet transmission with an error designator such as a framing error or a circular redundancy check (CRC) error. The error designator will cause the receiving device to reject and drop the packet.

The transfer processor210may continue to monitor the movement of the message data from the memory114to the transmit buffer206. When all of the message data is located in the transmit buffer206, the transfer processor210may cause the NIC physical interface208to re-transmit the packet. The re-transmit of the erroneous packet caused by the transfer processor210represents another significant reduction in latency over a software based re-transmission. The transfer protocol, such as EtherNet, takes a significant amount of time to declare the packet missing and request a re-transmission of the data. Upon the successful completion of the re-transmitted packet, the operation of the NIC physical interface208and the DMA204reverts back to the normal overlap of data fetch and data transmission.

Referring now toFIG. 3, therein is shown a block diagram of the transfer processor210, ofFIG. 2. The block diagram depicts a DMA communication bus302coupled to a DMA interface304. The DMA interface304is coupled to a buffer interface306, which receives input from a buffer communication bus308, and a NIC status manager310. The NIC status manager310is coupled to a transmission monitor312and to the NIC physical interface208, ofFIG. 2, through a status communication bus314.

The DMA interface304receives information from the DMA204, ofFIG. 2, through the DMA communication bus302. The DMA interface304stores the details about the message that is to be transmitted, such as the total number of bytes and load status. Some of the information is passed to the NIC status manager310for transfer monitoring. The DMA interface304also passes load status and timing information to the buffer interface306. The buffer interface306resolves any timing issues between the transmit buffer206, ofFIG. 2, being loaded by the DMA204while being read by the NIC physical interface208. These events may happen concurrently in different locations of the transmit buffer without presenting an error. The buffer interface306receives input, from the buffer communication bus308, regarding the buffer location being read by the NIC physical interface208. This information is compiled and passed to the NIC status manager310for processing.

Signals from the status communication bus314are passed through the NIC status manager310to the transmission monitor312for synchronization and checking. In the vast majority of the transfers, the DMA204will provide all of the data to the transmit buffer206before the NIC physical interface208is near the end of the packet transfer. In this case the NIC status manager310presents an error free status to the NIC physical interface208. A rare possibility is that the DMA304may not complete the transfer of data to the transmit buffer206prior to the NIC physical interface208reading beyond the current data pointer. In this case the buffer interface306would detect a buffer wrap error and flag the error to the NIC status manager310.

The NIC status manager310communicates an error to the NIC physical interface208, which terminates the packet transfer with an error designator such as a framing error or a CRC error. The framing error can be caused by sending an unusual number of bits, for example not an integer multiple of eight bits, and presenting an incorrect CRC at the end of the packet. The CRC error may also be presented without the framing error. When this packet is detected at the destination, it will be dropped without response. If the packet is delayed or never arrives, a packet timeout would occur at the TCP/IP software level causing the destination to again request the transmission of the packet. This could represent a substantial delay in the processing of the data.

In order to expedite the transfer and maintain the reduction in latency, the NIC status manager310may initialize the pointers in the NIC physical interface208and immediately enable the retransmission of the packet prior to a network timeout without an upper level software, such as a Transfer Control Protocol/Internet Protocol (TCP/IP), for support. Under normal circumstances the DMA304would resume the transfer and complete the loading of the transmit buffer206without additional interruption. If per chance a second occurrence of the buffer wrap error is presented, the packet will not be retransmitted without all of the data transferred into the transmit buffer. In the event of the second buffer wrap error, the buffer interface306continues to monitor the flow of data into the transmit buffer206.

When all of the expected data resides in the transmit buffer206, the buffer interface306signals the NIC status manager310to enable re-transmission of the entire packet. The NIC status manager310resets the transmit buffer pointers in the NIC physical interface208and once again enables the transfer of the data. The NIC physical interface208transfers all of the data in the packet then awaits the next transmission command. At the destination, the packet is received and is passed on without additional delay.

Referring now toFIG. 4, therein is shown a timing diagram of the transmit buffer206interface in a successful transmission of a message packet using the concurrent direct memory access. The timing diagram depicts a DMA transfer signal402, which in this example writes eight bytes of the data on each active edge of the signal. When the DMA transfer signal402has loaded a sufficient number of bytes into the transmit buffer206, ofFIG. 2, a NIC read signal404commences to transfer data from the transmit buffer206to the NIC physical interface208, ofFIG. 2. The NIC read signal404may transfer a single byte of data into the NIC physical interface208for transfer to the reliable network118, ofFIG. 1. A network driver signal406is activated shortly after the first byte of data is loaded into the NIC physical interface208. The network driver signal406cycles one time for each bit sent across the network interface. As such there will be eight cycles of the network driver signal for every byte of data that is transferred.

The DMA transfer signal402will complete the transfer of data to the transmit buffer206well before the NIC read signal404can transfer the contents of the transmit buffer206to the NIC physical interface208. The NIC read signal404may operate in a burst mode as well. Typically after an initial burst of the NIC read signal404that fills the pipeline, a single cycle of the NIC read signal404may occur for every eight cycles of the network driver signal406. The network driver signal406will run constantly without pause as long as data is being transferred.

By way of example only, the timing diagram shows a burst of the DMA transfer signal402indicating a burst transfer of 64 bytes to the transmit buffer and the NIC read signal404starting after 24 bytes have been loaded into the transmit buffer. The actual number of bytes transferred by the DMA204, ofFIG. 2, may be different. As well the number of bytes transferred by the DMA transfer signal402and the NIC read signal404may be different. The starting threshold for initiating the transfer to the NIC physical interface208may different from 24, as any combination that represents the lowest latency and best system performance may be chosen.

Referring now toFIG. 5, therein is shown a timing diagram of the transmit buffer interface in an error condition during the transmission of a message packet using the concurrent direct memory access. The timing diagram depicts the DMA transfer signal402, which in this example writes eight bytes of the data on each active edge of the signal. When the DMA transfer signal402has loaded a sufficient number of bytes into the transmit buffer206, ofFIG. 2, the NIC read signal404commences to transfer data from the transmit buffer206to the NIC physical interface208, ofFIG. 2. The NIC read signal404may transfer a single byte of data into the NIC physical interface208for transfer to the reliable network118, ofFIG. 1. The network driver signal406is activated shortly after the first byte of data is loaded into the NIC physical interface208.

In this example, the DMA transfer signal402is paused for an extended period of time after the initial data is loaded into the transmit buffer206, ofFIG. 2. The pause of the DMA transfer signal may be caused by the memory114, ofFIG. 1, entering a refresh cycle or contention on the processor interface bus202, ofFIG. 2. As the NIC read signal404transfers data from the transmit buffer206, it will eventually deplete the data in the transmit buffer206and stall the transfer of data to the network driver signal406. When the transmit buffer206is depleted of data, an error is detected and the transmission of the current packet is abruptly ended with an error designator such as a framing error or a CRC error.

When this packet is received at the destination, the error designator will be detected and the packet will be discarded without further action. The computer node102, ofFIG. 1, at the destination relies on the TCP/IP software level timeout to monitor expected responses from the reliable network118, ofFIG. 1.

The DMA transfer signal402will eventually resume transfer and complete the transfer of the data to the transmit buffer206. Once all of the data has been transferred into the transmit buffer206, the NIC physical interface208is signaled to re-transmit the packet. The re-transmission of the packet may occur prior to the TCP/IP software level timeout on the computer node102at the destination, thus preventing any additional packet exchanges across the reliable network118, ofFIG. 1.

Referring now toFIG. 6, therein is shown a flow chart of an embodiment of a method600for performing a concurrent direct memory access in a computer system with concurrent direct memory access. The method embodiment600is discussed in the context ofFIGS. 1,2and3for illustrative purposes. As discussed with respect toFIG. 1, when the application108is to transfer a large amount of data, the flow-control NIC112assembles a message which to initiate the transfer of data between the memory114through the cut-through network interface controller104. Via signals from the processor interface bus202, the direct memory access circuit204moves602data from a memory114to a transmit buffer206in the cut-through network interface controller104. The network interface controller physical interface208transmits604the data, to the computer node102attached to a reliable network118, before all of the data is in the transmit buffer206. Responsive to overrun of data in the transmit buffer206, the transfer processor210signals606an error causing the NIC physical interface208to send the message packet transmission with an error designator.

In one aspect, the cut-through NIC of the present invention delivers a significant reduction in latency and provides enhanced performance in an aggregate data transfer. Additionally, the present invention provides lower latency, lower overhead and increased performance as compared to prior art network interface controllers.

In another aspect of the present invention, by carefully positioning the network interface system, for example by coupling it to the front side bus, a dramatic performance increase is possible.