Method and system for network communication

A method and system for network communication is provided. The method for network communication comprises setting a data size for a network connection, wherein the data size represents an amount of network data a network adapter can send to a host system for the network connection before the network adapter waits for an application to accept any data that has been sent to the host system; monitoring the amount of network data that is received by a host system driver; monitoring the amount of network data that is sent by the network adapter; and suspending transfer of network data to the host system, if the amount of network data sent by the network adapter is similar to the set data size.

FIELD OF THE INVENTION

The present invention relates to network systems, and more particularly, to offloading host system tasks for managing network related operations.

BACKGROUND OF THE INVENTION

Computer networks are commonly used in various applications. Computing systems typically use a network adapter to communicate with other networked devices. Continuous efforts are being made to efficiently improve processing of network data.

SUMMARY

In one embodiment, a method for network communication is provided. The method comprises setting a data size for a network connection, wherein the data size represents an amount of network data a network adapter can send to a host system for the network connection before the network adapter waits for an application to accept any data that has been sent to the host system; monitoring the amount of network data that is received by a host system driver; monitoring the amount of network data that is sent by the network adapter; and suspending transfer of network data to the host system, if the amount of network data sent by the network adapter is similar to the set data size.

In another embodiment, a method for network communication is provided. The method comprises setting a data size for a network connection, wherein the data size represents an amount of network data a network adapter can send to a host system for the network connection before the network adapter waits for an application accept any data that has been sent to the host system; monitoring an amount of network data that is received by a host system driver; wherein a value for a first counter for a host system memory indicates the amount of network data received by the host system driver; monitoring an amount of network data that is sent by the network adapter; wherein a value for a second counter for a network adapter memory indicates the amount of data that is sent to the host system; and suspending transfer of network data to the host system, if the amount of network data sent by the network adapter is similar to the set data size.

In yet another embodiment, a system for network communication is provided. The system comprises a host system executing an application for communicating with at least one networked device; and a network adapter interfacing with the host system and receiving network data from the at least one network device; wherein a data size is set for a network connection and the data size represents an amount of network data the network adapter can send to the host system for the network connection before the network adapter waits for the application to accept any data that has been sent to the host system; and the network adapter suspends transfer of network data to the host system, if the amount of network data sent by the network adapter is similar to the set data size after monitoring the amount of network data that is received by a host system driver and monitoring the amount of network data that is sent by the network adapter.

This brief summary has been provided so that the nature of the invention may be understood quickly. A more complete understanding of the invention can be obtained by reference to the following detailed description of the various embodiments thereof concerning the attached drawings.

DETAILED DESCRIPTION

To facilitate the understanding of the various embodiments, a top-level description of common network protocols/standards and the general architecture/operation of a host system will be described. The specific architecture and operation of the various embodiments will then be described with reference to the general architecture.

Computing systems and devices use various protocols/standards for network communication. Computer networks typically use a layered protocol structure to manage network traffic. One common model that is typically used is an ISO model that includes a physical layer, a data link layer that includes a MAC layer, a network layer and other layers also. Upper level protocol layers (ULPs) (for example, iSCSI and RDMA) described below, interface with a network layer to send and receive data from the network.

FIG. 1Ashows an example of various protocol layers (10) that may be used in networking systems. Data link layer (which includes a MAC layer)20interfaces with an IP (Internet Protocol) layer30. TCP (Transmission Control Protocol) layer40typically sits on top of the IP layer30. Upper Layer Protocols (ULPs)50may include plural layers, for example, iSCSI layer70, RDMA layer60and others. The following provides a brief introduction to some of these standards/protocols:

Transmission Control Protocol/Internet Protocol (“TCP/IP”): TCP (40) is a standard network protocol (incorporated herein by reference in its entirety) that provides connection-oriented, reliable, byte stream service. This means that two nodes establish a logical connection before sending data and TCP maintains state information regarding the data transfer. Reliable means that data is delivered in the same order that it was sent. A byte stream service means that TCP views data to be sent, as a continuous data stream that is sent in any way it sees fit and delivers it to a remote node as a byte stream.

IP (30) is a standard protocol (incorporated herein by reference in its entirety) that provides a datagram service whose function is to enable routing of data through various network subnets. Each of these subnets could be a different physical link such as Ethernet, ATM, or others. IP layer30is responsible for fragmentation of transmit data to match a local link's maximum transmission unit (MTU). IP layer30fragments data at a source node or at any intervening router between a source and a destination node.

iSCSI Protocol: Internet SCSI (iSCSI) protocol as defined by the Internet Engineering Task Force (IETF) maps the standard SCSI protocol on top of the TCP/IP protocol. iSCSI (incorporated herein by reference in its entirety) is based on the Small Computer Systems Interface (“SCSI”) standard, which enables host computer systems to perform block level input/output (“I/O”) operations with a variety of peripheral devices including disk and tape devices, optical storage devices, as well as printers and scanners. The iSCSI and TCP/IP protocol suite consist of four protocol layers: the application layer (of which iSCSI is one layer); the transport layer (TCP), the network layer (IP) and the link layer (i.e. Ethernet).

A traditional SCSI connection between a host system and peripheral device is through parallel cabling and is limited by distance and device support constraints. For storage applications, iSCSI was developed to take advantage of network architectures based on Ethernet standards.

The iSCSI architecture is based on a client/server model. Typically, the client is a host system such as a file server that issues a read or write command. The server may be a disk array that responds to the client's request. Typically, the client is an initiator that initiates a read or write command and a disk array is a target that accepts a read or write command and performs the requested operation.

In a typical iSCSI exchange, an initiator sends a “read” or “write” command to a target. For a read operation, the target sends the requested data to the initiator. For a write command, the target sends a “Ready to Transfer Protocol Data Unit (“PDU”)” informing the initiator that the target is ready to accept the write data. The initiator then sends the write data to the target.

Once the data is transferred, the exchange enters the response phase. The target then sends a response PDU to the initiator with the status of the operation. Once the initiator receives this response, the exchange is complete. The use of TCP guarantees the delivery of the PDUs.

Typically, logical units in a target process commands. Commands are sent by a host system in Command Descriptor Blocks (“CDB”). A CDB is sent to a specific logical unit and may include a command to read a specific number of data blocks. The target's logical unit transfers the requested data block to the initiator, terminating with a status message, indicating completion of the request. iSCSI encapsulates CDB transactions between initiators and targets over TCP/IP networks.

As iSCSI becomes popular various software solutions to execute the iSCSI layer in software are emerging. Host system software typically executes the iSCSI layer in software. However, this process is slow and may consume host system processor time and resources, especially for generating digests and checking cyclic redundancy code (CRC).

Operating systems, for example, Microsoft Chimney support offloading of TCP/IP protocol stack from a host system to a network adapter, but do not solve digest and data copy problems for the iSCSI layer. For example, Microsoft defines a data receive algorithm for interoperation between a TCP/IP Offload Engine (TOE) Driver (described below) and the Microsoft Chimney Operating System to deliver a limited amount of data (up to a Maximum Segment Size) when data becomes available. The TOE Driver is blocked from delivering any more data to an Upper layer Protocol (ULP) until the ULP accepts or rejects the delivered data. The ULP may post additional data that may be received after the delivered data.

This process has significant latency between arrival of data and availability of a buffer (memory storage) to store data

A TOE Module (described below) holds off delivery of data to the TOE Driver while the TOE Driver is waiting for the ULP to post a buffer. The ULP only posts a buffer large enough for the delivered data and the TOE Driver notifies the TOE Module to deliver the next data. In this case, the TOE Driver/TOE module interaction adds to the latency of data delivery to the ULP.

In another case when the TOE Module does not hold off delivery of data while the TOE Driver is waiting for a response from the ULP, the TOE Driver holds the received data in an anonymous buffer and then copies the data to the buffers posted by the ULP. This unnecessarily uses host system processor time and resources since data that is buffered by the TOE Driver has to be copied to ULP buffers consuming CPU cycles. The present adaptive aspects, described below tackle the latency issue.

RDMA: Remote Direct Memory Access (RDMA) is a standard upper layer protocol (incorporated herein by reference in its entirety) that assists one computer to directly place information in another computer's memory with minimal demands on memory bus bandwidth and CPU processing overhead. RDMA over TCP/IP defines interoperable protocols to support RDMA operations over standard TCP/IP networks.

FIG. 1Bshows an example of networking system100, used according to one embodiment. System100includes host system102, which typically includes several functional components. These components may include central processing unit (CPU)104, host memory (or main/system memory)106, system bus108, an input/output (“I/O”) device (not shown), read only memory (not shown), and other components.

Host memory106is coupled to CPU104via system bus108. Host memory106provides CPU104access to data and program information that is stored in host memory106at execution time. Typically, host memory106is composed of random access memory (RAM) circuits.

Host System102includes adapter interface110, which couples host system102to network adapter114via bus/connection112and host interface116. The structure of host interface116depends on bus/connection112. For example, if bus112is a PCI bus, then host interface116includes logic and structure to support PCI bus based communication.

Adapter114connects host system102to network122via network interface118and network connection120. The structure of network interface118depends on the type of network, for example, Ethernet, Fibre Channel and others.

FIG. 1Cshows a block diagram for adapter114interfacing with host system102via link (for example, a PCI bus)112and host interface116. Adapter114may be on a PCI development board with a Field Programmable gate Array (FPGA). Adapter114may also be integrated into an Application Specific Integrated Circuit (ASIC).

Adapter114includes a TCP/IP accelerator module (also referred to as TCP Offload Engine (TOE)132) that executes the TCP/IP protocol stack in the hardware, instead of a software stack at host system102. Details of a TOE Module are provided in co-pending patent application Ser. No. 10/620,040, filed on Jul. 15, 2003, incorporated herein by reference in its entirety.

Adapter114includes processor124that has access to adapter memory128. Processor124controls overall adapter114functionality by executing firmware instructions from memory128.

Adapter114also includes a direct memory access (“DMA”) engine126, which performs direct memory access functions in sending data to and receiving data from host system102.

Adapter114also includes iSCSI module130which includes a dedicated processor or state machine to accomplish this purpose. Instead of the software layer in host system102, iSCSI module130performs various iSCSI layer operations in adapter114, including processing digests, performing data copy offload and large PDU transmission offload operations.

RDMA offload module134executes the RDMA protocol functionality in adapter114.

FIG. 1Dshows a block diagram of a top-level software architecture for implementing the various embodiments disclosed herein. CPU104executes operating system (“OS”)136in host system102. In one example, OS136may be based on Microsoft Chimney. Application layer (may also be referred to as ULP)138is executed in host system102. Application138can send read and write requests via driver142(may also be referred to as TOE Driver142). OS interface140interfaces between TOE Driver142and application138. Adapter firmware144is executed by processor124out of memory128to control overall adapter114functionality.

FIG. 1Eshows a top-level block diagram of buffer pools that are created by TOE Driver142in host memory106to facilitate efficient TOE operation. TOE Driver142creates Anonymous Buffer Pool103with anonymous buffers (or kernel buffers)1-N107to temporally store incoming data from the network until application138is ready to accept data and move it to a named memory buffer109from amongst named memory buffer pool105. Named buffers109are allocated to store information for a particular connection/application (for example,138). Push Timers are used to make buffers available for a predetermined time only. Push Timers apply to Named buffers and are used to return Named Buffers109to an issuing application. Once a push timer has expired the associated buffer is released to anonymous buffer pool103. The usage of buffer pools103and105are described below.

Before describing the details of buffer pools103and105operations the following defines certain terms that are used to explain the functionality of various embodiments described herein: “indicateWindowSize” is a parameter used by TOE Driver142and adapter114specify a window size in bytes, representing the amount of data, which can be delivered by adapter114to TOE Driver142, before adapter114suspends data transfer to wait for an application to pass a named buffer to adapter114or to acknowledge that data has been stored in anonymous buffer107.

The value for “indicateWindowSize” may be set globally (i.e. for every interaction between TOE Driver142and adapter114) or on a per network connection basis. TOE Driver142may import the “indicateWindowSize” value from a User Interface, a driver parameter file, flash read only memory, NVRAM, as a default value or from TOE Module132. Generally, TOE Driver142passes the indicateWindowSize value to TOE Module132during initialization of adapter114. TOE Driver142may also specify a unique value for indicateWindowSize for a connection when the connection is offloaded to TOE Module132. The term “indicatewindowSize is open” means that data can be accepted from TOE module132.

An “indicatedBytes” parameter value indicates the number of bytes that have been received by TOE Driver142from adapter114(via TOE module132) at any given time. As shown inFIG. 1F, the “indicatedBytes” parameter is based on the value of counter148that monitors the number of bytes received by TOE Driver142from adapter114and placed in host memory106.

TOE Driver142resets “indicatedBytes” value to zero when “indicatedBytes” equals “indicateWindowSize” and received data has been copied to a named buffer or accepted by application138. When the “indicatedBytes” value (i.e. counter148) is reset to zero and there are no named buffers pending, TOE Driver142issues an “indicateAcknowledge” command to TOE Module132to open a new “indicateWindow” that allows TOE Module132to resume sending data to TOE Driver142. TOE Driver142may optionally send an “indicateAcknowledge” command while data is being accepted to keep the indicate window open.

An “indicatedBytesPending” parameter is based on the value of counter150(FIG. 1F). Counter150is incremented by TOE Module132when a segment is sent to TOE Driver142. The value of Counter150is incremented by the amount of data in the segment sent to TOE Driver142. TOE Module132suspends sending data to TOE Driver142when counter150value is equal to “indicateWindowSize”.

When TOE Module132receives an “indicateAcknowledge” command from TOE Driver142, TOE Module132decrements counter150by a value specified in an “AcknowledgedBytes” field of the “indicateAcknowledge” command. When TOE Module132receives a Named Buffer from TOE Driver142, TOE Module132resets counter150to the value 0.

Once TOE Module132has sent a number of bytes equal to or similar to a value specified by the “indicateWindowSize” to TOE Driver142, TOE Module132stops sending data received from the network. TOE Module132holds pending data until TOE Driver142posts a buffer for application138for pending data or future received data, or acknowledges that some or all of data has been received. At that time, TOE Module132resumes sending pending data or future data to TOE Driver142.

TOE Driver142receives network data sent by TOE Module132(in adapter14). When TOE Driver142receives a buffer from application138, TOE Driver142passes the buffer to TOE Module132. In one example, the size of the buffer may be greater than the “indicateWindowSize”. In this example, TOE Module132DMAs data outside of “indicateWindow” to the buffer passed by TOE Driver142. TOE Driver142will first copy data in “indicateWindow” to the buffer. The buffers passed to TOE Module132are returned when the buffer is filled or a timeout expires.

If the buffer received from application138is smaller than the amount of data in “indicateWindow”, then TOE Driver142copies the amount of data based on the “indicateWindow” size and returns the buffer to application138. TOE Driver142then indicates to application138that more data is available.

If TOE Driver142does not receive a buffer from application138and application138signals that it has accepted the sent data, TOE Driver142then sends the next data based on the “indicateWindow” size to application138.

As TOE Driver142moves data based on the “indicateWindow” size, it updates counter148(FIG. 1F). TOE Driver142may also inform TOE Module132that data has moved by issuing an “indicateAcknowledgement” command to TOE Module132.

FIG. 2shows a process flow diagram for initializing system100for network operations. The process starts in step S200, when adapter114is initialized.

In step S202, TOE Driver142loads default operating parameters including a default value for “indicateWindowSize”.

In step S204, TOE Driver142allocates host memory106for anonymous buffer pool103based on the default value for “indicateWindowSize”.

In steps S206, TOE Driver142assigns certain anonymous buffers107to TOE Module132.

In step S208, initialization of adapter114is complete and system100is ready for network communication, i.e. to receive and send network data.

FIG. 3illustrates a process for establishing and managing a network connection, according to one embodiment. The process starts in step S300when host system102using adapter114establishes a network connection.

In step S302, host system102notifies Adapter114via TOE Driver142to process the connection (i.e. the connection is offloaded).

In step S304, TOE Driver142creates the appropriate structures for managing the offloaded connection.

In step S306, TOE Driver142passes the “indicatewindowsize” to adapter114in general and to TOE module132in particular.

In step S308, TOE Driver142also indicates to Adapter114that no data has been received i.e. the “indicatedBytes” value is zero (based on counter148value).

In step S310, TOE Module132processes the network connection and notifies TOE Driver142upon its completion.

In step S312, TOE Driver142notifies host system102network stack that the connection has been completed and in step S314, the process ends.

FIGS. 4A-4Hshow process flow diagrams for moving network data to a host system, according to one embodiment.

The process starts in step S400, when TOE Driver142receives the “IndicateData” parameter from adapter114for a connection.

In step S402, TOE Driver142queues (or assigns) Anonymous Buffers107for the connection. In one embodiment, every connection has a context and data is queued for every connection.

In step S404, TOE Driver142updates counter148by an amount of data that it has received from adapter114at any given time. It is assumed that adapter114has previously received data from the network prior to passing it to TOE Driver142in step S400.

In step S406, TOE Driver142determines if application138has completed a given task at any given instance, application138has been notified of data availability, or if a named buffer from buffer pool105has been returned (i.e. cleared). If yes, then the process exits in step S408. Otherwise in step S410, TOE Driver142determines if a Named Buffer109is available from Application138for the connection. If a Named Buffer109is available, then in step S412TOE Driver142copies data from Anonymous Buffer107to Named Buffer109, otherwise the process moves to step S428, described below.

In step S414, TOE Driver142releases Anonymous Buffer107and in step S416, the released Anonymous Buffer107is returned to buffer pool103that is available for TOE Module132.

In step S418, TOE Driver142determines if a Named Buffer109for application138is full. If yes, then in step S420, the Named Buffer109is returned to buffer pool105. If Named Buffer109is not full the process moves to step S426, as described below.

In step S422, TOE Driver142checks if a Push Timer is running for the returned Named Buffer109. If yes, then in step S424, TOE Driver142stops the Push Timer, otherwise the process continues to step S426.

In step S426, TOE Driver142determines whether all the received data has been copied to a named Buffer109. If TOE Driver142determines that not all buffered data is copied to Named Buffer109(which means, that there is data but there are no Named Buffers109available), the process moves to step S428. If all the buffers are copied (which means, there are available buffers but there is no more data), the process moves to step S456, described below.

In step S428, TOE Driver142determines that Application138has been notified of data availability or Named Buffer109has been returned. If TOE Driver142determines that Application138has been notified, the process exits in step S440. If not, then in step S430, TOE Driver142notifies Application138of data availability.

In step S432, if application138provides a Return Code, then in step S436, TOE Driver142releases Anonymous Buffers107, otherwise the process continues in step S434.

In step S434, if TOE Driver142determines that Named Buffer109is available, then the process moves to step S412, otherwise the process exits in step S440.

In step S438, TOE Driver142releases Anonymous Buffer107to pool103making it available for TOE Module132.

In step S442, if TOE Driver142determines that there is more queued data, then in step S444, TOE Driver142checks if there is a Named Buffer109available for the queued data. If there is no queued data in step S442, then the process moves to step S446.

In step S444, if there is no Named Buffer109available then the process loops back to step S428. If a Named Buffer is available, then the process continues to step S412.

In step S446, TOE Driver142determines if “indicateWindow” is full, i.e. the amount of data that can be received has been received. This is inferred when “indicatedBytes”=“indicateWindowSize”. If yes, then in step S448, TOE Driver142resets counter148by setting “indicatedBytes”=0, otherwise the process exits in step S454.

In step S450, TOE Driver142determines if a Named Buffer109has been posted (or allocated) for TOE Module132, if yes, then in step S454, the process ends, otherwise in step S452, TOE Driver142issues an “indicateAcknowledge” parameter for TOE Module132. The “indicateAcknowledge” parameter instructs TOE Module132to open a new “indicateWindow” which enables TOE Module132to resume sending data to TOE Driver142.

In step S456, TOE Driver142determines whether “indicatedBytes”=“indicateWindowSize”. If yes, then in step458, TOE Driver142resets counter148and the process ends in step S464, otherwise the process moves to step S460.

In step S460, TOE Driver142determines if a Named Buffer109is partially full. If not, then the process ends in step S464, otherwise, TOE Driver142determines if a Named Buffer109is in Push Mode. If a Named Buffer109is not in a Push Mode, the process ends in step S464.

The term “Push Mode” as used herein applies to Named Buffers. A Named Buffer is determined to be in a Push Mode when an Application sets a PUSH MODE flag in an application programming interface (API) that is used to pass the Named Buffer to the TOE Driver142. When the Push Mode flag is set TOE Driver142and adapter114monitor received TCP segments that have been DMAed (using DMA engine126) or copied to a Named buffer. If the TOE Driver114or adapter114detect that the TCP PUSH flag is set in a processed TCP Segment, TOE Driver142or adapter114return the Named Buffer within a PUSH Timer timeout period regardless of how much data has been placed into the Named Buffer. If the Push Mode flag is not set, then adapter114and TOE Driver142do not monitor the TCP Push flag in received TCP segments.

If the Named Buffer109is in a Push Mode, then in step S466, TOE Driver142determines if a push bit is set in a copied TCP segment. If the bit is set, then in step S470, the Named Buffer109is returned. If the push bit is not set, then in step S468, TOE Driver142determines if the push timer is running. If the Push Timer is not running, then the timer is started in step S476and the process ends in step S478. If the push timer is running then the process ends in step S478.

If the push timer is running in step S472, then the timer is stopped in step S474and the process ends in step S478. If the push timer is not running in step S472, then the process ends in step S478.

FIG. 4Fshows a process flow diagram for handling Named Buffers, according to one embodiment. The process starts in step S480, when Application138posts (or allocates) a Named Buffer109for TOE Driver142. The term allocates or posts as used throughout this specification means that Named Buffer109is assigned to store data for a connection.

In step S482, TOE Driver142queues a Named Buffer109for TOE Module132for a connection.

In step S484, TOE Driver142determines if some Named Buffers have been submitted to TOE Module132. If yes, then the process continues to step S412, otherwise TOE Driver142provides a Named Buffer109for TOE Module132, starting from the head of a Named Buffer queue (or pool103).

In step S488, if TOE Driver142determines that Named Buffer Size is not larger than “indicateWindowSize”, then the process moves to step S412, otherwise TOE Driver142in step S490, notifies TOE Module132of Named Buffer109's size and address where data can be placed.

In step S494, if TOE Driver142determines if the push timer is running. If yes, then in step S496, TOE Driver142stops the push timer. Once the push timer expires or is stopped, in step S498, Named Buffer109is returned and the process ends in step S499.

The foregoing adaptive aspects reduce latency because the “indicateWindow” parameter allows data to be buffered while application138is determining whether to accept data or post a buffer. This reduces latency for data availability to a socket application. This results in better application performance. If the application is transaction based and operates primarily with small amounts of data the foregoing embodiments significantly increase performance by reducing latency that is incurred on every transaction.

In another embodiment, the foregoing aspects also assist applications that operate primarily on large amounts of data by reducing latency and by providing simultaneous transfer of data by the TOE Driver (via data copy) and the TOE Device (via DMA engine126).

Although the present invention has been described with reference to specific embodiments, these embodiments are illustrative only and not limiting. Many other applications and embodiments of the present invention will be apparent in light of this disclosure and the following claims.