Packet data placement in a processor cache

Packet data received by a network controller is parsed and at least a portion of a received packet is stored by the network controller in both a host memory of a system and also in a cache memory of the central processing unit of the system. Other embodiments are described and claimed.

BACKGROUND

In a network environment, a network controller or adapter on a host computer, such as an Ethernet controller, Fibre Channel controller, etc., will receive Input/Output (I/O) requests or responses to I/O requests initiated from the host. Often, the host computer operating system includes a device driver to communicate with the network controller hardware to manage I/O requests to transmit over a network. Data packets received at the network controller are often stored in an available allocated packet buffer in the host memory. The host computer may implement a protocol to process the packets received by the network controller that are stored in the packet buffer, and access any I/O commands or data embedded in the packet.

For instance, the computer may implement the Transmission Control Protocol (TCP) and Internet Protocol (IP) to decode and extract the payload data in the TCP/IP packets received at the network adapter. IP specifies the format of packets, also called datagrams, and the addressing scheme. TCP is a higher level protocol which establishes a virtual connection between a destination and a source. Another protocol, Remote Direct Memory Access (RDMA) establishes a higher level connection and permits, among other operations, direct placement of data at a specified memory location at the destination.

In many systems, the central processing unit of the host computer may have a cache in which data may be stored in anticipation that the cached data may satisfy an upcoming processor operation. The central processing unit can frequently read data more quickly from the central processing unit cache as compared to the host memory. As a consequence, if the proper data has been cached in the central processing unit cache, processing of that data can often be facilitated.

There are a number of data caching techniques for selecting the data to be cached. Many of these techniques are based on a “hit” or “miss” strategy. If target data requested by the central processing unit is found in the central processing unit cache, a “hit” occurs which provides positive feedback to continue selecting data for caching using the current criteria. Conversely, if target data requested by the central processing unit is not found in the central processing unit cache, a “miss” occurs which provides negative feedback. Once a certain number of misses occurs, the criteria used to select data for caching may be changed in an attempt to increase the frequency of cache hits.

If processing of data is initiated with an empty cache, a significant number of cache misses may be incurred as the cache is filled. One technique for increasing cache efficiency is to “warm” the cache by placing data in the cache prior to initiating processing of the data. A cache may be warmed by placing prefetch instructions in the network controller driver. For example, a driver for the network controller may provide prefetch instructions to the central processing unit to place headers of selected packets in the central processing unit cache in anticipation of that header information being needed by the processor. However, in many systems, the central processing unit is not obligated to act on such prefetch instructions from the driver. As a consequence, significant data access latency may occur as the cache is filled as processing of the data packets is initiated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1illustrates one example of a computing environment in which aspects of the description provided herein may be employed. A computer2includes one or more central processing units (CPU)4(only one is shown), a volatile host memory6, nonvolatile storage8, an operating system10, and a network controller12. An application program14further executes in host memory6and is capable of directing the transmission and reception of packets over a network18.

The computer2may comprise any suitable computing device, such as a mainframe, server, personal computer, workstation, laptop, handheld computer, telephony device, network appliance, virtualization device, storage controller, network controller, etc. The CPU4may comprise any suitable microprocessor, controller, or logic circuit. The operating system10may comprise any suitable operating system. Programs and data in memory6may be swapped into storage8as part of memory management operations.

The CPU4has a cache15in which data may be stored in anticipation of that data being needed by the CPU4for processing. The cache15may be a part of the integrated circuit chip on which the CPU4is formed. Alternatively, the cache15may be one or more separate integrated circuit chips of the chipset which includes the CPU4. Lines of data from the host memory6may be cached by the CPU cache15in accordance with various caching techniques. The architecture of the CPU4and cache15is such that, in one embodiment, it provides substantially faster access by the CPU4to the cache15as compared to access by the CPU4to the host memory6. For example, a private bus may interconnect the CPU4and the cache15wherein a system bus may interconnect the CPU4to the host memory6. The details of the caching architecture and technique will vary, depending upon the particular application.

The computer2provides a protocol stack which includes lower protocol layers16and upper protocol layers22. The operations of each of the various protocol layers may be implemented in hardware, firmware, drivers, operating systems, applications or other software, in whole or in part, alone or in various combinations thereof. In the illustrated embodiment, certain lower protocol layers are implemented in hardware and firmware of the network controller12and certain upper protocol layers22are implemented by system resources such as CPU4and system software in the memory6of the computer2.

The lower protocol layers16of the network controller12of the illustrated embodiment include a network protocol layer implementing a network protocol such as the IP protocol, for example, to send and receive network packets to and from remote devices over the network18. The network18may comprise a Local Area Network (LAN), the Internet, a Wide Area Network (WAN), Storage Area Network (SAN), etc. The embodiments may be configured to transmit data over a wireless network or connection, such as wireless LAN, Bluetooth, etc.

In certain embodiments, one or more of the lower protocol layers16or upper protocol layers22may implement the Ethernet protocol (IEEE std. 802.3, published Mar. 8, 2002) over unshielded twisted pair cable, TCP/IP (Transmission Control Protocol/Internet Protocol), Remote Direct Memory Access (RDMA), token ring protocol, Fibre Channel (IETF RFC 3643, published December, 2003), Infiniband, or any other suitable networking protocol. Details on the TCP protocol are described in “Internet Engineering Task Force (IETF) Request for Comments (RFC) 793,” published September, 1981, details on the IP protocol are described in “Internet Engineering Task Force (IETF) Request for Comments (RFC) 791, published September, 1981, and details on the RDMA protocol are described in the technology specification “Architectural Specifications for RDMA over TCP/IP” Version 1.0 (October, 2003).

The network controller12may be integrated into circuits on the motherboard carrying the CPU chipset either as part of the CPU chipset or other integrated circuits of the motherboard. The circuits of the motherboard can include various controllers including a system controller, peripheral controller, memory controller, hub controller, I/O bus controller, etc. Alternatively, the network controller12may comprise separate integrated circuits disposed in a separate chassis or on an expansion board which is connected to a system bus in an expansion slot.

A device driver20executes in memory6and includes network controller12specific commands to communicate with the network controller12and interface between the operating system10, applications14and the network controller12. In the illustrated embodiment, the network layer of the lower protocol layers16handles network communication and stores received packets in a packet buffer21prior to being processed by a transport layer of the upper protocol layers22.

The lower protocol layers16of the illustrated embodiment further include a data link layer which includes two sublayers: a Media Access Control (MAC) layer and a Logical Link Control (LLC) layer. The MAC sublayer controls how a computer on the network gains access to the data and permission to transmit it. The LLC layer controls frame synchronization, flow control and error checking. In the illustrated embodiment, the packet buffer21is located in the MAC portion of the network controller. It is appreciated that the buffer21may be located in other portions of the network controller12as well as other portions of the computer2. A physical layer of the lower protocol layers16includes hardware such as a data transceiver. In an embodiment employing an Ethernet protocol, the data transceiver could be an Ethernet transceiver.

A transport layer of the upper protocol layers22interfaces with the device driver20, or operating system10or application14and performs various transport protocol layer operations on the received packets. The operations include sending to the packet sender acknowledgments of the receipt of packets in accordance with the appropriate protocol. In addition, the transport layer can process the content of messages included in the packets received at the network controller12that are wrapped in a transport layer, such as TCP and/or IP, the Internet Small Computer System Interface (iSCSI), Fibre Channel SCSI, parallel SCSI transport, or any other transport layer protocol in the art. The transport layer can unpack the payload from the received packet and transfer the data to the device driver20, operating system10or application14.

In certain implementations, the upper protocol layers22can further include an RDMA protocol layer as well as the transport protocol layer. Thus, an application14transmitting messages over an RDMA connection can transmit the message through the device driver20and the RDMA protocol layer of the upper protocol layers22. The data of the message can be sent to the transport protocol layer of the layers22to be packaged in a TCP/IP packet. The transport protocol layer can further encrypt the packet before transmitting it over the network18through the lower protocol layers16of the network controller12.

The upper protocol layers22and the lower protocol layers16of the protocol stack can each include additional or fewer protocol layers, depending upon the particular application. One or more layers implemented in hardware, firmware, software or any combination thereof in the lower protocol layers16of the network controller12, may be implemented in hardware, firmware, software or any combination thereof in the upper protocol layers22of the computer2, and vice versa.

The memory6further includes file objects24, which also may be referred to as socket objects, which include information on a connection to a remote computer over the network18. The application14uses the information in the file object24to identify the connection. The application14may use the file object24to communicate with a remote system. The file object24may indicate the local port or socket that will be used to communicate with a remote system, a local network (IP) address of the computer2in which the application14executes, how much data has been sent and received by the application14, and the remote port and network address, e.g., IP address, with which the application14communicates. Context information26comprises a data structure including information the device driver20, operating system10or application14maintains to manage requests sent to the network controller12as described below.

FIG. 2illustrates a format of a network packet50received at the network controller12. The network packet50is implemented in a format understood by the network protocol layer of the lower protocol layers16, such as the IP protocol. The network packet150may include an Ethernet frame that would include additional Ethernet components, such as a header and error checking code (not shown).

A transport packet52is included in the network packet50. The transport packet52is capable of being processed by the transport layer of the host stack in accordance with a transport protocol such as the TCP protocol. The packet52may be processed by other layers in accordance with other protocols including Internet Small Computer System Interface (iSCSI) protocol, Fibre Channel SCSI, parallel SCSI transport, etc. The transport packet52includes payload data54as well as other transport layer fields, such as a header and an error checking code. Included in the header of each packet is the packet sequence number. The payload data52includes the underlying content being transmitted, e.g., commands, status and/or data. The driver20, operating system10or an application14may include a layer, such as a SCSI driver or layer, to process the content of the payload data54and access any status, commands and/or data therein. The payload data54may include RDMA message segments or data formatted in accordance with other protocols. The RDMA message segments include RDMA headers as well as RDMA message data. Accordingly, each received packet may include one or more headers in accordance with the various protocols of the lower protocol layers16and the upper protocol layers22.

FIG. 3shows operations of a network controller such as the network controller12which can facilitate CPU cache management. In a first operation (block300), a packet is received from a network18and is stored (block302) in a system host memory such as the memory6. In accordance with one aspect of the description provided herein, the network controller12can also store (block304) at least a portion of the received packet directly in the CPU cache15to warm the cache15for CPU processing. For example, the network controller12can store header portions of received packets in the CPU cache15in anticipation of that information being needed by the CPU4to process the received packets in accordance with the upper protocol layers22and one or more of the operating system10, applications14, etc.

FIG. 4illustrates another example of operations of a network controller such as the network controller12which can facilitate CPU cache management.FIG. 5is a schematic diagram illustrating portions of the network controller12of the illustrated embodiment in greater detail. In one operation, a packet received from the network18is stored (block400) in a network controller buffer such as the input packet buffer21. In the embodiment ofFIG. 5, the operations of the lower protocol layers16are performed by a lower protocol processor500which may include hardware, software, firmware or a combination thereof. In another operation, the lower protocol processor500parses (block402) a received packet to identify selected portions of the received packet. In the illustrated embodiment, the lower protocol processor500identifies a header portion of each received packet and stores identified header portions in a receive ring502for header buffers. Similarly, the lower protocol processor500identifies data payload portions of received packets and stores data payload portions in a receive ring504for data buffers. It is appreciated that the header and data payload portions of each received packet may be stored in other types of buffers as well. It is further appreciated that the network controller12may have fewer or greater numbers of buffers, depending upon the particular application.

In the illustrated embodiment, the lower protocol processor500parses the received packets for header and payload portions. These parsed headers may be TCP/IP or RDMA headers or headers of any other protocol type. Similarly, the parsed payloads may be TCP/IP or RDMA payloads or payloads of any other protocol type. It is appreciated that other packet portions may be parsed from the received packets. It is appreciated that packet portions may be parsed and separated using a variety of techniques, depending upon the particular application.

In another operation, a direct memory access (DMA) controller506of the network adapter12stores (block404) a packet header parsed from a received packet, into a host memory, such as a portion6aof the host memory6. The DMA controller506may include hardware, software, firmware or a combination thereof. It is appreciated that in other embodiments, the host CPU4may transfer packet headers from the network controller12to the host memory6a. This may be achieved directly or using a suitable host memory controller such as the controller508of the computer2. Similarly, the DMA controller506of the network controller12may transfer packet headers from the network controller12to the host memory6adirectly or using a host memory controller508of the computer2.

In another operation, the packet header which was stored in the host memory6is also stored (block406) in the cache15of the CPU4of the computer2. Such an operation can “warm” the CPU cache15to facilitate packet processing. In the illustrated embodiment, the DMA controller506of the network controller12uses the host memory controller508to access the CPU cache15and store the packet header from the receive ring for header buffers502to the CPU cache15. It is appreciated that packet portions may be transferred by a network controller12both to a CPU cache15and a host memory6ausing a variety of techniques, the details of which will vary, depending upon the particular application. For example, in alternative embodiments, the DMA controller506of the network controller12may transfer packet headers from the network controller12to the CPU cache15directly or using a host memory controller508of the computer2.

In the illustrated embodiment, the network controller12stores packet headers in the CPU cache15. The headers stored by the network controller12in the CPU cache15may be TCP/IP or RDMA headers or headers of any other protocol type. It is appreciated that other packet portions may be stored by the network controller12directly into the CPU cache15as well as into other locations such as the host memory6.

In another operation, the direct memory access (DMA) controller506of the network adapter12stores (block408), a packet payload parsed from a received packet, into a host memory, such as a portion6bof the host memory6. It is appreciated that in other embodiments, the host CPU4may transfer packet payloads from the network controller12to the host memory6b. This may be achieved directly or using a suitable host memory controller such as the controller508of the computer2. Similarly, the DMA controller506of the network controller12may transfer packet payloads from the network controller12to the host memory6directly or using a host memory controller508of the computer2.

As received packets are processed by the computer2in accordance with the upper protocol layers22, the system CPU4reads headers of the received packets. If a targeted packet header is first found in the CPU cache15, a cache hit occurs. It is believed that in many applications, the frequency of cache hits and hence the efficiency of cache utilization may be increased by warming the CPU cache15as described above. It is appreciated that other aspects of the description provided herein may be utilized, depending upon the particular application. If a targeted packet header is not found in the CPU cache15, a cache miss occurs. The targeted packet header may then be read from the host memory6a.

In one embodiment, warming of the CPU cache15may be undertaken as packets are received and prior to substantial processing of received packets by the CPU4. In other applications, warming of the CPU cache15may be undertaken as appropriate, depending upon the particular application.

Additional Embodiment Details

In the described embodiments, various protocol layers and operations of those protocol layers were described. The operations of each of the various protocol layers may be implemented in hardware, firmware, drivers, operating systems, applications or other software, in whole or in part, alone or in various combinations thereof.

In certain implementations, the device driver and network controller embodiments may be included in a computer system including a storage controller, such as a SCSI, Integrated Drive Electronics (IDE), Redundant Array of Independent Disk (RAID), etc., controller, that manages access to a nonvolatile storage device, such as a magnetic disk drive, tape media, optical disk, etc. Such computer systems often include a desktop, workstation, server, mainframe, laptop, handheld computer, etc. In alternative implementations, the network controller embodiments may be included in a system that does not include a storage controller, such as certain hubs and switches.

In certain implementations, the network controller may be configured to transmit data across a cable connected to a port on the network adapter. Alternatively, the network controller embodiments may be configured to transmit data over a wireless network or connection, such as wireless LAN, Bluetooth, etc.

In certain implementations, the buffer21used by the network controller12was described as being separate from the host memory6and being physically located in the network controller12. In other embodiments, the buffer21may be a part of he host memory6or a part of other controller circuits on a separate card or on a motherboard.

FIG. 6illustrates one implementation of a computer architecture600of the network components, such as the hosts and storage devices shown inFIG. 1. The architecture600may include a processor602(e.g., a microprocessor), a memory604(e.g., a volatile memory device), and storage606(e.g., a nonvolatile storage, such as magnetic disk drives, optical disk drives, a tape drive, etc.). The storage606may comprise an internal storage device or an attached or network accessible storage. Programs in the storage606are loaded into the memory604and executed by the processor602in a suitable manner. The architecture further includes a network card608to enable communication with a network, such as an Ethernet, a Fibre Channel Arbitrated Loop, etc. Further, the architecture may, in certain embodiments, include a video controller609to render information on a display monitor, where the video controller609may be implemented on a video card or integrated on integrated circuit components mounted on the motherboard. As discussed, certain of the network devices may have multiple network cards. An input device610is used to provide user input to the processor602, and may include a keyboard, mouse, pen-stylus, microphone, touch sensitive display screen, or any other suitable activation or input mechanism. An output device612is capable of rendering information transmitted from the processor602, or other component, such as a display monitor, printer, storage, etc.

The network controller12,608may be implemented on a network card, such as a Peripheral Component Interconnect (PCI) card or some other I/O card, or on integrated circuit components mounted on the motherboard. Details on the PCI architecture are described in “PCI Local Bus, Rev. 2.3”, published by the PCI-SIG.

The foregoing description of various embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit to the precise form disclosed. Many modifications and variations are possible in light of the above teaching.