NIC with stateful connection tracking

Network interface apparatus includes packet processing circuitry, comprising hardware logic coupled between a network interface and a host interface for connection to a host processor. The hardware logic accesses a list of active connections established between the local processes running on the host processor and corresponding processes on other computers on the network and maintains context information with respect to each of the active connections. Upon receiving a packet from the network having a header identifying the packet as having been transmitted to a local process in accordance with a predefined transport protocol, the hardware logic checks the list to find a connection to which the packet belongs and upon finding the connection, verifies that the packet conforms to the respective state indicated by the context information for the connection and, if so, updates the context information and passes the packet to the local process.

FIELD OF THE INVENTION

The present invention relates generally to data communications, and specifically to apparatus and methods for filtering data packets received from a network.

BACKGROUND

A network interface controller (NIC) is a device that manages and transfers communications between a host computer (referred to alternatively simply as a “host”) and a network, such as a local area network or switch fabric. The NIC directs packets from the network to their destination in the computer, for example by placing the packets in a buffer of a destination application in the computer memory, and directs outgoing packets to the network.

Some NICs are capable of filtering packets received from the network, and can thus perform access control and firewall functions on behalf of the host computer. For example, U.S. Patent Application Publication 2015/0358290 describes a method for stateful packet classification that uses hardware resources for performing stateful connection flow handshaking. To classify an incoming packet from a network, some embodiments perform stateless look-up operations for the incoming packet in hardware and forward the result of the stateless look-up to the software. The software in turn uses the result of the stateless look-up to perform the stateful connection flow handshaking and to determine the result of the stateful packet classification.

SUMMARY

Embodiments of the present invention that are described hereinbelow provide improved network interface devices and methods for processing packets received by a host computer from a network.

There is therefore provided, in accordance with an embodiment of the invention, network interface apparatus, including a host interface for connection to a host processor, and a network interface, which is configured to receive over a network incoming data packets destined for one or more local processes running on the host processor. Packet processing circuitry includes hardware logic, which is coupled between the network interface and the host interface and is configured to access a list of active connections established between the local processes running on the host processor and corresponding processes on other computers on the network in accordance with a predefined transport protocol and to maintain context information with respect to a respective state of each of the active connections. The hardware logic is configured, upon receiving a packet from the network having a header identifying the packet as having been transmitted to a local process in accordance with the predefined transport protocol, to check the list to find a connection to which the packet belongs and upon finding the connection, to verify that the packet conforms to the respective state indicated by the context information for the connection and, if so, to update the context information and pass the packet to the local process.

In some embodiments, the hardware logic is configured, when the connection is not found on the list, to pass the packet to a privileged software process running on the host processor for handling by the privileged software process. Typically, the privileged software process is configured to establish a new connection in response to the packet, whereupon the new connection is added to the list and the context information maintained by the hardware logic for use by the hardware logic in handling subsequent packets received from the network on the new connection.

Additionally or alternatively, the hardware logic is configured, when the packet does not conform to the respective state of the connection to which the packet belongs, to identify the packet as invalid, wherein invalid packets are not passed to the local process. Typically, the hardware logic is configured to pass the invalid packet to a privileged software process running on the host processor for handling by the privileged software process.

In a disclosed embodiment, the hardware logic is further configured to monitor outgoing data packets transmitted to the network by the local processes over the active connections and to update the context information for the active connections responsively to the transmitted data packets.

In some embodiments, the predefined transport protocol includes a Transmission Control Protocol (TCP). Typically, the context information includes a state machine maintained for each of the active connections. Additionally or alternatively, the context information includes a TCP connection window, wherein the hardware logic is configured to check that a sequence number of the packet falls within the TCP connection window and to update the TCP connection window responsively to the sequence number of the packet.

In one embodiment, when the local process is running on a virtual machine running over a virtual machine monitor on the host processor, the hardware logic is configured, upon verifying that the packet conforms to the respective state, to pass the packet to the local process on the virtual machine while bypassing the virtual machine monitor.

There is also provided, in accordance with an embodiment of the invention, a method for communication, which includes receiving over a network in a network interface controller (NIC) incoming data packets destined for one or more local processes running on a host processor that is coupled to the network by the NIC. A list is maintained of active connections established via the NIC between the local processes running on the host processor and corresponding processes on other computers on the network in accordance with a predefined transport protocol, and context information is maintained with respect to a respective state of each of the active connections. Upon receiving in the NIC a packet from the network having a header identifying the packet as having been transmitted to a local process in accordance with the predefined transport protocol, the NIC checks the list to find a connection to which the packet belongs. Upon finding the connection, the NIC verifies that the packet conforms to the respective state indicated by the context information for the connection and, if so, updates the context information and passes the packet to the local process.

DETAILED DESCRIPTION OF EMBODIMENTS

Filtering of incoming packets received from a network is crucial in maintaining security against attacks. Stateless filtering—in which packets are simply checked against certain valid and invalid patterns of header and payload content—is useful as a first line of defense, but has been found to provide insufficient protection against more sophisticated types of attacks, such as distributed denial of service (DDoS) attacks and probes used by persistent attackers. Attacks of these sorts take advantage of connections made using stateful transport protocols, such as the Transmission Control Protocol (TCP). Stateful filtering, based on monitoring the states of connections and rejecting packets that do not conform to the state parameters expected of legitimate traffic, is therefore an important tool in mitigating possible damage due to such attacks.

Stateful connection monitoring and filtering involve complex decision logic and are therefore generally carried out in software, for example using the Connection Tracking software module of the Linux operating system kernel. In a server handling large volumes of TCP or other connection-oriented traffic, this sort of connection tracking imposes a heavy burden on the central processing unit (CPU), which must check and update context information for every packet that is received or transmitted over many different connections concurrently. This processing load can also have a serious negative impact on message throughput and latency.

Embodiments of the present invention that are described herein address these limitations by offloading at least a portion of these connection tracking and filtering functions to hardware logic in the NIC that is used to connect the host computer to a network. Because of the complexities of connection-oriented transport protocols, such as TCP, it is generally not practical to offload all the control flows of connection tracking to hardware logic. The large majority of connection-oriented traffic, however, is made up of legitimate message flows over existing connections. Therefore, in the disclosed embodiments, connection set-up and termination are still carried out in software, typically by a privileged process in or connected with the kernel of the host operating system. At the same time, hardware logic in the NIC performs stateful filtering and context updates for existing connections, while passing exceptions to the software for handling.

In the embodiments that are described hereinbelow, network interface apparatus, such as a NIC, comprises a host interface for connection to a host processor and a network interface, which receives incoming data packets from a network that are destined for local processes running on the host processor. (The local process to which an incoming packet is destined, as indicated by the address information in the packet header, is also referred to herein as the destination process.) In order to handle the incoming packets that belong to a certain connection-oriented transport protocol (or protocols), such as TCP, hardware logic in the NIC accesses a list of active connections that have been established between the local processes running on the host processor and corresponding processes on other computers in accordance with the protocol. The hardware logic also maintains context information with respect to the respective state of each of the active connections, and updates this context information based on the incoming and outgoing packets handled by the NIC.

Upon receiving a packet from the network having a header identifying the packet as having been transmitted to a local process in accordance with the protocol that is being monitored, the hardware logic checks the list of active connections to find the connection to which the packet belongs. Upon finding the connection, the hardware logic verifies that the packet conforms to the state indicated by the context information for this connection. If it does conform, the hardware logic updates the context information while passing the packet to the local process. Packets can be passed in this manner to and from non-privileged, user-level application software, including applications running on virtual machines running on the host computer, without involvement by the host operating system kernel or other privileged software. The hardware may also monitor outgoing packets on the active connections in order to update the context information as required.

Typically, the hardware logic in the NIC passes the packet to the target local process indicated by the packet header only when the packet conforms as expected to the state of an existing connection on the list. When the connection is not found on the list, or the packet does not conform to the state of the connection to which the packet belongs, the hardware logic applies other processing rules. For example, the hardware logic can pass the packet to a privileged software process running on the host processor for the purpose of possibly establishing a new connection or tearing down an existing connection. When such a new connection is established by software, it is added to the list and to the context information maintained by the hardware logic for use in handling subsequent packets. In other cases, the hardware logic may identify the packet as invalid, and discard the packet or pass it to a privileged software process for special handling.

For the sake of concreteness and clarity or illustration, the embodiments that are described herein relate specifically to filtering of TCP traffic. The techniques described herein are similarly applicable, mutatis mutandis, to other connection-oriented transport protocols, such as the Stream Control Transmission Protocol (SCTP) and the File Transfer Protocol (FTP), as well as to connection tracking with respect to stateless protocols, such as the User Datagram Protocol (UDP). Furthermore, the principles of hardware-based connection tracking that are described herein may also be applied in stateful packet handling for other purposes, such as offload to a NIC of functions associated with network address translation (NAT). All such alternative implementations and applications are considered to be within the scope of the present invention.

FIG. 1is block diagram that schematically illustrates the operation of a server20with a NIC26that functions as a stateful firewall (SFW)32, in accordance with an embodiment of the invention. Server20comprises a host complex22, which is connected by NIC26to a packet data network24, such as an Internet Protocol (IP) network. Processes running on other computers on network24, such as a client process28, interact with processes running on server20, such as an application30, by opening and using TCP connections through the network. The principles and requirements of TCP in this regard are described, for example, in Request for Comments (RFC) 793 of the Internet Engineering Task Force (IETF).

Stateful firewall32comprises hardware logic, as described further hereinbelow, which is integrated as a part of the packet processing logic of NIC26and is configured to track TCP connections that are in use by host complex22, such as the connection between client process28and application30shown inFIG. 1. Once this TCP connection has been set up, stateful firewall32tracks and updates the state of the connection based on the packets that are transmitted and received via the connection over network24. On this basis, NIC26is able to filter incoming TCP packets from client process28, to verify that they conform to the expected state, and if so, to update the context information while passing the packet to application30.

Stateful firewall32in NIC26will similarly filter TCP packets transmitted to server20by malicious entities, such as an attacker34. The attacker may, for example, attempt to bombard server20with requests to open TCP connections, as in a DDoS attack, or may attempt to send malformed or otherwise non-standard TCP packets over a connection to server20in order to exploit a software vulnerability of host complex22. In such cases, stateful firewall32will encounter incoming TCP packets that do not conform to the expected state of an existing connection. NIC26will not pass such packets to application30(or to other non-privileged processes), but will rather queue them for handling by a privileged software process running on host complex22. In the meanwhile, because the functions of stateful firewall32are performed by hardware logic, NIC26will continue passing legitimate incoming TCP packets that have been received on existing connections to application30without significant delay or reduction of bandwidth, even under attack conditions.

The functions performed by firewall32, as described herein, are “stateful” in the sense that their implementation requires NIC26to track and make use of the respective states of the packet flows involved. Thus, depending on the states of the various connections that firewall32monitors, NIC26may find a given set of header parameters in a certain packet to be legitimate at a given time, and thus will pass the packet through to its destination process. NIC26may reject a later packet with the same set of parameters because it does not conform to the current connection state. In addition to the novel stateful, hardware-based filtering functions that are described herein, NIC26may also carry out stateless filtering functions that are known in the art, such as application of predefined access control rules.

FIG. 2is a block diagram showing details of server20, and specifically of NIC26and stateful firewall32, in accordance with an embodiment of the invention. For the sake of brevity, only those aspects of host complex22and NIC26that are necessary to an understanding of the present embodiments are shown in the figure and described hereinbelow. Other features of the construction and operation of server20, as well as the integration of these features with those described herein, will be apparent to those skilled in the art. Typically, NIC26comprises one or more integrated circuit chips, in which the functional components that are shown inFIG. 2, along with other, standard elements, are implemented in hardware logic, which may be hard-wired or programmable. Additionally or alternatively, certain elements of NIC26may be implemented in software or firmware running on an embedded processor.

Host complex22comprises a host processor in the form of a central processing unit (CPU)38, with a memory40, typically comprising random-access memory (RAM). NIC26is connected to CPU38and memory40via a bus42, such as a Peripheral Component Interconnect Express® (PCIe®) bus, as is known in the art. In the pictured example, server20supports a virtual machine environment, in which one or more virtual machines50run on CPU38. The software running on CPU38, including both operating system and application programs, may be downloaded to server20in electronic form, over a network for example. Additionally or alternatively, the software may be stored on tangible, non-transitory computer-readable media, such as optical, magnetic or electronic memory media, which may be embodied in memory40.

CPU38operates a native domain46, with a host operating system44, which can support host user applications and other native processes. These processes include a privileged TCP connection handling module48, which may be a part of the kernel of operating system44. Module48is “privileged” in that it has access to and is able to make changes in the state context of TCP connections established over network24. This context includes parameters that are used in setting up and maintaining TCP connections, such as the current state, packet sequence numbers, and window values. Other privileged software modules running on CPU38include a virtual machine monitor (VMM)54, also referred to as a “hypervisor.”

In addition to these native domain functions, CPU38concurrently runs one or more virtual machines50, as noted above, each with its own guest operating system52and guest user applications, such as application30. VMM54in native domain46interacts with the kernels of guest operating systems52of virtual machines50in a manner that emulates the host processor and allows the virtual machines to share the resources of CPU38. A wide range of virtual machine software of this sort is available commercially, and further description is beyond the scope of the present disclosure.

NIC26comprises a host interface56, for connection to CPU38and memory40via bus42, and a network interface58, comprising one or more ports connected to network24. Network interface58transmits and receives data packets in multiple packet flows from and to processes running on CPU38. The packets are processed by packet processing circuitry60, which is coupled between host interface56and network interface58and comprises a receive (Rx) pipe62, for processing incoming data packets received from network24, and a transmit (Tx) pipe64, for processing outgoing packets for transmission to the network.

In received pipe62, packet steering logic66identifies, for each incoming packet, the flow to which the packet belongs and the local process running on CPU38to which the packet is to be delivered. In order to make this decision, steering logic66extracts a flow identifier from the packet, typically based on one or more packet header fields, such as the IP 5-tuple (source and destination addresses, source and destination ports, and protocol identifier) and/or a transport-layer value. Steering logic looks up the flow in a steering table80, which also indicates whether the packet is to be referred to stateful firewall32before forwarding to the destination process. If so, firewall32filters the packet and then may indicate to steering logic66the destination process to which the packet should be directed. For example, firewall32may indicate to steering logic66whether to forward the packet to application30or to connection handling module48. Steering logic66then directs a scatter engine68in receive pipe62to write the packet to an appropriate data buffer76in memory40, typically by direct memory access (DMA) over bus42. Receive pipe62notifies the destination process that the packet is available for reading, for example by placing a completion report in a completion queue in memory40.

In transmit pipe64, scheduling logic70arbitrates among packet transmission requests submitted by processes running on CPU38. When scheduling logic70has scheduled a packet for transmission, a gather engine72reads the packet data from memory40and passes the packet to packet steering logic73, which passes information to stateful firewall32regarding outgoing TCP packets that have been queued for transmission. This information enables firewall32to update the state context of existing TCP connections and to return corresponding state information to steering logic73. On this basis, steering logic73makes the appropriate steering decision, for example to pass the packet to network interface58for transmission over network24.

Processes running on CPU38, such as user application30and connection handling module48, interact with NIC26by manipulating a transport service instance, known as a “queue pair” (QP)74. To send and receive messages over network24using NIC26, processes running on CPU38submit work items, called work queue elements (WQEs), to the appropriate queues for execution by the NIC. In a model that is known as single-root I/O virtualization (SR-IOV), each virtual machine50interacts with NIC26in this manner as though the NIC was dedicated to that virtual machine, linking the virtual machine to other machines (virtual and/or physical) on network24. NIC26thus appears to each virtual machine50to be a dedicated I/O device, or vNIC, for use by that virtual machine in communicating directly over the network. This configuration minimizes the burden of communication on VMM54and on host operating system44.

When connection handling module48establishes a new TCP connection over network24, it opens a connection-tracking (CT) context78in memory40and notifies packet steering logic66and stateful firewall32of the connection. Portions of context78may also be cached in a dedicated memory (not shown) in NIC26for rapid access by firewall32. Context78contains information to be used and updated by firewall32in connection tracking and packet filtering, as well as by connection handling module48in handling exceptions detected by firewall32. When a connection is terminated by the participating processes, connection handling module48typically deletes the corresponding context from memory40.

The information held in CT context78for each TCP connection can include, for example:Header fields (such as source and destination addresses and ports) of incoming and outgoing data packets on the connection.Connection state (for example, TCP SYN packet sent or received, ACK packet sent or received, FIN packet sent or received, or connection reset in progress).TCP data sequence number (SN) and length of last TCP packet sent or received and corresponding TCP acknowledgment.Acknowledgment (ACK) sequence number. (References to checking of TCP sequence numbers in the description that follows and in the claims include, as appropriate, either or both of the data and acknowledgment sequence numbers.)Current TCP window size information.Connection performance data, such as the number and status of packet retransmissions.Control parameters, for example setting thresholds for protocol faults (such as window tracking deviations) at which firewall32should block packet reception and report an exception, and types of exceptions that the firewall should report to software.
The above context parameters are listed by way of example, and details and variations on the above will be apparent to those skilled in the art after reading the present description.

FIG. 3is a flow chart that schematically illustrates a method for handling packets received from a network in NIC26, in accordance with an embodiment of the invention. The method is initiated by packet steering logic66upon receiving an incoming packet from network24, at a packet input step82. Steering logic66checks the protocol field in the packet header in order to determine whether it is a TCP packet, at a protocol detection step84. If not, processing of the packet by NIC26proceeds in the usual stateless fashion, at a normal handling step86. This step may include packet filtering by firewall32, but these stateless modes of filtering are outside the scope of the present description.

If the protocol field identifies the packet as a TCP packet at step84, packet steering logic66checks the packet header against the list of existing TCP connections in steering table80, at a connection checking step88. For this purpose, packet steering logic66can check the IP 5-tuple against the entries in table80, for example using a hash-based lookup over the 5-tuple fields.

If the packet does not match any of the connections listed in table80, steering logic66passes the packet to a privileged software process running on CPU38, such as connection handling module48, at an exception handling step90. For this purpose, steering logic66can use a QP74(FIG. 2) that is dedicated to the privileged software process. Specifically, steering logic66reads a WQE from the receive queue of the QP and instructs scatter engine68to write the packet to a data buffer76indicated by the WQE.

The action taken at step90by connection handling module48depends upon the type of exception that has been encountered, and NIC26may signal different exception types to indicate to connection handling module48what sort of action is required. One exception type could be, for example, a TCP SYN packet received from network24, such as a TCP SYN packet sent by client process28(FIG. 1) in order to initiate a communication session with application30on server20. In this case, connection handling module48will establish a new connection in response to the packet, and will add the new connection to the list in steering table80. Connection handling module will also open a new CT context78in memory40for use by the hardware logic in handling subsequent packets received from client process28on the new connection.

When packet steering logic66finds at step88that the header of the current packet does match an entry listed in steering table80, it passes the packet to stateful firewall32, at a firewall invocation step92. Stateful firewall32looks up the appropriate CT context78for the packet, either in memory40or in an on-board cache, at a context lookup step96. The stateful firewall then checks whether the packet conforms to the current state of the connection, at a state checking step96. A state machine model that can be used for this purpose is shown inFIG. 4.

When the packet does not conform to the current state of the connection to which the packet belongs, stateful firewall32marks the packet as invalid, and passes the packet to connection handling module48(or to another privileged software process that is designated for the purpose), at exception handling step90. For security reasons, such invalid packets are not passed to the purported destination process of the connection (such as application30), and their processing in software may be subject to measures intended to detect and mitigate attacks on server20. In addition, stateful firewall32may also pass packets used in setting up and tearing down TCP connections (as opposed to packets carrying messages over an existing connection) to connection handling module48at this step.

When stateful firewall32finds at step96that the current state of the packet is valid and is not associated with connection set-up or tear-down, it checks the TCP sequence number against the current TCP window parameters held for this connection in CT context78, at a window checking step98. Specifically, stateful firewall32checks that the sequence number of the packet falls within the TCP connection window, in accordance with standard TCP requirements. If not, stateful firewall32again treats the packet as invalid and passes it on for handling by privileged software at step90.

Only when a TCP packet has passed all of the above tests does stateful firewall32return the packet to steering logic66with clearance to pass the packet to the destination process, such as application30, at a packet delivery step100. Stateful firewall32will also update the relevant state parameters in CT context78, including an update of the TCP connection window based on the sequence number of the packet. In a virtualized environment offering SR-IOV functionality, such as that shown inFIG. 2, steering logic66can instruct scatter engine68to pass the packet directly to application30running on virtual machine50, while bypassing VMM54.

FIG. 4is a state diagram that schematically illustrates a state machine model110used by stateful firewall32in TCP connection tracking, in accordance with an embodiment of the invention. Transitions among the states depend on the type of the latest TCP packet received by the stateful firewall (for example, SYN, ACK, SYNACK, FIN, RST, or none) and the direction of the packet (from the connection originator to the target, or in reply from the target to the originator). Stateful firewall32updates the state of model110at each pass through step100(FIG. 3) depending upon the current state and the packet type and direction.

Model110includes the following TCP connection states:A SYN received state112, in which connection set-up has been initiated in response to an incoming or outgoing TCP SYN packet.A connection established state114, following acknowledgment of the TCP SYN packet. The connection remains in this state as long as proper message exchange is going on in the expected manner over the established connection.A FIN received state116, in which connection termination has been initiated in response to an incoming or outgoing TCP FIN packet.A closure waiting state118, following acknowledgment of the TCP FIN packet.A connection termination state120, when the termination handshake has been completed.
In all other cases—for initial connection set-up and final tear-down, connection reset (RST), and when a packet has been received that does not conform to the expected state transitions listed above—stateful firewall32passes the connection to a software handling state122.