Patent Description:
<CIT> describes methods and systems for RDMA packet routing packets to virtual network addresses. The disclosure describes a field programmable gate array, FPGA, that includes an inbound processing path having an inbound packet buffer configured to receive an inbound packet from the computer network, a NIC buffer, and a multiplexer between the inbound packet buffer and the NIC, and between the NIC buffer and the NIC. In embodiments, the inbound processing path receives inbound RDMA packets from a computer network via a TOR switch.

It is the object of the present invention to provide a computer-readable memory device, a method, and a computing device for improving packet capture.

Remote direct memory access (RDMA) enables access to a memory resource on a computing device without involving the device's CPU (central processing unit). Data packets traversing a NIC (network interface controller/card) on a server in a network are efficiently captured by adapting an ASIC (application-specific integrated circuit) in a programmable TOR (top of rack) switch to modify headers of incoming data packets to indicate to the NIC that the packets are RDMA packets. Such modification enables the packets to be written directly to the server memory while bypassing the server's CPU which can typically act as a bottleneck when attempting full packet capture.

In some implementations, the TOR switch can be configured to use the SONiC (Software for Open Networking in the Cloud) network operating system (OS) software. SONiC can be extended to perform the packet modifications (in combination with ASIC firmware in some cases) to include the RDMA information. Leveraging RDMA can advantageously enable packet capture at line rates (e.g., <NUM> Gb/s) to facilitate network forensics and critical incident response which can require full packet capture.

It will be appreciated that the above-described subject matter may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as one or more computer-readable storage media. These and various other features will be apparent from a reading of the following Detailed Description and a review of the associated drawings.

Like reference numerals indicate like elements in the drawings. Elements are not drawn to scale unless otherwise indicated.

<FIG> shows an illustrative datacenter <NUM> environment in which multiple physically embodied networking devices, such as routers and switching devices <NUM>, are configured to route incoming network traffic <NUM> to servers <NUM> which may be operating as part of a cloud service infrastructure. For example, the servers may be configured to host virtual machines to remotely provide solutions to customers, such as analytics, artificial intelligence (AI) processing, data storage, etc. The network infrastructure may span from the customers <NUM> - e.g., the customer computing devices (not shown) that access the servers <NUM> - over a network <NUM> to switching devices and servers that are variously located in the datacenter in racks and bays including a regional spine <NUM>, spine <NUM>, row leaf <NUM>, and top of rack (TOR) <NUM>. In some embodiments, the switching devices <NUM> may be configured using a Clos topology. The datacenter devices and layout are illustrative and can vary from that shown according the needs of a particular implementation of packet capture.

As shown in <FIG>, the network traffic <NUM> includes data packets <NUM> that are carried using TCP/IP (transport control protocol/Internet protocol). Handling the packets at the server <NUM> typically requires copy operations, which add latency and consume significant CPU (central processing unit) and memory resources in the datacenter <NUM>. Utilization of the remote direct memory access (RDMA) protocol enables removal of data copy operations and enables reduction in latencies by allowing an application on a customer computing device to read or write data on a remote server's memory with minimal demands on memory bus bandwidth and CPU processing overhead, while preserving memory protection semantics. The RDMA protocol is described in the RFC <NUM> specification published by the Internet Engineering Task Force (IETF) and is built on the direct data placement (DDP) protocol as described in the RFC <NUM> specification.

A network interface controller (NIC) <NUM> in the server <NUM> provides an interface to receive the data packets <NUM> at some nominal line rate (e.g., <NUM> Gb/s, <NUM> Gb/s. For RDMA traffic, the NIC can write packets directly to a memory <NUM> in the server over an RDMA path <NUM> and thus bypass the CPU <NUM>. For other, non-RDMA traffic, the NIC interacts with CPU and/or buffers (not shown) as indicated by line <NUM> to write data to the memory.

<FIG> shows an illustrative top of rack (TOR) switch <NUM> that modifies incoming data packets <NUM> to be handled by NIC <NUM> in the server <NUM> using RDMA. The modification includes changes to the packet header, as discussed in more detail below, to identify the packets to the NIC as RDMA packets. Thus, when the NIC processes the modified packet header <NUM>, it will direct the packet to the memory <NUM> over the RDMA path <NUM>. Utilization of RDMA enables avoidance of the bottleneck that is ordinarily presented by operations of the CPU <NUM> in writing data to memory. Accordingly, as the NIC can process the incoming network traffic at the line rate, a packet capture (PCAP) appliance <NUM> can archive and analyze <NUM> percent of the traffic irrespective of packet processing functions such as filtering and classification. In alternative implementations the PCAP appliance is not utilized. Instead, a PCAP application <NUM> may be configured for operations with a PCAP API <NUM> or other suitable interface with the TOR switch hardware including the CPU and memory to implement various packet capture analyses and functionalities.

The PCAP appliance <NUM> or the application <NUM> can be supported in the datacenter <NUM> (<FIG>) to analyze and archive network traffic <NUM> (<FIG>) in full including both the headers and payload. Packet capture can be utilized, for example, for purposes of network troubleshooting, maintenance, forensics, and security. In some implementations, the PCAP appliance and application can be configured to capture a subset of the network traffic based on a set of user-definable filters, for example, IP address or MAC (media access control) address. However, analyses for network forensics and responses to incidents such as malicious intrusions typically require full packet capture.

As shown in <FIG>, the TOR switch <NUM> may, in some implementations, comprise an ASIC <NUM> that is configured to interoperate with an instance of SONiC (Software for Open Networking in the Cloud) that functions as a network operating system (OS) <NUM>, and a switch abstraction interface (SAI) <NUM>). The SONiC network OS and SAI are optionally utilized as indicated by the dashed rectangles in <FIG>. SONiC is an exemplary open source network OS based on Linux® which utilizes multiple containerized components that can facilitate extensibility, scalability, and customization. The SAI can provide a standard interface which enables disaggregation between the ASIC hardware and other software components in the TOR switch. The SAI may include a set of standardized application programming interfaces (APIs) to facilitate communication between the containers and other network applications using the network OS and switching devices <NUM> (<FIG>) to thereby enable disaggregation. In some implementations, the TOR switch may comprise a hardware platform that is operated using OEM (original equipment manufacturer) software <NUM>. For example, the OEM software may be configured using an OEM SDK (software development kit) <NUM> to implement the RDMA header modifications described herein.

<FIG> shows the TOR switch <NUM> performing data packet modifications in TCP/IP (Transport Control Protocol/Internet Protocol) and Ethernet scenarios. The TOR switch may perform the operations, for example, under control of the SONiC network OS alone, or in combination with firmware that operates on the ASIC <NUM>. For TCP/IP, the TOR switch can insert an RDMA header <NUM> into a data packet that includes a TCP/IP header <NUM> and payload <NUM> (the packet structures shown here are simplified for clarity of exposition). For Ethernet, the TOR switch can encapsulate an RDMA transport packet <NUM> in an Ethernet frame to accompany a header <NUM> and payload <NUM>.

RDMA may be implemented using various network protocols. For example, RDMA can be implemented using the TCP/IP protocol. RDMA over converged Ethernet (RoCE) is a network protocol that enables RDMA over an Ethernet network by defining how it will perform in such an environment. RoCE includes versions <NUM> and <NUM>. The later version <NUM> provides packet encapsulation to include IP and UDP (user datagram protocol) headers so that RDMA can be used in both L2 and L3 networks (for example, to implement Layer <NUM> routing). Internet wide area RDMA protocol (iWARP) leverages the TCP protocol or stream control transmission protocol (SCTP) to transmit data. The iWARP methodology was developed by the IETF to enable applications on a server to read or write directly to applications executing on another server without support from the operating system on either server. InfiniBand provides another standard RDMA protocol for high-speed InfiniBand network connections.

Two exemplary data packet structures, as modified by the TOR switch <NUM> (<FIG>), to leverage RDMA for packet capture are shown in <FIG> shows an illustrative data field of an Ethernet frame <NUM> transporting TCP/IP packets <NUM> with a modified packet header <NUM> that includes RDMA information. <FIG> shows an illustrative Ethernet frame that includes an encapsulated RDMA transport packet <NUM> using RoCEv2 which encapsulates an RDMA transport packet <NUM> within an Ethernet/IPv4/UDP packet/frame <NUM> that includes an L2 header <NUM> and a EtherType <NUM>. As shown, the encapsulated RMDA transport packet includes an IP header <NUM> and a UDP header <NUM>. These data packet structures are exemplary, and other structures may be utilized to meet the needs of a particular RDMA packet capture implementation.

<FIG> shows an overall view of the present leveraging of RDMA for packet capture. Network traffic <NUM> is received at the TOR switch <NUM> having an associated line rate. The TOR switch modifies the incoming data traffic to include the RDMA information. The modification is dynamically performed at the incoming data line rate. The modified data packets are received at the NIC <NUM> which writes the data packets directly to the memory <NUM> to avoid the bottleneck that is otherwise presented by the server CPU when RDMA is not utilized. Various packet capture functionalities, such as troubleshooting and other analyses, can be performed in full by either the PCAP application <NUM> or the PCAP appliance <NUM> on the data that is captured at the line rate.

<FIG> is a flowchart of an illustrative method <NUM> that may be performed by a switch (e.g., TOR switch <NUM> in <FIG>). Unless specifically stated, methods or steps shown in the flowcharts and described in the accompanying text are not constrained to a particular order or sequence. In addition, some of the methods or steps thereof can occur or be performed concurrently and not all the methods or steps have to be performed in a given implementation depending on the requirements of such implementation and some methods or steps may be optionally utilized.

At block <NUM>, a stream of data packets is received at a (TOR) switch, in which the received data packet stream has an associated line rate. At block <NUM> at the TOR switch, the received data packets are identified to a NIC for processing that is disposed in a computing device (e.g., server <NUM> in <FIG>) having a memory using RDMA. At block <NUM>, from the TOR switch, the identified data packets are transmitted to the computing device so that that the NIC writes the data to the memory using RDMA for packet capture at the line rate.

<FIG> is a flowchart of an illustrative method <NUM> that may be performed at a datacenter (e.g., datacenter <NUM> in <FIG>). At block <NUM>, packets of data are received at the switch. At block <NUM>, the received data packets are manipulated to transform them into RDMA data packets. At block <NUM>, the RDMA data packets are sent to a NIC disposed in a server that has at least CPU and at least one non-transitory memory. At block <NUM>, at the NIC, the RDMA packets are received from the switch and the NIC performs write operations of the RDMA packets to the non-transitory memory using RDMA to thereby bypass the CPU during the write operations.

<FIG> is a flowchart of an illustrative method <NUM> that may be performed by a switch (e.g., TOR switch <NUM> in <FIG>). At block <NUM>, a stream of data packets is received at a line rate. At block <NUM>, information is inserted into each of the received data packets to identify the data packet as processable using RDMA. At block <NUM>, the data packets with the RDMA identifying information are transmitted to the NIC for processing, in which the NIC is disposed in a computing device (e.g., server <NUM> in <FIG>) having a memory, so that the NIC writes the data to the memory using RDMA at the line rate.

<FIG> shows an illustrative architecture <NUM> for a device, such as a server, capable of executing the various components described herein for the present leveraging of RDMA for packet capture. The architecture <NUM> illustrated in <FIG> includes one or more processors <NUM> (e.g., central processing unit, dedicated AI chip, graphic processing unit, etc.), a system memory <NUM>, including RAM (random access memory) <NUM> and ROM (read only memory) <NUM>, and a system bus <NUM> that operatively and functionally couples the components in the architecture <NUM>. A basic input/output system containing the basic routines that help to transfer information between elements within the architecture <NUM>, such as during startup, is typically stored in the ROM <NUM>. The architecture <NUM> further includes a mass storage device <NUM> for storing software code or other computer-executed code that is utilized to implement applications, the file system, and the operating system. The mass storage device <NUM> is connected to the processor <NUM> through a mass storage controller (not shown) connected to the bus <NUM>. The mass storage device <NUM> and its associated computer-readable storage media provide non-volatile storage for the architecture <NUM>. Although the description of computer-readable storage media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it may be appreciated by those skilled in the art that computer-readable storage media can be any available storage media that can be accessed by the architecture <NUM>.

By way of example, and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable media includes, but is not limited to, RAM, ROM, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read only memory), Flash memory or other solid state memory technology, CD-ROM, DVDs, HD-DVD (High Definition DVD), Blu-ray, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the architecture <NUM>.

According to various embodiments, the architecture <NUM> may operate in a networked environment using logical connections to remote computers through a network. The architecture <NUM> may connect to the network through a network interface unit <NUM> connected to the bus <NUM>. It may be appreciated that the network interface unit <NUM> also may be utilized to connect to other types of networks and remote computer systems. The architecture <NUM> also may include an input/output controller <NUM> for receiving and processing input from a number of other devices, including a keyboard, mouse, touchpad, touchscreen, control devices such as buttons and switches or electronic stylus (not shown in <FIG>). Similarly, the input/output controller <NUM> may provide output to a display screen, user interface, a printer, or other type of output device (also not shown in <FIG>).

It may be appreciated that the software components described herein may, when loaded into the processor <NUM> and executed, transform the processor <NUM> and the overall architecture <NUM> from a general-purpose computing system into a special-purpose computing system customized to facilitate the functionality presented herein. The processor <NUM> may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the processor <NUM> may operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions may transform the processor <NUM> by specifying how the processor <NUM> transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the processor <NUM>.

Encoding the software modules presented herein also may transform the physical structure of the computer-readable storage media presented herein. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable storage media, whether the computer-readable storage media is characterized as primary or secondary storage, and the like. For example, if the computer-readable storage media is implemented as semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable storage media by transforming the physical state of the semiconductor memory.

As another example, the computer-readable storage media disclosed herein may be implemented using magnetic or optical technology. In such implementations, the software presented herein may transform the physical state of magnetic or optical media, when the software is encoded therein. These transformations may include altering the magnetic characteristics of particular locations within given magnetic media. These transformations also may include altering the physical features or characteristics of particular locations within given optical media to change the optical characteristics of those locations.

In light of the above, it may be appreciated that many types of physical transformations take place in the architecture <NUM> in order to store and execute the software components presented herein. It also may be appreciated that the architecture <NUM> may include other types of computing devices, including wearable devices, handheld computers, embedded computer systems, smartphones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the architecture <NUM> may not include all of the components shown in <FIG>, may include other components that are not explicitly shown in <FIG>, or may utilize an architecture completely different from that shown in <FIG>.

<FIG> is a high-level block diagram of an illustrative datacenter <NUM> that provides cloud computing services or distributed computing services that may be used to implement the present leveraging of RDMA for packet capture. Datacenter <NUM> may incorporate the features disclosed in <FIG>. A plurality of servers <NUM> are managed by datacenter management controller <NUM>. Load balancer <NUM> distributes requests and computing workloads over servers <NUM> to avoid a situation wherein a single server may become overwhelmed. Load balancer <NUM> maximizes available capacity and performance of the resources in datacenter <NUM>. Routers/switches <NUM> support data traffic between servers <NUM> and between datacenter <NUM> and external resources and users (not shown) via an external network <NUM>, which may be, for example, a local area network (LAN) or the Internet.

Servers <NUM> may be standalone computing devices, and/or they may be configured as individual blades in a rack of one or more server devices. Servers <NUM> have an input/output (I/O) connector <NUM> that manages communication with other database entities. One or more host processors <NUM> on each server <NUM> run a host operating system (O/S) <NUM> that supports multiple virtual machines (VM) <NUM>. Each VM <NUM> may run its own O/S so that each VM O/S <NUM> on a server is different, or the same, or a mix of both. The VM O/Ss <NUM> may be, for example, different versions of the same O/S (e.g., different VMs running different current and legacy versions of the Windows® operating system). In addition, or alternatively, the VM O/Ss <NUM> may be provided by different manufacturers (e.g., some VMs running the Windows® operating system, while other VMs are running the Linux® operating system). Each VM <NUM> may also run one or more applications (App) <NUM>. Each server <NUM> also includes storage <NUM> (e.g., hard disk drives (HDD)) and memory <NUM> (e.g., RAM) that can be accessed and used by the host processors <NUM> and VMs <NUM> for storing software code, data, etc. In one embodiment, a VM <NUM> may employ the data plane APIs as disclosed herein.

Datacenter <NUM> provides pooled resources on which customers or tenants can dynamically provision and scale applications as needed without having to add servers or additional networking. This allows tenants to obtain the computing resources they need without having to procure, provision, and manage infrastructure on a per-application, ad-hoc basis. A cloud computing datacenter <NUM> allows tenants to scale up or scale down resources dynamically to meet the current needs of their business. Additionally, a datacenter operator can provide usage-based services to tenants so that they pay for only the resources they use, when they need to use them. For example, a tenant may initially use one VM <NUM> on server <NUM><NUM> to run their applications <NUM>. When demand for an application <NUM> increases, the datacenter <NUM> may activate additional VMs <NUM> on the same server <NUM><NUM> and/or on a new server <NUM>N as needed. These additional VMs <NUM> can be deactivated if demand for the application later drops.

Datacenter <NUM> may offer guaranteed availability, disaster recovery, and back-up services. For example, the datacenter may designate one VM <NUM> on server <NUM><NUM> as the primary location for the tenant's application and may activate a second VM <NUM> on the same or a different server as a standby or back-up in case the first VM or server <NUM><NUM> fails. Database manager <NUM> automatically shifts incoming user requests from the primary VM to the back-up VM without requiring tenant intervention. Although datacenter <NUM> is illustrated as a single location, it will be understood that servers <NUM> may be distributed to multiple locations across the globe to provide additional redundancy and disaster recovery capabilities. Additionally, datacenter <NUM> may be an on-premises, private system that provides services to a single enterprise user or may be a publicly accessible, distributed system that provides services to multiple, unrelated customers and tenants or may be a combination of both.

Domain Name System (DNS) server <NUM> resolves domain and host names into IP addresses for all roles, applications, and services in datacenter <NUM>. DNS log <NUM> maintains a record of which domain names have been resolved by role. It will be understood that DNS is used herein as an example and that other name resolution services and domain name logging services may be used to identify dependencies. For example, in other embodiments, IP or packet sniffing, code instrumentation, or code tracing.

Datacenter health monitoring <NUM> monitors the health of the physical systems, software, and environment in datacenter <NUM>. Health monitoring <NUM> provides feedback to datacenter managers when problems are detected with servers, blades, processors, or applications in datacenter <NUM> or when network bandwidth or communications issues arise.

Access control service <NUM> determines whether users are allowed to access particular connections and services on cloud service <NUM>. Directory and identity management service <NUM> authenticates user credentials for tenants on datacenter <NUM>.

<FIG> is a simplified block diagram of an illustrative computer system <NUM> such as a PC, client machine, or server with which the present leveraging of RDMA for packet capture may be implemented. Computer system <NUM> includes a processor <NUM>, a system memory <NUM>, and a system bus <NUM> that couples various system components including the system memory <NUM> to the processor <NUM>. The system bus <NUM> may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, or a local bus using any of a variety of bus architectures. The system memory <NUM> includes read only memory (ROM) <NUM> and random access memory (RAM) <NUM>. A basic input/output system (BIOS) <NUM>, containing the basic routines that help to transfer information between elements within the computer system <NUM>, such as during startup, is stored in ROM <NUM>. The computer system <NUM> may further include a hard disk drive <NUM> for reading from and writing to an internally disposed hard disk (not shown), a magnetic disk drive <NUM> for reading from or writing to a removable magnetic disk <NUM> (e.g., a floppy disk), and an optical disk drive <NUM> for reading from or writing to a removable optical disk <NUM> such as a CD (compact disc), DVD (digital versatile disc), or other optical media. The hard disk drive <NUM>, magnetic disk drive <NUM>, and optical disk drive <NUM> are connected to the system bus <NUM> by a hard disk drive interface <NUM>, a magnetic disk drive interface <NUM>, and an optical drive interface <NUM>, respectively. The drives and their associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system <NUM>. Although this illustrative example includes a hard disk, a removable magnetic disk <NUM>, and a removable optical disk <NUM>, other types of computer-readable storage media which can store data that is accessible by a computer such as magnetic cassettes, Flash memory cards, digital video disks, data cartridges, random access memories (RAMs), read only memories (ROMs), and the like may also be used in some applications of the present user and device authentication for web applications. In addition, as used herein, the term computer-readable storage media includes one or more instances of a media type (e.g., one or more magnetic disks, one or more CDs, etc.). For purposes of this specification and the claims, the phrase "computer-readable storage media" and variations thereof, are intended to cover non-transitory embodiments, and does not include waves, signals, and/or other transitory and/or intangible communication media.

A number of program modules may be stored on the hard disk, magnetic disk <NUM>, optical disk <NUM>, ROM <NUM>, or RAM <NUM>, including an operating system <NUM>, one or more application programs <NUM>, other program modules <NUM>, and program data <NUM>. A user may enter commands and information into the computer system <NUM> through input devices such as a keyboard <NUM> and pointing device <NUM> such as a mouse. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, trackball, touchpad, touchscreen, touch-sensitive device, voice-command module or device, user motion or user gesture capture device, or the like. These and other input devices are often connected to the processor <NUM> through a serial port interface <NUM> that is coupled to the system bus <NUM>, but may be connected by other interfaces, such as a parallel port, game port, or universal serial bus (USB). A monitor <NUM> or other type of display device is also connected to the system bus <NUM> via an interface, such as a video adapter <NUM>. In addition to the monitor <NUM>, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. The illustrative example shown in <FIG> also includes a host adapter <NUM>, a Small Computer System Interface (SCSI) bus <NUM>, and an external storage device <NUM> connected to the SCSI bus <NUM>.

The computer system <NUM> is operable in a networked environment using logical connections to one or more remote computers, such as a remote computer <NUM>. The remote computer <NUM> may be selected as another personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer system <NUM>, although only a single representative remote memory/storage device <NUM> is shown in <FIG>. The logical connections depicted in <FIG> include a local area network (LAN) <NUM> and a wide area network (WAN) <NUM>. Such networking environments are often deployed, for example, in offices, enterprise-wide computer networks, intranets, and the Internet.

Claim 1:
One or more hardware-based non-transitory computer-readable memory devices storing computer-executable instructions which, upon execution by one or more processors in a top of rack, TOR, switch, cause the TOR switch (<NUM>) to:
receive a stream of data packets (<NUM>) at the TOR switch (<NUM>), in which the data packet stream has an associated line rate, wherein the data packets (<NUM>) are captured at the line rate;
at the TOR switch (<NUM>), identify the received data packets (<NUM>) to a network interface controller, NIC, (<NUM>) that is disposed in a computing device (<NUM>) having a memory (<NUM>) for processing using remote direct memory access, RDMA, wherein identifying the received data packets (<NUM>) to the network interface controller comprises manipulating the received data packets to transform them into RDMA data packets, wherein manipulating the received data packets comprises adding an RDMA header to one or more of the received data packets;
from the TOR switch (<NUM>), transmit the identified data packets (<NUM>) to the computing device (<NUM>) so that the NIC (<NUM>) writes the data to the memory (<NUM>) using RDMA for packet capture at the line rate.