Techniques for packet transmit scheduling

Techniques to schedule transmission of a packet from a computing platform include calculating adjustments to portions of the packet to cause corrections to at least one portion of the packet. An adjustment to a scheduled transmission of the packet is made based on the corrections.

TECHNICAL FIELD

Descriptions are generally related to scheduling packets for transmission from a computing platform.

BACKGROUND

A computing platform such as a server coupled to a network may include a network interface card (NIC) having circuitry or logic to schedule transmission of packets of data from the server. The server may be included as part of a communication network or part of a large data center. In some examples, the server may be deployed in a base station or Node B (e.g., base station transceiver (BTS)) or in other network access roles and often may need to “tunnel” or encapsulate traffic flows to different destinations using a variety of packet header formats. Receivers or intermediary transit points of tunneled or encapsulated traffic flows may receive these traffic flows with one or more of the original packet headers removed or replaced by intervening processing functions. Furthermore, some of these receivers or intermediaries may track or “police” the rates at which they receive traffic from a given sender as part of enforcing service level agreements (SLAs), general network and/or equipment performance measurement and management, or other reasons. Different receivers or intermediaries may therefore measure received/transited traffic rates differently, both because the packet headers themselves may be removed or replaced by intervening processing functions and because the receiver or intermediary transit point may include different amount of received header data or other overhead when measuring received rates. Applications sending traffic in this environment may therefore need traffic shaping capabilities that account for different amount of header or other overhead at different shaping hierarchy levels to account for the different ways that transit points and receivers may be measuring traffic rates.

DETAILED DESCRIPTION

In some examples, a server or computing platform may include network interface circuitry such as, but not limit to, a network interface card (NIC). The NIC may include logic and/or features to perform at least some speculative analysis associated with scheduling transmission of packets from the server or computing platform over a network. The logic and/or features of the NIC, in some instances, may have to schedule transmission of packets having multiple packet headers of various sizes associated with respective hierarchy layers of a protocol stack. Packet headers for these packets may vary in size due to protocol differences between hierarchy layers of the protocol stack. For example, the hierarchy layers may be associated with the Open Systems Interconnection (OSI) model. For the OSI model, a first header or content associated with a first physical (PHY) layer may include an interpacket gap (IPG) and a preamble that is a first size, a second header or content associated with a second data link or medium access control (MAC) layer may include an Ethernet header or content that is a second size, a third header or content associated with a third network layer may include internet protocol (IP) content that is a third size, or a fourth header or content associated with a fourth transport layer may include a transmission control protocol (TCP) content that is a fourth size.

According to some examples, a NIC scheduling packets for a server hosting BTS or other types of communication applications may have to schedule packets having headers or content, depending on the layer, that may or may not have a significant impact on an amount of bandwidth needed to transmit these scheduled packets. Different receivers or intermediaries may measure received/transmitted traffic rates differently when determining service level agreement (SLA) conformance, both because the packet headers themselves may be removed or replaced by intervening processing functions and because the receiver or intermediary transmit point may include a different amount of received header data or other overhead when measuring received rates (e.g. perhaps because it operates at a potentially different protocol layer). These SLA requirements may require that a given shaped traffic rate be met when transmitting packets associated with a given hierarchy layer. Thus, a need exists to consider what impact the different header or content sizes may have on packets to be scheduled for transmission and then adjust the content or header sizes accounted for in the rate shaping mechanism prior to transmission to increase a likelihood that SLA requirements are met. Because the SLA requirements may be established per hierarchy layer of traffic shaper, distinct shaping or adjusting of header or content sizes may be needed to enable the NIC per shaping hierarchy level “node”.

FIG. 1illustrates an example system100. In some examples, as shown inFIG. 1, system100includes a computing platform105coupled with a network170via one or more links160. Also, as shown inFIG. 1, computing platform105may include processing element(s)110, a system memory120, an operating system (OS)130, one or more applications (App(s))140or a NIC150.

According to some examples, NIC150includes circuitry154to support a transmit (Tx) scheduler155to facilitate scheduling of a packet to be transmitted from computing platform105via link(s)160. For these examples, packet data for packets to be scheduled for transmission may be pulled from system memory120and at least temporarily stored at a memory152at NIC150. Memory152, in some examples, may include a plurality of transmission queues to at least temporarily store packets scheduled for transmission. As described more below, Tx scheduler155may include logic and/or features to make multiple adjustments to various portions of the packet at different levels of the shaping hierarchy. Different application flows may configure different adjustment amounts for their use of a given shaping hierarchy level (so the adjustment may be distinct per shaping “node”). These adjustments may facilitate efficient scheduling of packet to be transmitted from computing platform105via link(s)160.

In some examples, elements of NIC150, link(s)160, or network170may utilize protocols and/or interfaces according to one or more Ethernet standards promulgated by the Institute of Electrical and Electronics Engineers (IEEE). For example, one such Ethernet standard promulgated by IEEE may include IEEE 802.3-2018, Carrier sense Multiple access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications, Published in August 2018 (hereinafter “the IEEE 802.3 specification”). Although examples are not limited to protocols and/or interface used in accordance with the IEEE 802.3 specification, other or additional standards or specification may be utilized.

According to some examples, computing platform105may be arranged as part of a server, a server array or server farm, a server for a base transceiver station (BTS), a web server, a network server, an Internet server, a work station, a mini-computer, a main frame computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, multiprocessor systems, processor-based systems, or combination thereof.

In some examples, processing element(s)110or circuitry154of NIC150may include various commercially available processors, including without limitation an AMD® Epyc®, Ryzen®, Athlon®, Duron® and Opteron® processors; ARM® application, embedded and secure processors; IBM® and Motorola® DragonBall® and PowerPC® processors; IBM and Sony® Cell processors; Intel® Atom®, Celeron®, Core (2) Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®, Xeon® or Xeon Phi® processors; and similar processors. According to some examples, processing element(s)110or circuitry154may also include an application specific integrated circuit (ASIC) and at least some elements, logic and/or features of processing element(s)110or circuitry154may be implemented as hardware elements of an ASIC. According to some examples, processing element(s)110or circuitry154may also include a field programmable gate array (FPGA) and at least some elements, logic and/or features of processing element(s)110or circuitry154may be implemented as hardware elements of an FPGA.

According to some examples, system memory120may be composed of one or more memory devices or dies which may include various types of volatile and/or non-volatile memory. Also, memory152at NIC150may also include one or more memory devices or dies which may include various types of volatile and/or non-volatile memory. Volatile memory may include, but is not limited to, random-access memory (RAM), Dynamic RAM (D-RAM), double data rate synchronous dynamic RAM (DDR SDRAM), static random-access memory (SRAM), thyristor RAM (T-RAM) or zero-capacitor RAM (Z-RAM). Non-volatile memory may include, but is not limited to, non-volatile types of memory such as three-dimensional (3-D) cross-point memory. The non-volatile types of memory may be byte or block addressable and may include, but are not limited to, memory that uses chalcogenide phase change material (e.g., chalcogenide glass), multi-threshold level NAND flash memory, NOR flash memory, single or multi-level phase change memory (PCM), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, or spin transfer torque MRAM (STT-MRAM), or a combination of any of the above, or other non-volatile memory types.

FIG. 2illustrates an example packet200. According to some examples, packet200may include example portions of a packet to be transmitted from a computing platform such as computing platform105shown inFIG. 1. For these examples, as shown inFIG. 1, portions210,220,230and240may be related to respective hierarchy layers of a protocol stack. The respective hierarchy layers may be associated with the OSI model. For example, portion210may be related to a PHY first layer, portion220may be related to a MAC/data link second layer, portion230may be related to network third layer and portion240may be related to a transport fourth layer. Examples are not limited to the OSI model for a protocol stack, other models may apply that include use of a hierarchy of layers for a protocol stack.

In some examples, as shown inFIG. 2, portions210to240, when combined, may represent a total or full length of a packet to be transmitted from a computing platform. For example, a full length of a packet to be transmitted from computing platform105via link(s)160to a destination located within network170. Packet200, in some examples, may be transmitted in accordance with one or more Ethernet standards such as the IEEE 802.3 specification.

According to some examples, as shown inFIG. 2, portion210includes an IPG and a preamble. For these examples, the IPG and preamble may be related to a first layer or PHY. As mentioned more below, the IPG may establish a minimum time interval between transmission of successive packets. The preamble may include PHY specific information that may be added at time of transmission to facilitate receiving of a transmitted packet200.

In some examples, as shown inFIG. 2, portion220includes a MAC (medium/media access control). For these examples, the MAC may be related to a second layer or MAC/data link layer. The MAC may include MAC destination/source information and other types of second layer related content. According to some examples, portions220,230and240of packet200may include packet data stored to system memory120prior to the scheduling of packet200. These portions of packet200may be referred to as a host memory packet length as it reflects that actual portion of a packet stored to a system memory (e.g., system memory120) prior to being sent to a NIC and then scheduled for transmission from a queue included in a memory at the NIC (e.g., memory152). Also, the combined portions220,230and240may be considered as an L2 length of packet200based on the inclusion of second layer as well as third and fourth layers. As mentioned more below, the L2 length may be applicable to determine whether or not a packet needs to be padded to meet minimum packet size requirements. In some examples, cyclic redundancy control (CRC) may also be considered as part of the L2 length.

According to some examples, as shown inFIG. 2, portion230includes an IP. For these examples, the IP may be related to a third layer or network layer. The IP may include either internet protocol version 4 (IPv4) content or internet protocol version 6 (IPv6) content. In some examples, the IPv4 or IPv6 content included in portion230may include an L3 IP header+an L3 IP payload. Also, when used, an IP security (IPsec) encapsulating security payload (ESP) trailer may be included in portion230.

In some examples, as shown inFIG. 2, portion240includes a TCP. For these examples, the TCP may be related to a fourth layer or transport layer. Examples are not limited to a TCP related to a fourth layer (L4) or transport layer. Other types of L4 or transport layer protocols may be included. The other types of L4 or transport layer protocols may include, but are not limited to, internet control message protocol (ICMP), user datagram protocol (UDP) or stream control transmission protocol (SCTP). According to some examples, portion240may also include a data payload for packet200.

FIG. 3illustrates an example format300. In some examples, format300may be an example format to show the various content or headers associated with or related to hierarchy layers of a protocol stack. For example, PHY content310may be related to a first layer, MAC content320may be related to a second layer, IPv4 content330may be related to a third layer, and TCP header340may be related to a fourth layer. Also, payload350may be related to the fourth layer as well. In other words, PHY content310may be similar to portion210of packet200shown inFIG. 2, MAC content320may be similar to portion220, IP content330may be similar to portion230and TCP header340/payload350may be similar to portion240.

According to some examples, adjustments may be made to content or headers included in example format300to facilitate scheduling of transmission of a packet. As described more below, these adjustments may be based, at least in part, on identified profile priorities that may rank each hierarchy layer in terms of importance. The adjustments may include predetermined or default offsets for one or more of the headers and/or packet structure for the entire packet or portions of the packet.

FIG. 4illustrates an example table400. In some examples, table400may include a list of four configurable protocol priority lists that may rank different priorities for various operator identifiers (IDs). The various operator IDs may be associated with different sources and/or destinations of packets to be transmitted. The different sources and/or destination associated with operator IDs may be for types of applications, types of users, types of customers, etc. According to some examples, a given operator ID may be assigned to be transmitted from one or more assigned Tx queues at a NIC or may be assigned to a given packet type. As mentioned more below, Tx queue context information for a packet to be transmitted may include an indication of an operator ID.

In some examples, operator IDs may separately include four configurable protocol priorities that may be ranked to indicate relative priorities. As shown inFIG. 4for table400, the four configurable protocol priorities may include data, MAC, network and transport priority ranks for respective operator IDs401-1to401-n, where “n” may represent any positive, whole integer greater than 2. For these examples, based on an operator ID's protocol priorities a given offset and packet structure recipe may be used to calculate adjustments to cause corrections to one or more portions of a packet being scheduled for transmission. As described in more detail below, the calculated adjustments may cause one or more corrections to portions of the packet related to hierarchy layers of a protocol stack such as portions210,220,230or240as mentioned above for packet200. For example, offset & packet structure recipe405may be based on operator ID401-1's protocol priorities that place a highest rank for data throughput and a lowest rank for a network layer of the protocol stack. As a result of these priorities, calculated adjustments may be made to cause an increase in data bandwidth (e.g., more data with smaller headers and/or less padding) and less emphasis may be placed on adjustments to cause corrections to a network (e.g., IP) layer of the packet.

According to some examples, if an Operator ID is not available or was not identified, a set of default protocol priorities may be used. For example, as shown inFIG. 4, default protocol priorities for data, MAC, network and transport may have default rankings of 1, 2, 3 and 4, respectively. These default rankings may result in use of offset & packet structure recipe420to calculate adjustments.

FIG. 5illustrates an example IPG table500. In some examples, as shown inFIG. 5, IPG table500may include a list of interpacket gaps (IPGs) for port IDs501-1to501-m, where “m” is any whole, positive integer greater than 6. As mentioned above for packet200and example format300, IPGs may be included in a portion of a packet to be transmitted related to a PHY or first layer of hierarchy layers of a protocol stack. A given IPG for a port ID may be based, at least in part, on a minimum idle period between transmission of packets from a physical port of a NIC coupled to a network. A relatively smaller sized IPG for a given port may indicate higher data transmission rates from the given port. For example, port ID501-3's IPG of 1 byte may indicate a data transmission rate capability of up to 100 gigabits/second (Gbs). As mentioned more below, IPGs for a port ID may be included in corrections to be made to a packet scheduled for transmission from a NIC coupled to a network.

FIG. 6illustrates an example process600. According to some examples, process600depicts an example process of how recipe405and other adjustments (e.g., IPG and L2 padding) may be applied to calculate adjustments to different hierarchy layers of a protocol stack to provide corrections610. As mentioned above forFIG. 4, recipe405may be used for Operator ID401-1. For example, offset and structure for an entire packet scheduled for transmission as well as IPG (e.g., for port501-1) and L2 padding may be calculated to adjust a first layer. Also, offset and structure for a MAC portion of the packet to the end of the packet (including L2 padding) may be calculated to adjust a second layer. Also, offset and structure for an L3 IP header plus L3 IP payload (including an ESP trailer) may be calculated to adjust a third layer. Also, offset and structure for a TCP header plus a TCP data payload may be calculated to adjust a fourth layer. Following these calculated adjustments, corrections610may be provided to logic and/or features of a NIC that schedule transmission of a packet.

FIG. 7illustrates an example scheme700. In some examples, scheme700may illustrate an example of how corrections610may be applied to a transmit scheduling tree720. For these examples, portions of packet200include separate patterns to show how hierarchy layers of a protocol stack relate to transmit scheduling tree720.

According to some examples, nodes of transmit scheduling tree720including a same pattern represent a node or group of nodes targeted for all or portions of packet200. For example, adjustment722of corrections610may be made to impact scheduling of packet200to groups of nodes that may be part of a network hop and adjustments to all of packet200may be of interest to determine bandwidth calculations. Adjustment724may be made to impact scheduling of packet200to groups of nodes interested in second layer or L2 packet traits of packet200. Adjustment726may be made to impact a node interested in third layer or L3 packet traits of packet200. Adjustment728may be made to impact scheduling of packet200to groups of nodes interested in fourth layer traits of packet200.

FIG. 8illustrates an example process800. According to some examples, process800may be implemented by logic and/or features of Tx scheduler155. As mentioned above and shown inFIG. 1, Tx scheduler155may be supported by circuitry154included in NIC150coupled with computing platform105. For these examples, as shown inFIG. 8, Tx scheduler155includes a schedule logic805, an initial adjust logic810, an ID logic815, a protocol & offset ID logic820, a match logic825, an L2 pad logic830, a final adjust logic835and an accumulation logic840. Examples are not limited to the logic shown inFIG. 8, more or less logic and/or features of a Tx scheduler155may be utilized to facilitate scheduling of a packet to be transmitted from computing platform105.

Process800begins at8.1where schedule logic805may initially schedule a packet to be transmitted from computing platform105using a packet size reported by an application causing the packet to be transmitted. In some examples, the application may report the packet size in a doorbell to NIC150/Tx scheduler155or the packet size may be reported based on a pre-configured queue used for transmitting scheduled packets for the application. In some examples, the packet size may be an L2 packet size.

Moving to8.2, an initial adjust logic810may cause an initial adjustment to the scheduled packet based on descriptor data associated with the packet. In some examples the descriptor data may indicate an ESP trailer length and a cyclic redundancy check (CRC) that may cause some initial adjustment to the scheduled packet. For example, the packet may have been initially scheduled to be transmitted via a first port coupled to link(s)160. The initial adjustment may cause the packet to be scheduled for transmission via a second port coupled to link(s)160that has a greater data bandwidth capability to handle the additional ESP trailer length and the CRC. Without this initial adjustment, SLAs associated with minimum data bandwidth requirements for transmitting the packet may have not been met using the first port.

Moving to8.3, ID logic815may use Tx queue information to identify an operator ID for the packet. According to some examples, the operator ID may be assigned to a given Tx queue and selection of the given Tx queue for transmission of the packet may enable ID logic815to identify the operator ID. For these examples, as shown inFIG. 8, the operator ID is identified as operator ID401-1.

Moving to8.4, protocol & offset ID logic820may receive packet metadata that has been retrieved from host memory (e.g., system memory120) and was parsed from packet data by a Tx parser (not shown). In some examples, the packet metadata may be used by protocol & offset ID logic820to identify what protocols and offsets for these respective protocols are included in the packet scheduled for transmission. For these examples, the identified protocols may be for hierarchy layers of a protocol stack. For example, PHY protocols, MAC protocols, IP protocols or TCP protocols. Protocol & offset ID logic820may also determine an L2 packet size.

Moving to8.5, match logic825may use table400that includes a list of four configurable protocol priority lists to see if a match for operator ID401-1is found in table400. In some examples, operator ID401-1is located in table400and table400indicates an offset and packet structure recipe405for operator ID401-1. Offset and packet structure recipe405is then used to calculate adjustments to portions of the packet based, at least in part, on protocol priorities.

Moving to8.6, L2 pad logic830may calculate any needed L2 padding for the packet. For example, if the packet is an Ethernet IPv4 packet smaller than 64 bytes, then the L2 padding would need to adjust the packet to have a size of at least 64 bytes. If the packet is an Ethernet IPv6 packet smaller than 84 bytes, the L2 padding would need to adjust the packet to a size of at least 84 bytes. Examples are not limited to IPv4 or IPv6 L2 padding. Other types of padding to cause the packet to reach at least a minimum packet size requirement are contemplated.

Moving to8.7, final adjust logic835may use recipe405and L2 padding (if needed) to calculate final adjustments to the packet scheduled for transmission. For some examples, an IPG based on port501-2may be provided from IPG table500. This IPG for port501-2may also be used to calculate final adjustments. For these examples, process600as shown inFIG. 6may be followed to calculate the final adjustments that may cause corrections610to one or more hierarchy layers of a protocol stack for the packet scheduled for transmission from computing platform105.

Moving to8.8, accumulation logic840may accumulate the calculated adjustments for the one or more hierarchy layers of the protocol stack as corrections610.

Moving to8.9, accumulation logic840may forward corrections610to schedule logic805. Schedule logic805may then make the final adjustments to the packet based on corrections610. The final adjustments may cause changes to the initial scheduling of the packet due to possible packet size expansions or reductions caused by corrections610to one or more hierarchy layers of a protocol stack for the packet scheduled for transmission from computing platform105. Process800then comes to an end.

FIG. 9illustrates an example logic flow900. Logic flow900may be representative of some or all of the operations executed by one or more logic, features, or devices described herein, such as Tx scheduler155. More particularly, logic flow900may be implemented by schedule logic805, protocol & offset ID logic820, final adjust logic835or accumulation logic840.

According to some examples, logic flow900at block902may schedule transmission of a packet from a computing platform. For these examples, the computing platform may be computing platform105coupled with NIC150and schedule logic805of Tx scheduler155may schedule transmission of the packet.

In some examples, logic flow900at block904may receive packet metadata for the packet. For these examples, protocol & offset ID logic820may receive the packet metadata that was parsed from packet data.

According to some examples, logic flow900at block906may identify protocols and respective offsets of the protocols based on the packet metadata, the protocols and respective offsets separately included in four portions of the packet, each portion separately related to respective layers of hierarchy layers of a protocol stack. For these examples, protocol & offset ID logic820may identify the protocols and respective offsets of the protocols.

In some examples, logic flow900at block908may calculate adjustments to the four portions to cause corrections to at least one of the four portions. For these examples, final adjust logic835may calculate the adjustments to the four portions.

According to some examples, logic flow900at block910may adjust the scheduled transmission of the packet based on the corrections. For these examples, accumulation logic840may accumulate the corrections made by final adjust logic835and forward the corrections to schedule logic805for schedule logic805to adjust the scheduled transmission.

FIG. 10illustrates an example storage medium1000. In some examples, storage medium1000may be an article of manufacture. Storage medium1000may include any non-transitory computer readable medium or machine readable medium, such as an optical, magnetic or semiconductor storage. Storage medium1000may store various types of computer executable instructions, such as instructions to implement logic flow900. Examples of a computer readable or machine readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of computer executable instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like. The examples are not limited in this context.

FIG. 11illustrates and example system1100. In some examples, as shown inFIG. 11, system1100may include a host1102. Host1102may be any computing platform with compute, interconnect, memory, storage, and network resources (not shown). For example, host1102may include one or more processors, interconnects, one or more memory, one or more storage devices, and one or more network interfaces. Host1102may support one or more virtual machines (VMs)1104to1104-n. VMs1104-1to1104-N may be any VM supported by host1102. Also, VM queues1106-1to1106-nmay be associated with respective VMs1104-1to1104-N and may be included in memory resources maintained by host1102.

According to some examples, for a packet transmission, virtual switch1110may detect that a transmit packet and/or descriptor is formed in a VM queue and a virtual switch1110supported by host1102may request the packet header, payload, and/or descriptor be transferred to a NIC1150using a direct memory access (DMA) engine1152located at NIC1150. For these examples, descriptor queues1158may receive the descriptor for the packet to be transmitted. NIC1150may transmit the packet. For example, a packet may have a header that identifies the source of the packet, a destination of the packet, and routing information of the packet. A variety of packet protocols may be used, including, but not limited to Ethernet, FibreChannel, Infiniband, or Omni-Path. Host1102may transfer a packet to be transmitted from a VM queue from among VM queues1106-1to1106-nto NIC1150for transmission without use of an intermediate queue or buffer.

In some examples, a virtual switch1110supported by host1102may monitor properties of the transmitted packet header to determine if those properties are to be used to cause an update to a mapping table1156or add a mapping in mapping table1156. According to some examples, to program a mapping table, a source IP address of a packet may be transmitted from VM1104-1. For these examples, a mapping is created in mapping table1156between that source IP address and VM queue1106-1is assigned for that mapping. A packet received by NICT1150with a destination IP address equal to the value of the source IP address of VM1104-1is placed in mapped VM queue1106-1. Also, for these examples, the source IP address is used to program the mapping, but it is the destination IP address that is an inspected characteristic or property of packets received on the network card, to determine where to route these packets. Thereafter, a received packet having a property or properties that match the mapping rule is transferred from network interface1150to VM queue1106-1using DMA engine1152. For example, if VM1104-1requests packet transmission from a source IP address of 2.2.2.2, and if no mapping rule for VM1104-1is in mapping table1156, then virtual switch1110may add a mapping of a received packet with a destination IP address of 2.2.2.2 to VM queue1106-1, which is associated with VM1104-1.

Virtual switch1110may be any software and/or hardware device that provides one or more of: visibility into inter-VM communication; support for Link Aggregation Control Protocol (LACP) to control the bundling of several physical ports together to form a single logical channel; support for standard 802.1Q VLAN model with trunking; multicast snooping; IETF Auto-Attach SPBM and rudimentary required LLDP support; BFD and 802.1ag link monitoring; STP (IEEE 802.1D-1998) and RSTP (IEEE 802.1D-2004); fine-grained QoS control; support for HFSC qdisc; per VM interface traffic policing; network interface bonding with source-MAC load balancing, active backup, and L4 hashing; OpenFlow protocol support (including many extensions for virtualization), IPv6 support; support for multiple tunneling protocols (GRE, VXLAN, STT, and Geneve, with IPsec support); support for remote configuration protocol with C and Python bindings; support for kernel and user-space forwarding engine options; multi-table forwarding pipeline with flow-caching engine; and forwarding layer abstraction to ease porting to new software and hardware platforms. A non-limiting example of virtual switch1110is Open vSwitch (OVS), described at https://www.openvswitch.org/.

An orchestrator, cloud operating system, or hypervisor (not shown) may be used to program virtual switch1110. For example, OpenStack, described at https://www.openstack.org/can be used as a cloud operating system. The orchestrator, cloud operating system, or hypervisor can be executed on or supported by host1102or may be executed on or supported by a different physical computing platform.

According to some examples, for a received packet, NIC1150may use packet mapper1154to route received packets and/or associated descriptors to a VM queue supported by host1102. Descriptor queues1158may be used to store descriptors of received packets. Packet mapper1154may use mapping table1156to determine which characteristics of a received packet to use to map to a VM queue. A VM queue can be a region of memory maintained by host1102that is able to be accessed by a VM. Content maintained or stored in the VM queue may be accessed in first-received-first-retrieved manner or according to any order that a VM requests. For example, a source IP address of 2.2.2.2 specified in a header of a received packet can be associated with VM queue1106-1in mapping table1156. Based on mapping in mapping table1156, NIC1150may use DMA engine1152to copy a packet header, packet payload, and/or descriptor directly to VM queue1106-1, instead of copying the packet to an intermediate queue or buffer.

In some examples, as shown inFIG. 11, NIC1150may also include a transceiver1160, processor(s)1166, a transmit queue1168, a receive queue1170, a memory1172, and a bus interface1174. Transceiver1160may be capable of receiving and transmitting packets in conformance with applicable protocols such as Ethernet as described in IEEE 802.3, although other protocols may be used. Transceiver1160may receive and transmit packets from and to a network via a network medium or link. Transceiver1160may include PHY circuitry1162and MAC circuitry1164. PHY circuitry1162may include encoding and decoding circuitry (not shown) to encode and decode data packets. MAC circuitry1164can be configured to assemble data to be transmitted into packets, that include destination and source addresses along with network control information and error detection hash values. Processors1166can be any processor, core, graphics processing unit (GPU), or other programmable hardware device that facilitates programming of NIC1150. For example, processor(s)1166may execute packet mapper1154. Memory1172may be any type of volatile or non-volatile memory device and may at least temporarily store instructions used to program one or more elements of NIC1150. Transmit queue1168may include data or references to data for transmission by NIC1150. Receive queue1170may include data or references to data that was received by NIC1150. Descriptor queues1158may reference data or packets in transmit queue1168or receive queue1170. A bus interface1174may provide an interface with host1102. For example, bus interface1174can be compatible with PCI, PCI Express, PCI-x, Serial ATA, and/or USB compatible interface (although other interconnection standards may be used).

Some examples may be described using the expression “coupled” and “connected” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, descriptions using the terms “connected” and/or “coupled” may indicate that two or more elements are in direct physical or electrical contact with each other. The term “coupled” or “coupled with”, however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

The follow examples pertain to additional examples of technologies disclosed herein.

An example apparatus may include circuitry at a NIC coupled with a computing platform. The circuitry may schedule transmission of a packet from the computing platform. The circuitry may also receive packet metadata for the packet. The circuitry may also identify protocols and respective offsets of the protocols based on the packet metadata. The protocols and respective offsets may be separately included in four portions of the packet. Each portion may be separately related to respective layers of hierarchy layers of a protocol stack. The circuitry may also calculate adjustments to the four portions to cause corrections to at least one of the four portions and adjust the scheduled transmission of the packet based on the corrections.

The apparatus of example 1, the circuitry to schedule transmission of the packet from the computing platform may include the circuitry to initially schedule transmission based on a size of the packet as reported by an application hosted by the computing platform that caused the transmission of the packet from the computing platform.

The apparatus of example 2, the circuitry to adjust the scheduled transmission of the packet based on the corrections may include the circuitry to adjust the scheduled transmission responsive to the size of the packet increasing or decreasing based on the corrections to the at least one of the four portions of the packet.

The apparatus of example 1, the respective layers of the hierarchy layers of the protocol stack may include a PHY, a MAC layer, a network layer and a transport layer. The PHY, the MAC, the network and the transport layers may be related to respective first, second, third and fourth portions of the four portions of the packet.

The apparatus of example 4, the circuitry may also identify an operator ID for a transmit queue at the NIC used to at least temporarily store the packet scheduled for transmission. The circuitry may also determine whether the operator ID matches an operator ID for a protocol priority list that relatively ranks a data priority, a MAC priority, a network priority and a transport priority. The circuitry may also select an offset and packet structure recipe based on the determination and use the offset and packet structure recipe to calculate adjustments to the four portions to cause the corrections to at least one of the four portions.

The apparatus of example 5 may also include a memory. The transmit queue at the NIC may be included in the memory.

The apparatus of example 5, the first portion related to the PHY layer may include the first portion including an IPG and a preamble for the packet. The circuitry to calculate adjustments to the first portion may include the circuitry to determine an entire packet size when transmitted from the computing platform.

The apparatus of example 7, the second portion related to the MAC layer may include the packet formatted as an Ethernet packet. The second portion may include MAC source and MAC destination information. The circuitry to calculate adjustments to the second portion may include the circuitry to determine a packet size for the packet that excludes the IPG and the preamble included in the first portion.

The apparatus of example 8, the third portion related to the network layer may include the third portion including IPv4 or IPv6 content. The circuitry to calculate adjustments to the third portion may include the circuitry to determine an offset and structure for an IP header plus IP payload included in the third portion and determine a length of an IPsec ESP trailer to be transmitted with packet.

The apparatus of example 9 may also include the circuitry to use descriptor data for the packet to determine the length of the IPsec ESP trailer and determine a length of a CRC. The circuitry may also cause an initial adjustment to the scheduled transmission of the packet based on the determined lengths of the IPsec ESP trailer and the CRC. The initial adjustment may occur prior to the circuitry calculating adjustments to the four portions.

The apparatus of example 9, the third portion may include IPv4 content, wherein the circuitry to separately calculate adjustments to the first and second portions includes the circuitry to determine whether padding is needed to meet an IPv4 minimum packet size and then use any needed padding to determine the packet size for an entire packet or to determine the packet size for the packet that excludes the IPG and the preamble.

The apparatus of example 8, the fourth portion related to the transport layer may include the fourth portion including TCP content. the circuitry to calculate adjustments to the fourth portion may include the circuitry to determine an offset and structure for a TCP header plus TCP payload included in the fourth portion.

An example method may include scheduling, at circuitry for a NIC coupled with a computing platform, transmission of a packet from the computing platform. The method may also include receiving packet metadata for the packet. The method may also include identifying protocols and respective offsets of the protocols based on the packet metadata. The protocols and respective offsets may be separately included in four portions of the packet. Each portion may be separately related to respective layers of hierarchy layers of a protocol stack. The method may also include calculating adjustments to the four portions to cause corrections to at least one of the four portions. The method may also include adjusting the scheduling of the packet based on the corrections.

The method of example 13, scheduling transmission of the packet from the computing platform may include initially scheduling transmission based on a size of the packet as reported by an application hosted by the computing platform that caused the transmission of the packet from the computing platform.

The method of example 14, adjusting the scheduling of the packet based on the corrections may include adjusting the scheduling responsive to the size of the packet increasing or decreasing based on the corrections to the at least one of the four portions of the packet.

The method of example 13, the respective layers of the hierarchy layers of the protocol stack may include a PHY, a MAC layer, a network layer and a transport layer. the PHY, the MAC, the network and the transport layers may be related to respective first, second, third and fourth portions of the four portions of the packet.

The method of example 16 may also include identifying an operator ID for a transmit queue at the NIC used to at least temporarily store the packet scheduled for transmission. The method may also include determining whether the operator ID matches an operator ID for a protocol priority list that relatively ranks a data priority, a MAC priority, a network priority and a transport priority. The method may also include selecting an offset and packet structure recipe based on the determination. The method may also include using the offset and packet structure recipe to calculate adjustments to the four portions to cause the corrections to at least one of the four portions.

The method of example 16, the first portion related to the PHY layer may include the first portion including an IPG and a preamble for the packet. Calculating adjustments to the first portion may include determining an entire packet size when transmitted from the computing platform.

The method of example 18, the second portion related to the MAC layer may include the packet formatted as an Ethernet packet. The second portion may include MAC source and MAC destination information. Calculating adjustments to the second portion may include determining a packet size for the packet that excludes the IPG and the preamble included in the first portion.

The method of example 19, the third portion related to the network layer may include the third portion including IPv4 or IPv6 content. Calculating adjustment to the third portion may include determining an offset and structure for an IP header plus IP payload included in the third portion and determining a length of an IPsec ESP trailer to be transmitted with packet.

The method of example 20 may also include using descriptor data for the packet to determine the length of the IPsec ESP trailer and determine a length of a CRC. The method may also include causing an initial adjustment to the scheduling of the packet based on the determined lengths of the IPsec ESP trailer and the CRC. The initial adjustment may occur prior to calculating adjustments to the four portions.

The method of example 20, the third portion may include IPv4 content. Separately calculating adjustments to the first and second portions may include determining whether padding is needed to meet an IPv4 minimum packet size and then using any needed padding when determining the packet size for an entire packet or when determining the packet size for the packet that excludes the IPG and the preamble.

The method of example 19, the fourth portion related to the transport layer may include the fourth portion including TCP content. Calculating adjustments to the fourth portion may include determining an offset and structure for a TCP header plus TCP payload included in the fourth portion.

An example at least one machine readable medium may include a plurality of instructions that in response to being executed by a system may cause the system to carry out a method according to any one of examples 13 to 23.

An example apparatus may include means for performing the methods of any one of examples 13 to 23.

An example system may include a memory that includes transmit queues. The system may also include circuitry. The circuitry may schedule transmission of a packet from a computing platform. The circuitry may also receive packet metadata for the packet. The circuitry may also identify protocols and respective offsets of the protocols based on the packet metadata. The protocols and respective offsets may be separately included in four portions of the packet. Each portion may be separately related to respective layers of hierarchy layers of a protocol stack. The circuitry may also calculate adjustments to the four portions to cause corrections to at least one of the four portions and adjust the scheduled transmission of the packet based on the corrections.

The system of example 26, the circuitry to schedule transmission of the packet from the computing platform may include the circuitry to initially schedule transmission based on a size of the packet as reported by an application hosted by the computing platform that caused the transmission of the packet from the computing platform.

The system of example 27, the circuitry to adjust the scheduled transmission of the packet based on the corrections may include the circuitry to adjust the scheduled transmission responsive to the size of the packet increasing or decreasing based on the corrections to the at least one of the four portions of the packet.

The system of example 27, the respective layers of the hierarchy layers of the protocol stack may include a PHY, a MAC layer, a network layer and a transport layer. The PHY, the MAC, the network and the transport layers may be related to respective first, second, third and fourth portions of the four portions of the packet.

The system of example 29 may also include the circuitry to identify an operator ID for a transmit queue from among the transmit queues. The transmit queue may be used to at least temporarily store the packet scheduled for transmission. The circuitry may also determine whether the operator ID matches an operator ID for a protocol priority list that relatively ranks a data priority, a MAC priority, a network priority and a transport priority. The circuitry may also select an offset and packet structure recipe based on the determination. The circuitry may also use the offset and packet structure recipe to calculate adjustments to the four portions to cause the corrections to at least one of the four portions.

An example at least one machine readable medium may include a plurality of instructions that in response to being executed by a system at a NIC may cause the system to schedule transmission of a packet from a computing platform coupled with the NIC. The instructions may also cause the system to receive packet metadata for the packet. The instructions may also cause the system to identify protocols and respective offsets of the protocols based on the packet metadata. The protocols and respective offsets separately included in four portions of the packet. Each portion may be separately related to respective layers of hierarchy layers of a protocol stack. The instructions may also cause the system to calculate adjustments to the four portions to cause corrections to at least one of the four portions. The instructions may also cause the system to adjust the scheduled transmission of the packet based on the corrections.

The at least one machine readable medium of example 31, the instructions to cause the system to schedule transmission of the packet from the computing platform may include the instructions to cause the system to initially schedule transmission based on a size of the packet as stored in system memory of the computing platform.

The at least one machine readable medium of example 32, the instructions to cause the system to schedule transmission of the packet from the computing platform may include the instructions to cause the system to initially schedule transmission based on a size of the packet as reported by an application hosted by the computing platform that caused the transmission of the packet from the computing platform.

The at least one machine readable medium of example 31, the respective layers of the hierarchy layers of the protocol stack may include a PHY, a MAC layer, a network layer and a transport layer. The PHY, the MAC, the network and the transport layers may be related to respective first, second, third and fourth portions of the four portions of the packet.

The at least one machine readable medium of example 34, the instructions to further cause the system to identify an operator ID for a transmit queue at the NIC used to at least temporarily store the packet scheduled for transmission. The instructions may also cause the system to determine whether the operator ID matches an operator ID for a protocol priority list that relatively ranks a data priority, a MAC priority, a network priority and a transport priority. The instructions may also cause the system to select an offset and packet structure recipe based on the determination. The instructions may also cause the system to use the offset and packet structure recipe to calculate adjustments to the four portions to cause the corrections to at least one of the four portions.

The at least one machine readable medium of example 34, the first portion related to the PHY layer may include the first portion including an IPG and a preamble for the packet. The instructions to cause the system to calculate adjustments to the first portion may include the instructions to cause the system to determine an entire packet size when transmitted from the computing platform.

The at least one machine readable medium of example 36, the second portion related to the MAC layer may include the packet formatted as an Ethernet packet. The second portion may include MAC source and MAC destination information. The instructions to cause the system to calculate adjustments to the second portion may include the instructions to cause the system to determine a packet size for the packet that excludes the IPG and the preamble included in the first portion.

The at least one machine readable medium of example 37, the third portion related to the network layer may include the third portion including IPv4 or IPv6 content. The instructions to cause the system to calculate adjustment to the third portion may include the instructions to cause the system to determine an offset and structure for an IP header plus IP payload included in the third portion and determining a length of an IPsec ESP trailer to be transmitted with packet.

The at least one machine readable medium of example 38, the instructions to further cause the system to use descriptor data for the packet to determine the length of the IPsec ESP trailer and determine a length of a CRC. The instructions may also cause the system to cause an initial adjustment to the scheduled transmission of the packet based on the determined lengths of the IPsec ESP trailer and the CRC. The initial adjustment may occur prior to calculating adjustments to the four portions.

The at least one machine readable medium of example 38, the third portion including IPv4 content. The instructions to cause the system to separately calculate adjustments to the first and second portions may include the instructions to cause the system to determine whether padding is needed to meet an IPv4 minimum packet size and then use any needed padding when determining the packet size for an entire packet or when determining the packet size for the packet that excludes the IPG and the preamble.

The at least one machine readable medium of example 37, the fourth portion related to the transport layer comprises the fourth portion including TCP content. The instructions to cause the system to calculate adjustments to the fourth portion may include the instructions to cause the system to determine an offset and structure for a TCP header plus TCP payload included in the fourth portion.