Patent Description:
Conventional network interface cards are typically implemented either as an application-specific integrated circuit (ASIC) die or as a field-programmable gate array (FPGA) die. The ASIC implementation provides performance, power, and area advantages but its customized nature for a specific use is oftentimes not flexible enough to support a wide range of different customer requirements. The FPGA implementation provides more configurability to meet different customer requirements but at the cost of decreased performance, more power consumption, and increased area/cost. <CIT> discloses integrated circuit devices that may include a network processor in a data processing die and an on-package memory in a base die. The data processing die may implement one or more network functionalities that may exchange data with low-latency memory, high capacity in the base die. The data processing die may be programmable fabric, which may be dynamically reconfigured during operation. The document discloses that the data processing die is stacked vertically with respect to the base die and that the base die includes basic network function circuits. However, the document does not disclose that the base die is an application-specific integrated circuit (ASIC) die comprising non-customizable network functions.

The document further does not disclose a flow rate control hardware accelerator circuit configured to drop packets that are taking up more than a predetermined chunk of bits on a network port. Furthermore, dropping or ignoring packets is not disclosed, as is the use of network ports.

It is within this context that the embodiments described herein arise.

The present embodiments relate to a network interface controller (NIC) implemented using a programmable integrated circuit die stacked on top of an application-specific integrated circuit (ASIC) die. This stacked die configuration combines the benefits of both ASICs and programmable logic devices (PLDs) by using the programmable interconnect fabric in the top programmable die to implement customer-specific requirements and forming static fixed functional blocks on the bottom ASIC die (e.g., a highly flexible packet processing pipeline can be implemented using the programmable fabric of the top die, whereas stable, high area, and high power consumption blocks can be implemented as hardened blocks on the bottom die without burdening the general purpose programmable fabric on the top die).

By stacking the programmable die directly on top of the ASIC die, close coupling between the two dies can be achieved, which enables efficient inter-block control and data communication between the two dies. The bottom (base) die may also include an extensive network of memory circuits that serve as lookup tables and intermediate packet buffers. Network interface controllers implemented in this way can provide superior customer configurability/customizability while also meeting power and form factor constraints.

<FIG> is a diagram of an illustrative system <NUM> of interconnected electronic devices. The system of interconnected electronic devices may have multiple electronic devices such as device A, device B, device C, device D, and interconnection resources <NUM>. Interconnection resources <NUM> such as conductive lines and busses, optical interconnect infrastructure, or wired and wireless networks with optional intermediate switching circuitry may be used to send signals from one electronic device to another electronic device or to broadcast information from one electronic device to multiple other electronic devices. For example, a transmitter in device B may transmit data signals to a receiver in device C. Similarly, device C may use a transmitter to transmit data to a receiver in device B.

The electronic devices may be any suitable type of electronic device that communicates with other electronic devices. Examples of such electronic devices include integrated circuits having electronic components and circuits such as analog circuits, digital circuits, mixedsignal circuits, circuits formed within a single package, circuits housed within different packages, circuits that are interconnected on a printed-circuit board (PCB), etc..

To facilitate communications between an integrated circuit and a computing network, a network interface controller (sometimes referred to as a network interface card or "NIC") is provided. Typical smart-NIC processing of packets received from the network involves pre-parsing Ethernet packets, validating the packets, extracting quality of service (QoS) requirements from the packets, and decrypting the packets. The decrypted packets are stored in a buffer and the associated header information is passed to a packet processing engine. The packet processing engine transforms the packet to enable network address and port translation services, connection tracking, policing, and gathering of per-entity statistics.

Smart-NICs implemented using only a programmable integrated circuit such as a field-programmable gate array (FPGA) die provides the desired flexibility to enable different open-virtual switching and virtual router implementations as well as being configurable to optimize for the network service. Network interface cards are also used to facilitate secure communication channels between different compute and storage servers by accelerating or offloading cryptographic functions, remote direct memory access, flowing pacing, and compression services. As the packet processing rate increases, however, these basic packet processing functions tend to consume a lot of FPGA resources, which translate to higher logic element usage and higher power consumption.

In accordance with an embodiment, a network interface controller (NIC) is provided where the basic primitive functions are implemented on a base (bottom) die i.e. an application-specific integrated circuit or "ASIC" die) while the configurable fabric and packet processing components are implement on a top (stacked) die (i.e., a programmable integrated circuit such as a field-programmable gate array or "FPGA"). The programmable fabric of the top die provides high bandwidth interconnect circuitry to efficiently access the primitive functions on the base die.

<FIG> is a diagram of an illustrative system <NUM> with a network interface controller such as NIC <NUM>. As shown in <FIG>, network interface controller <NUM> may be interposed between a host processor such as host processor <NUM> and a computing network such as network <NUM>. In general, system <NUM> may be part of a personal area network (PAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a storage area network (SAN), an enterprise private network (EPN), a datacenter network, a virtual private network (VPN), a system area network, or other suitable types of data storage and processing networks. Host processor <NUM> may for example be a central processing unit (CPU), a microprocessor, a microcontroller, or a graphics processing unit (GPU). Host processor <NUM> (sometimes referred to as a host CPU or simply CPU) may include one or more processing cores for processing instructions of a computer program or software application.

Host processor <NUM> may include a host direct memory access (DMA) component such as host DMA engine <NUM> for direct access of an external host memory <NUM>. Host memory <NUM> is sometimes referred to as a "main memory" device. Main memory refers to physical memory that can be directly accessed by host processor <NUM>. Main memory <NUM> may be implemented using volatile memory components such as dynamic random-access memory (DRAM). Main memory (sometimes also referred to as "primary storage") is distinguished from external mass storage devices such as disk drives, optical drives, and tape drives. CPU <NUM> can only manipulate data that is stored in main memory <NUM>. Thus, every program that is executed or every file that is accessed must be copied from an external mass storage device into main memory <NUM>. The amount of storage in memory main <NUM> may therefore determine how many programs can be executed at any point in time and the amount of data that can be made readily available to the program.

In general, the software running on host CPU <NUM> may be implemented using software code stored on non-transitory computer readable storage media (e.g., tangible computer readable storage media). The software code may sometimes be referred to as software, data, program instructions, instructions, script, or code. The non-transitory computer readable storage media may include non-volatile memory such as non-volatile random-access memory (NVRAM), one or more hard drives (e.g., magnetic drives or solid state drives), one or more removable flash drives or other removable media, or the like. Software stored on the non-transitory computer readable storage media may be executed on the processing circuitry of host processor <NUM>.

Still referring to <FIG>, network interface controller <NUM> may include a host interface such as host interface <NUM> operable to communicate with host processor <NUM>, a memory interface such as memory interface <NUM> operable to communicate with off-chip NIC-attached memory devices <NUM>, a network interface such as network interface(s) <NUM> operable to communicate with network <NUM>. Network interface controller <NUM> includes customizable networking circuits <NUM>, and primitive functional circuits <NUM>.

Host interface <NUM> may be a non-coherent computer bus interface such as the PCIe (Peripheral Component Interconnect Express) interface or a coherent computer bus interface such as UltraPath Interconnect (UPI), QuickPath Interconnect (QPI), Compute Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX), Gen-Z, Open Coherent Accelerator Processor Interface (OpenCAPI), Intel Accelerator Link (IAL), NVidia's NVLink, or other computer bus interfaces.

Memory interface <NUM> may be implemented as double data rate (DDR) interfaces such as DDR type-<NUM> (DDR3), low power DDR3 (LPDDR3), DDR type-<NUM> (DDR4), low power DDR4 (LPDDR4), DDR type-<NUM> (DDR5), graphics DDRx, quad data rate (QDR), Open NAND Flash Interface (ONFI), or other suitable interfaces for communicating with external memory. Memory interface <NUM> may include memory controllers for supporting a wide variety of external memory protocols. Network interface(s) <NUM> may implemented as a wired IEEE <NUM> Ethernet interface, a Fast Ethernet interface, a Gigabit Ethernet interface, a Token Ring interface (just to name a few), or a wireless connection (e.g., a IEEE <NUM> Wi-Fi or Bluetooth® connection).

The customizable networking circuits <NUM> include circuit blocks that might evolve from one product generation to another and/or that might be different depending on a customer's needs. In contrast, the "primitive" function circuits <NUM> represent basic/generic networking blocks that remain stable between product generations and/or remain fixed across different customer applications. As such, the customizable (and evolving) networking circuit blocks <NUM> are implemented using "soft" or reconfigurable resources on a programmable integrated circuit die such as a field-programmable gate array (FPGA), whereas the primitive (and stable) functional circuit blocks <NUM> is implemented as "hardened" or nonreconfigurable functional blocks on an application-specific integrated circuit (ASIC) die.

As demands on integrated circuit technology continue to outstrip even the gains afforded by ever decreasing device dimensions, an increasing number of applications demand a packaged solution with more integration than is possible in one silicon die. In an effort to meet this need, more than one IC die may be placed within an integrated circuit package (i.e., a multichip package). As different types of devices cater to different types of applications, more IC dies may be required in some systems to meet the requirements of high performance applications. Accordingly, to obtain better performance and higher density, a multichip package may include multiple dies arranged laterally along the same plane; the present invention,the multichip package includes multiple dies stacked on top of one another.

<FIG> is a cross-sectional side view of an illustrative network interface controller (NIC) such as NIC <NUM> of <FIG> implemented as a stacked multichip package such as multichip package <NUM>. As shown in <FIG>, multichip package <NUM> includes a top FPGA die <NUM> and associated external memory <NUM> stacked directly on top of a bottom ASIC die <NUM> (sometimes referred to as the "base die"). This configuration in which top die <NUM> is stacked vertically over base die <NUM> is sometimes referred to as a <NUM>-dimensional or "3D" die stack configuration.

The customizable (and evolving) networking circuit blocks <NUM> are implemented on the top die <NUM>, whereas the primitive (and stable/fixed) functional circuit blocks <NUM> are implemented on the bottom die <NUM>. This close coupling between top die <NUM> and bottom die <NUM> facilitates efficient inter-block control and data communication between the two dies. The primitive hardened function blocks on the base die <NUM> may be selectively accessed using flexible high-bandwidth interconnect fabric on the top die <NUM>. This arrangement in which the high power and high logic element consuming functions are allocated to the base die while the flexible interconnect circuitry are delegated to the top FPGA die creates a customizable packet processing pipeline while meeting desirable power and form factor targets.

The arrangement of <FIG> in which the programmable die <NUM> is stacked on top of the ASIC die <NUM> is merely illustrative. If desired, the ASIC die <NUM> may alternatively be stacked on top of the programmable die <NUM>.

<FIG> is a diagram showing illustrative circuit blocks that may be included within the top programmable die <NUM>. As shown in <FIG>, top programmable die <NUM> may include a host interface and associated direct memory access (DMA) engine (labeled collectively as circuits <NUM> and optionally corresponding to at least host interface <NUM> of <FIG>), network interface(s) <NUM> (optionally corresponding to network interface <NUM> of <FIG>), memory interfaces <NUM> (optionally corresponding to memory interface <NUM> of <FIG>), packet buffering and arbitration hardware acceleration circuits <NUM>, flexible packet processing pipeline and high bandwidth interconnects <NUM>, customer application intellectual property (IP) blocks <NUM>, and other suitable configurable network processing and storage circuitry.

Packet buffering and arbitration hardware (HW) acceleration (ACC) circuits <NUM> may include ingress arbiter and class of service (CoS) queue(s), collectively labeled as circuits <NUM>-A. Circuits <NUM> may also include egress arbiter and CoS queue(s), collectively labeled as circuits <NUM>-B. Ingress arbiter and CoS queue circuits <NUM>-A may be primarily response for absorbing short term high packet incoming rate and preventing important data packets from dropping when packets cannot be processed at a lower rate due to resource constraints. Circuits <NUM>-A may send flow rate control messages to the remote sender to ensure that no packets of such class of packets are getting dropped. Egress arbiter and CoS queue circuits <NUM>-B may be primarily responsible for handling receipt of remote flow control messages and stalling transmission of packets in response to receive such flow control messages. Circuits <NUM>-B may also be responsible for prioritizing the scheduling of important packets onto the network relative to traffic with lesser priority.

The flexible (reconfigurable) packet processing pipeline and high bandwidth interconnects <NUM> may serve as a programmable yet efficient conduit between the different functional blocks on the base die. If desired, the packet processing pipeline of the top die may optionally be configured to support packet parsing, packet validation, extracting flow keys for exact matching operations, longest prefix matching (LPM) operations, and range and wildcard match based lookup to extract actions on how to modify/remove/add bytes of a packet, connection tracking, policing and statistics gathering operations, flexible congestion control for remote DMA, and other suitable operations that might be desirable to be implemented using programmable logic resources. Customer application IP block <NUM> may be configured to support whatever application that is desired by the user of the overall NIC component.

The top die <NUM> may be implemented as a programmable integrated circuit such as a programmable integrated circuit <NUM> of the type shown in <FIG>. Programmable integrated circuit <NUM> may be a programmable logic device (PLD). Examples of programmable logic devices include programmable arrays logic (PALs), programmable logic arrays (PLAs), field programmable logic arrays (FPLAs), electrically programmable logic devices (EPLDs), electrically erasable programmable logic devices (EEPLDs), logic cell arrays (LCAs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs), just to name a few. As shown in <FIG>, device <NUM> may include a two-dimensional array of functional blocks, including logic array blocks (LABs) <NUM> and other functional blocks, such as random access memory (RAM) blocks <NUM> and specialized processing blocks such as digital signal processing (DSP) blocks <NUM> that are partly or fully hardwired to perform one or more specific tasks such as mathematical/arithmetic operations.

Functional blocks such as LABs <NUM> may include smaller programmable regions (e.g., logic elements, configurable logic blocks, or adaptive logic modules) that receive input signals and perform custom functions on the input signals to produce output signals. Device <NUM> may further include programmable routing fabric that is used to interconnect LABs <NUM> with RAM blocks <NUM> and DSP blocks <NUM>. The combination of the programmable logic and routing fabric is sometimes referred to as "soft" logic, whereas the DSP blocks are sometimes referred to as "hard" logic. The type of hard logic on device <NUM> is not limited to DSP blocks and may include other types of hard logic. Adders/subtractors, multipliers, dot product computation circuits, and other arithmetic circuits which may or may not be formed as part of a DSP block <NUM> may sometimes be referred to collectively as "arithmetic logic.

Programmable logic device <NUM> may contain programmable memory elements for configuring the soft logic. Memory elements may be loaded with configuration data (also called programming data) using input/output elements (IOEs) <NUM>. Once loaded, the memory elements provide corresponding static control signals that control the operation of one or more LABs <NUM>, programmable routing fabric, and optionally DSPs <NUM> or RAMs <NUM>. In a typical scenario, the outputs of the loaded memory elements are applied to the gates of metal-oxide-semiconductor transistors (e.g., pass transistors) to turn certain transistors on or off and thereby configure the logic in the functional block including the routing paths. Programmable logic circuit elements that may be controlled in this way include parts of multiplexers (e.g., multiplexers used for forming routing paths in interconnect circuits), look-up tables, logic arrays, AND, OR, NAND, and NOR logic gates, pass gates, etc. The logic gates and multiplexers that are part of the soft logic, configurable state machines, or any general logic component not having a single dedicated purpose on device <NUM> may be referred to collectively as "random logic.

The memory elements may use any suitable volatile and/or non-volatile memory structures such as random-access-memory (RAM) cells, fuses, antifuses, programmable read-only-memory memory cells, mask-programmed and laserprogrammed structures, mechanical memory devices (e.g., including localized mechanical resonators), mechanically operated RAM (MORAM), programmable metallization cells (PMCs), conductive-bridging RAM (CBRAM), resistive memory elements, combinations of these structures, etc. Because the memory elements are loaded with configuration data during programming, the memory elements are sometimes referred to as configuration memory, configuration RAM (CRAM), configuration memory elements, or programmable memory elements.

In addition, programmable logic device <NUM> may use input/output elements (IOEs) <NUM> to drive signals off of device <NUM> and to receive signals from other devices. Input/output elements <NUM> may include parallel input/output circuitry, serial data transceiver circuitry, differential receiver and transmitter circuitry, or other circuitry used to connect one integrated circuit to another integrated circuit. As shown, input/output elements <NUM> may be located around the periphery of the chip. If desired, the programmable logic device may have input/output elements <NUM> arranged in different ways.

The routing fabric (sometimes referred to as programmable interconnect circuitry) on PLD <NUM> may be provided in the form of vertical routing channels <NUM> (i.e., interconnects formed along a vertical axis of PLD <NUM>) and horizontal routing channels <NUM> (i.e., interconnects formed along a horizontal axis of PLD <NUM>), each routing channel including at least one track to route at least one wire. If desired, the functional blocks of device <NUM> may be arranged in more levels or layers in which multiple functional blocks are interconnected to form still larger blocks. Other device arrangements may use functional blocks that are not arranged in rows and columns.

<FIG> is a diagram showing illustrative circuit blocks that are included within the base ASIC die <NUM>.

As shown in <FIG>, bottom die <NUM> includes a flow rate control hardware accelerator circuit <NUM> and base memory circuits <NUM> and may further include primitive functional blocks including but not limited to: transportation encryption/decryption hardware accelerator circuit <NUM>, storage compression/decompression (at-rest encryption) hardware accelerator circuit <NUM>, network flow search hardware accelerator circuit <NUM>, distributed denial of service (DDoS) hardware accelerator circuit <NUM>, and other suitable high power or stable/fixed functional blocks that are better suited for an ASIC implementation. If desired, in an alternative example not covered by the claim, the bottom die <NUM> may also be provided with one or more configurable logic sector blocks to offer more flexibility.

Base memory circuits <NUM> may be volatile memory circuit blocks such as static random-access memory (SRAM) blocks, buffers, cache, and/or other memory circuit suitable for storing packet information.

Transportation encryption/decryption (security) hardware accelerator <NUM> may help secure in-transmit networking application traffic between applications/containers across different hosts connected through an Ethernet network with protocols like Internet Protocol Security (IPsec), Datagram Transport Layer Security (DTLS), Advanced Encryption Standard (AES), Secure Hash Algorithms (SHA), and other SHA-like algorithms. As an example, transportation encryption/decryption HW accelerator <NUM> may include an inline cryptographic engine (ICE) that accelerates security protocol processing by implementing IPsec or security functions for DTLS. For instance, the inline crypto engine may implement <NUM>-bit AES-GCM (Galois/Counter Mode), AES-CBC (Cipher Blocker Chaining), or SM4-like encryption algorithms to create a secure infrastructure. The inline crypto engine may also implement message authentication and anti-reply checking to prevent "man-in-the-middle" attacks.

Storage compression/decompression hardware accelerator <NUM> may compress and/or decompress network traffic, which helps reduce the size of stored data and therefore reduces the total cost of ownership for data center operators. Large data files can optionally be secured using "at-reset" encryption. As an example, storage compression/decompression HW accelerator <NUM> may include a remote DMA (RDMA) engine that enables reduction in CPU cycles in disaggregated storage applications. As another example, HW accelerator <NUM> may also include a look-aside cryptographic/compression engine (LCE) that enables compression and other cryptographic functions.

Network flow search hardware accelerator <NUM> may be used to search packet fields. A network flow is typically characterized by N different unique packet fields. Oftentimes, N-type packet fields can be hundreds or even thousands of bits. Network flow search HW accelerator <NUM> may perform such search using complex hashing algorithms. As an example, network flow search HW accelerator <NUM> may include a cryptographic transport parser (CXP) that identifies a security association corresponding to a particular application packet. The CXP may extract different packet fields that are unique for an application and then perform a flow search operation to identify the security key(s) associated with the application. As another example, accelerator <NUM> may include a small exact match (SEM) engine or a large exact match (LEM) engine that is used to search a specific flow to extract flow keys from incoming packets. As yet another example, accelerator <NUM> may include a longest prefix match (LPM) engine that performs a longest prefix match for Internet Protocol version <NUM> (IPv4) and Internet Protocol version <NUM> (IPv6) addresses (as examples) to identify the final destination of a packet.

Flow rate control hardware accelerator <NUM> is used to ensure smooth delivery of service for all tenants on the network by controlling the traffic flow of an "aggressive" tenant. In other words, some networking application flows corresponding to one tenant or a group of tenants over-utilizing the network may be rate-controlled or throttled to ensure service level agreement (SLA) guarantees are met for all tenants hosted on the same host processor. As an example, accelerator <NUM> may include a traffic shaper engine that is used to regular the amount of traffic on a per application basis to guarantee a smoother delivery as well as reducing packet drops in the network. As another example, accelerator <NUM> may include a meters/policing and statistics engine that regulates the application traffic to enable SLA guarantees to other containers and applications while enabling gathering of accounting, billing, and other desirable networking analytics.

Distributed denial of service (DDoS) hardware accelerator <NUM> may be used to block service attacks originating from unauthorized network end points. As an example, DDoS HW accelerator <NUM> may include a range match (RM) engine and/or a wild card match (WCM) engine that block inappropriate application packets from suspicious applications to prevent potential DDoS attacks.

If desired, the base ASIC die <NUM> may also include accelerator circuits for performing connection tracking, key extraction, and other potentially high power or high logic resource consuming operation that might be more suitably offloaded to the base die.

<FIG> is a top plan (layout) view of bottom ASIC die <NUM>. As shown in <FIG>, ASIC die <NUM> has a variety of primitive network function blocks which may be distributed throughout an array of memory circuits <NUM> (see shaded blocks). For example, a transportation encryption/decryption hardware acceleration circuit block <NUM> (e.g., an inline cryptography engine) may be formed in a first location on die <NUM> surrounded by neighboring memory blocks <NUM>, a first storage compression/decompression hardware accelerator circuit block <NUM>-<NUM> (e.g., an RDMA engine) may be formed in a second location on die <NUM> surrounded by neighboring memory blocks <NUM>, a second storage compression/decompression hardware accelerator circuit block <NUM>-<NUM> (e.g., a look-aside cryptography/compression engine) may be formed in a third location on die <NUM> surrounded by neighboring memory blocks <NUM>, a network flow search hardware accelerator circuit block <NUM> (e.g., a crypto transport parser, an SEM/LEM engine, or LPM engine) and DDoS hardware accelerator circuit block <NUM> may both be formed in a fourth location on die <NUM> surrounded by neighboring memory blocks <NUM>, a first flow rate control hardware accelerator circuit block <NUM>-<NUM> (e.g., a traffic shaper engine) may be formed in a fifth location on die <NUM> surrounded by neighboring memory blocks <NUM>, and a second flow rate control hardware accelerator circuit block <NUM>-<NUM> (e.g., a RM/WCM engine) may be formed in a sixth location on die <NUM> surrounded by neighboring memory blocks <NUM>.

The example of <FIG> in which six hardened primitive network function blocks are formed in a sea of base memory blocks <NUM> is merely illustrative If desired, each non-memory function block need not be completely surrounded by memory blocks <NUM> (e.g., a given primitive block might only be surrounded by memory blocks <NUM> on three sides, on two sides, or on only one side), non-memory function blocks can be formed at any desired interval on die <NUM>, any number of accelerator blocks can be grouped at any particular location, etc. If desired, the base die may also be implemented as multiple chiplets or smaller integrated circuit dies stacked below the top FPGA die. Each chiplet may include one or more of the primitive functional blocks and/or base memory circuits.

<FIG> and <FIG> show illustrative steps for operating a network interface controller of the type shown in connection with <FIG> in accordance with an embodiment. At step <NUM>, a packet is received at network interface <NUM> from the network, which can then be fed to ingress arbiter and CoS queue circuits <NUM>-A in the top die. Circuits <NUM>-A may be primarily response for absorbing short term high packet incoming rate and preventing important data packets from dropping when packets cannot be processed at a lower rate due to resource constraints. Circuits <NUM>-A may send flow rate control messages to the remote sender to ensure that no packets of such class of packets are getting dropped.

At step <NUM>, the packet may be pre-classified to identify the desired class of service (e.g., by extracting CoS bits from the packet header or label field). At step <NUM>, the packet may optionally be dropped or a transmit rate control message may be send to a remote transmitter to avoid drops.

At step <NUM>, the NIC may arbitrate across network interface ports and CoS queues to select a next packet to be processed. The selected packet may then be send to a security protocol processing engine, which may include a cryptographic transport parser (CXP) and an inline cryptographic engine (ICE) as an example. In general, the security protocol processing engine may include circuits from transportation encryption/decryption HW accelerator <NUM>, network flow search HW accelerator <NUM>, storage compression/decompression HW accelerator <NUM>, and/or DDoS HW accelerator <NUM>. At step <NUM>, the security protocol processing engine may process the packet to extract security flow specific fields (e.g., to extract one or more security keys from the packet).

At step <NUM>, the inline cryptographic engine (ICE) may use the extracted security key to decrypt and authenticate the packet. The ICE may help accelerate security protocol processing by implementing IPsec or security functions for DTLS. For example, the inline crypto engine may implement <NUM>-bit AES-GCM (Galois/Counter Mode), AES-CBC (Cipher Blocker Chaining), or SM4-like encryption algorithms to create a secure infrastructure. The inline crypto engine may also implement message authentication and anti-reply checking to prevent "man-in-the-middle" attacks.

At step <NUM>, flow matching operations may be performed on the decrypted packet. As examples, different fields are matched against security or denial-of-service policies using the range matching (RM) engine and/or the wild card matching (WCM) engine. The RM/WCM engines may block inappropriate application packets from suspicious applications to prevent potential DDoS attacks. At step <NUM>, the packet is optionally dropped from further processing if a security policy is violated.

Additional steps are shown in <FIG>. At step <NUM>, if a security policy is not violated, the packet is sent to network flow search hardware accelerator <NUM> to determine where to send the packet. For example, the extracted flow keys may be used to perform exact or longest prefix matches to extract actions on how to process the packet (e.g., matching operations may be performed using the small exact match engine, the large exact match engine, and/or the longest prefix match engine).

At step <NUM>, actions are extracted from the flow matching operations and metering is performed to prevent a particular flow from taking up an excessive amount of bandwidth. Packets that are taking up more than a predetermined chunk of bits on a network port are selectively dropped. The metering is managed using flow rate control hardware accelerator <NUM> (e.g., a traffic shaper engine or a meters/policing and statistics engine), which is used to ensure smooth delivery of service for all tenants on the network by controlling the traffic flow of an aggressive tenant. In other words, some networking application flows corresponding to one tenant or a group of tenants over-utilizing the network may be rate-controlled or throttled to ensure service level agreement (SLA) guarantees are met for all tenants hosted on the same host processor.

At step <NUM>, the packet may be classified using the extracted actions, and the packet may either be sent to a storage processing flow/thread (proceeding to step <NUM>) or to a non-storage processing flow/thread (proceeding to step <NUM>).

When forking to the storage processing thread, the packet may be sent from the packet processor to an RDMA engine assuming an RDMA enabled flow (step <NUM>). The packet processor may include circuits in only the top die (e.g., the flexible pipeline), in only the bottom die (e.g., the SEM/LEM engine, LPM engine, meters and statistics engine, etc.), or in both the top and bottom dies. At step <NUM>, the packet may be sent from the RDMA engine to a storage processor. The storage processor may include circuits in only the top die, in only the bottom die, or in both the top and bottom dies. At step <NUM>, the packet may be sent from the storage processor to application/software (SW) buffers (e.g., SW buffers within the customer IP blocks in the top die or application buffers in the host processor).

When forking to the non-storage processing thread, the packet may be sent from the packet processor to the DMA engine (e.g., a DMA engine on the top die or a DMA engine in the host processor). At step <NUM>, the packet may be sent from the DMA engine directly to application buffers (e.g., application buffers within the customer IP blocks in the top die or software buffers in the host processor). For instance, the DMA engine may fetch descriptors to move the packet into the SW/application buffers and may send an interrupt notification to the application.

Follow either step <NUM> or <NUM>, processing may proceed to step <NUM>. At step <NUM>, the application software may process the application data and may then release the application buffer back to the descriptor pool. Operated in this way, future packets may use the released application buffer pointed by such descriptor.

Although the methods of operations are described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or described operations may be distributed in a system which allows occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in a desired way.

Claim 1:
A network interface controller (<NUM>), comprising:
a first integrated circuit die (<NUM>) that includes customizable networking circuits (<NUM>), wherein the first integrated circuit die comprises a programmable integrated circuit die; and
a second integrated circuit die (<NUM>) that includes basic non-customizable network (<NUM>) function circuits (<NUM>) wherein the second integrated circuit die is an application-specific integrated circuit, ASIC, die, wherein the basic non-customizable network function circuits in the second integrated circuit die comprises memory circuits (<NUM>) and a flow rate control hardware accelerator circuit (<NUM>) configured to drop packets that are taking up more than a predetermined chunk of bits on a network port, and wherein the first integrated circuit die is stacked vertically with respect to the second integrated circuit die.