In-network failure indication and recovery

A programmable switch includes a plurality of ports for communicating with a plurality of network devices. A packet for a distributed system is received via a port and at least one indicator is identified in the received packet. Reliability metadata associated with a network device used for the distributed system is generated using the at least one indicator. The generated reliability metadata is sent to a controller for the distributed system for predicting or determining a reliability of at least one of the network device and a communication link for the network device and the programmable switch.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. application Ser. No. 16/548,116 titled “DISTRIBUTED CACHE WITH IN-NETWORK PREFETCH”, filed on Aug. 22, 2019, and published as U.S. Patent Application Publication No. 2020/0349080 on Nov. 5, 2020, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 16/697,019 titled “FAULT TOLERANT DATA COHERENCE IN LARGE-SCALE DISTRIBUTED CACHE SYSTEMS”, filed on Nov. 26, 2019, and published as U.S. Patent Application Publication No. 2020/0351370 on Nov. 5, 2020, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 16/914,206 titled “DEVICES AND METHODS FOR MANAGING NETWORK TRAFFIC FOR A DISTRIBUTED CACHE”, filed on Jun. 26, 2020, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 16/916,730 titled “DEVICES AND METHODS FOR FAILURE DETECTION AND RECOVERY FOR A DISTRIBUTED CACHE”, filed on Jun. 30, 2020, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 17/174,681, titled “DEVICES AND METHODS FOR NETWORK MESSAGE SEQUENCING”, filed on Feb. 12, 2021, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 17/175,449, titled “MANAGEMENT OF NON-VOLATILE MEMORY EXPRESS NODES”, filed on Feb. 12, 2021, which is hereby incorporated by reference in its entirety. This application is also related to U.S. application Ser. No. 17/331,453, titled “DISTRIBUTED MEMORY SYSTEM MANAGEMENT”, filed on May 26, 2021, which is hereby incorporated by reference in its entirety.

BACKGROUND

Current trends in cloud computing, big data, and Input/Output (I/O) intensive applications have led to greater needs for high performance distributed systems in data centers in terms of low latency, high throughput, and bandwidth. Although protocols such as Non-Volatile Memory express (NVMe) have been extended over networks, such as with NVMe over Fabrics (NVMeOF), to support access to high performance NVMe devices, such as NVMe Solid-State Drives (SSDs), distributed systems are susceptible to network errors due to unreliable networks, such as with an Ethernet network, and errors at various devices in the network. Such errors can cause significant data loss and service down time that can greatly affect system performance.

Existing data center fault-tolerance approaches are typically based on reactive failure detection and recovery techniques such as erasure coding to recover data after a failure occurs. Replication is often used in distributed systems to provide fault tolerance for hardware failures. Existing error detection and recovery relies heavily on the end-hosts to detect errors in received data and to correct the errors to recover the data. If the end-host fails to recover the data, the original sender will usually need to retransmit the lost or corrupted data. In this case, the latency for error recovery depends on multiple factors such as the number of links and nodes between the original sender and the end-host, and the links' speed and packet processing delays of nodes between the original sender and the end-host.

The latency overhead of reconstructing lost or corrupted data by the end-host and retransmitting lost or corrupted data negatively affects the performance benefits that can be achieved by using high performance devices, such as NVMe SSDs in a distributed system. In addition, these fault-tolerance approaches require storing extra data for the recovery process, which affects the amount of data that can be stored in the distributed system and reduces network data transfer efficiency. Moreover, reactively dealing with failures after they occur can cause a significant degradation in service and even system downtime.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth to provide a full understanding of the present disclosure. It will be apparent, however, to one of ordinary skill in the art that the various embodiments disclosed may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail to avoid unnecessarily obscuring the various embodiments.

SYSTEM EXAMPLES

FIG.1illustrates an example network100for implementing a distributed system according to one or more embodiments. As shown inFIG.1, server racks101A,101B, and101C use Top of Rack (ToR) switches102A,102B, and102C, respectively, to communicate with other devices in network100. Each server rack101includes one or more network devices, such as network device108inFIG.2, that can access memory blocks, and/or processing resources in other network devices in network100. The network devices in server racks101can include, for example, servers or processing nodes, such as Reduced Instruction Set Computer (RISC)-V cores, and memory devices, such as Solid-State Drives (SSDs) or Hard Disk Drives (HDDs). In some implementations, network100inFIG.1may be used as at least part of a data center and/or for distributed processing, such as for distributed machine learning or big data analysis.

Network100can include, for example, a Storage Area Network (SAN), a Local Area Network (LAN), and/or a Wide Area Network (WAN), such as the Internet. In this regard, one or more of server racks101, ToR switches102, aggregated switch104, and/or network controller120may not be physically co-located. Server racks101, ToR switches102, aggregated switch104, and/or network controller120may communicate using one or more standards such as, for example, Ethernet.

As shown in the example ofFIG.1, each of server racks101A,101B, and101C is connected to a ToR or edge switch102. In other implementations, each rack101may communicate with multiple ToR or edge switches102for redundancy. ToR switches102can include programmable switches, such as 64 port ToR P4 programmable switches that route messages to and from nodes or network devices located in server racks101. Such programmable switches can include, for example, a Barefoot Networks Tofino Application Specific Integrated Circuit (ASIC) with ports configured to provide 40 Gigabit Ethernet (GE) frame rates. Other types of programmable switches that can be used as a ToR switch102can include, for example, a Cavium Xpliant programmable switch or a Broadcom Trident 3 programmable switch. As discussed in more detail below, each ToR switch102can generate metadata for predicting or determining the reliability of one or more devices for a distributed system, such as for a distributed memory system or a distributed processing system.

Aggregated switch104routes messages between the ToR switches102and network controller120. In some implementations, server racks101A,101B, and101C with ToR switches102A,102B, and102C, and aggregated switch104may be viewed as a cluster of devices on network100. In this regard, those of ordinary skill in the art will appreciate that the network100can include many more network devices than those shown in the example ofFIG.1. For instance, network100may include other clusters of server racks101, ToR switches102, and aggregated switches104. As another example, network100may include additional levels such as with one or more core switches located between network controller120and aggregated switch104.

In this regard, different paths between the network devices of server racks101form different communication links or paths. In some implementations, multiple communication links may be available for sending and receiving data between the network devices. For example, some implementations may include backup ToR switches102for each rack101to provide a different communication link and/or additional aggregated switches104that provide more than one communication link between network devices, switches, and/or network controller120.

Aggregated switch104can include a programmable switch, such as a 64 port ToR P4 programmable switch that routes messages to and from ToR switches102and network controller120. Such a programmable switch can include, for example, a Barefoot Networks Tofino ASIC with ports configured to provide 40 Gigabit Ethernet (GE) frame rates. Other types of programmable switches that can be used as an aggregated switch104can include, for example, a Cavium Xpliant programmable switch or a Broadcom Trident 3 programmable switch.

Network controller120can include a Software Defined Networking (SDN) controller. As discussed in more detail below, network controller120can store global reliability metadata24for a distributed system implemented by different nodes or network devices in network100. Global reliability metadata24can be updated based on reliability metadata received from programmable switches, such as ToR switches102, and used by failure indication module22to determine or predict the reliability of different network devices and/or communication links used for the distributed system.

In this regard, ToR switches102are configured to inspect packets received by the ToR switch to identify indicators in packets for the distributed system and generate reliability metadata using the identified indicators that can be used by network controller120to predict or determine a reliability of at least one of the network devices and communication links. The identified indicators can include, for example, a Cyclic Redundancy Check (CRC) value, a timestamp, a message acknowledgment, and/or a message sequence number. The ToR switch102can inspect the packets it receives using inspection module12and generate reliability metadata16using indicators14. ToR switch102can then send the reliability metadata16, or a portion thereof, to network controller120. Reliability metadata16can include, for example, an indication of a transmission time for one or more packets, a count of corrupted packets, and/or a number of out-of-sequence packets.

In some implementations, ToR switch102may also use monitoring module10to monitor operations for at least one network device of the distributed system to generate metadata for reliability metadata16. In such implementations, monitoring module10may include, for example, an extended Berkeley Packet Filter (eBPF) executed by circuitry of ToR switch102(e.g., circuitry132inFIG.2). Monitoring module10may be used to generate additional reliability metadata based on monitored operations of the least one network device, such as operations for accessing a cache memory stored at network device108that is shared with other network devices. The monitored operations can include, for example, changes in data traffic for the at least one network device, packet drops for packets received from the at least one network device, and corrupted messages received from the at least one network device.

ToR switches102may also use monitoring module10to determine a port status or interface status associated with one or more network devices and generate metadata for reliability metadata16based on the determined port status or interface status. For example, ToR switch102A may determine that the status of a port is unavailable or cycles between being on and off more than a threshold number of state changes during a period of time. Such a high frequency cycling in port or interface status can indicate that the corresponding communication link is not reliable between the ToR switch102A and the network device. ToR switch102A may then add an indication of this unreliability to reliability metadata16for the network device120to use in determining reliability.

In addition, network device108shown inFIG.1may execute I/O monitoring module20to generate reliability metadata such as, for example, a number of reads and/or writes to a shared memory of the network device and an indication of errors, such as unrecoverable errors and/or parity errors. Network device108can provide such metadata to ToR switch102A to add to reliability metadata16, which may be provided to network controller120for predicting or determining a reliability of network device108.

In some implementations, ToR switches102and aggregated switch104can include, for example, programmable switches that can be programmed to handle different custom protocols. Programmable switches102and104can include programmable match-action pipelines to provide a configurable data plane and customized packet processing capability with L1/L2 packet switching18. Examples of such programmable switches can be found in co-pending U.S. application Ser. Nos. 17/174,681, 16/914,206, and 16/916,730, and U.S. Patent Application Publication Nos. 2020/0349080 and 2020/0351370, each of which are incorporated by reference above.

Data planes of programmable switches102and104in the example ofFIG.1can control point-to-point packet forwarding behavior of the programmable switch, such as with L1/L2 Ethernet packet switching, packet admission control, and scheduling or queuing. Data planes of programmable switches102and104are programmable and separate from higher-level control planes that determine end-to-end routes for packets or messages between devices in network100.

In some implementations, ToR switches102may serve as Non-Volatile Memory express (NVMe) controllers for NVMe nodes in their respective server racks101. In such implementations, ToR switches102can update available namespaces in an NVMe mapping for the server rack and/or perform an NVMe discovery process to determine whether there are one or more newly available namespaces. Such implementations are discussed in more detail in co-pending U.S. application Ser. No. 17/175,449, which is incorporated by reference above.

In addition, the use of programmable switches102and104can enable the configuration of high-performance and scalable memory centric architectures by defining customized packet formats and processing behavior. Programmable switches102and104enable a protocol-independent switch architecture and the use of off-the-shelf switches, as opposed to specially designed Networks on a Chip (NoCs). The processing resources of programmable switches102and104, such as the use of Content Addressable Memory (CAM) or Ternary CAM (TCAM) tables, or other types of match-action tables, can ordinarily provide faster processing and deep packet inspection, such as inspection of NVMe messages within a packet, than can occur at the end nodes. As discussed in more detail below, this can enable faster identification of failures or unreliability in the distributed system.

Those of ordinary skill in the art will appreciate with reference to the present disclosure that other implementations may include a different number or arrangement of server racks101, ToR switches102, and aggregated switches104than shown in the example ofFIG.1. In this regard, network100shown inFIG.1is for the purposes of illustration, and those of ordinary skill in the art will appreciate that network100may include many more server racks101, switches or routers than shown in the example ofFIG.1. Other implementations may include additional levels in network100that may include core switches, additional servers and/or other programmable switches. In some variations, aggregated switch104may be omitted.

In addition, some implementations may include a different arrangement of modules, such as a single module executed by a ToR switch102for inspecting packets, generating metadata, monitoring operations of at least one network device, and/or port or interface statuses. In yet other implementations, reliability metadata16may be stored in different locations than shown inFIG.1, such as at a node within a server rack101instead of at a ToR switch102. Similarly, global reliability metadata24collected by network controller120may be stored at a different location than at network controller120in other implementations.

FIG.2is a block diagram of example components included in network100ofFIG.1according to one or more embodiments. As shown inFIG.2, network device108includes processor116, memory118, storage device121, and interface122for communicating on network100. Network device108may be included as part of server rack101A, for example, inFIG.1. Although only network device108is shown in the example ofFIG.2, other nodes in network100may have similar or different components as network device108.

Processor116can execute instructions, such as instructions from I/O monitoring module20and application(s)28, which may include an Operating System (OS) and/or other applications used by network device108. Processor116can include circuitry such as a Central Processing Unit (CPU), one or more RISC-V cores, a Graphics Processing Unit (GPU), a microcontroller, a Digital Signal Processor (DSP), an ASIC, a Field Programmable Gate Array (FPGA), hard-wired logic, analog circuitry and/or a combination thereof. In some implementations, processor116can include a System on a Chip (SoC), which may be combined with one or both of memory118and interface122.

Memory118can include, for example, a volatile Random Access Memory (RAM) such as Static RAM (SRAM), Dynamic RAM (DRAM), a non-volatile RAM, or other solid-state memory that is used by processor116as an internal main memory to store data. Data stored in memory118can include data read from storage device121, data to be stored in storage device121, instructions loaded from I/O monitoring module20or application(s)28for execution by processor116, and/or data used in executing such applications. In addition to loading data from memory118, processor116may also load data from shared memory locations of other network devices as an external memory or distributed memory system. Such data may also be flushed after modification by processor116or evicted without modification back to memory118or an external network device via programmable switch102.

As shown inFIG.2, memory118stores cache26, which can be a shared cache that is shared with other network devices in network100. In some implementations, I/O monitoring module20may collect information on usage of cache26and/or error information related to data accessed in cache26, such as errors in reading data from cache26or in writing data to cache26.

Storage device121serves as secondary storage that can include, for example, one or more rotating magnetic disks or non-volatile solid-state memory, such as flash memory. While the description herein refers to solid-state memory generally, it is understood that solid-state memory may comprise one or more of various types of memory devices such as flash integrated circuits, NAND memory (e.g., single-level cell (SLC) memory, multi-level cell (MLC) memory (i.e., two or more levels), or any combination thereof), NOR memory, electrically erasable programmable read only memory (EEPROM), other discrete Non-Volatile Memory (NVM) chips, or any combination thereof.

Interface122is configured to interface network device108with programmable switch102. Interface122may communicate using a standard such as, for example, Ethernet. In this regard, network device108, programmable switch102, and network controller120may not be physically co-located and may communicate over a network such as a LAN or a WAN. As will be appreciated by those of ordinary skill in the art, interface122can be included as part of processor116.

As discussed above with reference toFIG.1, programmable switch102can be a ToR switch for a server rack101including network device108. In the example ofFIG.2, programmable switch102includes ports130, circuitry132, and memory134. Ports130provide a connection and are configured to communicate with devices, such as nodes, network controller120, and aggregated switch104. For example, ports130may include Ethernet ports.

Memory134of programmable switch102can include, for example, a volatile RAM such as DRAM, or a non-volatile RAM or other solid-state memory such as register arrays that are used by circuitry132to execute instructions loaded from cache monitoring module10, inspection module12, or firmware of programmable switch102, and/or data used in executing such instructions, such as indicators14or reliability metadata16. Circuitry132can include circuitry such as an ASIC, a microcontroller, a DSP, an FPGA, hard-wired logic, analog circuitry and/or a combination thereof. In some implementations, circuitry132can include an SoC, which may be combined with memory134.

As discussed in more detail below, cache monitoring module10and inspection module12can include instructions for implementing processes such as those discussed with reference toFIGS.4and6to generate reliability metadata and to enable network controller120to predict or determine which network devices in the distributed system are unreliable or more error prone. Network controller120may then adjust usage of network devices based on the determined or predicted reliability of one or more network devices, as discussed in more detail below with reference toFIGS.7and8.

Network controller120in the example ofFIG.2maintains global reliability metadata24, which may include a table or other type of data structure, such as a Key Value Store (KVS). Controller120receives reliability metadata updates or notifications from programmable switches102and/or aggregated switch104via interface128indicating updates or changes to the reliability metadata maintained by the programmable switches, such as reliability metadata16.

Processor124of network controller120executes failure indication module22to determine or predict a reliability of network devices and communication links based on global reliability metadata24and notify the programmable switches of adjustments to the usage of different network devices or communication links, as needed. Processor124can include circuitry such as a CPU, a GPU, a microcontroller, a DSP, an ASIC, an FPGA, hard-wired logic, analog circuitry and/or a combination thereof. In some implementations, processor124can include an SoC, which may be combined with one or both of memory126and interface128. Memory126can include, for example, a volatile RAM such as DRAM, a non-volatile RAM, or other solid-state memory that is used by processor124to store data. Network controller120communicates with programmable switches102via interface128, which is configured to interface with ports of programmable switches102, and may interface according to a standard, such as Ethernet.

As will be appreciated by those of ordinary skill in the art with reference to the present disclosure, other implementations may include a different arrangement or number of components, or modules than shown in the example ofFIG.2. For example, in some implementations, network device108may not include storage device121, or two programmable switches102may be used for a single server rack for redundancy. In addition, the arrangement shown for programmable switch102inFIG.2may also apply to aggregated switch104in some implementations.

FIG.3illustrates an example of reliability metadata16at programmable switch102according to one or more embodiments. In the example ofFIG.3, reliability metadata16may be stored as a table or other type of data structure such as a KVS. For example, reliability metadata16can include a single data structure or may be formed of multiple data structures stored at a programmable switch102, which may include a memory directly connected to and used by programmable switch102(e.g., memory134inFIG.2). Global reliability metadata24stored by network controller120may include similar information as that shown for reliability metadata16ofFIG.3, but with reliability metadata associated with network devices throughout the distributed system.

As shown inFIG.3, reliability metadata16includes device identifiers that identify different network devices used for the distributed system (e.g., a distributed memory system and/or a distributed processing system) and that communicate with the programmable switch102directly or indirectly. In implementations where reliability metadata is generated or stored by aggregated switches104, the device identifiers can identify different network devices that communicate with the aggregated switch104directly or indirectly. In some implementations, the programmable switches can identify the network devices using a network address for the network device. The network address may be used in some implementations as the device identifier in reliability metadata16.

In the example ofFIG.3, reliability metadata16includes metadata for each network device communicating with the programmable switch, such as an average transmission time for packets received from the network device, a percentage of packets that are received out of order, a port status for the port that communicates with the network device (e.g., on, off, rapidly changing between on and off), a percentage of packets received from the network device that have corrupted data, an average roundtrip time for a response to be returned from the network device for a request sent to the network device, a percentage of I/O errors reported by the network device, and a change in traffic for the network device, such as a sudden drop in traffic. Other implementations of reliability metadata16can include different reliability metadata, such as a total count of metrics over a certain period of time instead of percentages for out of sequence messages, corrupted data, or I/O errors.

The transmission time can be determined or generated by the programmable switch by identifying a timestamp in a packet received from a network device and subtracting the timestamp time from a current time. As noted above, the programmable switch can parse the headers of a packet to identify indicators in the packet, such as a timestamp indicating when the packet was sent. In some implementations, a match-action table of the programmable switch may be used to quickly calculate the transmission time and store the transmission time or to average the transmission time with other transmission times for the network device for storage in reliability metadata16. A long transmission time can indicate a reliability issue with the network device sending the packet or the communication link between the network device sending the packet and the programmable switch.

The out of sequence percentage or count can represent a number of messages or packets that are received out of order. Protocols such as NVMe can include a sequence number in messages that are encapsulated in the packet to indicate an order for data that is sent that exceeds a maximum data size and needs to be broken up into a series of messages. The sequence number can be used by a receiving node or network device to ensure the payload or data sent in the series of messages is properly assembled. The programmable switch can inspect the packets to determine if the series of messages are received in the proper order. When messages are received out of order, the programmable switch can increment or adjust the percentage or average of sequenced messages received from the network device that are received out of sequence. The receipt of out of sequence messages can indicate that the network device or the communication link between the programmable switch and the network device may not be reliable since packets may be dropped or delayed in route to the programmable switch.

The port status shown in reliability metadata16inFIG.3can represent a status of a port or interface for a communication link with the network device. For example, a port status of “1” inFIG.3can indicate that the port or interface is operating with a continuous “on”, ready, or powered state. A port status of “0” on the other hand, can indicate that the port or interface is “off”, unavailable, or not powered. In the example ofFIG.3, a port status of “2” can indicate that the port or interface is fluctuating between “on” and “off” states, which may indicate that the port or communication link is having connectivity problems. Other implementations may classify the fluctuating or power cycling status with the “off” status (i.e., “0” inFIG.3).

The corrupted indicator in reliability metadata16can indicate a percentage or count of packets that have data that has been corrupted or otherwise modified from an original value. In some implementations, the programmable switch can identify a CRC value in a packet from the network device and calculate a new CRC value using data in the packet. The programmable switch may then compare the CRC value identified in the packet with the new CRC value calculated by the programmable switch to determine if the CRC values match. If the CRC values do not match, a count for a number of corrupted packets received from the network device can be incremented. The example of reliability metadata16inFIG.3represents this count of corrupted packets as a percentage of the total packets received from the network device. Other implementations may instead use a total count over a predetermined period of time or another indicator such as a level to indicate a relative number of packets received with corrupted data. In addition, the programmable switch may correct the data using the CRC value in some implementations where corrupted data is received. In other cases, the programmable switch may drop the packet if the data cannot be corrected.

In cases where the programmable switch forwards a request from an originating network device, such as a write request or a read request to a destination network device, the programmable switch can keep track of when the request was sent and wait for a response from the destination network device acknowledging performance of the request. The programmable switch may then compare the time the request was sent from the programmable switch with the time the response was received by the programmable switch to calculate the round trip time. The requests sent by the programmable switch to the destination network device can come from other originating network devices that are routed via the programmable switch. In this regard, the programmable switch may first identify an indicator such as an operation code or op code indicating whether the packet to be sent to the destination network device include a request that will result in an acknowledgment from the destination network device, such as a read or write request. The programmable switch can associate the request that is sent with the acknowledgment received by identifying the source and destination addresses for the network devices before forwarding the request and the acknowledgment to their intended locations.

The I/O errors in reliability metadata16can indicate an amount of errors in reading and/or writing data in a memory of the network device. This information can come from the network device itself, such as with the use of I/O monitoring module20shown inFIGS.1and2. For example, a network device may keep track of a number of reads and/or writes to a shared memory of the network device that encounter an error, such as an unrecoverable error or a parity error. The network device may periodically provide this information to the programmable switch to update reliability metadata16. The example of reliability metadata16inFIG.3represents this as a percentage of a total number of reads and writes by the network device, but other implementations may indicate this count of I/O errors differently, such as with a total number of errors or a total number of errors within a predetermined period of time. In some implementations, the I/O error metadata may be collected by the network device using an eBPF filter.

The traffic change metadata provided inFIG.3can represent whether the traffic (i.e., number of packets) sent by or received from the network device has decreased below a threshold amount. This may indicate that the network device or the communication link between the programmable switch and the network device is not functioning properly. In other implementations, the programmable switch may monitor other operations of the network device, such as a number of packets that are dropped from the network device as a result of corrupted data or messages received from the network device. In this regard, the programmable switch may use an eBPF program to monitor the operations of the network device.

Reliability metadata16can be updated by the programmable switch to add new network devices communicating with the programmable switch or to remove network devices that have not communicated with the programmable switch during a predetermined period of time (e.g., 5 minutes). In some implementations, reliability metadata16may include metadata indicating when the reliability metadata16for a network device was last updated by the programmable switch to remove reliability metadata for inactive network devices.

As discussed in more detail below, the programmable switch sends some or all of reliability metadata16to network controller120so that the network controller120can update its own global reliability metadata24and use failure indication module22to predict or determine the reliability of network devices and communication links used for the distributed system. The controller120can then adjust the usage of certain network devices and/or communication links to shift usage away from unreliable network devices and/or communication links toward more reliable network devices and/or communication links.

As will be appreciated by those of ordinary skill in the art with reference to the present disclosure, reliability metadata16may include different information than shown inFIG.3. For example, some implementations of reliability metadata16may include other metadata for monitored operations, such as a count of dropped packets received from the network device or corrupted messages received from the network device. As another example variation, a last updated column can be included for indicating when reliability metadata for a network device was last updated to identify inactive network devices and free up memory for storing reliability metadata16.

EXAMPLE PROCESSES

FIG.4is a flowchart for a reliability metadata generation process according to one or more embodiments. The process ofFIG.4may be performed by, for example, circuitry132of programmable switch102executing inspection module12inFIG.2.

In block402, the programmable switch receives a packet for a distributed system, such as for a distributed memory system or a distributed processing system implemented at nodes or network devices of network100inFIG.1. The packet may be identified by the programmable switch as being for the distributed system by parsing the packet to identify a header or other field indicating an address for the distributed system, such as an address for a shared memory.

In block404, the programmable switch inspects the packet to identify at least one indicator for generating reliability metadata. In some implementations, the programmable switch may use inspection module12to perform deep packet inspection to identify indicators such as, for example, a timestamp for when the packet was sent, a CRC value or other error detection values for data included in the packet, a message acknowledgment, or a message sequence number indicating an order for data in the packet relative to data sent in other packets. The indicators may be temporarily stored by the programmable switch, as with indicators14inFIGS.1and2, for generating reliability metadata associated with a network device.

In block406, the programmable switch generates reliability metadata associated with a network device based on the inspection in block404using at least one indicator. In some implementations, the programmable switch may have programmed pipelines or match-action tables that perform operations using the identified indicators, such as calculating packet transmission time, determining if a message included in the packet has been received out of sequence, calculating and comparing a CRC value or other error checking value, calculating a round trip time for an acknowledgment to be received in response to an earlier packet sent by the programmable switch, or identifying a data error included in the packet. As discussed in more detail below with reference toFIGS.5and6, the programmable switch may generate additional reliability metadata based on monitored operations of a network device or of a port or interface of the programmable switch.

In block408, the programmable switch sends reliability metadata generated in block406to network controller120for predicting or determining a reliability of at least one of the network device and a communication link for the network device and the programmable switch. The programmable switch may send the reliability metadata as part of a background activity periodically, in response to receiving a request from the controller for reliability metadata, and/or in response to certain changes in the reliability metadata, such as changes in transmission time or traffic that exceed a threshold level. The dashed line inFIG.4indicates that the sending of the reliability metadata to the controller may occur at a different time from when the reliability metadata is generated. In some implementations, the programmable switch may send the most recently updated reliability metadata or only the reliability metadata that has changed since reliability metadata was last sent to the controller.

Those of ordinary skill in the art will appreciate with reference to the present disclosure that the order of blocks for the reliability metadata generation process ofFIG.4may differ in other implementations. For example, in some implementations, many packets may be inspected in block404before reliability metadata is generated in block406, or reliability metadata may be generated multiple times in block406before sending the reliability metadata to the controller in block408.

FIG.5is a flowchart for a reliability metadata generation process based on monitored operations of a network device according to one or more embodiments. The process ofFIG.5may be performed by, for example, circuitry132of programmable switch102executing monitoring module10and/or a processor116of a network device108executing I/O monitoring module20, which may be implemented as an eBPF.

In block502, the programmable switch monitors operations of at least one network device using its programmed pipeline. The monitored operations can include, for example, changes in data traffic for the network device, packet drops for packets received from the network device, and corrupted messages received from the network device.

In block504, the programmable switch generates additional reliability metadata based on the monitored operations. For example, the programed pipeline may update reliability metadata16if a packet received from a network device is dropped due to corrupted data, if the number of packets received from the network device drops below a threshold number of packets per minute, or if an error message is received from the network device. In some implementations, the program executed by the programmable switch may work in conjunction with an eBPF executed by the network device that may report error data to the programmable switch to generate additional reliability metadata associated with the network device.

In block506, the programmable switch sends the additional reliability metadata generated in block504to the network controller for predicting or determining a reliability of at least one of the network device and a communication link for the network device and the programmable switch. The programmable switch may send the additional reliability metadata as part of a background activity periodically, in response to receiving a request from the controller for reliability metadata, and/or in response to a change in the reliability metadata. In some implementations, the programmable switch may send the most recently updated reliability metadata or only the reliability metadata that has changed since reliability metadata was last sent to the controller.

Those of ordinary skill in the art will appreciate with reference to the present disclosure that the order of blocks for the reliability metadata generation process ofFIG.5may differ in other implementations. For example, in some implementations, metadata may be generated repeatedly in block504before sending metadata to the network controller in block506.

FIG.6is a flowchart for a reliability metadata generation process using port status or interface status of a programmable switch according to one or more embodiments. The process ofFIG.6may be performed by, for example, circuitry132of programmable switch102executing monitoring module10inFIG.2.

In block602, the programmable switch determines a port or interface status for one or more network devices and corresponding communication links. In some implementations, monitoring module10may monitor the power state of the connections to the ports of the programmable switch (e.g., ports130inFIG.2). A P4 program, for example, may be used to monitor the hardware state of the ports in some implementations.

In block604, the programmable switch generates additional reliability metadata based on the port status or interface status determined in block602. For example, the programmable switch may update the reliability metadata to represent a power state of the ports, such as whether the connection remains powered, loses power, or repeatedly cycles between having power and no power within a predetermined period of time. In other cases, the programmable switch may update the reliability metadata to represent an interface status, such as whether a ready status or an unavailable status is determined by the programmable switch for a communication link used for communicating with the network device.

In block606, the programmable switch sends the additional metadata generated in block604to the network controller for the distributed system for predicting or determining a reliability of the one or more network devices and corresponding communication links. The programmable switch may send the additional reliability metadata as part of a background activity periodically, in response to receiving a request from the controller for reliability metadata, and/or in response to a change in the port status or interface status. The dashed line inFIG.6indicates that the sending of the additional reliability metadata to the controller may occur at a different time from when the additional reliability metadata is generated. In some implementations, the programmable switch may send the most recently updated reliability metadata or only the reliability metadata that has changed since reliability metadata was last sent to the controller.

FIG.7is a flowchart for a usage reduction notification process according to one or more embodiments. The process ofFIG.7may be performed by, for example, circuitry132of programmable switch102inFIG.2.

In block702, the programmable switch receives an indication from the network controller to reduce usage of at least one of a network device and a communication link. The indication may be included in a message from the controller using a custom protocol for the distributed system. In some cases, the indication may indicate an address of a network device that should not be used in the distributed system or that should be used less in the distributed system in favor of other network devices in the distributed system. In other cases, the indication may indicate a communication link that should be avoided, such as with an address for another ToR or aggregate switch. In some implementations, a backup ToR switch may be used in place of a primary ToR switch if the communication link with the primary ToR switch is determined or predicted by the controller to be unreliable. The message from the controller in some implementations may indicate whether a particular network device or communication link is to be avoided completely, as in the case of a network device that is no longer available, or whether its usage is to only be reduced.

In block704, the programmable switch notifies at least one other network device of the indication from the controller to reduce usage of at least one of the network device and the communication link. The programmable switch may create a new message for the at least one other network device, such as for the network devices in its respective rack101, using a custom protocol, or may forward the message received from the controller in block702. In some implementations, the programmable switch may send messages to particular network devices that are known by either the programmable switch or the controller to communicate with the network device or using the communication link that is to have reduced usage. For example, the programmable switch and/or the controller may include a directory, such as for NVMe namespaces, that may indicate addresses for active network devices that communicate with a network device that is to have reduced usage.

FIG.8is a flowchart for a reliability prediction or reliability determination process according to one or more embodiments. The process ofFIG.8may be performed by, for example, processor124of network controller120executing failure indication module22.

In block802, the controller receives reliability metadata from a plurality of programmable switches indicating reliability of one or more network devices and communication links used fora distributed system. As discussed above with reference toFIGS.4to6, the programmable switches in network100may generate and/or collect their own reliability metadata (e.g., reliability metadata16inFIG.3) based on the communication links used by the programmable switch and the network devices that communicate with the programmable switch. The controller can receive updates from each of the programmable switches that send reliability metadata to update a global reliability metadata for the distributed system (e.g., global reliability metadata24inFIG.2). In some implementations, the controller may periodically request reliability metadata from the programmable switches. In other cases, the programmable switches may send their reliability metadata to the controller without a request from the controller.

In block804, the controller determines or predicts a reliability of one or more network devices and communication links based on the reliability metadata received from the programmable switches. In some implementations, the controller may use some or all of the reliability metadata as inputs to a function that has been determined by machine learning to predict the reliability of the one or more network devices and communication links. For example, failure indication module22can apply previously learned failure patterns to the reliability metadata and predict whether there is more than a threshold likelihood of a future failure of a network device or communication link. The failure indication module22may use different classification and prediction algorithms, such as a random forest, a neural network, or a decision tree algorithm.

In block806, the controller adjusts usage of network devices and communication links based on the determined or predicted reliability of one or more network devices and communication links. The adjustment in usage can include a load balancing that is performed by the controller to redistribute or transfer traffic and/or workloads from network devices and communication links that are determined to have more than a threshold level of unreliability. In cases where a network device or communication link has been determined to be completely unreliable, as in the case of an unavailable network device, the controller may adjust traffic to prevent the use of the network device.

In block808, the controller notifies at least one network device to reduce usage of at least one of a network device and a communication link based on the determined or predicted reliability of one or more network devices and communication links. The notifications can be sent to the network devices via a programmable switch as discussed above for the notification process ofFIG.7. The notifications may use a custom protocol for management of the distributed system, which may provide indications for whether the use of a network device or communication link should be completely stopped or whether its usage should be reduced.

Those of ordinary skill in the art will appreciate with reference to the present disclosure that the order of blocks for the reliability prediction or reliability determination process ofFIG.8may differ in other implementations. For example, the adjustment of block806may instead or additionally occur at programmable switches or network devices that perform load balancing or scheduling of requests for the distributed system.

As discussed above, the foregoing generation and collection of reliability metadata by in-line programmable switches can improve the fault tolerance of distributed systems by identifying potential failures before they cause data loss or downtime. In addition, the collection of reliability data from programmable switches throughout the network by the network controller can enable the use of predictive algorithms.

OTHER EMBODIMENTS

Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, and processes described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. Furthermore, the foregoing processes can be embodied on a computer readable medium which causes processor or controller circuitry to perform or execute certain functions.

To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, and modules have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those of ordinary skill in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The activities of a method or process described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by processor or controller circuitry, or in a combination of the two. The steps of the method or algorithm may also be performed in an alternate order from those provided in the examples. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable media, an optical media, or any other form of storage medium known in the art. An exemplary storage medium is coupled to processor or controller circuitry such that the processor or controller circuitry can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to processor or controller circuitry. The processor or controller circuitry and the storage medium may reside in an ASIC or an SoC.

The foregoing description of the disclosed example embodiments is provided to enable any person of ordinary skill in the art to make or use the embodiments in the present disclosure. Various modifications to these examples will be readily apparent to those of ordinary skill in the art, and the principles disclosed herein may be applied to other examples without departing from the spirit or scope of the present disclosure. The described embodiments are to be considered in all respects only as illustrative and not restrictive. In addition, the use of language in the form of “at least one of A and B” in the following claims should be understood to mean “only A, only B, or both A and B.”