Patent Description:
In packet processing devices such as network switches and routers, transitioning to smaller processing nodes was often sufficient to meet ever increasing performance targets. However, as the feature size of processing nodes approaches physical limitations, performance improvements become harder to achieve from process shrinkage alone. Meanwhile, high performance computing and other demanding scale out applications in the datacenter continue to require higher performance that is not met by conventional packet processing devices. Latency sensitive applications further require specialized hardware features, such as ternary content addressable memory ("TCAM"), which in turn imposes performance constraints that raise further hurdles in meeting performance targets.

Various objects, features, and advantages of the present disclosure can be more fully appreciated with reference to the following detailed description when considered in connection with the following drawings, in which like reference numerals identify like elements. The following drawings are for the purpose of illustration only and are not intended to be limiting of this disclosure, the scope of which is set forth in the claims that follow.

While aspects of the subject technology are described herein with reference to illustrative examples for particular applications, it should be understood that the subject technology is not limited to those particular applications. Those skilled in the art with access to the teachings provided herein will recognize additional modifications, applications, and aspects within the scope thereof and additional fields in which the subject technology would be of significant utility.

Disclosed herein are related to systems and methods for scheduling network operations. In one aspect, a network system includes a first data path to provide a first set of packets and a second data path to provide a second set of packets. The network system also includes an arbiter to arbitrate the first set of packets and the second set of packets. In one aspect, the arbiter is configured to receive a request for a task. The task is scheduled to occur or to be performed during a clock cycle. Based on the request, the arbiter generates a command to cause a scheduler to schedule a first idle slot for the first data path, and schedule a second idle slot for the second data path. An idle slot may be a null packet, or a packet with no data. According to the first idle slot, a pipe coupled between the first data path and the arbiter and between the second data path and the arbiter may bypass reading a packet from the first data path during the clock cycle to provide the first idle slot. Similarly, according to the second idle slot, the pipe may bypass reading a packet from the second data path during the clock cycle to provide the second idle slot. The arbiter may receive the first idle sot and the second idle slot from the pipe, and provide or output the first idle slot and the second idle slot during the clock cycle.

In one aspect, the disclosed network device (or network system) can reduce or avoid packet collisions to improve performances. For example, packet collisions from different data paths can increase power consumption and reduce throughput due to retransmission. In one aspect, an arbiter may provide or output a data packet from one data path, while enforcing synchronized idle slots for other data paths, such that the other data paths may bypass providing or outputting any packet. Accordingly, packet collisions can be avoided to lower power consumption and increase throughput.

In one aspect, the disclosed network device can improve a hardware learn rate. In one aspect, the disclosed network device allows learning or detecting a certain number (e.g., over <NUM> million) of features (e.g., MAC address, hash on any number of fields, source address, source IP address, etc.) of the network device for a given time period. In one example, learning a hardware feature includes extracting a certain field in a packet received, and checking if a matching entry of a table exists. Often, data from one or more data paths can interfere with the hardware learning process. By applying synchronized idle slots, hardware learning can be performed with less interference, such that a larger number of features of the network device can be determined for a given time period.

In one aspect, the disclosed network device can operate in a reliable manner, despite one or more erroneous processes. An erroneous process may exist, due to a false design by an engineer, or due to a hardware failure. For example, an unintended operation may be performed, or an operation may be performed at an unintended clock cycle. Such erroneous process may render the network device to be unreliable or unusable. Rather than discarding the network device, synchronized idle slots can be implemented for known erroneous processes. For example, idle slots can be enforced for a process from a faulty component, such that the process may be not executed or performed. Although the device may not perform intended processes associated with the erroneous processes, the disclosed network device can still perform other processes in a reliable manner and may not be discarded.

In one aspect, the disclosed network device can support a warm boot. In one aspect, various operations may be performed during a wake up sequence. In one example, a command or indication indicating no packet traffic can be provided. In response to the command or indication, the arbiter may ignore a packet spacing rule, and process data to support the wake up sequence, because there may be no data traffic from the data paths. By ignoring the packet spacing rule or other rules associated with data traffic, the disclosed network device can perform rigorous wake up sequence within a short time period (e.g., <NUM>).

In one aspect, the disclosed network device can achieve power savings by implementing idle slots. In one example, the device can detect or monitor power consumption of the device. In response to the power consumption exceeding a threshold value, the device may enforce idle slots. By enforcing idle slots, the arbiter or other components may not process data, such that power savings can be achieved.

The network environment <NUM> includes one or more electronic devices 102A-C connected via a network switch <NUM>. The electronic devices 102A-C may be connected to the network switch <NUM>, such that the electronic devices 102A-C may be able to communicate with each other via the network switch <NUM>. The electronic devices 102A-C may be connected to the network switch <NUM> via wire (e.g., Ethernet cable) or wirelessly. The network switch <NUM>, may be, and/or may include all or part of, the network switch discussed below with respect to the ingress / egress packet processing <NUM> of <FIG> and/or the electronic system discussed below with respect to <FIG>. The electronic devices 102A-C are presented as examples, and in other implementations, other devices may be substituted for one or more of the electronic devices 102A-C.

For example, the electronic devices 102A-C may be computing devices such as laptop computers, desktop computers, servers, peripheral devices (e.g., printers, digital cameras), mobile devices (e.g., mobile phone, tablet), stationary devices (e.g. set-top-boxes), or other appropriate devices capable of communication via a network. In <FIG>, by way of example, the electronic devices 102A-C are depicted as network servers. The electronic devices 102A-C may also be network devices, such as other network switches, and the like.

The network switch <NUM> may implement hyperscalar packet processing, which refers to a combination of several features that optimize circuit integration, reduce power consumption and latency, and improve performance for packet processing. Packet processing may include several different functions such as determining a correct port to forward a packet to its destination, gathering diagnostic and performance data such as network counters, and performing packet inspection and traffic categorization for implementing quality of service (QoS) and other load balancing and traffic prioritizing functions. Some of these functions may require more complex processing than other functions. Thus, one feature of hyperscalar packet processing is to provide two different packet processing blocks and arbitrate packets accordingly: a limited processing block (LPB) and a full processing block (FPB). Since packets may vary widely in the amount of required processing, it is wasteful to process all types of packets using a one size fits all packet processing block. By utilizing LPBs, smaller packets with less processing requirements can be quickly processed to provide very low latency. Further, since the LPBs may support a limited feature set, the LPBs can be configured to process more than one packet during a clock cycle compared to FPBs that process one packet, improving bandwidth and performance.

The number of LPBs and FPBs can be adjusted according to workload. The LPBs and FPBs may correspond to logical packet processing blocks in the Figures. However, in some implementations, the LPBs and FPBs may correspond to physical packet processing blocks or some combination thereof. For example, latency sensitive applications and transactional databases may prefer designs with a larger number of LPBs to handle burst traffic of smaller control packets. On the other hand, applications requiring sustained bandwidth of large packets such as content delivery networks or cloud backup may prefer designs with a larger number of FPBs.

Another feature is to organize processing blocks into physical groups providing a single logical structure with circuitry, such as logic and lookups, shared between the processing blocks to optimize circuit area and power consumption. Such grouped processing blocks may be able to process packets from multiple data paths, with corresponding data structures provided to allow coherent and stateful processing of packets. This may also enable an aggregate processing block to provide greater bandwidth to better absorb burst traffic and provide reliable response time in comparison to individual processing blocks with independent pipes that may become easily saturated, especially with increasing port speed requirements.

Another feature is to use a single shared bus and one or more arbiters for interfaces, allowing efficient utilization of available system bus bandwidth. The arbiter may enforce packet spacing rules and allow auxiliary commands to be processed when no packets are processed during a cycle.

Another feature is to provide slot event queues for data paths and a scheduler to enforce spacing rules and control the posting of events. By providing these features, events are not blocked by worst case data path latency, helping to further reduce latency and improve response time.

<FIG> is a block diagram of a logical block diagram of ingress / egress packet processing within an example network switch, according to one or more embodiments. While ingress packet processing is discussed in the below examples, ingress / egress packet processing <NUM> may also be adapted to egress packet processing. Ingress / egress packet processing <NUM> includes group 120A, group 120B, group 140A, first in first out (FIFO) queues <NUM>, shared bus 180A, shared bus 180B, post 190A, post 190B, post 190C, and post 190D. Group 120A includes LPB 130A and LPB 130B. Group 120B includes LPB 130C and LPB 130D. Group 140A includes FPB 150A and FPB 150B. It should be understood that the specific layout shown in <FIG> is exemplary, and in other implementations any combination, grouping, and quantity of LPBs and FPBs may be provided.

As shown in <FIG>, data path 110A, data path 110B, data path 110C, and data path 110D may receive data packets that are arbitrated via shared bus 180A and shared bus 180B through various packet processing and posting blocks. The shared bus 180A and 180B may allow for more efficient bandwidth utilization across high speed interconnects compared to separate individual buses with smaller bandwidth capacities. Packets may, for example, be analyzed based on packet size. If a packet is determined to be at or below a threshold packet size, such as <NUM> bytes, <NUM> bytes, or another value, then the packet may be arbitrated to one of the limited processing blocks, or LPB 130A-130D. This threshold packet size may be stored as a rule of an arbitration policy. Besides packet size, the arbitration policy rules may also arbitrate based on fields in the packet headers such as a packet type field, a source port number, or any other field. For example, if a type field indicates that a packet is a barrier or control packet rather than a data packet, then the packet may be arbitrated to one of the limited processing blocks.

If the packet is determined to exceed the threshold packet size or if the arbitration policy rules otherwise indicate that packet should be sent to a full processing block, then the packet may be arbitrated to one of the full processing blocks, or FPB 150A-150B. The arbitration policy may also assign data paths to specific processing blocks. For example, data path 110A is assigned to either LPB 130A or FPB 150A in <FIG>. However, in other implementations, a data path may be arbitrated to any available processing block. The enforcement of arbitration policy may be carried out by an arbiter of shared bus 180A and 180B, as described below in <FIG>.

As discussed above, each LPB 130A-130D may be capable of processing multiple packets in a single clock cycle, or two packets in the particular example shown. For example, each LPB 130A-130D may support a limited set of packet processing features, such as by omitting deep packet inspection and other features requiring analysis of packet payloads. Since the data payload does not need to be analyzed, the data payload may be sent separately outside of LPB 130A-130D. In this manner, the processing pipeline may be simplified and reduced in length and complexity, allowing multiple limited feature packet processing pipelines to be implemented within a physical circuit area that may be equal to a single full feature packet processing pipeline. Thus, up to <NUM> packets may be processed by LPB 130A-130D, wherein each LPB 130A-130D may send two processed packets to respective post 190A-190D.

On the other hand, each FPB 150A-150B may process a single packet in a single clock cycle. Thus, up to <NUM> packets may be processed by FPB 150A-150B, wherein FPB 150A may send a processed packet to post 190A or post 190B, and FPB 150B may send a processed packet to post 190C or 190D. Post 190A-190D may perform post-processing by e.g. reassembling the processed packets with the separated data payloads, if necessary, and further preparing the assembled packets for sending on a data bus, which may include serializing the data packets. After post 190A-190D, the serialized and processed packets may be sent on respective data buses <NUM>-<NUM>, which may further connect to a memory management unit (MMU).

Data paths 110A-110D may specifically correspond to ingress data buses in <FIG>. However, a similar design may be utilized for outputting to egress buses. Thus, when ingress / egress packet processing <NUM> corresponds to egress packet processing, data paths 110A-110D may correspond to post buses from the MMU, and post 190A-190D may output to respective egress data buses, which may further connect to upstream network data ports.

Groups 120A, 120B, and 140A may be organized to more efficiently share and utilize circuitry between and within the processing blocks contained in each group. In this way, circuit integration can be optimized, power consumption and latency can be reduced, and performance can be improved. For example, groups 120A, 120B, and 140A may share logic and lookups within each group to reduce total circuit area, as described in <FIG>. The reduced circuit area may consume less power. Group 140A may provide data structures to allow coherent and stateful processing of packets in an aggregate pipe, as described in <FIG>. Groups 120A-120B and 140A may further utilize separate data and processing pipelines described in <FIG>. Shared bus 180A and 180B may include arbiter <NUM> described in <FIG> or <FIG>.

<FIG> depicts an example system for processing a single packet from a single data path, according to one or more embodiments. As shown in <FIG>, a single data path, or data path 110A, is processed by a single full processing block, or FPB 150A. FPB 150A includes single packet processing <NUM>, which is able to process a single packet of any size for each clock cycle. Data path 110A and single packet processing <NUM> may share the same clock signal frequency. In a packet processing device, the system of <FIG> may be duplicated for a number of data paths to support, which may correspond to a number of network ports.

Packets to be processed may include a head of packet (HOP) that includes a start of packet (SOP) indication and a number of bytes to be processed, a payload, and a tail of packet (TOP) that includes packet size and error information. The portions of the packet to be processed may be referred to the start and end of packet (SEOP), whereas the payload may be bypassed using a separate non-processing pipe.

<FIG> depicts an example system for processing dual packets from data paths 110A and 110B, according to one or more embodiments. As discussed above, a key insight is that packets may vary widely in the amount of required processing. When a packet is below a processing threshold, which can correspond to a packet size threshold, then the packet may be processed using a limited processing block such as LPB 130A. LPB 130A may be implemented using a far less complex circuit design compared to FPB 150A, which supports all possible functionality of all packets. Thus, LPB 130A can provide dedicated hardware to process multiple packets from multiple data paths in a single clock cycle. Dual packet processing <NUM> may process a packet from each of data paths 110A and 110B in a single clock cycle. Further, since LPB 130A is a separate block from FPB 150A, packets processed through LPB 130A can be completed quicker for lower latency. For example, as discussed above, the processing pipeline for LPB 130A may be significantly shorter than for FPB 150A. In one implementation, a minimum latency for processing a packet through LPB 130A may be approximately <NUM> ns, whereas a minimum latency for processing a packet through FPB 150A may be approximately <NUM> ns. While two data paths are shown in <FIG>, the concept of <FIG> may be extended to multiple data paths, such as eight data paths as shown in <FIG>.

<FIG> depicts an example system for logically grouping dual packet processing 212A and 212B together, according to one or more embodiments. Group 120A includes dual packet processing 212A and 212B, which may be physically in proximity in a circuit layout. This proximity allows dual packet processing 212A and 212B to share logic and lookups for optimizing circuit area. At the same time, group 120A may also be logically grouped together to present a single logical processing block, for example by sharing logical data structures such as table structures. The incoming data packets from data paths 110A-110D may be arbitrated through a shared bus, such as shared bus 180A of <FIG>. To determine which processing block to route a data packet, an arbiter may be used, such as arbiter <NUM> of <FIG>. While four data paths 110A-110D are shown in <FIG>, the concept of <FIG> may be extended to multiple data paths, such as eight data paths as shown in <FIG> and <FIG>.

<FIG> depicts an example system for routing data paths 110A-110D through individual packet processing pipes, or pipes 260A-260D arbitrating into packet processing (PP) 262A-262B, according to one or more embodiments. Pipes 260A-260D may correspond to FIFO queues <NUM> from <FIG>. Each PP 262A-262B may include a full processing block, similar to FPB 150A.

<FIG> depicts an example system for arbitrating data paths 110A-110D through an aggregate packet processing pipe, or pipe 260E, according to one or more embodiments. As shown in <FIG>, rather than processing through independent pipes 260A-260D, a single aggregate pipe 260E is provided, which may support combined bandwidth corresponding to the sum of pipes 260A-260D. This allows pipe 260E to better handle burst traffic from any of data paths 110A-110D, helping to avoid latency and dropped packets. However, this may result in multiple packets from the same flow or data path to be processed in a single cycle by group <NUM>. To support this, data structures may be provided to enable coherent and stateful processing of packets in group <NUM>.

For example, hardware data structures may be provided such that counters, meters, elephant traps (ETRAPs) and other structures may be accessible for concurrent reads and writes across PP 262A-262B, even when processing packets from the same data path. Such hardware data structures for group <NUM> may include four <NUM> read, <NUM> write structures, or two <NUM> read, <NUM> write structures, or one <NUM> read, <NUM> write structure.

<FIG> depicts an example system combining the logical grouping of <FIG> with the aggregate packet processing pipe of <FIG>, according to one or more embodiments. As shown in <FIG>, any of data paths 110A-110D may be processed by either single packet processing 210A or 210B. For example, arbiter <NUM> as shown in <FIG> may be provided in a shared bus to arbitrate the packets into group 140A. As with group <NUM> in <FIG>, group 140A may receive packets from an aggregate pipe. Thus, group 140A may include similar hardware data structures to support coherent and stateful processing.

<FIG> depicts an example system combining the features shown in <FIG>, according to one or more embodiments. As shown in <FIG>, four data paths 110A-110D may be processed through ingress / egress packet processing <NUM> of network switch <NUM>, which may implement the features described in <FIG>. For example, referring to <FIG>, up to <NUM> packets may be processed by network switch <NUM> in a single cycle.

<FIG> is a block diagram of an example system for processing multiple packets from eight data paths 110A-<NUM> through <NUM> threads of packet processing, according to one or more embodiments. As shown in <FIG>, data paths 110A, 110B can be grouped as a first group, and data paths 110C, 110D can be grouped as a second group, where the first group and the second group can be provided to a first packet processing 262A. Similarly, data paths 110E, 110F can be grouped as a third group, and data paths <NUM>, <NUM> can be grouped as a fourth group, where the third group and the fourth group can be provided to a second packet processing 262B. In this structure, multiple packets from eight data paths 110A-<NUM> can be provided and processed through packet processing 262A, 262B. In one aspect, packet processing 262A, 262B may share logic circuits or various components to reduce area circuit area.

<FIG> is a block diagram of an example system for processing multiple packets from eight data paths 110A-<NUM> through four threads of packet processing 262A-262D, according to one or more embodiments. As shown in <FIG>, data paths 110A, 110B can be grouped and provided to a packet processing 262A through a pipe 260A, and data paths 110C, 110D can be grouped and provided to a packet processing 262B through a pipe 260B. Data paths 110E, 110F can be grouped and provided to a packet processing 262C through a pipe 260C, and data paths <NUM>, <NUM> can be grouped and provided to a packet processing 262D through a pipe 260D. The incoming data packets from data paths 110A-<NUM> may be arbitrated through a shared bus, such as shared bus 180A of <FIG>. To determine which processing block to route a data packet, an arbiter may be used, such as arbiter <NUM> of <FIG>.

In one aspect, the system shown in <FIG> can achieve high bandwidth (e.g., <NUM> TBps) with low power consumption. In one example, packet processing 262A-262D may share logic circuits or various components to reduce area circuit area. For example, multiples or combinations of systems shown in <FIG> can be implemented to achieve the same bandwidth (e.g., <NUM> TBps) as the system shown in <FIG>, but may consume a larger power or may be implemented in a larger area than the system shown in <FIG>.

<FIG> is a block diagram of an arbiter <NUM> providing synchronized idle slots, according to the invention. While the arbiter <NUM> is shown to include two input interfaces 330A, 330B and two output interfaces 332A, 332B, it should be understood that the number of interfaces can be scaled according to the bus arbitration requirements, e.g. as in shared bus 180A and 180B. Thus, shared bus 180A and 180B includes a respective arbiter <NUM>. Arbiter <NUM> receives packets from multiple data paths, or interfaces 330A and 330B. Arbiter <NUM> is therefore used to arbitrate multiple data paths through a single, shared bus for improved interconnect bandwidth utilization. Based on packet size arbitration rules and packet spacing rules defined in an arbitration policy, arbiter <NUM> may output packets for processing via interfaces 332A and 332B, which may further connect to packet processing blocks. The packet spacing rules may be enforced on a per-group basis. For example, the packet spacing rules may enforce a minimum spacing between certain packets according to data dependency, traffic management, pipelining rules, or other factors. For example, to reduce circuit complexity and power consumption, pipelines may be simplified to support successive commands of a particular type, e.g. table initialization commands, only after a full pipeline is completed, e.g. <NUM> cycles. Thus, when such a table initialization command is encountered, the packet spacing rules may enforce a minimum spacing of <NUM> cycles before another table initialization command can be processed. The arbitration policy may also enforce assignment of data paths to certain interfaces, which may allow table access structures to be implemented in a simplified manner, e.g. by reducing multiplexer and de-multiplexer lines.

When no packets are to be processed in a group, such as during idle slots 334A, 334B, and 334C, arbiter <NUM> may output ancillary or auxiliary commands received from command input <NUM>, which may be received from a centralized control circuit. For example, the ancillary commands may perform bookkeeping, maintenance, diagnostics, warm boot, hardware learn, power control, packet spacing, and other functions outside of the normal packet processing functionality.

<FIG> is a block diagram of an example system <NUM> for processing multiple packets from multiple paths with one or more schedulers, according to one or more embodiments. In some embodiments, the system <NUM> can be a part of the shared bus 180A or the system shown in <FIG>. In some embodiments, the system <NUM> includes schedulers 410A-<NUM>, event FIFOs 420A-<NUM>, read control circuit 430A, 430B and arbiters 350A, 350B. These components may be embodied as a field programmable gate array (FPGA), application specific integrated circuit (ASIC), one or more logic circuits, or any combination of them. These components may operate together to route packets or data streams from data paths 110A-<NUM> to packet processing 262A-262D based on synchronized idle slots (e.g., idle slots <NUM>). In one aspect, the system <NUM> includes a first pipe 455A encompassing the read control circuit 430A and the arbiter 350A, and a second pipe 455B encompassing the read control circuit 430B and the arbiter 350B. In some embodiments, the system <NUM> includes more, fewer, or different components than shown in <FIG>.

In some embodiments, the arbiters 350A, 350B are components that route or arbitrate packets or data streams from the data paths 110A-<NUM> to packet processing 262A-262D. In one example, the arbiters 350A, 350B may operate separately or independently from each other, such that the arbiter 350A may route or arbitrate packets or data streams from the data paths 110A, 110B to packet processing 262A, 262B through outputs 495A, 495B and the arbiter 350B may route or arbitrate packets or data streams from the data paths 110C, 110D to packet processing 262C, 262D through outputs 495C, 495D. In one example, the arbiters 350A, 350B may exchange a synchronization command <NUM>, and operate together in a synchronized manner, according to the synchronization command <NUM>. For example, the arbiters 350A, 350B may provide idle slots at outputs 495A-495D simultaneously to reduce power consumption or perform other ancillary operations.

In some embodiments, the schedulers 410A-<NUM> are circuits or components to schedule the FIFOs <NUM> to provide packets. Although the schedulers 410A-<NUM> are shown as separate circuits or components, in some embodiments, the schedulers 410A-<NUM> may be embodied as a single circuit or a single component. In one aspect, each scheduler <NUM> may schedule operations for a corresponding data path <NUM>, for example, according to an instruction or command from the arbiter <NUM>. For example, each scheduler <NUM> may provide a packet <NUM> (or a start of packet) from a respective data path <NUM> to a respective event FIFO <NUM>.

In some embodiments, the event FIFOs 420A-420D are circuits or components that provide packets <NUM> to the pipe 455A or the read control circuit 430A, and the event FIFOs 420E-<NUM> are circuits or components that provide packets <NUM> to the pipe 455B or the read control circuit 430B. Each event FIFO <NUM> may be associated with a corresponding data path <NUM>. Each event FIFO <NUM> may implement a queue to provide or output packets <NUM> in the order that the packets <NUM> are received.

In some embodiments, the read control circuits 430A and 430B are circuits or components to receive packets <NUM> from event FIFOs <NUM>, and provide packets to corresponding arbiters <NUM>. For example, the read control circuit 430A receives packets <NUM> from event FIFOs 420A-420D, and provides packets to the arbiter 350A. For example, the read control circuit 430B receives packets <NUM> from event FIFOs 420E-<NUM>, and provides packets to the arbiter 350A. In one aspect, the read control circuit <NUM> may apply randomization or round robin function to provide packets from FIFOs <NUM> to the arbiter <NUM>.

In one aspect, the arbiters 350A, 350B may request idle slots. An idle slot may be a null packet, or a packet with no data. The arbiters 350A, 350B may receive a command or an instruction from a centralized control unit (or a processor) for one or more operations of a task. Examples of a task may include power saving, warm boot, hardware learning, time spacing, etc. In response to the command or instruction, the arbiter 350A may provide an idle slot request command 438A to one or more corresponding schedulers 410A-410D and the read control circuit 430A, and the arbiter 350B may provide an idle slot request command 438B to one or more corresponding schedulers 410E-<NUM> and the read control circuit 430B. In response to the idle slot request command <NUM>, the scheduler <NUM> may provide an idle slot (or packet with no data) to the read control circuit 430A to generate an idle slot. In response to the idle slot (or packet with no data) from a FIFO, the read control circuit <NUM> may provide the idle slot (or packet with no data) to the arbiter <NUM> through one or more interfaces <NUM>. In response to the idle slot request command <NUM>, the read control circuit <NUM> may bypass reading packets from corresponding FIFOs <NUM>, such that an idle slot (or packet with no data) can be provided to the arbiter <NUM> through one or more interfaces <NUM>.

In one aspect, the read control circuit <NUM> indicates or marks whether idle slots are generated in response to the idle slot request command <NUM> or not. According to the indication or mark, the arbiter <NUM> may determine that the idle slot or a packet with no data is explicitly generated in response to the idle slot request command <NUM>. Accordingly, the arbiter <NUM> may avoid erroneously responding to incidental packets with no data.

In one aspect, the system <NUM> can improve a hardware learn rate. In one aspect, the system <NUM> allows learning or detecting a certain number (e.g., over <NUM> million) of features (e.g., MAC address, hash on any number of fields, source address, source IP address, etc.) of the system <NUM> for a given time period. In one example, learning a hardware feature includes extracting a certain field in a packet received, and checking if a matching entry of a table exists. Often, data from one or more data paths (e.g., data paths 110A-<NUM>) can interfere with the hardware learning process. The arbiters 350A, 350B can enforce synchronized idle slots, such that hardware learning can be performed with less interference and a set number of features of the system <NUM> can be determined for a given time period.

In one aspect, the system <NUM> can operate in a reliable manner, despite one or more erroneous processes. An erroneous process may exist, due to a false design by an engineer, or due to a hardware failure. For example, an unintended operation may be performed, or an operation may be performed at an unintended clock cycle. Such erroneous process may render the system <NUM> unreliable or unusable. Rather than discarding the system <NUM>, the arbiters 350A, 350B can enforce idle slots for known erroneous processes. For example, the arbiters 350A, 350B may identify or determine that an instruction from a particular component is associated with processes from faulty components, and can enforce the idle slots, in response to identifying that the instruction is from a faulty component. Accordingly, erroneous processes due to such instruction may not be performed. Although the system <NUM> may intentionally not perform erroneous processes, the system <NUM> can perform other processes in a reliable manner and may not be discarded.

In one aspect, the system <NUM> can support a warm boot. In one aspect, various operations may be performed during a wake up sequence. In one example, the wake up sequence involves: resetting the chip, configuring phase locked loop, enabling IP/EP clock, bringing MMU or processors out of reset, setting program registers, accessing TCAM, etc. In one example, the arbiters 350A, 350B may receive a command or indication indicating no packet traffic. In response to the command or indication, the arbiters 350A, 350B may ignore or bypass a packet spacing rule, and process data to support the wake up sequence, because there may be no data traffic from the data paths (or data paths 110A-<NUM>). By ignoring or bypassing the packet spacing rule or other rules associated with data traffic, the system <NUM> can perform a rigorous wake up sequence within a short time period (e.g., <NUM>).

In one aspect, the system <NUM> can achieve power savings by implementing idle slots. In one example, the system <NUM> can detect or monitor power consumption of the system <NUM>. For example, the system <NUM> may include a power detector that detects or monitors power consumption of the system <NUM>. In response to the power consumption exceeding a threshold value or threshold amount, the power detector or a centralized control circuit can provide an instruction or a command to the arbiters 350A, 350B to reduce power consumption. In response to the instruction or command provided, the arbiters 350A, 350B may enforce idle slots. By enforcing idle slots, the arbiters 350A, 350B or other components may not process data, such that power consumption can be reduced.

In one aspect, the system <NUM> can support various operation modes or operating conditions. In one example, two arbiters 350A, 350B of two pipes (e.g., pipe 455A, 455B) can provide data packets simultaneously at outputs 495A, 495B, 495C, 495D. In one example, the first arbiter 350A of the pipe 455A can provide data packets at outputs 495A, 495B, while the second arbiter 350B of the pipe 455B can support ancillary operations, which may access macros shared within the pipe 455B. In one example, the first arbiter 350A of the pipe 455A can provide idle slots at outputs 495A, 495B, while the second arbiter 350B of the pipe 455B can support ancillary operations, which may access macros shared across the pipes 455A, 455B.

<FIG> show example waveforms for generating synchronized null slots, according to one or more embodiments. In the example shown in <FIG>, the arbiter <NUM> may generate idle slot request command <NUM> requesting idle slots for a zeroth clock cycle, a second clock cycle, a third clock cycle, a seventh clock cycle, and an eighth clock cycle. According to the idle slot request command <NUM>, the arbiter <NUM> may provide or enforce idle slots at the requested clock cycles. In one example, a centralized control circuit (or processor) may provide an instruction or command with respect to a particular clock cycle, and request to generate one or more idle slots for other clock cycles with respect to the particular clock cycle. For example, centralized control circuit may provide an instruction or command with respect to a third clock cycle, and may also indicate to generate idle slots for three and one clock cycles before the third clock cycle, and four and five clock cycles after the third clock cycle. In response to the command or the instruction, the arbiter <NUM> may generate the idle slot request command <NUM> to cause the scheduler <NUM> and the read control circuit <NUM> to provide idle slots at corresponding clock cycle (e.g., zeroth clock cycle, a second clock cycle, a third clock cycle, a seventh clock cycle, and an eighth clock cycle). Advantageously, the arbiter <NUM> may provide multiple idle slots for a single instruction or command (e.g., an instruction or command provided in response to an erroneous request or associated with an erroneous request). In one example, an erroneous request from a known source (e.g., processor) due to false design or errors can be bypassed, according to the single instruction or command causing idle slots for multiple clock cycles.

<FIG> is a block diagram of a circuit <NUM> to provide different clocks for a scheduler <NUM>, according to one or more embodiments. In one aspect, the circuit <NUM> is included in the system <NUM> or coupled to the system <NUM>. The circuit <NUM> may provide adaptive clock signals CLK_OUT1, CLK_OUT2 to the schedulers 410A-<NUM>. In some embodiments, the circuit <NUM> includes FIFOs 650A, 650B.

The FIFO 650A may receive a clock control signal CLK_CTRL1, for example, from the arbiter 350A. In response to the clock control signal CLK_CTRL1, the FIFO 650A circuit may provide a selected one of a data path clock signal DP_CLK or a packet processing clock signal PP_CLK to corresponding schedulers <NUM> (e.g., schedulers 410A-410D) as a clock output CLK_OUT1, according to the clock control signal CLK_CTRL1. The data path clock signal DP_CLK may be a clock signal of a data path <NUM>, and the packet processing clock signal PP_CLK may be a clock signal of a packet processing <NUM>.

Similarly, the FIFO 650B may receive a clock control signal CLK_CTRL2, for example, from the arbiter 350B. In response to the clock control signal CLK_CTRL2, the FIFO 650B circuit may provide a selected one of the data path clock signal DP_CLK or the packet processing clock signal PP_CLK to corresponding schedulers <NUM> (e.g., schedulers 410E-<NUM>) as a clock output CLK_OUT2, according to the clock control signal CLK_CTRL2.

In one aspect, the arbiters 350A, 350B may provide clock control signals CLK_CTRL1, CLK_CTRL2, to allow the schedulers <NUM> to adaptively operate. In some cases, a frequency of the data path clock signal DP_CLK may be higher than a frequency of a packet processing clock signal PP_CLK. In some cases, a frequency of the data path clock signal DP_CLK may be lower than the frequency of the packet processing clock signal PP_CLK. The circuit <NUM> can be configured, such that one of the data path clock signal DP_CLK and the packet processing clock signal PP_CL having a higher frequency can be provided to the schedulers <NUM> as clock outputs CLK_OUT1, CLK_OUT2. By selectively providing the clock outputs CLK_OUT1, CLK_OUT2, the system <NUM> can support operations in different modes or configurations with different clock frequencies of the data path clock signal DP_CLK and the packet processing clock signal PP_CLK.

<FIG> is a flow chart of a process <NUM> to schedule synchronized idle slots, according to one or more embodiments. In some embodiments, the process <NUM> is performed by a network system (e.g., system <NUM> shown in <FIG> or other systems shown in <FIG>, <FIG>, <FIG>). In some embodiments, the process <NUM> is performed by other entities. In some embodiments, the process <NUM> includes more, fewer, or different steps than shown in <FIG>.

In one approach, an arbiter <NUM> receives <NUM> a request to perform one or more operations of a task. The task may be performed or scheduled to be performed during a clock cycle. Examples of a task may include power saving, hardware learning, time spacing, etc. The request may be generated by a centralized control unit (or a processor).

In one approach, the arbiter <NUM> generates <NUM> a command for a scheduler <NUM>, based on the request. For example, the arbiter <NUM> generates an idle slot request command <NUM>. The arbiter <NUM> provides the idle slot request command <NUM> to the scheduler <NUM> and/or the read control circuit <NUM>.

In one approach, the scheduler <NUM> schedules <NUM> a first idle slot for a first data path (e.g., data path 110A), and schedules <NUM> a second idle slot for a second data path (e.g., data path 110B). For example, in response to the idle slot request command <NUM>, the scheduler 410A may generate a first idle slot or a packet with no data according to the schedule for the first data path, and provide the first idle slot or packet with no data to an event FIFO 420A. For example, in response to the idle slot request command <NUM>, the scheduler 410B may generate a second idle slot or a packet with no data according to the schedule for the second data path, and provide the second idle slot or packet with no data to an event FIFO 420B.

In one approach, the arbiter <NUM> provides <NUM> the first idle slot and the second idle slot during the time slot. For example, the read control circuit 430A may receive the idle slots or packets with no data from the FIFOs 420A, 420B, and provide the idle slots to the arbiter 350A during the clock cycle. In one example, the read control circuit <NUM> may receive an idle slot request command <NUM> from the arbiter <NUM>, and bypass reading packets from corresponding FIFOs <NUM>, in response to the idle slot request command <NUM>. By bypass reading packets from corresponding FIFOs <NUM>, idle slots (or packets with no data) can be provided to the arbiter <NUM>. The arbiter <NUM> may provide the first idle slot and the second idle slot from the read control circuit <NUM> at its outputs. By providing the synchronized idle slots as disclosed herein, various operations of the task can be supported.

<FIG> is a flow chart of a process <NUM> to reduce power consumption by scheduling idle slots, according to one or more embodiments. In some embodiments, the process <NUM> is performed by a network system (e.g., system <NUM> shown in <FIG> or other systems shown in <FIG>, <FIG>, <FIG>). In some embodiments, the process <NUM> is performed by other entities. In some embodiments, the process <NUM> includes more, fewer, or different steps than shown in <FIG>.

In one approach, the system <NUM> monitors <NUM> power consumption of the system <NUM>. For example, the system <NUM> may include a power detector that detects or monitors power consumption of the system <NUM>.

In one approach, the system <NUM> determines <NUM> whether the power consumption of the system is larger than a threshold value or a threshold amount. If the detected power consumption is less than the threshold value, the system <NUM> may proceed to the step <NUM>.

If the detected power consumption is larger than the threshold value, the system <NUM> may proceed to the step <NUM>. For example, the arbiter <NUM> may enforce idle slots, in response to determining that the power consumption exceeding the threshold value. The arbiter <NUM> may cause the scheduler <NUM> to schedule idle slots for a predetermined number of clock cycles. By enforcing idle slots, the arbiter <NUM> or other components may not process data, such that power consumption of the system <NUM> can be reduced. After the predetermined number of clock cycles, the process <NUM> may proceed to the step <NUM>.

<FIG> is a flow chart of a process <NUM> to synchronize operations of two arbiters to prevent a packet collision, according to one or more embodiments. The process <NUM> is performed by a network system (e.g., system <NUM> shown in <FIG> or other systems shown in <FIG>, <FIG>, <FIG>). In some embodiments, the process <NUM> includes more, fewer, or different steps than shown in <FIG>.

In one approach, a processor (e.g., processor or a centralized control circuit of the system <NUM>) determines <NUM> to support or provide a packet collision avoid mode. The processor may determine to support or provide the packet collision avoid mode, in response to a user instruction or in response to detecting that a packet collision rate has exceeded a predetermined threshold.

In one approach, the processor selects <NUM> the first arbiter 350A. In one example, the processor may select the first arbiter 350A to provide a first data packet, based on a priority, where the master arbiter 350A may have a higher priority than the slave arbiter 350B. In one example, the processor may select the first arbiter 350A, in response to the data path 110A associated with the first arbiter 350A receiving a packet before data paths 110E-<NUM> associate with the second arbiter 350B.

In one approach, the processor causes the first arbiter 350A to provide <NUM> the first data packet from the data path 110A during a first clock cycle, while the second arbiter 350B provides idle slots. For the example, the processor may generate a command to cause the first arbiter 350A and the second arbiter 350B to synchronize with each other through the synchronization command <NUM>. In addition, the processor may generate a command to cause the first arbiter 350A to provide the first data packet from the data path 110A at an output 495A and to provide a no data packet at an output 495B during the first clock cycle. The processor may also generate a command to cause the second arbiter 350B to provide or enforce idle slots at its outputs 495C, 495D during the first clock cycle.

In one approach, after providing the first packet, the processor selects <NUM> the second arbiter 350B, and causes the second arbiter 350B to provide <NUM> a second data packet from the data path 110E during a second clock cycle, while the first arbiter 350A provides idle slots. For example, the processor may generate a command to cause the arbiter 350B to provide the second data packet from the data path 110E at an output 495C and to provide a no data packet at an output 495D during the second clock cycle. The processor may also generate a command to cause the arbiter 350A to provide or enforce idle slots at its outputs 495A, 495B during the second clock cycle.

Accordingly, the arbiters 350A, 350B may operate in a synchronized manner to avoid a packet collision. By avoiding packet collisions, power consumption of the system <NUM> can achieve lower power consumption and higher throughput.

Many aspects of the above-described example process <NUM>-<NUM>, and related features and applications, may also be implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium), and may be executed automatically (e.g., without user intervention). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

The term "software" is meant to include, where appropriate, firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some implementations, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while remaining distinct software aspects of the subject disclosure. In some implementations, multiple software aspects can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software aspect described here is within the scope of the subject disclosure. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

<FIG> illustrates an electronic system <NUM> with which one or more implementations of the subject technology may be implemented. The electronic system <NUM> can be, and/or can be a part of, the network switch <NUM> shown in <FIG>. The electronic system <NUM> may include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system <NUM> includes a bus <NUM>, one or more processing unit(s) <NUM>, a system memory <NUM> (and/or buffer), a ROM <NUM>, a permanent storage device <NUM>, an input device interface <NUM>, an output device interface <NUM>, and one or more network interfaces <NUM>, or subsets and variations thereof.

Finally, as shown in <FIG>, the bus <NUM> also couples the electronic system <NUM> to one or more networks and/or to one or more network nodes, through the one or more network interface(s) <NUM>. In this manner, the electronic system <NUM> can be a part of a network of computers (such as a LAN, a wide area network ("WAN"), or an Intranet, or a network of networks, such as the Internet. Any or all components of the electronic system <NUM> can be used in conjunction with the subject disclosure.

Claim 1:
A method comprising:
receiving, by an arbiter (<NUM><NUM><NUM>) of a network system to arbitrate a first set of packets from a first data path (<NUM>) and a second set of packets from a second data path (<NUM>), a request for a task, the task to be performed during a clock cycle;
generating, by the arbiter (<NUM><NUM><NUM>) based on the request, a command to cause a scheduler (<NUM>) of the network system to:
schedule a first idle slot (<NUM>) for the first data path (<NUM>), path, and
schedule a second idle slot (<NUM>) for the second data path (<NUM>); and
providing, by the arbiter, the first idle slot and the second idle slot during the clock cycle.