Speculative loading of buffers within a port of a network device

In one embodiment a method for processing an incoming packet in a port of an interconnect device includes speculatively forwarding the incoming packet to multiple buffers within the port prior to determining which of the multiple buffers is a target buffer for the packet, decoding the packet, and determining which of the multiple buffers is the target buffer for the packet using the decoded packet. The method further includes notifying an agent associated with the target buffer that the target buffer is an intended owner of the data packet. In one embodiment, agents associated with the multiple buffers are designated to process packets that are not subject to a credit-based flow control method.

FIELD OF THE INVENTION

The present invention relates generally to the field of data communications and, more specifically, to processing incoming packets in a port of an interconnect device using speculative loading of buffers within the port.

BACKGROUND OF THE INVENTION

Existing networking and interconnect technologies have failed to keep pace with the development of computer systems, resulting in increased burdens being imposed upon data servers, application processing and enterprise computing. This problem has been exasperated by the popular success of the Internet. A number of computing technologies implemented to meet computing demands (e.g., clustering, fail-safe and 24×7 availability) require increased capacity to move data between processing nodes (e.g., servers), as well as within a processing node between, for example, a Central Processing Unit (CPU) and Input/Output (I/O) devices.

With a view to meeting the above described challenges, a new interconnect technology, called the InfiniBand™, has been proposed for interconnecting processing nodes and I/O nodes to form a System Area Network (SAN). This architecture has been designed to be independent of a host Operating System (OS) and processor platform. The infiniBand™ Architecture (IBA) is centered around a point-to-point, switched IP fabric whereby end node devices (e.g., inexpensive I/O devices such as a single chip SCSI or Ethernet adapter, or a complex computer system) may be interconnected utilizing a cascade of switch devices. The InfiniBand™ Architecture is defined in the InfiniBand™ Architecture Specification Volume 1, Release 1.0, released Oct. 24, 2000 by the InfiniBand Trade Association. The IBA supports a range of applications ranging from back plane interconnect of a single host, to complex system area networks, as illustrated inFIG. 1(prior art). In a single host environment, each IBA switched fabric may serve as a private I/O interconnect for the host providing connectivity between a CPU and a number of I/O modules. When deployed to support a complex system area network, multiple IBA switch fabrics may be utilized to interconnect numerous hosts and various I/O units.

Within a switch fabric supporting a System Area Network, such as that shown inFIG. 1, there may be a number of devices having multiple input and output ports through which data (e.g., packets) is directed from a source device to a destination device. Such devices include, for example, switches, routers, repeaters and adapters (exemplary interconnect devices). In addition to multiple communication ports directing external data packets, an interconnect device such as a switch typically includes a management port which handles InfiniBand™ management packets. Management packets are used to implement management functions and may include Sub-Network Management Packets, Performance Management Packets, and Baseboard Management Packets. Further details regarding various management packets are provided in the InfiniBand™ Architecture Specification, Volume 1, Oct. 24, 2000.

Processing of management packets requires additional resources and bandwidth, thereby affecting performance of the interconnect device. Accordingly, it is important to process management packets in an efficient manner.

SUMMARY OF THE INVENTION

Methods and systems for processing an incoming packet in a port of an interconnect device are described. According to one aspect of the present invention, an exemplary method includes speculatively forwarding an incoming packet to multiple buffers within the port prior to determining which of the multiple buffers is a target buffer for the packet, decoding the packet, and determining which of the multiple buffers is the target buffer for the packet using the decoded packet. The method further includes notifying an agent associated with the target buffer that the target buffer is an intended owner of the data packet. In one embodiment, agents associated with the multiple buffers are designated to process packets that are not subject to a credit-based flow control method.

According to another aspect of the present invention, an exemplary method includes speculatively forwarding an incoming packet to each buffer from a first group of buffers and to an intermediary buffer associated with a second group of buffers prior to determining which buffer from the first second groups of buffers is a target buffer for the packet, decoding the packet, and determining which buffer from the first and second groups of buffers is the target buffer for the packet using the decoded packet. The method further includes notifying an agent associated with the target buffer that the target buffer is an intended owner of the data packet. In one embodiment, agents associated with buffers from the first group are designated to process packets that are not subject to a credit-based flow control method, and agents associated with buffers from the second group is designated to process packets that are subject to the credit-based flow control method. A flow controller associated with the intermediary buffer is responsible for managing credits for packets received by the second group of buffers. In one embodiment, when the target buffer is from the second group of buffers, the agent associated with the target buffer obtains the packet from the intermediary buffer.

DETAILED DESCRIPTION

Methods and systems to process incoming packets in a port of an interconnect device are described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without these specific details.

For the purposes of the present invention, the term “interconnect device” shall be taken to include switches, routers, repeaters, adapters, or any other device that provides interconnect functionality between nodes. Such interconnect functionality may be, for example, module-to-module or chassis-to-chassis interconnect functionality. While an exemplary embodiment of the present invention is described below as being implemented within a switch deployed within an InfiniBand architectured system, the teachings of the present invention may be applied to any interconnect device within any interconnect architecture.

FIGS. 2A and 2Bprovide a diagrammatic representation of a datapath20, according to a prior art embodiment, implemented within an interconnect device (e.g., a switch). The datapath20is shown to include a crossbar22that includes ten 36-bit data buses30, a 66-bit request bus32and a 64-bit grant bus34. Coupled to the crossbar are eight communication ports24that issue resource requests to an arbiter36via the request bus32, and that receive resource grants from the arbiter36via the grant bus34.

The arbiter36implements a central arbitration scheme within the datapath20, in that all requests and resource information are brought to a single location (i.e., the arbiter36). The arbiter36includes a request preprocessor38to receive resource requests from the request bus32and to generate a modified resource request42to a resource allocator40. The resource allocator40then issues a resource grant on the grant bus34. Specifically, when the request preprocessor38receives a request, the request's destination address is used to find an output port number for the request in a unicast and multicast routing tables (not shown). Based on the output port number and an input port identifier, a lookup in a virtual lane map (not shown) is performed to determine a virtual lane (VL) for this request. VLs are, in one embodiment, independent data streams that are supported by a common physical link. A VL may represent a set of transmit and receive buffers in a port. VL15is reserved exclusively for sub-network management packets (SMPs). There are 15 other VLs (VL0through VL14) referred to as data VLs. All ports support VL15and at least one data VL. Data VLs are subject to a credit-based flow control mechanism which is used to prevent the loss of packets due to buffer overflow by the receiver at each end of the link. Flow control is applied to each VL independently, except for VL15which is not subject to flow control. Further details regarding the concept of “virtual lanes” are provided in the InfiniBand™ Architecture Specification, Volume 1, Oct. 24, 2000.

A modified request42including such additional information as an output port identifier and a VL identifier is received at the resource allocator40from the request preprocessor38. If sufficient flow control credits for a virtual lane identified by the virtual lane identifier of the request are available and an output port identified by the output port identifier132of the request is available, then both the virtual lane and output port identified within the relevant request are allocated to the request by the resource allocator40. If either of the above entity is unavailable, the request is put on the back of a corresponding request queue. As flow control credits and output ports become available, the resource allocator36chooses among pending requests in the queues and issues a grant64, on the grant bus34.

In addition to the eight communication ports, a management port26and a functional Built-In-Self-Test (BIST) port28are also coupled to the crossbar22. The functional BIST port28supports stand-alone, at-speed testing of an interconnect device embodying the datapath20; The functional BIST port28includes a random packet generator, a directed packet buffer and a return packet checker.

The management port26includes a Sub-Network Management Agent (SMA)74that is responsible for network configuration, a Performance Management Agent (PMA)72that maintains error and performance counters, and a Baseboard Management Agent (BMA)70that monitors environmental controls and status. Each of these agents includes one or more buffers to store packets processed by the corresponding agents. Specifically, the SMA74may include asset of VL15buffers, and the PMA72and BMA70may include VL0buffers. When the arbiter36grants a request to a packet having the management port26as a destination, the packet is received by a grant controller76via the grant bus34. Then, according to a conventional approach, the grant controller76forwards the packet to a temporary buffer (not shown) where the packet is stored while a packet decoder78parses and decodes the packet. Once the parsing and decoding activities are completed, the packet is transmitted to a buffer of the rightful target agent for execution. This approach, however, delays the execution of the packet and fails to take into account the nature of the InfiniBand VL15architecture as will be described in greater detail below.

FIG. 3is a block diagram illustrating a system300for processing an incoming packet in a port of an interconnect device, according to one embodiment of the present invention. System300is located in a port of an interconnect device and includes a set of agents302that are responsible for processing packets received at the port. Each agent302includes one or more buffers304to store packets being processed by the agents. Agents302process packets that are not subject to a specific flow control mechanism (e.g., a credit-based flow control mechanism) utilized to prevent the loss of packets due to buffer overflow. Instead, the overflow of buffers304is controlled by discarding a packet when the packet transmitted to the port does not allow enough room for reception. In one embodiment, this port is an InfiniBand management port, buffers304are InfiniBand VL15buffers, and agents302include a set of SMAs and a processor bus interface. Alternatively, the mechanism of the present invention can be used in any other port if this port includes two or more agents responsible for processing incoming packets which are architecturally allowed to be discarded when sent without sufficient room for reception.

When a packet is received at the port, it is speculatively forwarded to all buffers304before the packet is parsed and decoded by a packet decoder306. The packet can be forwarded to all buffers in parallel or in any other sequence. In one embodiment, the packet is received from an arbiter by a grant controller via a grant bus. Alternatively, the packet may be received from a port of this or other interconnect device or endnode by a set of registers (e.g., Serializer-Deserializer circuits) via a link. In one embodiment, the packet is forwarded to the buffers304through the packet decoder306. In another embodiment, the packet is forwarded to the packet decoder306and the buffers304in parallel or in any other sequence.

When the packet decoder306determines a target buffer from the decoded packet, it notifies an agent associated with the target buffer. This target agent then begins processing the packet. In one embodiment, the packet decoder306sends an indicator to all agents302notifying the agents302whether their corresponding buffers304are rightful owners of the packet. Alternatively, the packet decoder306only sends an indicator of rightful ownership to the target agent. Accordingly, the speculative load approach of the present invention saves the time and effort required for loading the packet from a temporary buffer to a target buffer subsequent to decoding, when a prior art method described above is used.

In one embodiment, the size of each buffer304is between the size of one incoming packet and the size of two incoming packets, i.e., the buffer304can only fit one full packet. In this embodiment, when the target agent receives the indicator of rightful ownership from the packet decoder306, the target agent blocks the loading of subsequent packets into the target buffer. That is, the next packet cannot be speculatively loaded into the target buffer until processing of the current packet is completed. As a result, if the target buffer is a rightful owner of the next packet which is sent while the processing of the current packet has not been completed, the next packet is discarded and will not be processed. This outcome is desirable because it prevents the target buffer from overflowing. Thus, the speculative approach of the present invention can be used as a flow control mechanism with respect to packets received by the buffers304. In one embodiment, the target agent does not resume the buffer load until ensuring that no partial buffer load can occur.

In another embodiment, in which each buffer304can fit two full packets, the next packet can be speculatively loaded into the target buffer before processing of the current packet is completed. If the same agent is a rightful owner of both the current packet and the next packet, blocking of the target buffer occurs when the target agent receives an indicator of rightful ownership with respect to the next packet and continues until the current packet is processed entirely.

FIG. 4is a block diagram illustrating another embodiment of a system for processing incoming packets in a port of an interconnect device. According to this embodiment, system400includes one or more agents302and one or more agents402. Agents302process incoming packets that are not subject to a flow control mechanism (e.g., a credit-based flow control mechanism). Agents402process incoming packets that are subject to a flow control mechanism such as a credit-based flow control mechanism. A flow controller408is responsible for managing credits for packets processed by agents402.

Each agent302includes one or more buffers304, and each agent402includes one or more buffers404. All buffers304are referred to as a first group412of buffers. Similarly, all buffers404are referred to as a second group414of buffers. The flow controller408includes an intermediary buffer406. In one embodiment, the port where system400resides is an InfiniBand management port, buffers304are VL15buffers and buffers404are non-VL15(e.g., VL0) buffers. Alternatively, system400may reside in any other port that includes one or more buffers for processing packets that are not subject to flow control and one or more buffers that are subject to flow control.

When a packet is received at the port, it is forwarded to a packet decoder410, to the intermediary buffer406and speculatively to each buffer304from the first group412. Subsequently, when the packet decoder410completes the packet parsing and decoding operations, it notifies a target agent that its corresponding buffer is a rightful owner of the packet. In one embodiment, the packet decoder410notifies each agent302,402and flow controller408whether their corresponding buffers are rightful owners of the packet. Only one (or none) buffer can be a target buffer for the packet. Alternatively, the packet decoder410notifies only the target agent. If the target buffer is from the second group414, the packet decoder410may also send an indicator of rightful ownership to the flow controller408.

In one embodiment, the target buffer is from the first group412. In this embodiment, the target agent blocks the loading of subsequent packets into the target buffer upon receiving the indicator of rightful ownership. In one embodiment, the target agent does not resume the buffer load until ensuring that no partial buffer load can occur.

In another embodiment, the target buffer is from the second group414. In this embodiment, the target agent obtains the packet from the intermediary buffer406upon receiving the indicator of rightful ownership. In response, the flow controller408returns the credits associated with the packet to the arbiter. In one embodiment, the target agent receives an indicator of rightful ownership with respect to the current packet while processing a prior packet. Then, the target agent sets a pending indicator to serve as a reminder that the target agent needs to obtain the current packet from the intermediary buffer406, after completing the processing of the prior packet.

In yet another embodiment, the packet decoder410determines that the target buffer is from the second group414but the packet is invalid (e.g., has invalid format). In this embodiment, the packet decoder410notifies the flow controller408that the packet is invalid. The flow controller408then returns the credits associated with this packet to the arbiter.

FIGS. 5 and 6are block diagrams of a method for processing an incoming packet in a port of an interconnect device, according to two exemplary embodiments of the present invention.

Referring toFIG. 5, method500begins with speculatively (and prior to decoding to determine an actual target buffer) forwarding the packet to a plurality of buffers within a port of an interconnect device (processing block504). In one embodiment, each buffer is associated with an agent that is responsible for processing certain packets received at the port. In one embodiment, the packets processed by these agents are not subject to a credit-based flow control method or any other similar flow control mechanism. In one embodiment, the port is an InfiniBand management port, the buffers are VL15buffers, and the agents associated with the buffers can include a set of SMAs and a processor bus interface.

At processing block506, the packet is decoded. Based on the information in the decoded packet, it is determined which of the plurality of buffers is a target buffer for this packet (processing block508). Further, at processing block510, a target agent associated with the target buffer is notified that the target buffer is an intended owner of the packet. In one embodiment, an indicator of rightful ownership is sent only to the target agent. Alternatively, each agent is notified whether the packet resides in the corresponding buffer properly or not.

In one embodiment, upon receiving the indicator of rightful ownership, the target agent blocks the buffer loading until the processing of the current packet is completed. As a result, the next packet cannot be speculatively loaded into the target buffer while the current packet is being processed. In one embodiment, the target agent does not resume the buffer loading until ensuring that no partial buffer load can occur.

Referring toFIG. 6, method600begins with speculatively (and prior to decoding to determine an actual target buffer) forwarding a packet received at the port to each buffer from a first group of buffers and to an intermediary buffer associated with a second group of buffers (processing block604). In one embodiment, agents associated with the first group of buffers process packets that are not subject to a credit-based flow control method or any other similar flow control method. Agents associated with the second group of buffers process packets that are subject to flow control such as a credit-based flow control method. In one embodiment, the port is an InfiniBand management port, the first group of buffers includes one or more VL15buffers and the second group of buffers include two or more non-VL15(VL0) buffers. In this embodiment, the agents associated with the first group of buffers may include a set of SMAs and a processor bus interface, and the agents associated with the second group of buffers may include a PMA, a BMA and the processor bus interface.

At processing block606, the packet is decoded. Based on the information in the decoded packet, it is determined which buffer from the first and second groups of buffers is a target buffer for this packet (processing block608). At processing block610, an agent associated with the target buffer is notified that the target buffer is an intended owner of the packet. In one embodiment, each agent (i.e., agents associated with the first and second groups of buffers and a flow controller associated with the intermediary buffer) is notified whether its corresponding buffer is a rightful owner of the packet. Alternatively, only the target agent is notified that the target buffer is a rightful owner of the packet. In one embodiment, when the target buffer is from the second group of buffers, an indicator of rightful ownership may also be sent to the flow controller.

In one embodiment, the target buffer is from the first group of buffers. In this embodiment, upon receiving the indicator of rightful ownership, the target agent blocks the buffer loading until the processing of the current packet is completed.

In another embodiment, the target buffer is from the second group of buffers. In this embodiment, upon receiving the indicator of rightful ownership, the target agent obtains the packet from the intermediary buffer unless a pending indicator associated with the target buffer is set. A pending indicator is set when the indicator of rightful ownership is sent to the target agent before the processing of the prior packet has completed. When the processing of the prior agent is completed, the target agent resets the pending indicator and obtains the current packet from the intermediary buffer. This causes the flow controller to return the credits associated with the current packet to the arbiter.

FIG. 7is a block diagram illustrating a system700for processing an incoming packet in a management port of a switch, according to an exemplary embodiment of the present invention. System700includes a set of agents with corresponding buffers. Specifically, an SMA708includes a buffer710, an SMA Passthrough712includes a buffer714, a processor bus interface716includes buffers718and720, a BMA722includes a buffer724, and a PMA726includes a buffer728. Buffers710,714and718are used for VL15packets. Buffers720,724and728are used for VL0packets. A flow controller705manages the credits for VL0packets. The flow controller705includes a flow control buffer706to store packets before transferring them to buffers720,724or728.

When a grant controller702receives a packet via a grant bus, it forwards the packet to buffers706,710,714and718via a packet decoder704. The packet decoder704is responsible for parsing and decoding the packet to find a target buffer for this packet. Various fields in the packet may be used to identify the packet's target buffer depending on the packet format.FIG. 9illustrates a format of an exemplary packet that is decoded by the packet decoder704according to one embodiment of the present invention.

Referring toFIG. 9, fields relevant to determining a target buffer may include a destination address904, a virtual lane (VL) identifier902, a destination queue pair (DQP) identifier906, a packet version identifier908, a management class910, a method912, an attribute identifier914, and an attribute modifier916. The positions of each field within the packets is provided in bits words. When there are two numbers, the number in parenthesis is given for a packet without a global router header (GRH), and the other number is given for a packet that does not include a GRH.

The destination address904or Destination Local Identifier (DLID) identifies a destination port. The VL identifier902specifies whether this packet is a VL15packet or non-VL15(e.g., VL0) packet. The DQP identifier906identifies a target destination queue pair. A queue pair is used to queue up a set of instructions or data that the hardware executes. A queue pair consists of a queue for send operations and a queue for receive operations. VL15packets must use queue pair0and non-VL15packets can use any other queue pairs except queue pair0. Further details regarding the concept of “queue pairs” are provided in the InfiniBand™ Architecture Specification, Volume 1, Oct. 24, 2000.

The packet version identifier908defines the version of the architecture that can be handled by a management agent implemented in hardware. If the packet cannot be handled by any hardware-implemented management agents, a target agent for the packet is the processor bus interface716. If the packet is a VL15packet, its target buffer is buffer718. Alternatively, if the packet can be handled by a hardware-implemented management agent, the packet's target buffer is buffer710or714. The management class910identifies a particular agent that should process the packet. If an agent identified by the identifier910does not exist in the management port, the packet should be forwarded to the processor bus interface716. In one embodiment, all hardware agents are disabled and their functions are performed in software. In this embodiment, the processor bus interface716sends signals to the packet decoder704delivering this information. The packet decoder704will then use either VL15buffer718or VL0buffer720as a target buffer for the packets being decoded.

The method912identifies an operation (e.g., read or write) requested by the packet. The attribute identifier914and the attribute modifier916are used to identify the location of the requested operation.

It should be noted that various other fields in the packet can be used to extract the information required by the system700. In addition, incoming packets may have a variety of other formats and fields that the packet decoder704may use to extract the required information. Further, several fields not described above may be used to determine whether the packet being decoded is valid.

Returning toFIG. 7, the packet decoder704determines whether the packet is valid and which buffer is a target buffer, based on the decoded information such as the information described above in conjunction with FIG.9. Upon making this determination, in one embodiment, the packet decoder704sends indicators to the flow controller705and the agents708,712,716,722and726. Each indicator may consist of two bits. One bit may be designated to specify whether the corresponding buffer is a rightful owner of the packet and the other bit may be designated to specify whether the packet is invalid. In another embodiment, the indicator is sent only to the target agent and, when the target buffer is one of the buffers720,724and728, to the flow controller705.

If the packet is valid and a target buffer is one of the buffer710,714and718, a corresponding agent708,712or716blocks the loading of its buffer and begins processing the packet. When the packet is processed, the loading is unblocked. Accordingly, the next packet cannot be speculatively loaded to the target buffer unless the packet stored in the target buffer is processed. If the target buffer is a rightful owner of the next packet as well, this next packet will be discarded. However, this situation satisfies the IBA requirements for VL15packets. According to IBA, the VL15packets should not generate heavy traffic. Instead, one of the IBA requirements is to discard a VL15packet when it is sent without providing enough room for reception. Thus, the speculative load approach compliments the nature of the InfiniBand Virtual Lane15Architecture.

In one embodiment, the agent should ensure that no partial buffer load occurs, by starting the buffer load at the beginning of the packet.

If the packet is invalid and a target buffer is one from the group of buffers720,724and728, the flow controller705returns the credits associated with the packet to the arbiter. If the packet is valid and a target buffer is one from the group of buffers720,724and728, then the target agent verifies that a corresponding pending indicator is not set and reads the packet from the flow control buffer706. If the pending indicator is set, the target agent does not begin reading the packet from the flow control buffer706until completing the execution of a prior packet. Once the data transfer from the flow control buffer706to the target agent starts, the flow controller705begins returning credits associated with the packet to the arbiter.

FIGS. 8A-8Care flow diagrams of a method800for processing an incoming packet in a management port of a switch, according to an exemplary embodiment of the present invention. Method800begins with speculatively forwarding the incoming packet to all VL15buffers (e.g., a SMA buffer, a SMA Passthrough buffer, and a processor interface VL15buffer) and a flow control buffer (processing block804).

At processing block806, the packet is decoded. At processing block808, the information in the decoded packet is used to set an indicator for each of the VL15buffers, VL0buffers (e.g., a processor interface VL0buffer, a PMA, and a BMA) and the flow control buffer. Each indicator specifies whether the packet is valid and whether the corresponding buffer is a target buffer for the packet.

If the packet is invalid (decision box810) and the packet is a non-VL15packet (e.g., a VL0packet) (decision box816), a flow controller returns the credits associated with the packet to the arbiter (processing block818).

If the packet is valid (decision box810) and the packet is a VL15packet (decision box812), the target agent blocks the loading of the target buffer until the packet is processed (processing block814).

If the packet is valid (decision box810) and the packet is a non-VL15packet (e.g., VL0packet) (decision box812), the target agent determines whether a pending indicator is set (decision box820). If the pending indicator is set, the target agent completes the processing of a prior packet and resets the pending indicator (processing block822). Method800then proceeds to processing block824.

If the pending indicator is not set (decision box820), the target agent obtains the packet from the flow control buffer for execution (processing block824) and the flow controller returns the credits associated with the packet to the arbiter (processing block826).

Thus, methods and systems to process incoming requests within a port of an interconnect device have been described. Although the present invention has been described with reference to specific exemplary embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.