Intelligent network resource manager

A method and apparatus for intelligent network resource manager for distributed computing systems is provided. A first priority is assigned to a first virtual channel set that includes at least two virtual channels of a plurality of virtual channels associated with a physical communication channel. A second priority is assigned to a second virtual channel set that includes at least one virtual channel of the plurality of virtual channels. The first virtual channel set has more virtual channels than the second virtual channel set. Outbound messages of the first priority are directed to virtual channels of the first virtual channel set. Outbound messages of the second priority are directed to virtual channels of the second virtual channel set. The virtual channels are processed in a round-robin order, where processing includes sending the outbound messages over the physical communication channel.

FIELD OF THE INVENTION

The present invention relates to distributed systems.

BACKGROUND

As Internet technologies advance, e-commerce, multimedia, business-to-business and other applications must reliably support an increasing volume of data. Large amounts of traffic must be routed while maintaining specified Quality of Service (QoS) levels. QoS refers to the capability of a network to provide consistent levels of service to selected network traffic or applications. QoS levels are particularly important in Online Transaction Processing (OLTP) systems designed to immediately respond to user requests. OLTP involves obtaining input data, processing the data, and performing updates in the system to reflect the processed input data. Most applications use a database management system (DBMS) to support OLTP.

One important factor for database I/O performance is latency. Latency refers to the amount of time it takes to store or retrieve data. Latency is an issue even if the amount of data involved is small. For example, latency is affected as a request travels across switches, within a server's operating system I/O stack, and other hardware in a database system.

Different types of database input and output (I/O) have different sensitivities to latency. For example, in the Oracle™ Relational Database Management System (RDBMS) environment, redo logs are latency sensitive. Redo logs include redo records that provide a history of all changes made to the corresponding database. Redo log I/O is latency sensitive because transactions typically wait for return confirmation that the redo records have been persisted. Therefore, redo log I/O has a large impact on the performance of OLTP systems. Other types of database I/O, such as batch, reporting, and backup-related I/O, are less latency sensitive. Nevertheless, some types of database I/O that are not latency sensitive are significant consumers of network bandwidth. When non-latency sensitive I/O requires high network bandwidth, latency may increase. Database systems must handle both latency sensitive and non-latency sensitive transactions while taking latency sensitivity into consideration. Typical solutions for minimizing latency range from simply rescheduling bandwidth-intensive jobs (e.g. to off hours) to maintaining dedicated OLTP systems.

Thus, there is a need for a solution that effectively utilizes storage network bandwidth while at the same time, meeting the latency requirements for latency-sensitive Database I/O.

DETAILED DESCRIPTION

General Overview

An intelligent network resource manager is described herein. A set of virtual channels are available on a physical communication channel. The set of virtual channels allow multiple data flows to share the same physical communication channel. Each virtual channel manages an independent data flow and may use distinct resources, such as buffering and flow control resources. The virtual channels are processed in a circular order to send the outbound messages over the physical communication channel. At each turn, an equal portion of the data flow of each virtual channel is processed. In one embodiment, one or more virtual channels may be weighted, and a weighted portion of the data flow is processed.

A plurality of virtual channels is selected from the set of available virtual channels. A selected priority is assigned to each virtual channel of the plurality of virtual channels. In one embodiment, the priority for each virtual channel is selected from a first priority and a second priority. Outbound messages of the first priority are directed to virtual channels of the first priority, and outbound messages of the second priority are directed to virtual channels of the second priority. In one embodiment, more virtual channels are assigned the first priority than the second priority. In other words, more virtual channels are available for messages of the first priority. When the virtual channels are processed in circular order, this leads to more bandwidth being available for the high priority outbound messages.

A selected category may also be assigned to each virtual channel of the plurality of virtual channels. In one embodiment, the category for each virtual channel is selected from a first category and a second category. Outbound messages of the first category are directed to virtual channels of the first category, and outbound messages of the second category are directed to virtual channels of the second category. In one embodiment, the first category is a first message size category, and the second category is a second message size category that is larger than the first message size category. This prevents messages of the first message size from being queued after a larger message of the second message size in a virtual channel. In one embodiment, more virtual channels are assigned the first category than the second category such that more virtual channels are available for messages of the first category.

One or more available virtual channels may be reserved (i.e. not assigned a priority and/or category). Infiniband™ provides QoS functionality including low-priority Infiniband™ Virtual Lanes and high-priority Infiniband™ Virtual Lanes. In one embodiment, the virtual channels assigned a priority and/or category are low-priority Infiniband™ Virtual Lanes. One or more Infiniband™ Virtual Lanes may be reserved, including one or more high-priority Infiniband™ Virtual Lanes.

Virtual Channel Priority and Category

In one embodiment, a first virtual channel set comprising at least two virtual channels is assigned a first priority. A second virtual channel set comprising at least one virtual channel is assigned a second priority. The first virtual channel set has more virtual channels than the second virtual channel set. Not all available virtual channels associated with the physical communication channel must be used.

In one embodiment, REDO Log I/O messages are assigned the first priority. Thus, outbound REDO Log I/O messages are directed to virtual channels of the first virtual channel set. In the Oracle™ RDBMS environment, redo logs are generated for all changes made to a corresponding database. Each redo record of a redo log file includes change vectors that describe changes made in the database. REDO Log I/O will be described in greater detail hereafter.

Categories may also be assigned to the virtual channels and used to directed outbound messages. In one embodiment, two categories are used: a first category for a small message size and a second category for a large message size. The two categories may be determined based on a size threshold. This prevents messages of the first message size category from being queued after a larger message of the second message size category in a virtual channel. In one embodiment, the size threshold is determined based on one or more database properties, such as a data block size. The size threshold may be chosen such that most outbound REDO Log I/O messages are expected to fall into the first message size category.

In one embodiment, each virtual channel of the first virtual channel set (i.e. the virtual channels assigned the first priority) is assigned one of the first category and the second category. Outbound messages corresponding to the first category are directed to virtual channels of the first category, and outbound messages corresponding to the second category are directed to virtual channels of the second category. In this case, three classes of virtual channels are created:

FIRST PRIORITY, FIRST CATEGORY;

FIRST PRIORITY, SECOND CATEGORY; and

When (1) REDO Log I/O messages are assigned the first priority, (2) the first category and the second category are based on size, and (3) most outbound REDO Log I/O messages are expected to fall into the first message size category, most outbound REDO Log I/O messages will be directed to virtual channels assigned the first priority and the first category.

In one embodiment, each virtual channel of the second virtual channel set (i.e. the virtual channels assigned the second priority) is assigned one of the first category and the second category. In this case, three classes of virtual channels are created:

FIRST PRIORITY

SECOND PRIORITY, FIRST CATEGORY; and

When (1) REDO Log I/O messages are assigned the first priority, outbound REDO Log I/O messages will be directed to virtual channels assigned the first priority.

In one embodiment, each virtual channel of both the first virtual channel set (i.e. the virtual channels assigned the first priority) and the second virtual channel set (i.e. the virtual channels assigned the second priority) is assigned one of the first category and the second category. In this case, four classes of virtual channels are created:

FIRST PRIORITY, FIRST CATEGORY;

SECOND PRIORITY, FIRST CATEGORY; and

When (1) REDO Log I/O messages are assigned the first priority, (2) the first category and the second category are based on size, and (3) most outbound REDO Log I/O messages are expected to fall into the first message size category, most outbound REDO Log I/O messages will be directed to virtual channels assigned the first priority and the first category, and larger REDO Log I/O messages will be directed to virtual channels assigned the first priority and the second category.

FIG. 1is a block diagram depicting an embodiment of an assignment of a priority and a category to plurality of virtual channels associated with a physical communication channel.FIG. 1shows a plurality of virtual channels associated with a physical communication channel. The virtual channels are shown using unique identifiers 0, 1, 2, 3, 4, 6, and 7. The virtual channels are logical channels associated with the physical communication channel.

The virtual channels are assigned a priority selected from a set of possible priorities. For example, virtual channels 1, 2, 4, 6, and 7 are assigned a first priority102(“PRIORITY 1”), and virtual channels 0 and 3 are assigned a second priority104(“PRIORITY 2”). Outbound messages of the first priority are directed to virtual channels of the first priority102, while outbound messages of the second priority are directed to virtual channels of the second priority104.

The physical communication channel is configured to process the virtual channels in a round-robin order. QoS may be tailored by changing the number of virtual channels assigned to a particular priority. In one embodiment, more virtual channels are assigned the first priority102than the second priority104.

The virtual channels may also be assigned a category selected from a set of possible categories. For example, virtual channels 1, 2, 3, and 7 are assigned a first category106(“CATEGORY 1”), and virtual channels 0, 4, and 6 are assigned a second category108(“CATEGORY 2”). Outbound messages corresponding to the first category106are directed to virtual channels of the first virtual channel set, while outbound messages corresponding to the second category108are directed to virtual channels of the second virtual channel set.

In one embodiment, the set of possible categories are defined based on message size. For example, first category106may be a first message size category and second category108may be a second message size category that is larger than the first message size category. When larger messages are directed to different virtual channels than smaller messages, smaller latency sensitive messages will typically reach the head of a virtual channel more predictably due to the elimination of the potential of incurring the transfer time for larger messages that were previously enqueued.

In one embodiment, for virtual channels assigned the first category106, more virtual channels are assigned the first priority102than the second priority104. For example, inFIG. 1, of the four virtual channels assigned the first category106, three virtual channels are assigned the first priority102and one virtual channel is assigned the second priority104.

In one embodiment, for virtual channels assigned the second category108, more virtual channels are assigned the first priority102than the second priority104. For example, inFIG. 1, of the three virtual channels assigned the second category106, two virtual channels are assigned the first priority102and one virtual channel is assigned the second priority104.

Virtual Channel Configuration

FIG. 2is a block diagram depicting an embodiment of an assignment of a priority, a category and a weight to a plurality of virtual channels associated with a physical communication channel. Table250includes data corresponding to a plurality of virtual channels associated with a physical communication channel. In one embodiment, available virtual channels, such as Infiniband™ Virtual Lanes, are configured on one or more devices. Such configurations may be propagated across other connected devices in the same network.

In table250, the Lane ID252field contains a unique identifier for each virtual channel in a set of virtual channels. The virtual channels specified by a particular lane ID250may be assigned at least one of a priority, category and weight. The priority254field contains the priority assigned to the corresponding virtual channel. The category256field contains the category assigned to the corresponding virtual channel. The weight258contains the weight assigned to the corresponding virtual channel. When, the virtual channels are weighted, the amount of data processed and sent from a particular virtual channel during its turn is weighted based on the weight assigned to the particular virtual channel.

In one embodiment, one or more virtual channels in a set of available virtual channels is reserved. Reserved virtual channel260is not assigned a priority254or category256because it is not used in the described QoS scheme that is based on priority and/or category. However, reserved virtual channel260may be used in another QoS scheme. For example, the Infiniband™ architecture provides 8 virtual lanes, which may be each designated as a high-priority Infiniband™ virtual lane or a low-priority Infiniband™ Virtual Lane. Reserved virtual channel260may be reserved for potential or actual use as a high-priority Infiniband™ Virtual Lane, while the non-reserved virtual channels may be implemented as low-priority Infiniband™ Virtual Lanes. In one embodiment, a mechanism is implemented to prevent unauthorized software from using reserved virtual channel260.

The assignment data shown inFIG. 2is presented as a single table. The actual assignment data may be stored as one or more tables or as one or more other data structures. The assignment data in table250may be used to configure one or more network elements, including one or more switches, routers and/or other hardware in an underlying network architecture, such as an Infiniband™ switch, router and/or other hardware.

The configuration may be performed automatically, such as based on a configuration file, automatic optimization tests or procedures, a default setting, or other configuration settings. In one embodiment, a user, such as an administrator, manager, or other user, may perform one or more portions of the configuration, including specifying how many Virtual Lanes map to a particular priority and/or category. The specific Virtual Lanes that map to a particular priority and/or category may also be mapped. In addition, the priority assigned to different messages, the category types, the category ranges, and the weights of the Virtual Lanes may be configured.

Virtual Channel Message Processing

Outbound messages in the virtual channels are processed and sent over the physical communication channel. In one embodiment, the virtual channels are processed in a circular order. The virtual channels may be processed using a round-robin scheme. In this case, bandwidth is assigned to each process in equal portions and in circular order, and messages from the virtual channels are processed. In one embodiment, the virtual channels are processed using a weighted round-robin scheme. In this case, bandwidth is assigned to each process in proportion to the weight and in circular order. If a particular virtual channel is empty or becomes empty during its turn before processing and sending the allocated amount of data, processing proceeds to the next virtual channel. No additional priority scheme is necessary other than the virtual channel configuration by priority, category and/or weight.

In one embodiment, the messages are segmented into packets based on size, and an equal number of packets are processed and sent from a virtual channel during its turn in the round-robin order. In one embodiment, the virtual channels are weighted, and the number of packets that are processed and sent from a virtual channel during its turn up is weighted based on the weight assigned to the virtual channel.

One or more embodiments described herein may be implemented over any network architecture, including but not limited to a switch-based I/O fabric architecture. The implementation may use one or more features of the network architecture. In one embodiment, the implementation is designed for an InfiniBand™ fabric architecture. InfiniBand™ is a switched-based I/O fabric architecture typically used for high-performance and/or enterprise solutions.

A switched fabric network topology includes network nodes interconnected by one or more network switches. The switched fabric network spreads network traffic across multiple physical links. Features of InfiniBand™ include high throughput, low latency, QoS, failover, and scalability. The switched fabric architecture provides scalability which can be accomplished by adding switches to the fabric. Unlike a shared bus architecture, the aggregate bandwidth increases as additional switches are added to the network. Multiple paths between devices keep the aggregate bandwidth high and provide failsafe, redundant connections.

InfiniBand™ transmits data as messages or packets of up to 4 KB. Examples of messages include a direct memory access read from or, write to, a remote node, a channel send, a channel receive, a transaction-based operation, a multicast transmission and an atomic operation. All InfiniBand™ transmissions begin or end at a channel adapter. For example, processors have host channel adapters (HCAs) and peripherals have target channel adapters (TCAs). These channel adapters can also exchange information for security or QoS.

Communications are transmitted as messages that are segmented into packets. The payload of each packet is limited by the maximum transmit unit (MTU) negotiated for the path between a first channel adapter and a second channel adapter. Segmentation and reassembly of packets is done in hardware. Typical MTUs are 256 bytes and 2048 bytes.

In one embodiment, one or more embodiments described herein are implemented on an Infiniband™ architecture by configuring Infiniband™ Virtual Lanes (VL). For example, priority tables, Service Level (SL) to Virtual Lane (VL) mapping and the QoS policy rules are configured in the Infiniband™ devices. This information will get propagated to new nodes when they join the fabric.

The port side of a channel adapter implements InfiniBand™ Virtual Lanes. InfiniBand™ Virtual Lanes enable multiple independent data flows to share same physical communication link. Buffering and flow control may be separately implemented for each Virtual Lane. Currently, 8 Virtual Lanes are available on each InfiniBand™ link. QoS is supported via low-priority Infiniband™ Virtual Lane and high-priority Infiniband™ Virtual Lane configurations. Each Virtual Lane has its own dedicated set of associated buffering resources. The InfiniBand™ fabric can be regarded as an overlay of multiple logical fabrics that only share the physical links.

Virtual lane arbitration is the mechanism an output port utilizes to select from which Virtual Lane to transmit. The InfiniBand™ architecture specifies a dual-priority weighted round robin scheme. In a round-robin scheme for scheduling, bandwidth is assigned to each process in equal portions and in circular order, handling all processes without priority. In a weighted round-robin scheme, bandwidth is assigned to each process in proportion to the weight.

Packets from high-priority InfiniBand™ Virtual Lanes are always transmitted ahead of those from low-priority InfiniBand™ Virtual Lanes. Within a given priority, data is transmitted from InfiniBand™ Virtual Lanes in approximate proportion to their assigned weights (excluding InfiniBand™ Virtual Lanes that have no outbound data to be transmitted).

InfiniBand™ is further described in Crupnicoff, Diego et al., “Deploying Quality of Service and Congestion Control in InfiniBand-based Data Center Networks,” (Document No. 2379, Rev. 1.0), available at “http://www.mellanox.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf,” the contents of which are incorporated herein by reference in its entirety. However, one or more embodiments described herein may use the architecture described in the cited reference in a manner that deviates from the stated purpose in the cited reference. For example, although InfiniBand™ provides for low-priority and high-priority Infiniband™ Virtual Lane configurations, using the built-in high-priority protocols does not achieve the desired performance, such as with respect to redo log I/O latency in the example described below.

In the Oracle™ RDBMS environment, redo logs are generated for all changes made to a corresponding database. Each redo record of a redo log file includes change vectors that describe changes made in the database. For example, redo records are generated when a database operation is performed that makes changes to the database. The redo logs may be used in many ways, including but not limited to database recovery, replay, rollback, backup, and other functions.

When a transaction is committed, the redo records corresponding to the transaction are written to an available redo log file, along with data identifying the transaction. Oracle™ LoG WRiters (LGWRs) is a background process that writes the redo log buffers to the on-line redo log files. Whenever a transaction is committed, LGWR writes the transaction redo records from the redo log buffer to a redo log file, and assigns a system change number (SCN) to identify the redo records for each committed transaction. The process that submitted the transaction is notified that the transaction has committed only after all the redo records are persistently written. This is one point where reducing latency is important.

An OLTP workload can benefit greatly from fast response times for database log writes. With respect to redo log I/O, latency strongly affects the performance of OLTP systems. If there are no application-related bottlenecks or contention for database locks and resources, one limiting factor in database system performance may be an amount of time spent waiting for redo log writes. In one embodiment, virtual channels are assigned priorities and categories to prioritize LGWR issued I/O to minimize redo log I/O latency, which shall be described in greater detail hereafter.

Redo log I/O is described in greater detail in: U.S. application Ser. No. 13/346,656, Utilizing Multiple Storage Devices To Reduce Write Latency For Database Logging, filed by Kesavan P. Srinivasan, et al. on Jan. 9, 2012; U.S. application Ser. No. 13/485,557, Rapid Recovery From Loss Of Storage Device Cache, filed by Juan R. Loaiza, et al. on May 31, 2012; U.S. application Ser. No. 13/801,319, Rapid Recovery From Downtime Of Mirrored Storage Device, filed by Juan R. Loaiza, et al. on May 13, 2013; U.S. application Ser. No. 13/288,785, Write-Back Storage Cache Based On Fast Persistent Memory, filed by Bharat Chandra Baddepudi, et al. on Nov. 3, 2011; and U.S. application Ser. No. 12/871,805, Controlling Data Lag In A Replicated Computer System, filed by Jia Shi, et al. on Aug. 30, 2010; the contents of which are incorporated herein by reference in their entirety.

Example Configuration to Prioritize Redo Log I/O

The configuration described in this section is an example of a configuration for prioritizing redo log I/O to reduce latency. This configuration is deployable in an Oracle™ Exadata™ database server.

In an Oracle™ database, I/Os issued by one or more LGWR processes are prioritized. 3 QoS levels are defined:

NORMAL for other processes; and

DEFAULT for external clients that do not specify any QoS level,

I/O messages with a QoS level of NORMAL and DEFAULT are treated equally. I/O messages with a QoS level of VIP are prioritized over those with a QoS level of NORMAL or DEFAULT.

Two size categories are created:

SMALL for messages with up to 32K of payload; and

LARGE for messages with greater than 32K of payload.

The 32K size threshold is based on one or more database block size characteristics.

7 of the 8 available Virtual Lanes in the Infiniband™ architecture are configured as low-priority Infiniband™ Virtual Lanes. One Virtual Lane is reserved, such as for potential use as a high-priority Infiniband™ Virtual Lane. Seven virtual channels are configured as follows:

Given the QoS level and typical size of redo log I/O, redo log I/O messages are typically directed to the VIP, SMALL Virtual Lanes. When multiple LGWR processes are running, one or more LGWR processes may be hashed to one of the three VIP/SMALL Virtual Lanes.

Process for Directing Outbound Messages Based on Priority

FIG. 3is a flow diagram that illustrates an embodiment of a process for directing outbound messages to a plurality of virtual channels associated with a physical communication channel based on priority. Process300may be performed by one or more computing devices. For example, one or more blocks of process300may be performed by computer system500.

In block302, a first priority is assigned to a first virtual channel set. The first virtual channel set includes at least two virtual channels selected from a plurality of virtual channels associated with a physical communication channel.

In block304, a second priority is assigned to a second virtual channel set. The second virtual channel set includes at least two virtual channels selected from the same plurality of virtual channels associated with the same physical communication channel. The first virtual channel set has more virtual channels than the second virtual channel set, so more virtual channels of the plurality of virtual channels are assigned the first priority than the second priority.

In block306, outbound messages of the first priority are directed to the virtual channels belonging to the first virtual channel set. In one embodiment, one or more specific message types are identified as having first priority. The one or more specific message types may be pre-determined or dynamically specified, such as by a user. In one embodiment, REDO log I/O messages are identified as having the first priority.

In block308, outbound messages of the second priority are directed to the virtual channels belonging to the second virtual channel set. In one embodiment, one or more specific message types are identified as having the second priority. The one or more specific message types may be pre-determined or dynamically specified, such as by a user. Alternatively or in addition, a set of prioritized message types are identified as having the first priority outbound messages, and remaining messages types (e.g. outbound messages of a type not included in the set of prioritized message types) are treated as having the second priority.

In block310, the virtual channels are processed in round-robin order. Outbound messages in the virtual channels are sequentially processed and sent over the physical communication channel. In one embodiment, the messages are segmented into packets based on size, and an equal number of packets are processed and sent from a virtual channel during its turn in the round-robin order. In one embodiment, the virtual channels are weighted, and the number of packets that are processed and sent from a virtual channel during its turn is weighted based on the weight assigned to the virtual channel. If a particular virtual channel is empty or becomes empty during its turn, processing proceeds to the next virtual channel.

Process for Directing Outbound Messages Based on Priority and Category

FIG. 4is a flow diagram that illustrates an embodiment of a process for directing outbound messages to a plurality of virtual channels associated with a physical communication channel based on priority and another category. Process400may be performed by one or more computing devices. For example, one or more blocks of process400may be performed by computer system500.

In block402, a selected priority and a selected category are assigned to each virtual channel of a plurality of virtual channels associated with a physical communication channel. The selected priority is selected from two or more valid priorities, which may be pre-determined or dynamically specified, such as by a user. For example, the selected priority may be selected from a first priority and a second priority. In one embodiment, the first priority is considered a higher priority than the second priority. More virtual channels of the plurality of virtual channels may be assigned the first priority than the second priority.

The selected category is selected from two or more valid categories, which may be pre-determined or dynamically specified, such as by a user. In one embodiment, the category is based on message size. For example, the selected category may be selected from a small message size category and a large message size category. In one embodiment, the size categories are defined by one or more thresholds such that REDO log I/O messages typically fall under the small message size category.

In block404, outbound messages are directed to the plurality of virtual channels based on category and priority. For example, if the categories include a first category and a second category, then outbound messages of the first category are directed to a virtual channel assigned the first category, while outbound messages of the second category are directed to a virtual channel assigned the second category. In one embodiment, the two or more valid categories are based on size. For example, the valid categories may include a small message size category and a large message size category. When the valid categories include one or more size-based categories, the categories may be defined by one or more size thresholds.

Outbound messages are also directed to a virtual channel based on priority. Outbound messages of a particular priority are directed to one of the virtual channels assigned the particular priority. In one embodiment, one or more specific message types are assigned the particular priority.

In one embodiment, the valid priorities include at least a first priority and a second priority, where more virtual channels are assigned the first priority than the second priority. In one embodiment, one or more first specific message types are identified as having the first priority, and one or more second specific message types are identified as having the second priority. The one or more first specific message types and/or the one or more second specific message types may be pre-determined or dynamically specified, such as by a user. In one embodiment, REDO log I/O messages are identified as having the first priority.

Alternatively or in addition, a specific set of message types are identified as having the first priority, and remaining outbound messages (e.g. outbound messages of a type not included in the specific set of message types) are assigned the second priority.

In block406, the virtual channels are processed in a circular order to send the outbound messages over the physical communication channel. In one embodiment, the messages are segmented into packets based on size, and an equal number of packets are processed and sent from a virtual channel during its turn in the round-robin order. In one embodiment, the virtual channels are weighted, and the number of packets that are processed and sent from a virtual channel during its turn is weighted based on the weight assigned to the virtual channel. If a particular virtual channel is empty or becomes empty during its turn, processing proceeds to the next virtual channel.

Hardware Overview

Computer system500further includes a read only memory (ROM)508or other static storage device coupled to bus502for storing static information and instructions for processor504. A storage device510, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus502for storing information and instructions.