Managing arbitration in mixed link rate wide ports

Systems and methods presented herein provide for the management of link rates for connecting targets devices (e.g., storage devices) to initiators (e.g., host systems). In one embodiment, an expander includes a plurality of PHYs including a PHY having a first link rate and a PHY having a second link rate that is different than the first link rate. The expander also includes a link manager communicatively coupled to the PHYs and operable to process a connection request from an initiator for the first link rate, extract a timer from the connection request, and determine whether the first link rate is available. The link manager is also operable to start the timer when the link manager determines that the first link rate is unavailable and issue a response to the initiator to inform the initiator that the timer has started and that connection at the first link rate is delayed.

FIELD OF THE INVENTION

The invention generally relates to the field of link rate management among devices, such as those using the SAS protocol.

BACKGROUND

Some data systems employ multiple target devices which may be accessed by host systems or other “initiators” to read and write data. Examples of such systems include data storage systems with multiple storage devices virtualized into logical volumes. Expanders are operable to interconnect between the target devices and the initiators via physical interfaces, or “PHYs”, to increase the number of targets that the initiators may access. The connection rates of the PHYs, or “link rates”, to the targets can vary. And, some of the initiators consistently request specific link rates. Accordingly, once the desired link rates to the targets become occupied, certain initiators may be excluded from accessing PHYs that support the targets via the faster link rates that they request.

SUMMARY

Systems and methods presented herein provide for the management of link rates for connecting target devices (e.g., storage devices) to initiators (e.g., host systems). In one embodiment, an expander is operable to interconnect a plurality of target devices and an initiator. The expander includes a plurality of PHYs including a PHY having a first link rate and a PHY having a second link rate that is different than the first link rate. The expander also includes a link manager communicatively coupled to the PHYs and operable to process a connection request from the initiator for the first link rate, extract a timer from the connection request, and determine whether the requested first link rate is available. The link manager is also operable to start the timer when the link manager determines that the requested first link rate is unavailable and issue a response to the initiator to inform the initiator that the timer has started and that connection at the requested first link rate is delayed while the timer is running.

The various embodiments disclosed herein may be implemented in a variety of ways as a matter of design choice. For example, the embodiments may take the form of computer hardware, software, firmware, or combinations thereof. Other exemplary embodiments are described below.

DETAILED DESCRIPTION OF THE FIGURES

FIG. 1is a block diagram of an exemplary system100for managing link rates between an initiator101and target devices107and108. The system includes an expander102that is operable to link a plurality of initiators, such as the initiator101, to a plurality of target devices, such as the target devices107and108. The expander102is any device, system, software, or combination thereof operable to connect between target devices and initiators, including other expanders, to form a data network, or “switched fabric”106. One example of the expander102is a wide port Serial Attached Small Computer System Interface (SAS) expander that uses the SAS protocol to communicate between the initiator101and the target devices107/108and create the switched fabric106. Such systems can be found in Redundant Array of Independent Disks (RAID) storage systems and other storage networks employing disk drives. In this regard, the initiator101may include a storage controller, or Host Bus Adapter (HBA), that processes host Input/Output (I/O) operations that are routed or switched via the switch fabric106to communicate with the target devices107/108.

The expander102includes a link manager103that is operable to arbitrate communication requests from the initiators to the target devices107/108through a plurality of physical transceivers, or “PHYs”, (e.g., PHYs104and105) and to negotiate link rates for the initiators communicating with the target devices, as link rates to and among the target devices may vary. For example, the target device107may have a faster link rate than the target device108. This may be due to the available PHY or due to the connection speed of the target device itself. The initiator101may send a communication request to the expander102desiring the faster link rate of the target device107. The link manager103may process that communication request and attempt to link the initiator101with the target device107at the faster link rate. Alternatively, the initiator101may request communications at a slower link rate that the link manager103similarly arbitrates.

Target devices, such as target devices107and108, are any devices capable of receiving and processing a connection request as discussed herein. Examples of target devices include other expanders and computer disk drives as used in a storage system. PHYs comprise any combination of hardware, software, firmware, and other associated logic capable of operating as physical transceivers between elements disclosed herein. Also, the expander102may be operable to forward or otherwise route communications for the system100according to one or more protocols including SAS, FibreChannel, Ethernet, ISCSI, etc.

Discussion of the expander102and, more particularly, the link manager103and the arbitration/negotiation process thereof will now be directed to the flowchart ofFIG. 2. It should be noted that, while the system100is illustrated with a single initiator101and two target devices107and108, the invention is not intended to be limited to such. Rather,FIG. 1is merely intended to concisely illustrate the principles of the system100as expanders, such as expander102, may communicate with many initiators, many target devices, and even other expanders.

FIG. 2is a flowchart200of an exemplary method for managing link rates with the system100ofFIG. 1. The method begins with the link manager103processing a connection request from the initiator101for a particular link rate, in the process element201. The link manager103extracts a timer included in the connection request, in the process element202. The timer of the connection request is the amount of time that the initiator101is willing to wait for a particular link rate to a target device. For example, assuming that the link rate through the PHY104to the target device107is faster than the link rate through the PHY105to the target device108, the initiator101may desire to communicate at the faster link rate of the PHY104. If the link rate is not available, the initiator101may provide an amount of time within the timer that the initiator101is willing to wait until the link rate becomes available. In this regard, the link manager103determines whether the link rate to the target device107is available, in the process element203. If the link rate is available, the link manager allows the expander102to connect the initiator101through the PHY104, in the process element204, and allows the initiator101to communicate with the target107as long as the initiator101deems necessary, thereby ending the link rate arbitration, or “speed negotiation”, in the process element209.

If, however, the link rate is unavailable, the link manager103starts the timer provided by the initiator101in the communication request, in the process element205. Upon starting the timer, the link manager103issues a response to the initiator101to inform the initiator101that the timer has started and that connection to the target107via the requested link rate of the PHY104is delayed in the process element206. The link manager103continually monitors the timer to determine whether the timer has expired, in the process element207. If the timer has not expired, the link manager103continues to determine whether the requested link rate is available (e.g., the process element203) until the timer expires. If the timer expires, the link rate arbitration ends in the process element209(e.g., without connection).

With the timer determination returning to the process element203, it is merely intended to show that the link manager103monitors the requested link rate availability until the timer expires. It is not necessary for the link manager to issue a response to the initiator101each time the link manager103decrements the timer. Also, the timer may be implemented in multiple ways. In one embodiment, the timer establishes a maximum time for the initiator101to wait for a particular PHY having the requested link rate (e.g., PHY104). Alternatively, the timer may establish a maximum time for the initiator101to wait for the requested link rate to come available regardless of the PHY and/or target. Additionally, the initiator101is not limited to waiting for the timer to expire as it may abort the connection request. In such an embodiment, the link manager103may monitor the initiator101to determine whether an initiated timer is still needed, thus stopping the timer when not. It should also be noted that initiators may be operable to negotiate link rates as desired. Accordingly, the invention is not intended to be limited to initiators simply negotiating for any particular or faster link rates.

FIG. 3is an exemplary message diagram between the initiator101and the expander102. In this embodiment, the expander102is a SAS expander that is operable to process modified Open Address Frames (OAF) of the SAS protocol. For example, the OAF may be modified to include the timer described above such that the initiator101may intelligently negotiate with the expander102for a particular link rate. The initiator101transfers the modified OAF to the expander102. The expander102, in turn, translates the modified OAF to a command to a target and replies to the initiator101with a standard Arbitration in Progress (AIP) primitive of the SAS protocol (e.g., via the link manager103) informing the initiator101that the expander102is determining the availability of the link rate. As discussed above, if the link rate is indeed available, the expander102connects the initiator101with the target device through the PHY.

If, however, the link rate is not available, the expander102may inform the initiator101that another link is available with a different speed/rate giving the initiator101the opportunity to select the other link rate. If the other link rate is not selected, the expander102returns a modified AIP primitive of the SAS protocol to the initiator101indicating that the timer conveyed in the modified OAF has started. The modified AIP indicates that the expander102is Waiting for Rate Availability (WRA) and that the timer conveyed to the expander102has started. The expander102may periodically convey the AIP-WRA primitive to the initiator101to inform the initiator101that the requested link rate is still not available to give the initiator101the opportunity to cancel the OAF. If the requested link rate comes available before the end of the timer, the target device transfers an Open Accept (OA) to the initiator101by way of the expander102such that the initiator101can begin transmitting its frame data to the target device at the request link rate.

FIG. 4is an exemplary message diagram between the initiator101, the expander102, and the target devices107and108providing additional details to the messaging described above. In this embodiment, the initiator101transfers an unmodified OAF, or standard SAS OAF, to the expander102requesting a first rate connection to the target device108. Assuming that the first rate link of the target device108is available, the target device108transfers an OA to the expander102, which in turn forwards the OA to the initiator101such that the initiator101can transmit frames to the target device108. When the initiator101desires a different speed, the initiator101prepares and transmits a modified OAF to the expander102requesting the different or second link rate.

Again, in this request, the OAF is modified with a timer indicating an amount of time that the initiator101is willing to wait for the requested link rate (either on a per PHY basis or a link rate basis regardless of PHY). The expander102in turn checks to see if the target device107having the second link rate is available. If the target device107is busy, the expander102again provides the availability of the first rate link of the target device108to the initiator101for optional selection.

If the first rate link is not selected, the expander102transfers the AIP_WRA to the initiator101and starts the timer. As can be seen in the messaging diagram, the target device107comes available prior to expiration of the timer. The expander102then transfers the modified OAF to the target device107which in turn transfers an OA providing the requested second rate to the expander102and thus to the initiator101. The initiator101then transmits its frame(s) at the second rate to the expander102and to the target device107through the PHY of the expander102.

FIG. 5is a block diagram of an exemplary hardware modification to registers301for receiving or transmitting the modified OAF301of the SAS protocol (labeled OAF Registers 0-n, where “n” is an integer greater than 1).FIG. 6is a block diagram of the modified OAF304(i.e., a SAS frame) for requesting link rate availability, a Link Rate Availability OAF (LRA_OAF). For example, both the expander102and the initiator101may transmit modified OAFs. Generally, however, the expander102and the target devices107and108are the components that receive the modified OAFs. Accordingly, certain devices may reconfigure their registers to accommodate the timers of the modified OAFs. In this embodiment, devices that transmit or receive the modified OAFs are configured with registers that accommodate a Maximum Single PHY Timeout (MSPT) timer302and a Maximum Lifetime Timeout (MLT) timer303. The modified OAF304that is transmitted by either the initiator101or the expander102is configured with fields that correspond to the MSPT timer302and the MLT timer303(i.e., fields305and306, respectively) such that the timer value may be loaded into the registers of the expanders.

The MSPT timer302is started and decremented when waiting on a particular PHY with a requested link rate. As the expander102may be coupled to other expanders, this timer, once initiated, is decremented at each expander. For example, the expander102may search the availability of the target device107through another expander. The other expander having a PHY to support the requested link rate also starts the MSPT timer until the requested link rate of the available target device107coupled to the other expander is available.

The MLT timer303, on the other hand, is decremented at each expander when waiting on any PHY to a requested target device with a suitable negotiated link rate. This generally occurs as long as the field of the MLT timer303is not set to its maximum value. For example, setting the second field to its maximum value effectively disables the MLT timer303because other features of the SAS protocol cause any negotiation between an initiator and an expander to timeout. In any case, if either the MSPT timer302or the MLT timer303expires, the expander102returns “Open Reject—Rate not Supported” (OR_RNS), a primitive of the SAS protocol.

FIG. 7is a more detailed block diagram of the modified OAF304. In this embodiment, the 24thand 25thbytes of a standard OAF are modified with the MSPT timer305and the 26thand 27thbytes are modified with the MSPT timer306. The Address Frame Type 307 is denoted as 0x2 in element307of the modified OAF304which is used for detection of the modified OAF304.

In one embodiment, SAS devices recognize whether other attached devices support the functionality described above. For example, an available bit in an Identify Address Frame (IAF) may be used to indicate whether a particular device supports sending and receiving the modified OAF304frames. Generally, this bit would also be reflected in the Discover SMP response primitive data for each PHY. For example, when two PHYs are connected together, the PHYs exchange an IAF. The IAF contains information about the device which transmitted the frame. Thus, if the designated bit is set to logical zero, for example, such would indicate the device does not support the modified OAF304frames. Accordingly, the modified OAF304would be translated to a standard or “legacy” OAF. If the bit is set to logical one, for example, no translation of the modified OAF304frame would be necessary and could be passed directly to the device. The translation of the modified OAF304frame provides for backwards compatibility with the legacy OAFs. An example of a modified IAF350is shown in the block diagram ofFIG. 8.

InFIG. 8, the IAF350includes modifications to indicate to a device that it supports LRA-OAF. In this example, there are several “reserved” locations that may be used to implement the bit that indicates whether a device supports the modified OAF304frame. In one embodiment, the bit is placed in byte21of the modified IAF350(note element351at bit5, byte21of the IAF350). However, it generally does not matter where the bit is placed so long as the bit does not interfere with any already defined bits, as compatibility with prior generations of could be affected.

An expander arbitrating a connection over a PHY to a target device not supporting the modified OAF304converts the frame to a standard OAF of the SAS protocol. Otherwise the modified OAF304is forwarded without conversion. An example of such is shown and described below in the flowchart ofFIG. 9.

FIG. 9is a more detailed flowchart400of a process for managing connections according to link rate availability. In this embodiment, the expander102receives an OAF from the initiator101(or another transmitter), in the process element401, and the link manager103processes the modified OAF to extract a timer and determine the requested link rate. In doing so, the link manager103determines whether the MSPT timer is less than the MLT timer, in the process element402. If the MSPT timer is less than the MLT timer, the link manager103loads the value of the MSPT timer305into the register302of the expander102, in the process element404(seeFIGS. 5 and 6). If the MSPT timer305is greater than the MLT timer, then the link manager103loads the value of the MLT timer into the register303of the expander102, in the process element403. The link manager103then determines whether any links are available, in the process element405.

If no link with the requested link rate is available, the link manager103transmits a standard AIP primitive to the initiator101. If a link is available, the link manager103selects the link in the process element406and determines whether that link supports the requested link rate, in the process element407. For example, the link manager103may determine that the link rate of the PHY105to the target device108is available but that link rate does not support the link rate requested in the modified OAF. Thus, if the link rate is not supported, the link manager103transmits a modified AIP primitive to the initiator101, in the process element408, indicating to the initiator101that the link manager103is WRA. The link manager103then decrements the timer that was loaded into the register, in the process element409. The link manager103continually monitors the timer, in the process element410, to determine whether the timer has expired and accordingly continues to detect whether any links are available until the timer expires. In one embodiment, this allows the initiator101the opportunity to select another link if presented by the link manager103even though that link may not be the one specifically requested. Once the timer expires in the process element410, the link manager103transmits an OR_RNS primitive of the SAS protocol to the initiator101, in the process element412, and the link rate availability process ends, in the process element417.

If the link rate is supported by the link of the PHY and the target device, in the process element407, then the link manager103determines whether the link supports LRA_OAFs, in the process element413. For example, the target device or another expander coupled to the PHY of the expander102may not have the hardware registers configured to process LRA_OAFs with timers. Accordingly, the link manager103may transmit a standard OAF to the target device or other expander in the process element416. In this regard, the link manager103translates the LRA_OAF to a standard OAF and the link rate arbitration process ends in the process element417. That is, the link rate arbitration (e.g., SAS speed negotiation) between the link manager103and the initiator101ends because the link rate manager103has presented an acceptable link rate to the initiator101such that the initiator101may transmit frames to the target device through the appropriate PHY of the expander102. Thus, only the link rate arbitration ends.

If however, the target device or other expander coupled to the PHY of the expander102does support LRA-OAFs, then the target device or the other expander recalculates the MLT timer in the MLT timer register303based on the MLT timer306of the LRA-OAF304ofFIG. 6. The link manager103then transmits the LRA-OAF to the target device or other expander, in the process element415, and the link rate arbitration process ends in the process element417(again, the initiator101still transmits frames to the target device).

The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In one embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.FIG. 10illustrates a computing system500in which a computer readable medium506may provide instructions for performing any of the methods disclosed herein.

Furthermore, the invention can take the form of a computer program product accessible from the computer readable medium506providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, the computer readable medium506can be any apparatus that can tangibly store the program for use by or in connection with the instruction execution system, apparatus, or device, including the computing system500.

The computing system500, suitable for storing and/or executing program code, can include one or more processors502coupled directly or indirectly to memory508through a system bus510. The memory508can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code is retrieved from bulk storage during execution. Input/output or I/O devices504(including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, such as through host systems interfaces512, or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.