Adaptive SAS PHY configuration

A SAS expander adaptively configures a Serial-Attached-SCSI (SAS) PHY to accommodate varying lengths of a cable coupling the PHY to a remote PHY. The expander (a) configures the SAS PHY with settings of an entry of a table of PHY configuration settings, each entry in the table having different PHY configuration setting values; (b) clears a counter; (c) operates the PHY to communicate with the remote PHY for a monitoring period, after configuring the PHY and clearing the counter; (d) increments the counter when the PHY detects a PHY event during the monitoring period, and otherwise decrements the counter; (e) repeats steps (c) and (d) unless the counter rises above a threshold; and (f) when the counter rises above the threshold, repeats steps (a) through (e), wherein step (a) is performed with the settings of a different entry of the table.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates in general to the field of high data availability storage controllers that connect to storage devices via the Serial Attached SCSI (SAS) standard, and particularly to SAS PHY configuration.

Serial-Attached-SCSI (SAS) systems are becoming more and more common in modern computer systems. SAS systems include SAS initiator devices and SAS target devices as does its parent, the Small Computer Systems Interface (SCSI). SAS target devices are typically storage devices, such as disk drives, that receive commands from SAS initiator devices, such as SAS host bus adapters in host computers or SAS I/O controllers in Redundant Arrays of Inexpensive Disks (RAID) controllers.

Implementations and uses of SAS are described in detail in the following documents, each of which is incorporated by reference in its entirety for all intents and purposes:“Serial Attached SCSI-1.1 (SAS-1.1)”, Revision 10, Sep. 21, 2005. Working Draft, Project T10/1601-D, Reference number ISO/IEC 14776-151:200x. American National Standard Institute. (http://www.t10.org/ftp/t10/drafts/sas1/sas1r10.pdf)“Serial Attached SCSI-2 (SAS-2)”, Revision 6, Sep. 22, 2006. Working Draft, Project T10/1760-D, Reference number ISO/IEC 14776-152:200x. American National Standard Institute. (http://www.t10.org/ftp/t10/drafts/sas2/sas2r06.pdf)“Serial Attached SCSI-2 (SAS-2)”, Revision 10, May 15, 2007. Working Draft, Project T10/1760-D, Reference number ISO/IEC 14776-152:200x. American National Standard Institute. (http://www.t10.org/ftp/t10/drafts/sas2/sas2r10.pdf)

SAS systems are built on point-to-point serial connections between SAS devices. Each point-to-point connection is referred to as a link, or lane, and the two endpoints are referred to as a PHY. A PHY contains a transmitter device (TX) and receiver device (RX) and electrically interfaces to a link to communicate with another PHY at the other end of the link. The link, or lane, includes two differential signal pairs, one in each direction. A SAS port includes one or more PHYs. A SAS port that has more than one PHY grouped together is referred to as a wide port, and the more than one links coupling the two wide ports are referred to as a wide link. Wide ports and links provide increased data transfer rates between SAS endpoints and enable multiple simultaneous connections to be open between a SAS initiator and multiple SAS targets.

The simplest SAS topology is a single SAS initiator having a SAS port that is connected by a single SAS link to a SAS port of a single SAS target. However, it is desirable in many applications, such as a high data availability RAID system, to enable one or more SAS initiators to communicate with multiple SAS target devices. In addition to initiators and targets, SAS includes a third type of device, expanders, which are employed in SAS systems to achieve more complex topologies. SAS expanders perform switch-like functions, such as routing, to enable SAS initiators and targets to communicate via the SAS point-to-point connections.

The present inventors have observed various problems in complex topology SAS systems when a component is marginal or goes bad, such as a SAS device that generates logical errors, improper PHY analog settings, a bad or marginal PHY, or a bad or marginal link, which may include bad or marginal cables, connectors, or printed circuit board assembly traces. Some of the manifestations of the faulty components include intermittent communication errors between SAS devices, complete loss of a SAS link, or failure of an entire SAS domain. Another manifestation is the inability for an initiator to see a SAS target in the topology due to intermittent failures that cause a SAS device to work sufficiently well to be allowed into the topology, but to be sufficiently faulty to prevent effective communication between SAS devices.

One method of dealing with these problems is an initiator-based solution. The initiator may attempt to identify the faulty component and send a command through the SAS domain to disable, or bypass, various PHYs in the domain in a trial-and-error approach until the initiator has isolated the problem. However, the present inventors have observed some failure scenarios which cannot be satisfactorily remedied by the initiator-based approach. For example, assume a component fails in an intermittent fashion, such as a marginal PHY, that causes a SAS expander to first detect that a SAS link is operating properly, to subsequently detect that the link is not operating properly, and to continue this sequence for a relatively long time. According to the SAS standard, the SAS expander is required to transmit a BROADCAST primitive on each of its SAS ports to notify other SAS devices of the change of status within the SAS domain. Each time a SAS initiator receives the BROADCAST primitive it is required to perform a SAS discover process to discover the device type, SAS address, and supported protocols of each SAS device in the SAS domain and to configure routing tables within the SAS expanders as needed. The SAS discover process can take a relatively large amount of time. If the SAS expander transmits BROADCAST primitives due to the operational to non-operational link transitions according to a period that is comparable to the SAS discover process time, then consequently the SAS initiator may be unable to effectively send commands though the SAS domain to identify and remedy the problem. Or, even if the initiator is successful in identifying and fixing the problem, the SAS domain may have been effectively unavailable for providing user data transfers for an unacceptable length of time.

Another potential problem in SAS systems is the fact that the SAS standard allows cables that connect SAS PHYs to be anywhere within a relatively large range of lengths. For example, the SAS specification allows for cable lengths up to eight meters. The length of the SAS cable may significantly impact the quality of the signals transceived on the SAS link between two SAS PHYs.

Therefore, what is needed is a solution to improve the data availability in SAS systems, which are subject to the foregoing problems.

BRIEF SUMMARY OF INVENTION

In one aspect, the present invention provides a Serial-Attached-SCSI (SAS) expander. The SAS expander includes a plurality of SAS PHYs, each for coupling to a SAS cable for communication with a respective remote PHY. The SAS expander also includes a memory, configured to store a counter associated with each of the plurality of PHYs and a table of PHY configuration settings. Each entry in the table has different PHY configuration setting values. The SAS expander also includes a microprocessor, coupled to the memory and to the plurality of PHYs, configured to, for each of the plurality of PHYs: (a) configure the PHY with settings of an entry of the table; (b) clear the counter associated with the PHY; (c) operate the PHY to communicate with the remote PHY for a monitoring period, after configuring the PHY and clearing the counter; (d) increment the counter when the PHY detects a PHY event during the monitoring period, and otherwise decrement the counter; (e) repeat steps (c) and (d) unless the counter rises above a threshold; and (f) when the counter rises above the threshold, repeat steps (a) through (e), wherein the microprocessor performs step (a) with the settings of a different entry of the table.

In another aspect, the present invention provides a method for adaptively configuring a Serial-Attached-SCSI (SAS) PHY to accommodate varying lengths of a cable coupling the PHY to a remote PHY. The method includes: (a) configuring the SAS PHY with settings of an entry of a table of PHY configuration settings, each entry in the table having different PHY configuration setting values; (b) clearing a counter; (c) operating the PHY to communicate with the remote PHY for a monitoring period, after the configuring the PHY and the clearing the counter; (d) incrementing the counter when the PHY detects a PHY event during the monitoring period, and otherwise decrementing the counter; (e) repeating steps (c) and (d) unless the counter rises above a threshold; and (f) when the counter rises above the threshold, repeating steps (a) through (e), wherein step (a) is performed with the settings of a different entry of the table.

In another aspect, the present invention provides a Serial-Attached-SCSI (SAS) expander. The SAS expander includes a wide SAS port for coupling to a SAS cable for communication with a remote wide SAS port. The wide SAS port comprises a plurality of SAS PHYs. The SAS expander also includes a memory that stores a plurality of different PHY configuration settings. The SAS expander also includes a microprocessor coupled to the memory and to the wide SAS port. The microprocessor monitors for faults during operation of the wide SAS port. The microprocessor also adaptively re-configures the plurality of SAS PHYs with a different one of the plurality of PHY configuration settings, in response to detecting that all of the plurality of SAS PHYs of the wide SAS port are experiencing the same type of fault. The microprocessor also disables one of the plurality of SAS PHYs of the wide SAS port, in response to detecting that only the one of the plurality of SAS PHYs is experiencing faults.

In another aspect, the present invention provides a method for improving the reliability of a Serial-Attached-SCSI (SAS) domain including a SAS expander having a wide SAS port comprising a plurality of SAS PHYs. The method includes monitoring for faults during operation of the wide SAS port. The method also includes adaptively re-configuring the plurality of SAS PHYs with new PHY configuration settings, in response to detecting that all of the plurality of SAS PHYs of the wide SAS port are experiencing the same type of fault. The method also includes disabling one of the plurality of SAS PHYs of the wide SAS port, in response to detecting that only the one of the plurality of SAS PHYs is experiencing faults. The monitoring, adaptively re-configuring, and disabling are all performed by the SAS expander.

Advantageously, in addition to resolving varying SAS cable length problems, adaptive PHY configuration embodiments described herein may also be useful in resolving other SAS link-related problems, such as varying manufacturing quality of SAS cables and/or PHY circuitry.

DETAILED DESCRIPTION

Referring now toFIG. 1, a block diagram of a SAS system100according to the present invention is shown. The SAS system100includes two host computers108each coupled to two SAS-based RAID controllers104, via a host interconnect such as Ethernet, FibreChannel, or the like. Each RAID controller104is coupled to a corresponding SAS expander102via a wide SAS link112. In the embodiment ofFIG. 1, the wide SAS links112between the RAID controllers104and the SAS expanders102are 4× wide. Each of the SAS expanders102is coupled to a plurality of SAS disks106via a corresponding SAS link112. In the embodiment ofFIG. 1, the SAS links112between the SAS expanders102and the SAS disks106are narrow (i.e., 1×) SAS links112. The RAID controllers104, SAS expanders102, and SAS disks106are enclosed in an enclosure114. Additionally, the SAS expanders102within the enclosure114are connected via a SAS link112. In one embodiment, the inter-expander SAS link112is a 4× wide link. The inter-expander SAS link112advantageously provides a second SAS pathway for each RAID controller104to communicate with each of the SAS disks106through the SAS expander102that is directly connected to the other RAID controller104.

Advantageously, the SAS expanders102of the SAS system100are intelligent SAS expanders102that include the ability to identify faulty communications on a SAS link112connected to one of the SAS expander102PHYs. Furthermore, the intelligent SAS expanders102include the ability to disable the identified PHY to isolate the faulty component, which may be the PHY itself, from the rest of the SAS system100. Additionally, the intelligent SAS expanders102include the ability to report the disabled PHY. Still further, the intelligent SAS expanders102include the ability to recover from faulty condition. In one embodiment, a user notifies the SAS expander102that corrective action has been taken, such as replacing the faulty component (e.g., faulty cable, faulty SAS disk106, or other faulty component), and the SAS expander102responsively repairs the communication between the SAS expander102and the other device by re-enabling the previously disabled PHY. In one embodiment, the SAS expander102is intelligent enough to automatically detect that a user has remedied the fault, and responsively re-enables the PHY. In one embodiment, the SAS expander102is intelligent enough to automatically take action to remedy the fault, such as by adjusting the PHY analog settings (shown inFIG. 9) until reliable communications are restored. The SAS expander102includes a microprocessor that monitors status registers associated with the PHYs to identify faulty communications on a SAS link112, writes to control registers to disable and re-enable the PHYs, and performs the reporting function, as discussed in detail below.

Further advantageously, the intelligent SAS expanders102include the ability to automatically re-configure the analog settings (shown inFIG. 9) of one or more of their PHYs based on a history of PHY events in order to accommodate varying SAS cable lengths. The SAS expander102automatically re-configures the PHY on each attempt with settings from a table of entries with different settings. The settings for each entry in the table have been determined to be optimal for different cable length ranges. The automatic reconfiguring enables the PHY to adapt to different lengths or qualities of cables connecting the expander102PHY to a remote PHY.

The SAS system100ofFIG. 1also includes second and third enclosures114similar to the first enclosure114described above; however, the second and third enclosures114do not include the RAID controllers104, and are employed for enclosing only SAS disks106and two SAS expanders102. Each SAS expander102in the first enclosure114is linked to a corresponding one of the SAS expanders102in the second enclosure114via a corresponding wide SAS link112. Similarly, each SAS expander102in the second enclosure114is linked to a corresponding one of the SAS expanders102in the third enclosure114via a corresponding wide SAS link112. In one embodiment, each enclosure114may enclose up to 12 SAS disks106, in addition to the RAID controllers104, SAS expanders102, power supplies, cooling systems, management controllers, and other components as are well known in the storage system industry. The PHYs of the SAS expanders102in the enclosures114are connected by one or more SAS cables. In embodiments in which the link112between the enclosures114is a wide link112, typically a single cable encloses all of the differential signal pairs of the wide link112.

Advantageously, the SAS system100ofFIG. 1is arranged in a redundant manner to increase fault-tolerance of the SAS system100. In particular, each SAS disk106is accessible by each of the RAID controllers104so that if a RAID controller104, SAS link112, or SAS expander102fails, the hosts108may continue to access the SAS disks106via a surviving RAID controller104or SAS pathway. The SAS system100ofFIG. 1is intended as an example of a SAS system100in which the present invention may be employed. However, the present invention is not limited to the system configuration shown inFIG. 1. Rather, the present invention may be employed in various SAS topologies that include SAS expanders. In some embodiments, the hosts108may also be included in the first enclosure114.

As mentioned above, the SAS links112may include various components, such as cables, connectors, and printed circuit board assemblies that include signal conductors. In one embodiment, the SAS expander102comprises a PM8388 SXP 24×3G 24-port SAS expander available from PMC-Sierra, Inc., of Santa Clara, Calif., in which the present inventors have modified the code222(discussed below) to perform the fault identification, isolation, reporting, repairing, and adaptive PHY configuration steps described herein. In other embodiments, the SAS expander102comprises a modified version of the following PMC-Sierra models: PM8387 SXP 36×3G 36-port SAS expander, PM8399 SXP 24×3GSEC 24-port SAS expander, or PM8398 SXP 36×3GSEC 24-port SAS expander.

AlthoughFIG. 1illustrates a SAS system100including SAS disks106, the present invention may be employed in a SAS system100including SATA disks106, which are interoperable with SAS disks within a SAS domain. In particular, the SATA Tunneled Protocol (STP) provides a means for SAS/SATA initiators to communicate with SATA disks over the SAS hardware infrastructure.

Referring now toFIG. 2, a block diagram illustrating in more detail the SAS expander102ofFIG. 1according to the present invention is shown. The SAS expander102includes a microprocessor202coupled to a memory204and a plurality of SAS ports216. The memory204stores code222instructions that are fetched and executed by the microprocessor202to accomplish the fault identification, isolation, reporting, repairing, and adaptive PHY configuration steps described herein. The memory204also stores threshold values224which the microprocessor202compares with counter values392(described below with respect toFIG. 3) to detect faulty communications, as discussed below. The memory204also stores an analog setting table226, counters228, and thresholds232(described below with respect toFIGS. 9,10, and11, respectively) used to perform the adaptive PHY configuration described herein.

Each SAS port216includes one or more SAS PHYs208connected to one of the SAS links112ofFIG. 1. The links112may be included in SAS cables. As shown, some of the SAS ports216are wide SAS ports216and some are narrow SAS ports216. Each SAS port216also includes a SAS SERDES circuit (not shown).

The SAS expander102also includes a set of control and status registers (CSRs)206associated with each PHY208, which the microprocessor202reads and writes to monitor fault detection parameters300(described below with respect toFIG. 3) and to control the PHYs208, such as to disable and enable the PHYs208and to perform adaptive PHY configuration. The fault detection parameters300and their use are described in more detail below with respect to the remaining Figures. In addition, the fault detection parameters300include interrupt indicators218from the PHYs208that are provided to notify the microprocessor202of events related to communication on the SAS links112.

The SAS expander102also includes multiplexed data paths (such as a crossbar) and switching circuitry (not shown) that interconnect the various PHYs208to enable them to transfer commands and data from one PHY208to another to perform the switching function of the SAS expander102. The SAS expander102may also include buffering circuits associated with each of the PHYs208for buffering the commands and data when received in a port216and when waiting to be transmitted out a port216. The commands and data are routed through the network between the ports216based on routing table information, which in one embodiment is stored in the memory204.

Referring now toFIG. 3, a block diagram illustrating fault detection parameters300according to the present invention is shown. The fault detection parameters300include the interrupt indicators218, values stored in the CSRs206, and values stored in the memory204ofFIG. 2. The fault detection parameters300may be categorized generally as counters392, interrupt indicators218, states394, thresholds396stored in the CSRs206and the thresholds224stored in the memory204ofFIG. 2. The fault detection parameters300also include a performance monitoring period372stored in a CSR206, whose use is described below. In one embodiment, the monitoring period372is 100 milliseconds; however, embodiments are contemplated in which different monitoring period values372are used.

The microprocessor202maintains a corresponding threshold396for each of the counters392. Some of the thresholds396are stored in the CSRs206, namely the disparity error interval threshold362and the code violation error interval threshold364, and the SAS expander102hardware automatically compares them with the corresponding counter392value and generates an interrupt if the threshold is exceeded. The thresholds224corresponding to the other counter392values are stored in the memory204, and the microprocessor202periodically compares the counter392values, or accumulated counts derived from the periodically sampled counter392values, with the thresholds224to identify faulty communications on the SAS links112.

The counters392include an invalid DWORD count302, which indicates the number of invalid DWORDs received outside PHY reset sequences; a disparity error count304, which indicates the number of running disparity errors received outside PHY reset sequences; a code violation count306, which indicates the number of times a decode error was detected on a bit stream; a loss of DWORD synchronization count308, which indicates the number of times the PHY208has restarted the link reset sequence because it lost dword synchronization (i.e., the number of times the PHY208went from PHY ready state to COMINIT state); a PHY reset failed count312, which indicates the number of times the PHY208has failed to obtain dword synchronization during final SAS speed negotiation; a CRC error count314, which indicates the number of CRC DWORD errors detected for received IDENTIFY and OPEN address frames; an in connection CRC error count316, which indicates the number of in connection CRC errors; and a PHY change count318, which indicates the number of PHY change events that have been generated.

The interrupt indicators218include a PHY ready interrupt322, which indicates the PHY208has finished initialization and is ready to transmit and receive data (A PHY208becomes ready only after COMINIT has been detected); a COMINIT interrupt324, which indicates a valid COMINIT out of band (OOB) sequence has been successfully negotiated; a elastic store overflow interrupt326, which indicates a valid DWORD was received and the internal elastic store, or buffer, is full; a disparity error interrupt328, which indicates the disparity error interval threshold362has been exceeded during the number of clock cycles specified in the performance monitoring period372; a code violation error interrupt332, which indicates the code violation error interval threshold364has been exceeded during the number of clock cycles specified in the performance monitoring period372; a DWORD synchronization loss interrupt334, which indicates DWORD synchronization on the PHY208was lost and consequently the PHY208has restarted the link reset sequence.

The states394include a link connected state342, which indicates whether the port216is in a connected state; a DWORD synchronization lost state344, which indicates the PHY208has currently lost DWORD synchronization; an init passed state346, which indicates whether the port216has successfully completed the link initialization sequence; a device present state348, which indicates whether a device is connected to the PHY208; an attached device type state352, which indicates whether a SAS or SATA device was detected as being connected; a rate state354, which indicates whether the final negotiated line rate is 1.5 or 3.0 Gbits/sec; a PHY reset limit saturation state356, which indicates that the PHY208reset threshold has been reached.

In one embodiment, the SAS expander102is configured to receive from the RAID controllers104SCSI Enclosure Services (SES) pages that set and get the various fault detection parameters300, that get the status of the PHYs208, and that directly enable or disable individual PHYs208. In one embodiment, control and status information, such as SES pages, may be sent via an out-of-band communication path between the SAS expanders102within an enclosure114, such as an I2C connection or other communication path. The out-of-band communication path may be advantageously employed if the SAS expander102has disabled all PHYs208connecting the SAS expander102to an upstream SAS expander102, such as might occur if the SAS cable connecting them is faulty. The disabling SAS expander102may communicate to the other SAS expander102in the enclosure114status information indicating that it has disabled the PHYs208. In this situation, to avoid rebooting, the user may cause the other SAS expander102in the enclosure114to broadcast an SES page via the out-of-band communication path to the PHY-disabled SAS expander102instructing the SAS expander102to re-enable the disabled PHYs208after the cable has been replaced. The out-of-band communication path is particularly useful for the SAS expanders102within an enclosure114that do not have an inter-expander SAS link112, which may not be present because the SAS specification does not allow loops within the SAS topology. Furthermore, the SAS expander102includes default values of the fault detection parameters300that are stored in a non-volatile memory of the SAS expander102and that are employed at boot time of the SAS expander102. The default values may be modified by the RAID controllers104or by the microprocessor202during operation.

Referring now toFIG. 4, a flowchart illustrating operation of the SAS system100ofFIG. 1is shown. Flow begins at block402.

At block402, the microprocessor202ofFIG. 2of the SAS expander102ofFIG. 1periodically monitors the fault detection parameters300ofFIG. 3of each of its PHYs208based on the performance monitoring period register372value ofFIG. 3. The interrupt indicators218are polled by the microprocessor202. In one embodiment, the interrupts indicators218are received asynchronously by the microprocessor202as interrupt request signals. In one embodiment, in response to monitoring the fault detection parameters300, the microprocessor202also updates a database that it maintains for providing status information to the RAID controllers104. In one embodiment, the status information is provided via SES pages. In one embodiment, in response to monitoring the fault detection parameters300, the microprocessor202also maintains and updates accumulated error counts stored in the memory204over multiple monitoring periods, as described in more detail with respect toFIG. 12. In one embodiment, the monitoring period is determined by a timer interrupt to the microprocessor202whose period is specified in the monitoring period register372. Flow proceeds to block404.

At block404, the microprocessor202identifies faulty communications on a SAS link112connected to one of its PHYs208based on the monitoring at block402. The microprocessor202analyzes the fault detection parameters300according to isolation rules embodied in the code222for fault indications to determine whether there is a need to disable a PHY208or to adaptively configure a PHY208. The identification of the faulty communications may include various criteria as discussed herein. An isolation rule may be triggered by one or more of the various counts exceeding a threshold, by detection that a PHY208has reached one or more particular states, that one or more particular events have occurred as indicated by one or more of the interrupt indicators218, and various combinations thereof. In one embodiment, the microprocessor202only identifies faulty communications related to a PHY208if the PHY208is enabled. In one embodiment, the microprocessor202only identifies faulty communications related to a PHY208if isolation or adaptive configuration is allowed for the PHY208. In one embodiment, the SAS expander102receives SES pages from the RAID controllers104to selectively enable and disable individual PHYs208and to selectively allow and disallow isolation or adaptive configuration of individual PHYs208. Flow proceeds to block406.

At block406, the microprocessor202writes to a control register206to disable the PHY208identified at block404. Flow proceeds to block408.

At block408, the SAS expander102reports the fact that the PHY208was disabled to one or both of the RAID controllers104. In one embodiment, the SAS expander102also reports the reason the PHY208was disabled. In one embodiment, the SAS expander102also reports all threshold values used by the SAS expander102to make a determination to disable the PHY208. In one embodiment, the SAS expander102reports by transmitting an SES diagnostic page to the RAID controller104. In one embodiment, the SAS expander102reports by transmitting a Serial Management Protocol (SMP) message to the RAID controller104. In one embodiment, the SAS expander102provides an interface to the RAID controllers104to enable the RAID controllers104to obtain the status of each PHY208and the current error counts, state, and events described herein. Flow proceeds to block412.

At block412, the RAID controller104reports that the PHY208was disabled to a user. In one embodiment, the RAID controller104reports to the user via a management interface. In one embodiment, the RAID controller104reports to the user by reporting to one or both of the hosts108, which in turn notify the user. Flow proceeds to block414.

At block414, assuming the disabled PHY208is part of a wide port216, communications between the SAS expander102port and the SAS device connected to the port216continue via the remaining PHYs208of the port216and associated SAS links112that are functioning properly. It is noted that the SAS system100may experience a proportionally lower data throughput due to the disabled PHY208and its respective SAS link112. However, advantageously, by disabling the PHY208associated with the faulty SAS link112(or the PHY208itself may have been faulty), the likelihood that the SAS system100will continue functioning normally is increased, thereby improving the availability of the data on the SAS disks106to the hosts108, rather than experiencing the various problems discussed herein. Flow ends at block414.

Referring now toFIG. 5, a flowchart illustrating operation of the SAS system100ofFIG. 1according to an alternate embodiment is shown. The operation of the SAS system100as described inFIG. 5assumes that the disabled PHY208was part of a narrow port216(rather than a wide port216as assumed with respect toFIG. 4) such that communication between the SAS expander102and the SAS device that was connected to the PHY208that was disabled at block406is no longer possible via the narrow SAS link112between the SAS expander102and the SAS device. Additionally,FIG. 5does not describe attempts to recover normal functioning of the faulty SAS link112, such as is described with respect toFIGS. 6 through 8. In situations where recovery of normal functioning is performed, the SAS system100may continue to operate as described inFIG. 4or5(depending upon whether the disabled PHY208was part of a wide or narrow port216) until the recovery of normal functioning is achieved.

Flow begins at block402. Blocks402through412ofFIG. 5are the same as like-numbered blocks ofFIG. 4and for the sake of brevity are not described again here. Flow proceeds from block412ofFIG. 5to block514.

At block514, the hosts108continue to access the SAS disks106implicated by the PHY208disabled at block406via an alternate pathway that does not include the disabled PHY208. With respect to the SAS system100ofFIG. 1, the hosts108will communicate with the SAS disks106via the other RAID controller104. Flow ends at block512.

Referring now toFIG. 6, a flowchart illustrating operation of the SAS system100ofFIG. 1according to an alternate embodiment is shown.FIG. 6describes operation of the SAS system100ofFIG. 1in which the SAS expander102additionally re-enables the previously disabled PHY208in response to user input that the fault has been corrected.

Flow begins at block402. Blocks402through412ofFIG. 6are the same as like-numbered blocks ofFIG. 4and for the sake of brevity are not described again here. Flow proceeds from block412ofFIG. 6to block614.

At block614, the user takes action to correct the faulty component in response to the reporting of the disabled PHY208at block412. Examples of action that the user may take to correct the faulty component include, but are not limited to, replacing a cable, replacing a connector, replacing a SAS disk106, replacing a SAS expander102, replacing a RAID controller104, and reconfiguring a PHY208, such as to adjust its analog settings (shown inFIG. 9). Flow proceeds to block616.

At block616, the user notifies one of the RAID controllers104that he has taken the corrective action at block614. In one embodiment, the user notifies the RAID controller104via a management interface. In one embodiment, the user notifies one of the hosts108, which in turn notifies the RAID controller104. Flow proceeds to block618.

At block618, the RAID controller104notifies the SAS expander102that the corrective action was taken. In one embodiment, the SAS expander102is notified by receiving a SCSI Enclosure Services (SES) diagnostic page from the RAID controller104. In one embodiment, the SAS expander102is notified by receiving a Serial Management Protocol (SMP) message from the RAID controller104. In one embodiment, the RAID controller104notifies the SAS expander102by explicitly instructing the SAS expander102to re-enable the PHY208. Flow proceeds to block622.

At block622, the microprocessor202writes to a control register206to re-enable the PHY208that was previously disabled at block406, in response to the notification that the corrective action was taken. Flow ends at block622.

In one embodiment, the microprocessor202foregoes disabling the PHY208at block406if the PHY208is linked to another SAS expander102that is downstream from a RAID controller104linked to the SAS expander102that detected the fault. This advantageously simplifies recovery of certain failure modes on a SAS topology involving cascaded SAS expanders102, such as the SAS system100ofFIG. 1. For example, assume a cable is faulty that connects a SAS expander102in each of two of the enclosures114ofFIG. 1and both of the SAS expanders102detect the fault and disable their respective PHYs208. In this example, to recover operation of the SAS link112once the cable is replaced may require coordination between the two SAS expanders102and potentially between the two RAID controllers104. In contrast, by foregoing disabling the PHY208at block406if the PHY208is linked to a downstream SAS expander102, the requirement for coordination is avoided.

Referring now toFIG. 7, a flowchart illustrating operation of the SAS system100ofFIG. 1according to an alternate embodiment is shown.FIG. 7describes operation of the SAS system100ofFIG. 1in which the SAS expander102additionally re-enables the previously disabled PHY208in response to automatically detecting that the fault has been corrected.

Flow begins at block402. Blocks402through412and614ofFIG. 7are the same as like-numbered blocks ofFIG. 6and for the sake of brevity are not described again here. Flow proceeds from block614ofFIG. 7to block716.

At block716, the microprocessor202automatically detects that the user took the corrective action at block614. In one embodiment, the user corrective action automatically detected by the microprocessor202is a user replacing a cable. The microprocessor202automatically detects the cable replacement by detecting a change of state from link not connected to link connected via the link connected state342fault detection parameter300. In one embodiment, the user corrective action automatically detected by the microprocessor202is a user replacing a SAS disk106or a SATA disk. The microprocessor202automatically detects the disk replacement by detecting a change of state from device not present to device present via the device present state348fault detection parameter300and detects whether the replaced disk is a SAS disk or a SATA disk via the attached device type state352. In one embodiment, the user corrective action automatically detected by the microprocessor202is a user replacing a SAS expander102. The microprocessor202automatically detects the SAS expander102replacement by detecting a change of state from device not present to device present via the device present state348fault detection parameter300of a PHY208connected to the replaced SAS expander102via the inter-expander SAS link112ofFIG. 1or via receiving status on the out-of-band communication path discussed above with respect toFIG. 3. In one embodiment, the user corrective action automatically detected by the microprocessor202is a user replacing a RAID controller104. The microprocessor202automatically detects the RAID controller104replacement by detecting a change of state from device not present to device present via the device present state348fault detection parameter300of a PHY208connecting the SAS expander102to the replaced RAID controller104. The above are provided as examples of automatically detected user corrective action performed by the microprocessor202; however, embodiments are not limited to those described above, but may be employed for other user corrective action. The user corrective action, such as a cable replacement, may cause the SAS expander102to perform adaptive PHY configuration, as described with respect toFIGS. 9 through 14. Flow proceeds to block718.

At block718, the microprocessor202writes to a control register206to re-enable the PHY208that was previously disabled at block406, in response to the automatic detection at block716that the corrective action was taken by the user. Flow ends at block718.

Referring now toFIG. 8, a flowchart illustrating operation of the SAS system100ofFIG. 1according to an alternate embodiment is shown.FIG. 8describes operation of the SAS system100ofFIG. 1in which the SAS expander102automatically takes action to attempt to correct the faulty condition and recover communications on the previously disabled PHY208.

Flow begins at block402. Blocks402through412ofFIG. 8are the same as like-numbered blocks ofFIG. 4and for the sake of brevity are not described again here. Flow proceeds from block412ofFIG. 8to block814.

At block814, the microprocessor202automatically takes corrective action. In one embodiment, the automatic corrective action taken by the microprocessor202is to automatically adjust the PHY208analog settings (shown inFIG. 9) which may cause the SAS link112to start functioning properly if, for example, the cable length has been changed since the last time the PHY208analog settings were set. This adaptive PHY configuration is described in more detail with respect toFIGS. 9 through 14. Flow proceeds to block816.

At block816, the microprocessor202writes to a control register206to re-enable the PHY208that was previously disabled at block406. Flow proceeds to decision block818.

At decision block818, the microprocessor202determines whether normal communications have been restored on the SAS link112after re-enabling the PHY208. If so, flow ends; otherwise, flow proceeds to block822.

At block822, the microprocessor202disables the PHY208again. Flow returns from block822to block814.

In one embodiment, the microprocessor202maintains a retry count threshold, and once the microprocessor202has performed the steps in the loop at blocks814to822a number of times that exceeds the retry threshold, the microprocessor202leaves the PHY208disabled and stops trying to automatically repair the fault until it detects an event indicating that it should re-enable the PHY208.

In one embodiment, the microprocessor202increases the period of the steps performed in the loop at blocks814to822each time it disables the PHY208at block822in order to reduce the number of SAS discover processes that must be performed in response to the PHY208disabling/re-enabling. A management application client performs a SAS discover process to discover all the SAS devices and expander devices in the SAS domain (i.e., determining their device types, SAS addresses, and supported protocols). A SAS initiator device uses this information to determine SAS addresses to which it is able to establish connections. A self-configuring expander device uses this information to fill in its expander route table. Additionally, if there are multiple disabled PHYs208that need re-enabling, then the microprocessor202re-enables all of the disabled PHYs208at the same time in order to further reduce the number of SAS domain discover processes that must be performed.

Referring now toFIG. 9, a block diagram illustrating a PHY configuration setting table226ofFIG. 2used to perform adaptive PHY configuration according to the present invention is shown. The memory204stores the PHY configuration setting table226. The table226includes a plurality of entries for each of a corresponding plurality of cable length ranges902. Each entry in the table226includes PHY208analog setting912values that affect the quality of the signals received or transmitted by the PHY208. Each entry includes a RX equalization field904value, a TX pre-emphasis, or pre-compensation, field906value, and a TX voltage swing field908value, which are referred to collectively as analog settings912. The use of equalization, pre-emphasis, and voltage swing to improve the quality of transmitted signals is well-known. The microprocessor202programs the analog setting912values into CSRs206of each PHY208of the SAS expander102to control the equalization of the PHY208receiver, pre-emphasis of the PHY208transmitter, and voltage swing of the PHY208transmitter, respectively. The population of the table226and its use in performing adaptive PHY configuration are described below with respect toFIGS. 12 through 13. The embodiment shown inFIG. 9illustrates four different entries in the table226; however, embodiments are contemplated with different numbers of entries. Furthermore, the cable length ranges902may vary.

Referring now toFIG. 10, a block diagram illustrating adaptive PHY configuration counters1002and table indexes1004into the configuration setting table226used to perform adaptive PHY configuration according to the present invention is shown. The memory204stores a counter1002and table index1004for each PHY208of the SAS expander102. The adaptive PHY configuration code222executing on the microprocessor202updates the counters1002at the end of each monitoring period based on the presence or absence of PHY events during the period, and uses the counters1002to determine when to automatically re-configure one or more of the PHYs208, as described in more detail with respect toFIGS. 12 through 13below. Additionally, the adaptive PHY configuration code222updates the table index1004for a respective PHY208each time it adaptively re-configures the PHY208, as described in more detail with respect toFIGS. 12 through 13below.

Referring now toFIG. 11, a block diagram illustrating adaptive PHY configuration thresholds1102and1104ofFIG. 2used to perform adaptive PHY configuration according to the present invention is shown. In one embodiment, the memory204stores individual threshold values for each PHY208type, which for the SAS expander102includes an ingress PHY208type and an egress PHY208type. An ingress PHY208is a PHY208on the side of the SAS expander102closest to the SAS initiator (i.e., closest to the RAID controller104inFIG. 1), and an egress PHY208is a PHY208on the side of the SAS expander102closest to the SAS target (i.e., closest to the SAS disks106inFIG. 1). At the end of each monitoring period, the adaptive PHY configuration code222executing on the microprocessor202compares the counter1002ofFIG. 10for each PHY208with the appropriate threshold1102/1104to determine when to automatically re-configure one or more of the PHYs208, as described in more detail with respect toFIGS. 12 through 13below. In one embodiment, the ingress PHY type threshold1102value is 8, and the egress PHY type threshold1102value is 4.

Referring now toFIG. 12, a flowchart illustrating operation of the SAS system ofFIG. 1to perform adaptive PHY configuration according to the present invention. Flow begins at block1202.

At block1202, a designer, such as the RAID controller104designer or SAS expander102designer, determines the values with which the adaptive PHY configuration setting table226ofFIG. 2should be populated. In one embodiment, the values in the table226are determined empirically by connecting a PHY208of the SAS expander102to a PHY of the RAID controller104or of a SAS disk106with different cables of varying lengths. For each cable of different length, the designer adjusts the analog settings of the SAS expander102PHY208and observes, such as with the aid of an oscilloscope, the eye mask generated during operation, i.e., during transmission of data between the SAS expander102and the RAID controller104or SAS disk106. The designer then selects for storage into the table226the analog setting values912that generate the best eye mask. The designer observes the analog setting values912that appear to provide the best eye masks for various ranges of lengths of SAS cable. As an example, the designer might determine that the test data falls into four sets of analog setting values912that best accommodate the various cable length ranges902. For example, the ranges may be: 1) less than 1 meter; 2) between 1 meter and 3 meters; 3) between 3 meters and 6 meters; and 4) greater than 6 meters. The analog settings values912are programmed into the table226. In one embodiment, the analog settings values912are compiled as part of the code222executed by the microprocessor202. Flow proceeds to block1204.

At block1204, the microprocessor202boots up and the initialization code222initializes the table index1004ofFIG. 10for each PHY208to a default value that selects the entry in the table226that contains the default analog settings912and configures each of the SAS expander102PHYs208with analog settings912from the default entry in the table226. In one embodiment, the default entry in the table226is selected as the entry corresponding to the cable length range902encompassing the most common length cable used by the RAID controller104manufacturer to connect the enclosures114. Flow proceeds to block1206.

At block1206, the initialization code222clears to zero the counter1002ofFIG. 10for each PHY208of the SAS expander102. Flow proceeds to block1208.

At block1208, the microprocessor202executes the code222to operate the SAS expander102PHYs208, i.e., to communicate with the remote PHYs208of the RAID controller104, other SAS expander102, or SAS disk106to which each SAS expander102PHY208is connected, for the monitoring period372ofFIG. 3. Flow proceeds to decision block1212.

At decision block1212, at the end of operation during the monitoring period372, the code222commences checking all the SAS expander102PHYs208to determine whether it has performed steps1214through1238for all of the PHYs208of the SAS expander102. If the code222determines that all of the SAS expander102PHYs208have been checked, flow returns to block1208to operate for another monitoring period372; otherwise, flow proceeds to block1214.

At block1214, the code222reads the PHY event registers for the current PHY208, such as the interrupt indicators218, counters392, and/or states394ofFIG. 3. In one embodiment, reading the event registers comprises reading one or more registers of the CSRs206that indicate the state of one or more of the interrupt indicators218ofFIG. 3. Flow proceeds to decision block1216.

At decision block1216, the code222determines whether a PHY event occurred for the current PHY208during the monitoring period372based on the information obtained at block1214. A PHY event is defined herein as detection by the PHY receiver of any of the following events:1) a running disparity error (such as indicated by a disparity error interrupt328and/or a non-zero value of the disparity error count304);2) a loss of DWORD synchronization (such as indicated by a DWORD synchronization loss interrupt334and/or a non-zero value of the loss of DWORD synchronization count308and/or predetermined value of the DWORD synchronization lost state344);3) an elasticity buffer overflow (such as indicated by elastic store overflow interrupt326);4) an 8b10b code violation (such as indicated by a code violation error interrupt332and/or a non-zero value of the code violation count306);5) a CRC error (such as indicated by a non-zero value of the CRC error count314or the in connection CRC error count316).
If a PHY event occurred for the current PHY208during the monitoring period372, flow proceeds to block1222; otherwise, flow proceeds to block1218.

At block1218, the code222decrements the counter1002associated with the current PHY208, since no PHY event occurred during the monitoring period372; however, the code222does not decrement the counter1002below zero. Flow returns to decision block1212.

At block1222, the code222increments the counter1002associated with the current PHY208, since at least one PHY event occurred during the monitoring period372. In one embodiment, rather than incrementing the counter1002by one, the code222increments the counter1002by a value greater than one, and the threshold1102/1104values are significantly larger. Flow proceeds to decision block1224.

At decision block1224, the code222determines whether the counter1002for the current PHY208has reached the relevant egress PHY type threshold1102or ingress PHY type threshold1104associated with the current PHY208type. If so, flow proceeds to block1228; otherwise, flow proceeds to block1226.

At block1226, the code222clears the relevant event registers, such as the counters392or interrupt indicator registers218, which indicated at decision block1216that a PHY208event occurred. Flow returns to decision block1212.

At block1228, the code222updates the table index1004of the PHY208. In one embodiment, the table index1004is incremented and is wrapped back to the first entry of the table226if the table index1004value exceeds the highest index value of the table226. Flow proceeds to decision block1232.

At decision block1232, the code222determines whether it has attempted to adaptively re-configure the PHY208with the settings of all the entries in the table226. If not, flow proceeds to block1234; otherwise, flow proceeds to block1234.

At block1234, the code222disables the PHY208, since there appears to be a problem with the link112that is not remediable by adaptively configuring the PHY208. Flow proceeds to block1226.

At block1236, the code222re-configures the PHY208with the analog settings912in the table226entry indicated by the entry index1004value that was updated at block1228. In one embodiment, the SAS expander102reports the adaptive configuration performed at block1236similar to the manner in which the SAS expander102reports the disabling of a PHY described above with respect to blocks408and412ofFIG. 4. Flow proceeds to block1238.

At block1238, the code222clears the counter1002to zero. Flow proceeds to block1226.

Referring now toFIG. 13, a table illustrating an example of operation of adaptive PHY configuration according toFIG. 12according to the present invention is shown. The table illustrates operation for a single PHY208of the SAS expander102ofFIG. 2. The table includes a number of consecutive monitoring periods, indicated as 0 through 26, each monitoring period being the time indicated by the value stored in the monitoring period register372ofFIG. 3. For each monitoring period, the table indicates whether a PHY event occurred according to block1216ofFIG. 12. For each monitoring period, the table also indicates the value of the PHY's208counter1002value. In the example ofFIG. 13, the PHY208is an egress PHY208type, and the egress PHY type threshold1102value is 4.

As shown, in monitoring period 0 no PHY event occurs so the counter1002value remains at its initial value of 0, according to block1222ofFIG. 12. In monitoring period 1 a PHY event occurs so the counter1002value is incremented to 1, according to block1222. In monitoring period 2 a PHY event occurs so the counter1002value is incremented to 2. In monitoring period 3 no PHY event occurs so the counter1002value is decremented to 1, according to block1218ofFIG. 12. In monitoring period 4 a PHY event occurs so the counter1002value is incremented to 2. In monitoring period 5 a PHY event occurs so the counter1002value is incremented to 3. In monitoring period 6 a PHY event occurs so the counter1002value is incremented to 4. Since the egress PHY type threshold1102has now been reached, as determined at block1224ofFIG. 12, the code222performs steps1228through1238ofFIG. 12; in particular, the code222re-configures the PHY208with the analog settings912in the table226entry indicated by the updated index and clears the counter1002to 0. The microprocessor202then continues to operate the PHY208with the new analog settings912to communicate with the remote PHY to which the PHY208is coupled.

In monitoring period 7 a PHY event occurs so the counter1002value is incremented to 1. In monitoring period 8 a PHY event occurs so the counter1002value is incremented to 2. In monitoring period 9 a PHY event occurs so the counter1002value is incremented to 3. In monitoring period 10 a PHY event occurs so the counter1002value is incremented to 4. Since the egress PHY type threshold1102has now been reached again, the code222again performs steps1228through1238ofFIG. 12by again re-configuring the PHY208with the analog settings912in the table226entry indicated by the updated index and clearing the counter1002to 0. The microprocessor202then continues to operate the PHY208with the new analog settings912to communicate with the remote PHY to which the PHY208is coupled.

In monitoring periods 11 through 13 no PHY event occurs so the counter1002value remains 0. In monitoring period 14 a PHY event occurs so the counter1002value is incremented to 1. In monitoring period 15 no PHY event occurs so the counter1002value is decremented to 0. In monitoring periods 16 through 26 no PHY event occurs so the counter1002value remains 0.

As may be observed fromFIG. 13, advantageously the SAS expander102operated according toFIG. 12to adaptively re-configure the analog settings912of the PHY208until it found analog settings912that appear to cause the PHY208to reliably operate with the remote PHY, in particular with respect to the length and/or transmission properties of the SAS cable attaching the PHY208to the remote PHY.

Referring now toFIG. 14, a flowchart illustrating operation of the SAS system100ofFIG. 1to improve the reliability of a SAS domain including a SAS expander102ofFIG. 2having a wide SAS port216according to the present invention is shown. Flow begins at block1402.

At block1402, the microprocessor202executes the code222to operate the SAS expander102to communicate via one of its wide ports216ofFIG. 2with a remote wide SAS port of the RAID controller104, other SAS expander102, or SAS disk106to which each SAS expander102wide port216is connected and monitors for faults during operation of the wide SAS port216in a manner similar to that described above with respect to blocks402and404ofFIG. 4and blocks1208through1216ofFIG. 12. Flow proceeds to decision block1404.

At decision block1404, the microprocessor202executes the code222to determine whether only one of the PHYs208of the wide port216experienced faults during the monitoring period. If so, flow proceeds to block1406; otherwise, flow proceeds to decision block1408.

At block1406, the microprocessor202executes the code222to effectively proceed to block406to disable the faulting PHY208and to report and correct the fault as described above with respect to one ofFIGS. 4 through 8. Flow returns to block1402.

At decision block1408, the microprocessor202executes the code222to determine whether all of the PHYs208of the wide port216experienced the same type of fault during the monitoring period. For example, all of the PHYs208of the wide port216may have experienced one or more of the PHY receiver events discussed with respect to decision block1216ofFIG. 12during the monitoring period. If all of the PHYs208of the wide port216experienced the same type of fault, flow proceeds to block1412; otherwise, flow returns to block1402.

At block1412, the microprocessor202executes the code222to effectively proceed to block1214to adaptively re-configure the PHYs208of the wide port216with new PHY configuration settings to accommodate differing cable lengths as described above with respect toFIG. 12. Flow returns to block1402.

Although the present invention and its objects, features, and advantages have been described in detail, other embodiments are encompassed by the invention. For example, although embodiments have been described in which the PHYs are SAS PHYs, the adaptive PHY configuration embodiments may also be employed with respect to SATA PHYs. For example, the PHY configuration setting table may include, for each cable length range, separate values for SAS and SATA. Furthermore, although embodiments have been described in which the PHYs are SAS expander PHYs, the adaptive PHY configuration embodiments may also be employed with respect to SAS end device PHYs, such as SAS disk PHYs or SAS initiator PHYs, in which case a different threshold value may exist for each of these PHY types. Still further, although embodiments have been described in which the adaptively configured analog settings include equalization, pre-emphasis, and voltage swing, other analog settings that affect the ability to reliably transceive signals between the SAS expander PHY and a remote PHY are contemplated.

Finally, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.