Patent Publication Number: US-9424118-B2

Title: Change message broadcast error detection

Description:
BACKGROUND 
     Small computer system interface (SCSI)-attached storage (SAS) systems have become increasingly popular, especially for enterprise environments and other large environments in which large amounts of data storage space is desirable. A SAS system can include initiator devices, expander devices, switches, and target devices. An initiator device is a controller that may be part of a host computing device to which data storage space is to be made available. A target device is a storage device, such as a hard disk drive, or an array of storage devices. An expander device permits multiple target devices to be connected to initiator devices. Similarly, a switch permits multiple target devices to be connected to initiator devices, but may further able to segment the target devices and the initiator devices among different zone groups. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a diagram of an example system. 
         FIG. 2  is a flowchart of an example method to detect an error within a system. 
         FIG. 3  is a flowchart of an example method to determine a cause of an error that has been detected within a system. 
         FIG. 4  is a diagram of an example hardware device. 
     
    
    
     DETAILED DESCRIPTION 
     As noted in the background section, small computer system interface (SCSI)-attached storage (SAS) systems have become increasingly popular. A SAS system can be a hot-pluggable system, which means that devices can be connected or disconnected within the system without having to power down the system. Each time a device is electrically connected or disconnected within a SAS system, a change message is broadcast within the system. 
     Specifically, a SAS broadcast change message is broadcast within a SAS system each time a device is electrically connected or disconnected within the system. A SAS broadcast change message is more particularly referred to as a BROADCAST (CHANGE) message. A SAS broadcast change message is specifically broadcast by a physical interface of a device within a SAS system to which a physical interface of another device that is being electrically connected or disconnected within the system. The physical interface of a device within a SAS system is referred to as a PHY interface, or simply as a PHY. 
     For example, a SAS system may already include an expander device. If a PHY of a target device is electrically connected to a PHY of this expander device, such as via a SAS cable, the PHY of the expander device issues a SAS broadcast change message within the SAS system. Similarly, if the PHY of the target device is subsequently electrically disconnected from the PHY of the expander device, the PHY of the expander device also issues a SAS broadcast change message within the SAS system. 
     What is referred to as a broadcast change message storm can occur within a SAS system, however, due to an error occurring within the system. For example, the SAS cable electrically connecting the PHY of an expander device to the PHY of a target device may become faulty, such that electrical connection between the PHYs may become intermittent. As such, the PHY of the expander device may issue a large number of SAS broadcast change messages over a short period of time, each time the electrical connection in question is disrupted, and again each time the electrical connection is temporarily resumed. 
     A SAS broadcast change message generally indicates just that a change has occurred within a SAS system, and does not otherwise provide much, if any, information regarding the change. Therefore, each time a SAS broadcast change message is broadcast within a SAS system, one or more devices within the SAS system may initiate discovery of the devices that are now connected within the SAS system. This discovery process can entail significant overhead both in terms of processing bandwidth in each such device, and in terms of communication bandwidth within the SAS system as a whole. 
     As such, when a broadcast change message storm occurs within a SAS system, the system may undesirably slow down, and even become effectively nonfunctional. Each device that initiates discovery of the devices that are connected within the SAS system may not be able to perform other types of processing. The communication bandwidth within the SAS system may become overwhelmed by the various messages sent within the system for achieving such discovery. 
     Disclosed herein are example techniques for detecting an error within a system like a SAS system, such as that which results in a broadcast change message storm. Specifically, a hardware device within the system, like a switch or other hardware device, detects the change messages broadcast within the system. The hardware device determines whether the change messages were broadcast every first period of time or less for at least a second period of time. For instance, the hardware device may determine whether the change messages were broadcast every eight seconds or less for at least sixty seconds. If this is the case, then the hardware device signals that an error has been detected. 
     Also disclosed herein are example techniques for detecting where such an error that has been detected occurs within a system like a SAS system. The error may therefore be able to be resolved. Once an error has been detected by a hardware device within the system, the hardware device broadcasts a report command within the system, such as a SAS broadcast report command that is also referred to as a SAS REPORT BROADCAST command. In response, the hardware device receives a first number of change messages broadcast by each interface of each other device within the system. 
     When the next change message broadcast within the system occurs less than the first period of time since the previous change message was broadcast, the hardware device broadcasts another report command within the system. In response, the hardware device receives a second number of change messages broadcast by each interface of each other device within the system. For each such interface, the hardware device determines whether the first and second numbers are identical. The hardware device thus determines the location of the error based on the interface where a change from the first number to the second number has occurred. 
       FIG. 1  shows an example SAS system  100 . The SAS system  100  includes host devices  102 A and  102 B, collectively referred as to the host devices  102 ; a switch  104  having an expander device  116 ; other expander devices  106 A,  106 B, and  106 C, collectively referred to as the expander devices  106 ; and, target devices  108 A,  108 B,  108 C,  108 D, and  108 E, collectively referred to as the target devices  108 . The number of each type of device within the SAS system  100 , and how the devices are interconnected with one another, can vary; the number of and interconnection among the devices as in  FIG. 1  is just an example. 
     The host devices  102  can each be a computing device, such as a desktop computer. The host devices  102  include initiator devices  110 A and  110 B, which are collectively referred to as the initiator devices  110 . The initiator devices  110  are more generally referred to as controller devices. The initiator device  110 A includes four PHYs  112 A, whereas the initiator device  110 B includes two PHYs  112 B and two PHYs  112 C. 
     As noted above, each PHY  112  is a physical interface, which may also be referred to as a hardware interface. A PHY  112  is a smart interface, in that it is able to perform processing, such as sending and receiving messages, as well as storing values. In this respect, the terminology interface as used herein is not just a number of electrical connectors that permit a device to be connected to corresponding connectors of a cable or another device. Rather, an interface is itself able to perform processing, apart from its constituent device, as is described in more detail below. 
     The switch  104  includes the expander device  116  that includes four PHYs  112 D, two PHYs  112 E, PHY  112 F, and PHY  112 G. The PHYs  112 D,  112 E,  112 F, and  112 G of the expander device  116  are also referred to herein as the PHYs  112 D,  112 E,  112 F, and  112 G of the switch  104  itself. Via a multiple-SAS cable lane  114 A, the PHYs  112 D of the switch  104  are directly physically and electrically connected to the PHYs  112 A of the initiator device  110 A. Via a narrower multiple-SAS cable lane  114 B, the PHYs  112 E of the switch  104  are directly physically and electrically connected to the PHYs  112 E of the initiator device  110 A. 
     The expander device  106 A includes two PHYs  112 H, PHY  112 I, and PHY  112 J. Via a multiple-SAS cable lane  114 C, the PHYs  112 H of the expander device  106 A are directly physically and electrically connected to the PHYs  112 C of the initiator device  110 B. The expander device  106 B includes two PHYs  112 L and PHY  112 M. Via a multiple-SAS cable lane  114 F, the PHYs  112 L of the expander device  106 B are directly physically and electrically connected to the PHYs  112 I of the expander device  106 A. The expander device  106 C includes PHYs  112 N,  112 O, and  112 P. Via a single SAS cable  114 E, the PHY  112 N of the expander device  106 C is directly physically and electrically connected to the PHY  112 G of the switch  104 . 
     The target devices  108  can each be a storage device, or an array of storage devices. An example of such a storage device includes a hard disk drive. The target device  108 A includes a PHY  112 K that is directly physically and electrically connected to the PHY  112 F of the switch  104  via a single SAS cable  114 D. The target device  108 B includes a PHY  112 Q that is directly physically and electrically connected to the PHY  112 M of the expander device  106 B via a single SAS cable  114 H. 
     The target device  108 C includes a PHY  112 R that is directly physically and electrically connected to the PHY  112 O of the expander device  106 C via a single SAS cable  114 I. The target device  108 D includes a PHY  112 S that is directly physically and electrically connected to the PHY  112 J of the expander device  106 A via a single SAS cable  114 G. The target device  108 E includes a PHY  112 T that is directly physically and electrically connected to the PHY  112 P of the expander device  106 C via a single SAS cable  114 J. 
     The initiator devices  110  permit their host devices  102  to access the target devices  108  within the SAS system  100 . Specifically, the host device  102 A may be able to access the target devices  108 A,  108 C, and  108 E, but is unable to access the target devices  108 B and  108 D. By comparison, the host device  102 B may be able to access all the target devices  108 . 
     The switch  104  is able to divide the initiator devices  110  and the target devices  108 A,  108 C, and  108 E to which it is connected into different zone groups. For example, the switch  104  may permit the initiator device  110 A to access the target device  108 A, but not the target devices  108 C and  108 D. The switch  104  may also permit the initiator device  110 B to access all three target devices  108 A,  108 C, and  108 E. 
     In this example, the switch  104  defines the target devices  108 A,  108 C, and  108 E into two zone groups: a first zone group including the target device  108 A, and a second zone group including the target devices  108 C and  108 E. The initiator device  110 A is a member of just the first zone group. By comparison, the initiator device  110 B is a member of both zone groups. 
     Both the switch  104  and the expander devices  106  are each able to permit more than one target device  108  downstream to be connected to the same upstream PHY  112 . For example, the expander device  106 C permits two downstream target devices  108 C and  108 E to be connected to the same upstream PHY  112 G. In  FIG. 1 , downstream communication is from the top of the figure to the bottom of the figure, and upstream communication is from the bottom of the figure to the top of the figure. Unlike the switch  104 , the expander devices  106  are unable to divide their connected devices into different zone groups. 
     The PHYs  112  of the target devices  108  are typically receptive to connection of a single SAS cable  114 . For instance, the PHY  112 K of the target device  108  is connected to the PHY  112 F of the switch  104  via the single SAS cable  14 D. By comparison, the PHYs  112  of the initiator devices  110 , the switch  104 , and the expander devices  106  can be interconnected via multiple-SAS cable lanes  114  or by single SAS cables  114 . For instance, the PHYs  112 A of the initiator device  110 A are connected to the PHYs  112 D of the switch  104  via a four-SAS cable lane  114 A, whereas the PHYs  112 I of the expander device  106 A are connected to the PHYs  112 L of the expander device  106 B via a two-SAS cable lane  114 F. Furthermore, the PHY  112 G of the switch  104  is connected to the PHY  112 N of the expander device  106 C via a single SAS cable  114 E. 
     In general, the more SAS cables that interconnect two PHYs  112 , the greater the communication bandwidth between the two PHYs  112 . For instance, the communication bandwidth between the PHYs  112 I and  112 L of the expander devices  106 A and  106 B can be up to twice the communication bandwidth between the PHY  112 J of the expander device  106 A and the PHY  112 S of the target device  108 D. Similarly, the communication bandwidth between the PHYs  112 A of the initiator device  110 A and the PHYs  112 D of the switch  104  can be up to twice the communication bandwidth between the PHYs  112 B of the initiator device  110 B and the PHYs  112 E of the switch  104 , and up to four times the communication bandwidth between the PHY  112 G of the switch  104  and the PHY  112 N of the expander device  106 C. 
     It is noted that  FIG. 1  shows to at least some extent more of a theoretical or architectural depiction of the representative SAS system  100  than how such a SAS system  100  is implemented in actuality. In actuality, each expander device  106  is typically part of another physical device, and is not a separate physical device in and of itself. For instance, the expander device  106 C may be part of a target device enclosure, such as a storage device enclosure, which also includes the target devices  108 C and  108 E. The expander device  106 A may be part of another switch, and the expander device  106 B may be part of a target device enclosure that also includes the target device  108 B. In this example, the target device  108 D may be a stand-alone target device. Furthermore, each PHY  112  is typically part of a physical port, where there may be one PHY  112 , or more than one PHY  112 , for each physical port. 
     The SAS system  100  is a hot-pluggable system. As noted above, this means that devices within the SAS system  100  can be electrically connected and disconnected without first powering down the SAS system  100 . Each time a device is electrically connected or disconnected within the SAS system  100 , a SAS broadcast change message is broadcast within the system  100 . For example, if the SAS cable  114 I interconnecting the PHY  112 O of the expander device  106 C and the PHY  112 R of the target device  108 C is removed from either or both the PHY  112 O and the PHY  112 R, the PHY  112 O broadcasts a SAS broadcast change message within the SAS system  100 . As another example, if the PHY  112 S of the target device  108 D is newly connected to the SAS cable  114 F already connected to the PHY  112 J of the expander device  106 A, the expander device  106 A (such as a PHY thereof) broadcasts a SAS broadcast change message within the SAS system  100 . 
     If a PHY  112  or a SAS cable  114  is faulty, the corresponding electrical connection can become intermittent. For example, if the PHY  112 S of the target device  108 D or the SAS cable  114 G connecting this PHY  112 S with the PHY  112 J of the expander device  106 A is faulty, each time the electrical connection is disrupted, the PHY  112 J broadcasts a SAS broadcast change message. Similarly, each time the electrical connection is temporarily resumed, the expander device  106 A (such as a PHY thereof) broadcasts a SAS broadcast change message. A broadcast change message storm can result, which can impede the proper functioning of the SAS system  100 . 
       FIG. 2  shows an example method  200  for detecting such an error within a system like the SAS system  100  of  FIG. 1 . The method  200 , like other example methods disclosed herein, can be performed by a hardware device within the system. For example, the method  200  may be performed by the switch  104  of  FIG. 1 . 
     Furthermore, the method  200 , like other example methods disclosed herein, can be implemented as one or more computer programs stored on a non-transitory computer-readable data storage medium. Execution of the computer program by a processor of a hardware device, such by a processor of the switch  104  of  FIG. 1 , causes the method  200  to be performed. It is noted that such an implementation encompasses the computer program being implemented via an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA) within the hardware device. 
     The change messages broadcast within the system are detected ( 202 ). It is determined whether the change messages were broadcast every first period of time, or less, for at least a second period of time ( 204 ). For instance, it may be determined whether the change messages were broadcast every eight seconds or less for at least sixty seconds. In general, it can be determined whether the change messages were broadcast every M seconds or less for at least N seconds, where M is less than N. 
     If the change messages were broadcast every first period of time, or less, for at least a second period of time, then it is concluded that an error has been detected, and this error is signaled ( 206 ). For instance, the hardware device performing the method  200  may send a message that a broadcast change message storm has been detected. The hardware device may signal the error in other ways as well, such as by sounding an audible alarm. 
       FIG. 3  shows an example method  300  for determining the cause of an error within a system, such as the SAS system  100  of  FIG. 1 , once the error has been detected. That a broadcast change message storm has been detected does not provide the cause of this error. This is because change messages that are broadcast within the system do not indicate the PHY that broadcast these messages. As such, the method  300  provides a technique for determining the cause of this error within the system. 
     Responsive to detecting that the error has occurred within the system, such as resulting from the method  200  of  FIG. 2  being performed, a report command is broadcast ( 302 ). The report command may be a SAS REPORT BROADCAST command broadcast by the hardware device performing the method  300 . In response, a (first) number of change messages broadcast by each interface within the system, such as each PHY  112  within the SAS system  100  of  FIG. 1 , is received ( 304 ). The number of change messages broadcast by an interface may be reported via a message including a BROADCAST COUNT value of the interface. For example, each PHY  112  may report that it has broadcast one change message within the SAS system  100 , where this information is received by the switch  104  of the system  100 . 
     In response to detecting an additional change message has been broadcast within the system ( 306 ), if the additional change message was broadcast less than the first period of time since a previous change message was broadcast within the system, then an additional report command is broadcast within the system ( 308 ). For example, if the next change message was broadcast less than eight seconds since the previous change message was broadcast, then the hardware device performing the method  300  sends an additional report command. The additional report command may also be a SAS REPORT BROADCAST command broadcast by the hardware device performing the method  300 , such as the switch  104  of the SAS system  100  of  FIG. 1 . 
     In response, a (second) number of change messages broadcast by each interface within the system, such as each PHY  112  within the SAS system  100  of  FIG. 1 , is received ( 310 ). As before, the number of change messages broadcast by an interface may be reported via a message including a BROADCAST 
     COUNT value of the interface. In the prior example, each PHY  112  may have originally reported that it has broadcast one change message within the SAS system  100 . An additional change message may have been broadcast by the PHY  112 J of the expander device  106 A, due to the PHY  112 S of the target device  108 D no longer being electrically connected to the PHY  112 J. In response to the additional report command broadcast in part  308 , the PHY  112 J reports it has broadcast two change messages within the SAS system  100 . By comparison, each other PHY  112  still reports that it has just broadcast one change message within the SAS system  100 . 
     Therefore, for each interface, such as for each PHY  112  within the SAS system  100  of  FIG. 1 , it is determined whether the first number received in part  304  is identical to the second number received in part  310  ( 312 ). In this way, the hardware device performing the method  300 , such as the switch  104  of  FIG. 1 , is able to locate where the error occurred within the system. In the prior example, just the PHY  112 J sent a second number received in part  310  that was different from the first number sent by this PHY in part  304 . As such, the hardware device is able to determine the cause of the error, based on the interface where a change has occurred from the first number received in part  304  to the second number received in part  310  ( 314 ). In the prior example, the hardware device is able to determine the cause of the error based on the fact that the PHY  112 J is the interface where such a change has occurred. 
     Three example techniques for determining the cause of the error in part  314  are disclosed herein. First, if the change resulted from an interface of an expander device connected directly to an interface of another expander device by a single cable, it may be concluded that the cause of the error is that this single cable is faulty ( 316 ). In this respect, it may be assumed that the likelihood that either interface is faulty is low, such that the cause of the error is more likely to be the cable connecting the two interfaces together. This information may be sent by the hardware device performing the method  300 , so that a network administrator or other user can act upon it. The interface in question of the upstream expander device may be temporarily disabled, so that the error is temporarily suppressed. 
     Second, if the change resulted from an interface of an expander device connected directly to an interface of another expander device by a multiple-cable lane, it may be concluded that the cause of the error is that this multiple-cable lane is faulty ( 318 ), such that one or more cables of the lane are faulty. In this respect as well, it may be assumed that the likelihood that either interface is faulty is low, such that the cause of the error is more likely to be the multiple-cable lane connecting the two interfaces together. This information may likewise be sent by the hardware device performing the method  300 , so that a network administrator or other user can act upon it. The interface in question of the upstream expander device may be temporarily disabled, so that the error is temporarily suppressed. 
     Third, if the change resulted from an interface of an expander device connected directly to an interface of a target device by a single cable, it may be concluded that the single cable or the target device itself, such as the interface of this target device, is faulty ( 320 ). In this respect, it may be assumed that the likelihood that the interface of the expander device is faulty is low, such that the cause of the error is more likely to be the single cable or the target device. This information may also be sent by the hardware device performing the method  300 , so that a network administrator or other user can act upon it. The interface of the expander device may be temporarily disabled, so that the error is temporarily suppressed. 
     The example methods  200  and  300  of  FIGS. 2 and 3  thus work together to detect errors such as broadcast change message storms, and to determine the cause of and temporarily suppress these errors. The error detection of the method  200  does not provide the cause of an error, but the method  300  does. Once the cause of an error has been detected, the interface that is broadcasting enough change messages to cause a broadcast change message storm can be temporarily disabled, so that the remaining devices within the system can still function. 
     In conclusion  FIG. 4  shows an example hardware device  400 . The example hardware device  400  can perform the methods  200  and  300  of  FIGS. 2 and 3 . The switch  104  of  FIG. 1  may be one such hardware device  400 . 
     The hardware device  400  includes one or more hardware interfaces  402  and two components  404  and  406 . Each hardware interface  402  communicatively connects the hardware device  400  within a system that includes the hardware device, one or more controller devices, one or more expander devices, and one or more target devices. For instance, the system may be the SAS system  100  of  FIG. 1 , which includes initiator devices  110 , expander devices  106 , and target devices  108 . In this example, each hardware interface  402  is a PHY. 
     The components  404  and  406  are each implemented at least in hardware. For example, each component  404  and  406  may be a computer program that is stored on a non-transitory computer-readable data storage medium of the hardware device  400  and that is executed by a processor of the device  400 . The component  404  is an error detection and signaling component, and thus performs the method  200  of  FIG. 2  that has been described. The component  406  is an error cause determination and remediation component, and thus performs the method  300  of  FIG. 3  that has been described.