Patent Publication Number: US-7913023-B2

Title: Specifying lanes for SAS wide port connections

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Application Ser. No. 61/191,037, filed Sep. 5, 2008, which is incorporated herein by reference. The present application is related to copending applications “COMBINING MULTIPLE SAS EXPANDERS TO PROVIDE SINGLE SAS EXPANDER FUNCTIONALITY” Ser. No. 12/384,289, filed on the same date as the present application by inventors Stephen B. Johnson, Timothy E. Hoglund, and Louis H. Odenwald, Jr.; “METHOD FOR PROVIDING PATH FAILOVER FOR MULTIPLE SAS EXPANDERS OPERATING AS A SINGLE SAS EXPANDER” Ser. No. 12/384,291, filed on the same date as the present application by inventors Christopher McCarty and Stephen B. Johnson; and “SAS PAIRED SUBTRACTIVE ROUTING” Ser. No. 12/384,288, filed on the same date as the present application by inventors Stephen B. Johnson, William Petty, and Owen Parry. All of these applications are incorporated herein by reference. 
    
    
     TECHNICAL FIELD 
     The present disclosure generally relates to the field of Serial Attached SCSI, and more particularly to a system, method, and product for specifying allowable lanes of SAS wide port connections in a SAS topology. 
     BACKGROUND 
     Serial Attached SCSI (SAS) is a term referring to various technologies designed to implement data transfer between computer devices. The SAS protocol is a serial successor to the parallel Small Computer System Interface. In the SAS protocol, all SAS devices are either an initiator device, a target device, or an expander device. Initiator devices are devices that begin an SAS data transfer, while target devices are the devices to which initiator devices transfer data. Together, initiator devices and target devices are known as end devices. 
     SAS expanders are devices that facilitate data transfer between multiple initiator devices and multiple target devices. The SAS protocol utilizes a point-to-point bus topology. Therefore, if an initiator device is required to connect to multiple target devices, a direct connection must be made between the initiator device and each individual target device in order to facilitate each individual data transfer between the initiator device and each individual target device. SAS expanders manage the connections and data transfer between multiple initiator devices and multiple target devices. SAS expanders may contain SAS devices. 
     SUMMARY 
     A method for specifying allowable lanes of a Serial Attached Small Computer System Interface (SAS) wide port for a data connection between a SAS initiator and an SAS target in a SAS domain may include, but is not limited to: discovering the optimal lanes of at least one SAS wide port for the data connection; specifying the allowable lanes for the data connection within the SAS wide ports of each level of the SAS domain; checking for the specified allowable lanes for the data connection; and creating the data connection on the specified allowable lanes. 
     A system may include, but is not limited to: means for discovering the optimal lanes of at least one SAS wide port for the data connection; means for specifying the allowable lanes for the data connection within the SAS wide ports of each level of the SAS domain; means for checking for the specified allowable lanes for the data connection; and means for creating the data connection on the specified allowable lanes. 
     A circuit for creating a data connection a SAS initiator to a plurality of SAS targets may include, but is not limited to: a SAS initiator; a first SAS expander connected to the SAS initiator via a SAS wide port; a second SAS expander including at least a first phy; a third SAS expander including at least a second phy; at least one common SAS wide port for combining at least the first numbered phy and the second numbered phy; wherein the first SAS expander is connected to the second SAS expander and the third SAS expander via the at least one common SAS wide port; wherein the second SAS expander is directly connected to the third SAS expander via the phys of the second SAS expander and the phys of the third SAS expander for inter-expander communications; and a plurality of SAS targets connected to the second SAS expander and the third SAS expander. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not necessarily restrictive of the present disclosure. The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate subject matter of the disclosure. Together, the descriptions and the drawings serve to explain the principles of the disclosure. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The numerous advantages of the disclosure may be better understood by those skilled in the art by reference to the accompanying figures in which: 
         FIG. 1  is a block diagram illustrating a SAS topology illustrating a single, cohesive SAS expander; 
         FIGS. 2A through 2E  are block diagrams illustrating various configurations of a single, cohesive SAS expander; 
         FIG. 3  is block diagram illustrating a blade center switch configuration of a single, cohesive SAS expander; 
         FIG. 4  is a block diagram illustrating the inter-expander links of a single, cohesive SAS expander; 
         FIG. 5  is a block diagram illustrating a failed internal link within a single, cohesive SAS expander; 
         FIG. 6  is a flow diagram illustrating a method for combining multiple SAS expanders to operate as a single, cohesive SAS expander; 
         FIG. 7  is a flow diagram illustrating a method for providing path failover when a link fails within a single, cohesive SAS expander 
         FIG. 8  is a block diagram illustrating selecting lanes of SAS wide ports for a data connection with a single, cohesive SAS expander; 
         FIG. 9  is a flow diagram illustrating a method for specifying lanes of SAS wide ports for a data connection with a single, cohesive SAS expander. 
         FIG. 10  is a block diagram illustrating a cascaded SAS topology configured for performing paired subtractive routing; 
         FIGS. 11A-11D  are tables illustrating a vendor unique function, commands and descriptors for paired subtractive ports; and 
         FIG. 12  is a flow diagram illustrating a method of performing paired subtractive routing in a cascaded SAS topology. 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to the subject matter disclosed, which is illustrated in the accompanying drawings. 
     A SAS topology featuring a single, cohesive SAS expander in accordance with an exemplary embodiment of the present disclosure is shown. The topology  100  may include initiator  110  connected to expander  120  via narrow ports  115   a . . . d . Expander  120  may be connected to single, cohesive SAS expander  200  via narrow ports  125   a . . . d . Single, cohesive SAS expander  200  may contain a plurality of physically separate SAS expanders  202   a . . . d . SAS expanders  202   a . . . d  may be configured to share an identical SAS address to combine narrow ports  125   a . . . d  to behave as single common wide port  150 . As shown in SAS topology  100 , narrow ports  125   a . . . d  may combine to make x4-wide port  150 . SAS expanders  202   a . . . d  may be connected to expander  130  via narrow ports  135   a . . . d . Expander  130  may be connected to target  140  via narrow ports  145   a . . . d . SAS specifications require that all connections be point-to-point. However, SAS topology  100  is SAS specification compliant because wide port  150  is a true SAS wide port. 
     Referring to  FIGS. 2A-2E , cohesive SAS expander  200  may contain a plurality of physically separated SAS expanders  202   a . . . h . SAS expanders  202   a . . . h  may be connected to output ports  205 - 244  via links  255   a . . . h - 294   a . . . h . Widely varying configurations of single, cohesive SAS expander  200  are contemplated by the present disclosure, depending on the number of SAS expanders  202   a . . . h , the number of ports of SAS expanders  202   a . . . h , the number of output ports  205 - 244 , the width of output ports  205 - 244 , and the number of SAS expander ports used in links  255   a . . . h - 294   a . . . h.    
     Output ports  205 - 244  may be wide ports. Output ports  205 - 244  may be configured to share an identical SAS address. Under SAS specifications, this defines output ports  205 - 244  to be a single wide port of the same expander. Each of output ports  205 - 244  may be connected to every SAS expander  202   a . . . h  (Not shown). The width of output ports  205 - 244  may depend on how many SAS expander ports are used in links  255   a . . . h - 294   a . . . h  to connect to each of output ports  205 - 244 . Each of SAS expander  202   a . . . h  may have multiple SAS expander ports connected to each of output ports  205 - 244 . For example, if single, cohesive SAS expander  200  contains two SAS expanders  202   a - h  connected to each of output ports  205 - 244  via links  255   a . . . h - 294   a . . . h  using one SAS expander port, then output ports  205 - 244  will be a x2 wide port. Furthermore, if single, cohesive SAS expander  200  contains two SAS expanders  202   a - h  connected to each of output ports  205 - 244  via links  255   a . . . h - 294   a . . . h  using two SAS expander ports, then output ports  205 - 244  will be a x4 wide port. These configurations may allow for simultaneous access of any port to any other port of single, cohesive SAS expander  200  at full port bandwidth. 
     A blade center switch configuration  301  of single, cohesive SAS expander  300  is provided. Referring to  FIG. 3 , multiple CPU blades  302  may be connected to input ports  305 - 320 . Input ports  305 - 320  may be wide ports. Input ports  305 - 320  may be configured to share an identical SAS address. Under SAS specifications, this defines input ports  305 - 320  to be a single wide port of the same expander. Each of input ports  305 - 320  may be connected to each of SAS expanders  202   a . . . b  via links  355   a . . . b - 375   a . . . b  (Not shown). As shown in blade center switch configuration  301 , input ports  305 - 320  are x2 wide ports connected to two SAS expanders  202   a . . . b  via links  355   a . . . b - 375   a . . . b  using one SAS expander port, but other configurations are fully contemplated by these disclosures. Each of SAS expanders  202   a . . . b  may be connected to each of output ports  205 - 218  via links  255   a . . . b - 268   a . . . b  (Not shown). As shown in blade center switch configuration  301 , output ports  205 - 218  are x4 wide ports connected to two SAS expanders  202   a . . . b  via links  255   a . . . b - 268   a . . . b  using two SAS expander ports, but other configurations are fully contemplated by the current disclosure. Output ports  205 - 218  may be connected to multiple data storage devices  303 . 
     SAS expanders  202   a . . . h  may be configured to share an identical SAS address. Under SAS specifications, this defines SAS expanders  202   a . . . h  to be a single expander. Firmware may run on each of SAS expanders  202   a . . . h  so SAS expanders  202   a . . . h  behave and respond as a single expander. As shown in  FIG. 4 , each of SAS expanders  202   a . . . h  may be connected to each of  202   a . . . h  via inter-expander links (IEL)  430 - 435 . IEL  430 - 435  may allow SAS expanders  202   a . . . h  to communicate and coordinate to behave and respond as a single expander. IEL  430 - 435  may communicate via the SAS Management Protocol (SMP) or other communication methods, such as Inter-Integrated Circuit Bus Protocol (I2C), Enhanced Parallel Port (EPP), Ethernet, shared memory, and the like. IEL  430 - 435  may permit the IEL domain  420  to be zoned from primary switched domain  410 . Broadcasts may be disabled in IEL  430 - 435  to eliminate any SAS specification non-compliant loop issues. IEL  430 - 435  may use multiple SAS expander ports. The phys used in IEL  430 - 435  may be completely hidden from the primary switched domain. The numbering of the phys used in the primary switched domain of SAS expanders  202   a . . . h  may be remapped to a single, logical numbering. IEL  430 - 435  may allow any combination of SAS expanders  202   a . . . h  to fail while connectivity is maintained between any functional SAS expanders  202   a . . . h . Further, all input ports  305 - 320  and output ports  205 - 244  may maintain connectivity at a reduced bandwidth in the event of less than every SAS expander  202   a . . . h  failing. SMP target processing may be handled by a single master SAS expander of single, cohesive SAS expander  200 . Connection requests to single, cohesive SAS expander  200  may be routed to the single master SAS expander via IEL  430 - 435 . SAS expanders  202   a . . . h  may share SMP target processing. 
     Referring generally to  FIG. 6 , a method for combining multiple SAS expanders is shown. For example, the method may implement techniques for connecting and combining SAS expanders as described below (and as shown in  FIGS. 1 ,  2 A through  2 E,  3 , and  4 ). The method  600  may include the step of grouping at least one first numbered phy of a first SAS expander with at least one second numbered phy of a second SAS expander physically separate from the first SAS expander into at least one common SAS wide port  602 . For example, the grouping into at least one common SAS wide port ( 205 - 244 ) may include assigning an identical SAS address to the at least one common wide port ( 205 - 244 ). 
     The method  600  may further include the step of assigning an identical SAS address to the first SAS expander and the second SAS expander  604 . Assigning an identical SAS address to the first SAS expander and the second SAS expander may allow the first SAS expander and the second SAS expander to behave and respond as a single, cohesive SAS expander  200 . The assigning may be performed via firmware executing on the first SAS expander and the second SAS expander. Further, the phy numbering of the first SAS expander and the second SAS expander may be remapped to appear as a single, logically ordered phy numbering of a single, cohesive SAS expander. For example, a first numbered phy of the first SAS expander may be remapped to be in a first range of the single, logically ordered numbering and a second numbered phy of the second SAS expander may be remapped to be in a second range of the single, logically ordered numbering. 
     The method  600  may further include the step of connecting the first SAS expander directly to the second SAS expander for inter-expander communications  606 . The inter-expander communications may utilize SMP communications or other communication methods, such as Inter-Integrated Circuit Bus Protocol (I2C), Enhanced Parallel Port (EPP), Ethernet, shared memory, and the like. In additional embodiments, the connection between the first SAS expander and the second SAS for inter-expander communications ( 430 - 435 ) may be performed via the phys of the first SAS expander and the phys of the second SAS expander. Further, the phys of the first SAS expander for inter-expander communications and the phys of the second SAS expander for inter-expander communications may be hidden from the primary switched domain ( 410 ) of the single, cohesive SAS expander. At least two phys of the first SAS expander and at least two phys of the second SAS expander may be used for inter-expander communications between the first SAS expander and the second SAS expander. Further, if at least one SAS expander within the single, cohesive SAS expander remains operational (ex.—such as during an SAS expander failure), all common SAS wide ports remain operational, with all common SAS wide ports operating at a reduced bandwidth. 
     The single, cohesive SAS expander  500  may be configured to provide path failover when an internal link fails within the single, cohesive SAS expander. Referring to  FIG. 5 , and as described above, input ports  502 - 508  may be connected to each SAS expanders  202   a . . . d  via links  552   a . . . d - 558   a . . . d . Further, input ports  502 - 508  may be SAS wide ports. Output ports  510 - 516  may be connected to each SAS expanders  202   a . . . d  via links  560   a . . . d - 566   a . . . d . Further, output ports  510 - 516  may be SAS wide ports. Each of SAS expanders  202   a . . . d  may be connected to each other via SAS expander ports for inter-expander communications. As shown in single, cohesive SAS expander  500 , the link  566   b  between SAS expander  202   c  and output port  516  may fail. For example, a physical defect may cause link  566   b  to fail. As a further example, link  566   b  may fail to due a logical problem. 
     The data transfer intended to be transmitted via failed link  566   b  may be re-routed to another SAS expander connected to output port  516 . This data transfer may be re-routed to another SAS expander via the phys used for inter-expander communications between the SAS expanders. As shown in single, cohesive SAS expander  500 , the data transfer between SAS expander  202   c  and output port  516  via failed link  566   b  may be re-routed through SAS expander  202   d  via inter-expander communications link  434 . From SAS expander  202   d , the data transfer may be re-routed to output port  516  via link  566   c . Thus output port  516  may receive the data transfer as originally intended, but may receive the data transfer on a different phy. Further, if SAS expander  202   d  already has an active link to output port  516 , SAS expander  202   c  may respond to the SAS initiator with an arbitration in progress (AIP), as provided via SMP. The SAS initiator may attempt the data transfer at a later time after receiving an AIP or OPEN REJECT (RETRY) SMP response from SAS expander  202   c.    
     In a further aspect of the present disclosure, firmware executing on the SAS expanders may reprogram the SAS expander route tables to re-route connections through inter-expander communications links ( 430 - 435 ) instead of through the failed links. Further, a SAS initiator connected to the single, cohesive SAS expander may learn that link  566   b  has failed by receiving a CHANGE primitive. For example, SMP provides a CHANGE primitive. The SAS initiator may then perform a SAS Discovery, for example, as provided via SMP. Further, the SAS initiator may note the number of remaining active connections to the target, and may only initiate a number of simultaneous data transfers equal to the number of remaining active connections to the target. 
     Referring generally to  FIG. 7 , a method for providing path failover while combining multiple SAS expanders to act as a single, cohesive SAS expander is shown. For example, the method may implement techniques as shown in  FIG. 5 . In a current embodiment of the current invention, the method  700  includes the step of detecting a failed link between a first SAS expander and a device  702 . For example, the detected failed link may fail due to a physical problem. The detected failed link may fail due to a logical problem. 
     The method  700  may further include the step of re-routing a data transfer of the first SAS expander connected to the device via the failed link to a second SAS expander connected to the device via a functional link  704 . For example, the failed link of the single, cohesive SAS expander may fail due to a physical problem within the failed link. In another example, the failed link may fail due to a logical problem. The first SAS expander may be connected to the second SAS expander via the phys of the first SAS expander and the phys of the second SAS expander for inter-expander communications. In further embodiments, the re-routing a data transfer of a SAS expander connected to a device via the failed link to a second SAS expander connected to the device via the functional link may occur via the phys of the first expander for inter-expander communications and the phys of the second SAS expander for inter-expander communications. In exemplary embodiments, the re-routing a data transfer may include reprogramming a route table of the first SAS expander to re-route the data transfer from the failed internal link to the links between via the phys of the first expander for inter-expander communications and the phys of the second SAS expander for inter-expander communications. For example, the reprogramming of the route table of the first SAS expander may be performed via firmware executing on the first SAS expander. 
     The step of re-routing data transfers from a failed link within the single, cohesive SAS expander to a second SAS expander via a link for inter-expander communications may not indicate to SAS initiators and SAS targets connected to the single, cohesive SAS expander that such re-routing is occurring. However, if the second SAS expander currently has an active link to the device and data is re-routed from the failed link to the second SAS expander, the first SAS expander connected to the device via the failed link may respond with an AIP response. In such a case, a SAS initiator receiving the AIP response may retry the data transfer at a later time. A SAS initiator connected to the single, cohesive SAS expander may be notified of the failed link via receiving a CHANGE primitive. Upon receiving a CHANGE primitive, the SAS initiator may perform a SAS Discovery. For example, SMP provides a SAS Discovery, which includes a SAS Discover and a SAS Discover response. 
     A SAS-initiator specified data connection configuration  800  is provided. Referring to  FIG. 8 , SAS initiator  810  may be connected to SAS expander  820  via SAS wide port  880 . At least one phy of SAS expander  830  and at least one phy of SAS  840  may be combined in common SAS wide port  890 . In order for SAS expander  830  and SAS expander  840  to operate as a single, cohesive SAS expander, SAS expander  830  may share an identical SAS address with SAS expander  840 . SAS expander  820  may be connected to SAS expander  830  and SAS expander  840  via common SAS wide port  890 . SAS expander  830  may also be directly connected to SAS expander  840  via an inter-expander communications link via the phys of SAS expander  830  and the phys of SAS expander  840 . SAS expander  830  and SAS expander  840  may not operate as a single, cohesive SAS expander, and may be in a cascaded configuration. A plurality of SAS targets  850 - 875  may be connected to SAS expander  830  and SAS expander  840  via a SAS narrow port. The number of SAS targets  850 - 875  may be greater than the number of phys on SAS expander  830  or SAS expander  840 . For example, SAS targets  850 - 875  may be a hard disk array such as a Just a Bunch Of Disks (JBOD) SAS array. 
     SAS initiator  810  may specify the lanes of wide ports  880 ,  890  to be used to create a data connection from SAS initiator  810  to one of SAS targets  850 - 875 . Specifying the lanes of wide ports  880 ,  890 , may be used for fairness control, where some lanes of wide ports  880 ,  890  may be used to access SAS targets  850 - 875  and other lanes of wide ports  880 ,  890  may be used to pass through to other parts of a SAS configuration. SAS initiator  810  may further specify the lanes within an OPEN frame of a connection request. Such an OPEN frame may be provided via an SMP connection request. SAS initiator  810  may have learned the optimal lanes for a data connection from SAS initiator  810  to one of SAS targets  850 - 875  via a performing a standard SAS discovery process. SAS expander  830  and SAS expander  840 , along with SAS targets  850 - 875  may be configured to respond via the SMP Discover response with the optimal lanes for a data connection from SAS initiator  810  to one of SAS targets  850 - 875 . Each SAS expander in the data connection between SAS initiator  810  and one of SAS targets  850 - 875  may check the OPEN frame of the connection request from SAS initiator  810  for the allowed lanes for the data connection and only makes a data connection on those specified lanes. 
     Referring generally to  FIG. 9 , a method for specifying allowable lanes of a SAS wide port for a data connection between a SAS initiator and a SAS target in a SAS domain in accordance with an exemplary embodiment of the present disclosure is shown. In a current embodiment of the present disclosure, method  900  includes the step of discovering the optimal lanes of at least one SAS wide port for the data connection  902 . For example, SAS initiator  810  may discover the optimal lanes for the data connection via performing a standard SAS discovery process. Further, the SMP Discover response may be used to communicate the optimal lanes for the data connection to SAS initiator  810 . 
     The method  900  may further include the step of specifying the allowable lanes for the data connection within the SAS wide ports of each level of the SAS domain  904 . For example, the allowable lanes for the data connection may be specified via a SAS initiator connection request. The allowable lanes for the data connection may further be specified within an OPEN frame of a SAS initiator connection request. 
     The method  900  may further include the step of checking for the specified allowable lanes for the data connection  906 . For example, when a SAS expander ( 820 - 840 ) receives an OPEN frame of a SAS initiator connection request, the SAS expander may check the OPEN frame for the specified allowable lanes for the data connection. 
     The method  900  may further include the step of creating the data connection of the specified allowable lanes  908 . For example, a SAS expander may create the data connection on the specified allowable lanes. Further, the SAS expander may create the data connection after checking the OPEN frame of a SAS initiator connection request for the specified allowable lanes. 
     A SAS cascaded topology is provided which utilizes paired subtractive routing. In order to utilize paired subtractive routing, SAS devices in SAS topology  1000  may reference a vendor unique SMP Function. SAS Initiators  1002 - 1004  may check SMP REPORT MANUFACTURER INFORMATION response, as provided via SMP, to see if the vendor unique SMP Function is supported. If so, SAS Initiators  1002 - 1004  may use the vendor unique SMP Function to request a list of the phys of SAS expanders  1010 - 1024  which have a modified routing attribute for utilizing paired subtractive routing. For example, SAS Initiators  1002 - 1004  may issue SMP REPORT MODIFIED PHY ROUTING ATTRIBUTE LIST Request  1100  to each of SAS expanders  1010 - 1024  of SAS topology  1000 . Upon receiving SMP REPORT MODIFIED PHY ROUTING ATTRIBUTE LIST Request  1100 , each of SAS expanders  1010 - 1024  may respond with a list of the phys with modified routing attributes via SMP REPORT MODIFIED PHY ROUTING ATTRIBUTE LIST Response  1120 . In operation, SMP REPORT MODIFIED PHY ROUTING ATTRIBUTE LIST Response  1120  may contain several Modified Routing Descriptors  1140 . Each of Modified Routing Descriptor  1140  may indicate a single phy of SAS expanders  1010 - 1024  and Modified Routing Attribute  1160  for the single phy. For example, Modified Routing Attribute  1160  may identify a phy to be one of Self Configured (SC), Table Initiator Only IN (TIOI), or Table Initiator Only OUT (TIOO). TIOI may identify an input phy with only SAS initiators listed in the routing table for that phy. TIOO may identify an output phy with only SAS initiators listed in the routing table for that phy. 
     SAS initiators  1002 - 1004  may ignore the routing attributes for a phy provided by a SMP DISCOVER response if SMP REPORT MODIFIED PHY ROUTING ATTRIBUTE LIST Response  1120  identifies a Modified Routing Attribute  1160  for the phy. In an embodiment of the present disclosure, the vendor unique SMP function may only report modified routing attributes TIOI, TIOO, and SC. SAS expanders  1010 - 1024  may have a plurality of ports with some possessing standard routing attributes (D, S, T) and some possessing modified routing attributes (TIOI, TIOO, SC). 
     SAS expanders  1010 - 1024  may be configured to have a primary subtractive port defined to be an IN and an OUT pair. Both the input port and the output port of the subtractive port may require the addresses of SAS initiators  1002 - 1004  in the route table for that port. Further, SAS expanders may add a secondary subtractive port for direct attached devices. For example, an OPEN received on the input port may be sent out the output port for that subtractive port if the DEST of the OPEN, as provided via SMP, is not a direct attached device (ex—a device attached to a port with a D routing attribute) or if the DEST of the OPEN is not in the route table for the input port. For example, an OPEN received on the output port may be sent out the input port for that subtractive port if the DEST of the OPEN is not a direct attached device (ex—a device attached to a port with a D routing attribute) or if the DEST of the OPEN is not in the route table for the output port. The modified routing attribute TIOI may indicate the input port of a subtractive port. The modified routing attribute TIOO may indicate the output port of a subtractive port. OPENs may always default to the primary subtractive port. 
     SAS initiators  1002 - 1004  may be located at the top or the bottom of cascaded SAS topology  1000 . SAS initiators  1002 - 1004  may program only their own SAS addresses into the route tables of TIOI ports  1060  and TIOO ports  1050  of the SAS expanders  1010 - 1024  of cascaded SAS topology  1000 . Further, SAS initiators  1002 - 1004  may program only their own SAS addresses only into the route tables of TIOI ports  1060  or the route tables of TIOO ports  1050  of the SAS expanders  1010 - 1024  of cascaded SAS topology  1000 , depending on the position of the SAS initiators  1002 - 1004  within cascaded SAS topology  1000 . SAS expanders  1010 - 1024  may execute firmware preventing SAS initiators  1002 - 1004  from stepping on each other when programming address into SAS expander route tables (e.g.—position independent route table programming. 
     SAS target  1070  may issue an OPEN to one of SAS expanders  1010 - 1024 . One of SAS expanders  1010 - 1024  may locate the DEST address of the OPEN in the route table for one of SAS expanders  1010 - 1024 . The OPEN may be sent to the SAS initiator indicated by the DEST address. In another embodiment, one of SAS initiators  1002 - 1004  may issue an OPEN to one of SAS expanders  1010 - 1024 . One of SAS expanders  1010 - 1024  may first search for the DEST address of the OPEN in any directly attached devices. If found, the OPEN is sent to the appropriate directly attached device. If the DEST address of the OPEN is not found in any directly attached devices, one of SAS expanders  1010 - 1024  may then search for the DEST address of the OPEN in the SAS expander route table. If found, the OPEN is routed to the appropriate SAS initiator. If the DEST address of the OPEN is not found in the SAS expander route table, one of SAS expanders  1010 - 1024  may then send the OPEN out its paired subtractive port. 
     Further, in order to perform SAS zoning in SAS cascaded topology  1000 , only end expanders in the topology may require checking End expanders may compare the DEST of the phys for direct attached devices to the source zone group in the OPEN command. The comparing behaves as subtractive until reaching the end device expander of the zone. 
     Referring generally to  FIG. 12 , a method for performing paired subtractive routing in a SAS cascaded topology is shown. Method  1200  includes the step of assigning an input port to a SAS expander device  1210 . The method  1200  may further include the step of assigning an output port to the SAS expander device  1220 . In exemplary embodiments, the method  1200  may further include the step of defining the output port and the input port to be paired with each other to operate as a paired subtractive port  1230 . In addition, the method  1200  may include adding a second input port and a second output port to the SAS expander device. The second input port and the second output port may be paired with each other as a secondary subtractive port. 
     In exemplary embodiments, the method  1200  may further include the step of sending an OPEN command out the output port upon receiving the OPEN command into the input port if the DEST of the OPEN command is not a direct attached device of the SAS expander device and the DEST is not in the route table of the SAS expander device  1250 . The method  1200  may further include the step of sending an OPEN command out the input port upon receiving the OPEN command into the output port if the DEST of the OPEN command is not a direct attached device of the SAS expander device and the DEST is not in the route table of the SAS expander device  1260 . In addition, sending an OPEN command out the output port upon receiving the OPEN command into the input port if the DEST of the OPEN command is not a direct attached device of the SAS expander device and the DEST is not in the route table of the SAS expander device  1250 . The method  1200  may further include the step of sending an OPEN command out the second input port upon receiving the OPEN command into the second output port if the DEST of the OPEN command is not a direct attached device of the SAS expander device and the DEST is not in the route table of the SAS expander device. The method  1200  may further include the step of sending an OPEN command out the second input port upon receiving the OPEN command into the second output port if the DEST of the OPEN command is not a direct attached device of the SAS expander device and the DEST is not in the route table of the SAS expander device. Further, the secondary subtractive port may operate for communication with the direct attached device of the SAS expander device. In addition, an OPEN command may default to the primary subtractive port. 
     In exemplary embodiments, the method  1200  may further include the step of specifying the subtractive ports of the SAS expander via a modified routing attribute of a vendor unique SMP function. Further, the SAS initiator may ignore the SMP DISCOVER response attribute for the subtractive ports and may use the modified routing attribute of a vendor unique SMP if a modified routing attribute exists for the subtractive ports. 
     In exemplary embodiments, the method  1200  may further include the step of programming only a SAS initiator address in the route table of the SAS expander  1240 . For example, the step of programming only a SAS initiator address in the route table of the SAS expander  1240  may be performed by the SAS expander. Further, the route table of the SAS expander programmed by the SAS expander may be the route table for a self-configuration specified port. The step of programming only a SAS initiator address in the route table of the SAS expander  1240  may be performed by every SAS initiator in the SAS routing topology, each SAS initiator programming only the SAS address for that SAS initiator in the route table of the SAS expander. Further, each SAS initator may only program the SAS address for that SAS initiator in the route table of one type of modified routing attribute port of the SAS expander. The type of modified routing attribute port of the SAS expander may be determined by the position of the SAS initiator in the SAS routing topology. SAS initiators may be located at the top or the bottom of the SAS routing topology. 
     In the present disclosure, the methods disclosed may be implemented as sets of instructions or software readable by a device. Such software may a computer program product which employs a computer-readable storage medium including stored computer code which is used to program a computer to perform the disclosed function and process of the present invention. The computer-readable medium may include, but is not limited to, any type of conventional floppy disk, optical disk, CD-ROM, magnetic disk, hard disk drive, magneto-optical disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, or any other suitable media for storing electronic instructions. Further, it is understood that the specific order or hierarchy of steps in the methods disclosed are examples of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the method can be rearranged while remaining within the disclosed subject matter. The accompanying method claims present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented. 
     It is believed that the present disclosure and many of its attendant advantages will be understood by the foregoing description, and it will be apparent that various changes may be made in the form, construction and arrangement of the components without departing from the disclosed subject matter or without sacrificing all of its material advantages. The form described is merely explanatory, and it is the intention of the following claims to encompass and include such changes.