Abstract:
Method and system for optimizing DMA request processing is provided. The system includes a HBA that uses a dynamic DMA maximum write burst count sizing to optimize processing of write and read requests, wherein the HBA includes a DMA optimizer module that selects a certain write burst size to adjust performance when read and write DMA requests are being utilized. The DMA optimizer module can toggle between write and read request priority based on a maximum write request burst size. A shorter maximum write burst size provides more opportunity to issue read requests and a larger maximum burst size provides a better write request performance. The method includes, evaluating a read request throughput rate; evaluating a write request throughput rate; evaluating a read request utilization rate; evaluating a write request utilization rate; and adjusting a maximum write burst size.

Description:
BACKGROUND 
   1. Field of the Invention 
   The present invention relates to computing systems, and more particularly to optimizing direct memory access (“DMA”) channel performance. 
   2. Background of the Invention 
   Storage area networks (“SANs”) are commonly used where plural memory storage devices are made available to various host computing systems. Data in a SAN is typically moved from plural host systems (that include computer systems, servers etc.) to the storage system through various controllers/adapters. 
   Host systems typically include several functional components. These components may include a central processing unit (CPU), main memory, input/output (“I/O”) devices, and streaming storage devices (for example, tape drives). In conventional systems, the main memory is coupled to the CPU via a system bus or a local memory bus. The main memory is used to provide the CPU access to data and/or program information that is stored in main memory at execution time. Typically, the main memory is composed of random access memory (RAM) circuits. A computer system with the CPU and main memory is often referred to as a host system. 
   Host systems often communicate with storage systems via a host bus adapter (“HBA”, may also be referred to as a “controller” and/or “adapter”) using an interface, for example, the “PCI” bus interface. PCI stands for Peripheral Component Interconnect, a local bus standard that was developed by Intel Corporation®. The PCI standard is incorporated herein by reference in its entirety. 
   PCI-Express is another Input/Output (“I/O”) bus standard (incorporated herein by reference in its entirety) that is compatible with existing PCI cards using the PCI bus. PCI-Express uses discrete logical layers to process inbound and outbound information. The logical layers are a Transaction Layer, a Data Link Layer (“DLL”) and a Physical Layer (“PHY”). PCI-Express uses separate links to transmit and receive information. 
   PCI-Express uses a packet-based protocol to exchange information between Transaction layers. Transactions are carried out using Requests and Completions. 
   The Transaction Layer assembles and disassembles Transaction Layer Packets (“TLPs”). TLPs are used to communicate transactions, such as read and write and other type of events. 
   Various other standard interfaces are also used to move data from host systems to storage devices. Fibre channel is one such standard. Fibre channel (incorporated herein by reference in its entirety) is an American National Standard Institute (ANSI) set of standards, which provides a serial transmission protocol for storage and network protocols such as HIPPI, SCSI, IP, ATM and others. 
   DMA modules are used by HBAs to perform data transfers between memory locations, or between memory locations and an input/output port. DMA units provide address and bus control signals to and from a device for a read and/or write cycle. 
   A DMA read request is a request from a DMA module (or channel) to an arbitration module to transfer data from a host system to a storage device. A DMA write request is a request from a DMA module to an arbitration module to transfer data from the storage device to a host system. 
   Specific channels are implemented in a DMA unit to allow storage devices to transfer data directly to and from memory storage devices. A channel can be activated by a DMA request signal (DREQ) from a storage device or a host system. The DMA unit receives the DREQ, provides a DMA acknowledged signal (DACK), and transfers the data over the channel to or from the storage device. 
   HBAs typically use multiple DMA channels and have an arbitration module that arbitrates access to a PCI-Express link. This allows an HBA to arbitrate and switch contexts (between channels) by actively processing command, status and data. Multiple channels are serviced in periodic bursts. 
   Typically, DMA write requests may be processed by writing data using multiple PCI-Express write request packets. A new DMA request (read or write) can only be processed if a previous DMA read and/or DMA write request has been processed. DMA read requests (with no data) are issued to generate Read Transfer requests, which transfers data from the host to a storage device. Write requests from the storage device to a host system are generated using the same side of the PCI-Express interface. The same transmit link (in the PCI-Express interface) is shared for write transfers and read requests. 
   The time to service a write request can be longer compared to servicing the read request because write request packets (TLPs) also transfer payload data. Also, delay in issuing read request packets can stall data transfer from a host to a storage device. 
   Therefore, there is a need for a method and system to optimize DMA read and write request processing that allows both read and write data transfers to be conducted efficiently. 
   SUMMARY OF THE INVENTION 
   In one aspect of the present invention, a storage area network (“SAN”) is provided. The SAN includes a HBA that uses a dynamic DMA maximum write burst count sizing to optimize processing of write and read requests, wherein the HBA includes a DMA optimizer module that selects a certain write burst size to adjust performance when read and write DMA requests are being utilized. The DMA optimizer module can toggle between write and read request priority based on a maximum write request burst size. A shorter maximum write burst size provides more opportunity to issue read requests and a larger maximum burst size provides a better write request performance. 
   In yet another aspect of the present invention, a HBA is provided that includes a DMA optimizer module that selects a certain write burst size to adjust performance when read and write DMA requests are being utilized for sizing a maximum DMA write burst count to optimize processing of write and read requests. 
   In yet another aspect of the present invention, a method for processing direct memory access requests in a HBA are provided. The method includes, evaluating a read request throughput rate; evaluating a write request throughput rate; evaluating a read request utilization rate; evaluating a write request utilization rate; and adjusting a maximum write burst size. 
   The maximum write burst size is increased if the read request utilization rate and the write request utilization rate are higher than a threshold value and the read throughput rate is greater than the write throughput rate. The maximum write burst size is decreased if the read request utilization rate and the write request utilization rate are higher than a threshold value and the write throughput rate is greater than the read throughput rate. 
   This brief summary has been provided so that the nature of the invention may be understood quickly. A more complete understanding of the invention can be obtained by reference to the following detailed description of the preferred embodiments thereof concerning the attached drawings. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The foregoing features and other features of the present invention will now be described with reference to the drawings of a preferred embodiment. In the drawings, the same components have the same reference numerals. The illustrated embodiment is intended to illustrate, but not to limit the invention. The drawings include the following Figures: 
       FIG. 1A  is a block diagram showing various components of a SAN; 
       FIG. 1B  is a block diagram of a host bus adapter with a DMA optimizer, according to one aspect of the present invention; 
       FIGS. 2A ,  2 B and  2 C show block/logic diagrams for optimizing DMA read and write requests, according to one aspect of the present invention; and 
       FIG. 3  shows a flow diagram for optimizing DMA read and write requests, according to one aspect of the present invention. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   To facilitate an understanding of the preferred embodiment, the general architecture and operation of a SAN, and a HBA will be described. The specific architecture and operation of the preferred embodiment will then be described with reference to the general architecture of the host system and HBA. 
   SAN Overview: 
     FIG. 1A  shows a SAN system  100  that uses a HBA  106  (referred to as “adapter  106 ”) for communication between a host system with host memory  101  to various storage systems (for example, storage subsystem  116  and  121 , tape library  118  and  120 ) using fibre channel storage area networks  114  and  115 . Host memory  101  includes a driver  102  that co-ordinates all data transfer via adapter  106  using input/output control blocks (“IOCBs”). Servers  117  and  119  can also access the storage sub-systems using SAN  115  and  114 , respectively. 
   A request queue  103  and response queue  104  is maintained in host memory  101  for transferring information using adapter  106 . Host system communicates with adapter  106  via a PCI-Express bus  105 . 
   HBA  106 : 
     FIG. 1B  shows a block diagram of adapter  106 . Adapter  106  includes processors (may also be referred to as “sequencers”) “XSEQ”  112  and “RSEQ”  109  for receive and transmit side, respectively for processing data received from storage sub-systems and transmitting data to storage sub-systems. Transmit path in this context means data path from host memory  101  to the storage systems via adapter  106 . Receive path means data path from storage subsystem via adapter  106 . It is noteworthy, that only one processor is used for receive and transmit paths, and the present invention is not limited to any particular number/type of processors. Buffers  111 A and  111 B are used to store information in receive and transmit paths, respectively. 
   Beside dedicated processors on the receive and transmit path, adapter  106  also includes processor  106 A, which may be a reduced instruction set computer (“RISC”) for performing various functions in adapter  106 . 
   Adapter  106  also includes fibre channel interface (also referred to as fibre channel protocol manager “FPM”)  113 A that includes modules  113 B and  113  in receive and transmit paths, respectively (shown as “FC RCV” and “FC XMT”). Modules  113 B and  113  allow data to move to/from storage systems. 
   Adapter  106  is also coupled to external memory  108  and  110  via connection  116 A ( FIG. 1A ) (referred interchangeably, hereinafter) and local memory interface  122 . Memory interface  122  is provided for managing local memory  108  and  110 . Local DMA module  137 A is used for gaining access to move data from local memory ( 108 / 110 ). Adapter  106  also includes a serial/de-serializer  136  for converting data from 10-bit to 8-bit format and vice-versa. 
   Adapter  106  also includes request queue DMA channel ( 0 )  130 , response queue DMA channel  131 , request queue ( 1 ) DMA channel  132  that interface with request queue  103  and response queue  104 ; and a command DMA channel  133  for managing command information. DMA channels are coupled to arbiter  107  that receives requests and grants access to a certain channel. 
   Both receive and transmit paths have DMA modules “RCV DATA DMA”  129  and “XMT DATA DMA”  135  that are used to gain access to a channel for data transfer in the receive/transmit paths. Transmit path also has a scheduler  134  that is coupled to processor  112  and schedules transmit operations. 
   A host processor (not shown) sets up shared data structures in buffer memory  108 . A host command is stored in buffer  108  and the appropriate sequencer (i.e.,  109  or  112 ) is initialized to execute the command. 
   Various DMA units (or channels, used interchangeably throughout this specification) (for example,  129 ,  130 ,  131 ,  132 ,  133  and  135 ) send a request to arbiter  107 . When a request is granted, the DMA unit is informed of the grant and memory access is granted to a particular channel. 
   Arbiter  107  is coupled to a PCI-Express Transaction Handler (PTH)  137 . PTH  137  is coupled to PCI-Express port logic  137 B that moves information to/from a host system. PTH  137  has also been referred to as PCI-Express interface and includes a receive side and transmit side link that allows communication between the host system and adapter  106 . The transmit side receives information from adapter  106  and destined for the host and the receive side receives information from adapter  106  and destined for the host system. 
   Arbiter  107  is also coupled to a DMA optimizer module  107 A (may also be referred to as module  107 A) that is coupled to plural DMA units (for example,  129  and  135 ). Module  107 A is described below in detail with respect to  FIGS. 2A-2C  and  3 . 
   DMA Optimization: 
   In one aspect of the present invention, based on an incoming Fibre Channel frame size, and maximum payload size allowed by PCI-Express, DMA arbitration sizing/priorities are enabled, disabled or modified to control DMA write request sizing. This allows write requests to be processed efficiently without significantly reducing read request processing. 
   In order to optimize read/write request performance, a maximum burst size for processing write requests is selected. Various maximum burst size may be selected, for example, 512 bytes, 1024 bytes, 1536 bytes, 2048 bytes or any other size. 
   Using a smaller burst size (for example, 512 bytes) provides more opportunities to send read DMA requests. Larger burst sizes allow for more efficient write request processing. Module  107 A balances the need for sending read requests without unduly slowing the write request processing, in one aspect of the present invention as described below with respect to  FIGS. 2A ,  2 B,  2 C and  3 . 
   It is noteworthy that read requests are sent without data and read request processing is completed after data is transferred from host via adapter  106  to a storage device. 
     FIG. 2A  shows arbiter  107  functionally coupled with module  107 A. Plural DMA channels (for example,  129  and  135 ) are coupled with arbiter  107 . Each DMA channel has a request pipeline (for example,  129 B and  135 B) and a segmentation module (for example  129 A and  135 A). Sequencer  109  and  112  send channel task commands ( 129 C and  135 C) to DMA channels  129  and  135 , respectively. The commands are used to generate a request to arbiter  107  (for example,  107 D and  107 E). 
   Transmit side DMA channel  135  also includes a read channel  135 D that issues read requests  107 F to arbiter  107 . 
   Segmentation modules  129 A and  135 A segment a DMA transfer into segments (or blocks). Segmentation modules  129 A and  135 A also operate based on certain rules, for example, when and how data blocks should be segmented. These rules can be turned on or off by the firmware. 
   Module  107 A includes a DMA request monitoring logic  107 B (may also be referred to as “logic  107 B” or “module  107 B”) that monitors both read and write requests. Logic  107 B receives information regarding pending requests (for example,  107 D,  107 E and  107 F). Output from logic  107 B is sent to a maximum write request burst count generator (may also be referred to as “generator”)  107 C that sends the maximum burst count  107 G to segmentation modules  129 A and  135 A. 
   Arbiter  107  generates signal  137 C that indicates a currently active request (i.e. a request that has won arbitration). Signal  137 D indicates if the request is for a read or write operation. Signal  137 E shows the byte count and signal  137 F is the acknowledgement after a request is completed. 
   Read/Write Channel Utilization Rate Comparison: 
     FIG. 2B  shows a block diagram for determining the utilization rate for read and write requests. Utilization rate in this context means the number of clocks in a timer interval when a DMA read or write request is pending (or the ratio of clocks used per timer interval and the total clocks per timer interval). Logic for determining the read utilization rate is shown as  200 A and logic for determining write path utilization rate is shown as  200 B. 
   DMA read channels  212  (similar to  135 D) issue DMA read requests  212 A and  212 B (similar to  107 F). Pending read requests  212 A and  212 B are input into counter  202  that maintains a running count of all clocks where read requests are pending in a given timer interval, whose period is provided by external timer  213 . Counter  202  is reset by signal  214  after every timer interval. A pending read request clock count is compared by comparator  201  to a number of clocks per timer interval  218 . A request may be designated as “pending” if the request is pending (or unacknowledged) for multiple clocks and increments counter  202 . 
   Comparator  201  is enabled by signal  219  from timer  213 . Comparator  201  provides the number of clocks with pending requests in a given timer period. 
   Output  201 A from comparator  201  is sent to another comparator  204  that also receives input  205 A from a threshold register  205 . Comparator  201  compares  201 A with threshold value  205 A. Register  205  holds threshold value  205 A for pending read requests. 
   Based on the comparison, comparator  204  generates signal  203  that is sent generator  107 C ( FIG. 2C ) based on which the maximum write burst size may be adjusted, as described below with respect. 
   The write side logic in segment  200 B operates similar to the logic elements in  200 A. DMA channels  211  generate the pending requests ( 211 A and  211 B). Counter  210  is similar to counter  202  and is reset by signal  216 . 
   Comparator  206  (enabled by signal  215 ) is similar to comparator  201 . Comparator  207  (enabled by signal  215 ) is similar to comparator  204 . Threshold register  208  stores threshold values ( 208 A) for the write side similar to threshold register  205 . Comparator  207  compares output  206 A with a threshold value  208 A. Based on the comparison, comparator  207  generates signal  209  that is sent to generator  107 C ( FIG. 2C ) based on which the maximum write burst size may be adjusted, as described below with respect. 
   Read/Write Throughput Rate Evaluation: 
     FIG. 2C  shows a logic diagram for comparing read/write operation throughput rates. Throughput in this context means the number of bytes transferred for a read/write operation within a timer interval (provided by  213 ). The comparison is performed to determine the ratio between the read and write throughput rate on a per timer interval basis. The ratio is compared to a threshold value that provides a range of acceptable values. Based on the comparison, adjustments are made to the maximum write burst count at which data is written for a write request. 
   The threshold value may be represented as a percentage value, for example, a threshold value greater than 0 and less than 100%. If the ratio is within a certain value, for example, greater or equal to 1.0−threshold value and less or equal to 1.0+threshold value, then no adjustment is made. 
   Turning in detail to  FIG. 2C , counter  213 B counts the running read/write byte counts. Counter  213 B detects if a request is acknowledged (ACK  137 F) and if it is a read request (READ/WRITE signal  137 D), then the read byte count (BYTE/CNT  137 E) is increased. The same is performed for a write request. Counter  213 B is reset by signal  213 G from timer  213 . 
   The running read/write byte count from counter  213 B is input to comparator  213 D that also receives a threshold value  213 H from a register  213 C. Comparator  213 D is enabled by signal  213 A. Comparator  213 D performs a relative comparison of read/write throughput rates to determine how far apart the throughput rates for read and write requests can be, before any adjustments to maximum write burst count are made. 
   Firmware for HBA  106  loads the threshold range value  213 H (as described above) in register  213 C. It is noteworthy that the threshold value  213 H may be pre-set or loaded dynamically. 
   A read throughput rate greater than the write throughput rate is shown as signal  213 E, while a write throughput rate greater than the read throughput rate is shown as signal  213 F. Signals  203 ,  209 ,  213 E and  213 F are sent to generator  107 C. Based on the signal values, as described below, signal  107 G is generated. 
   The term “signal” as used throughout this specification includes a command and/or bit value. 
   Process Flow: 
     FIG. 3  shows a flow diagram for optimizing DMA read and write request processing, according to one aspect of the present invention. Steps S 300 , S 302 , S 304  and S 306  are performed simultaneously. 
   In step S 300 , read throughput rate (Rt) is evaluated, as described above with respect to  FIG. 2C . Signal  213 E is generated if Rt is greater than the write throughput rate (Wt). 
   In step S 302 , the write throughput is evaluated, as described above with respect to  FIG. 2C . Signal  213 F is generated if the Wt is greater than Rt. 
   In step S 304 , read utilization rate (Ru) is evaluated, as described above with respect to  FIG. 2B . Signal  203  is generated after the evaluation and sent to generator  107 C. 
   In step S 306 , write utilization rate (Wu) is evaluated, as described above with respect to  FIG. 2B . Signal  209  is generated after the evaluation and sent to generator  107 C. 
   Based on signals  213 E,  213 F,  203  and  209 , in step S 308  adjustments are made to the maximum write burst size. Signal  107 G is generated and sent to module  135 A to adjust segmentation, if needed. 
   If both Wu and Ru are higher than their respective threshold values and Rt is relatively greater than Wt, then the maximum write burst size is increased. This will process pending write requests faster. 
   If both Wu and Rt are higher than their respective threshold values, and Wt is relatively greater than Rt, then the maximum write burst size is decreased. This will allow read requests to be processed faster and optimize overall processing of DMA requests. 
   Besides the throughput and utilization rates, other factors may also be used to adjust DMA write requests to create time windows for processing read requests at write request boundaries. Module  107 A may be configured to use rules similar to those used by PTH  137  to size DMA requests. This can be used to limit the size of write requests and increases the frequency with which read requests are processed. 
   The following are some of the rules used to size the DMA write requests: 
   128 k-byte address boundary: PCI-Express specification recommends that 128 byte boundaries be used during write requests. If a first PCI-Express write packet does not begin on a 128-byte boundary, then the packets following the first packet are aligned such that subsequent packets are at 128-byte boundary. Module  107 A may be used to align the boundaries by sizing the DMA request such that two DMA requests are formed, and this allows an additional read request to be inserted when the 128 byte alignment occurs for a DMA write request. 
   4 KB address boundary: PCI-Express specification requires that read and write requests do not cross a 4 KB address boundary. If a DMA request crosses a 4 KB boundary, then PTH  137  splits the requests into two different packets. Module  107  is used to size DMA requests such that they are only split at the 4 KB boundary for write requests and allows for an additional DMA read request to be inserted when the alignment occurs. 
   Max Payload Size: PCI-Express requires that write requests and read responses do not contain more data than a specified payload size that can be programmed by a host. If a DMA write request exceeds the maximum payload size, then it can be split into two different packets by splitting the DMA requests into 2 DMA requests. In this case, DMA write requests are split by module  107 A and hence an additional read request may be inserted when maximum payload sizing occurs for DMA write requests. 
   It is noteworthy that processor  106 A can read PCI-Express configuration registers located at PCI port logic  137 B. DMA channel write request sizing registers (not shown) are programmable by processor  106 A. Hence, processor  106 A can program the register bits (not shown) based on maximum payload size or other system configuration settings. 
   It is noteworthy that the foregoing DMA processing may be adjusted statically or dynamically, and module  107 A may be enabled or disabled by processor  106 A. 
   For static implementation, firmware for HBA  106  is used to set the maximum write count to a certain size, for example, 512K, 1K, 1.5K, 2K or any other size, and is adjusted based on the adaptive aspects of the present invention, described above. 
   For a dynamic implementation, an initial burst count value is selected, which is based on firmware settings (for example, 512K, 1K, 1.5K, 2K or any other size). At any given time, the value is adjusted when read or write throughput needs to be increased, as described above. 
   Although the present invention has been described with reference to specific embodiments, these embodiments are illustrative only and not limiting. Many other applications and embodiments of the present invention will be apparent in light of this disclosure and the following claims.