Patent Publication Number: US-9430278-B2

Title: System having operation queues corresponding to operation execution time

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application is a continuation of commonly assigned copending U.S. patent application Ser. No. 12/268,037, which was filed on Nov. 10, 2008, by David Morgan Robles for a System and Method for Implementing (Or Utilizing) Prioritized Queues on a Computer, which application is hereby incorporated by reference. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates to computer processors, and more specifically, to utilizing a plurality of prioritized queues on computer processors. 
     BACKGROUND OF THE INVENTION 
     A storage system is a computer that provides storage service relating to the organization of information on writable persistent storage devices, such as memories, tapes, disks or solid state devices, e.g., flash memory, etc. The storage system is commonly deployed within a storage area network (SAN) or a network attached storage (NAS) environment. When used within a NAS environment, the storage system may be embodied as a file server including an operating system that implements a file system to logically organize the information as a hierarchical structure of data containers, such as files on, e.g., the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information, such as the actual data (i.e., file data) for the file. 
     A network environment may be provided wherein information (data) is stored in secure storage served by one or more storage systems coupled to one or more security appliances. Each security appliance is configured to transform unencrypted data (cleartext) generated by clients (or initiators) into encrypted data (ciphertext) destined for secure storage or “cryptainers” on the storage system (or target). As used herein, a cryptainer is a piece of storage on a storage device, such as a disk, in which the encrypted data is stored. In the context of a SAN environment, a cryptainer can be, e.g., a disk, a region on the disk or several regions on one or more disks that, in the context of a SAN protocol, is accessible as a logical unit (lun). In the context of a NAS environment, the cryptainer may be a collection of files on one or more disks. Specifically, in the context of the CIFS protocol, the cryptainer may be a share, while in the context of the NFS protocol, the cryptainer may be a mount point. In a tape environment, the cryptainer may be a tape containing a plurality of tape blocks. 
     Each cryptainer is associated with its own encryption key, e.g., a cryptainer key, which is used by the security appliance to encrypt and decrypt the data stored on the cryptainer. An encryption key is a code or number which, when taken together with an encryption algorithm, defines a unique transformation used to encrypt or decrypt data. Data remains encrypted while stored in a cryptainer until requested by an authorized cliffs ent. At that time, the security appliance retrieves the encrypted data from the cryptainer, decrypts it and forwards the unencrypted data to the client. 
     One noted disadvantage that may arise during use of a security appliance is that certain operations may be long running and may generate a backlog within a processor of the security appliance. For example, execution of performing compression/decompression operations on, e.g., a tape data stream, by the processor may require significant amounts of time. Conversely, execution of single block encryption/decryption operations for data access requests directed to a disk drive may proceed rapidly. However, should a long-running tape compression/decompression operation be loaded onto an operations queue associated with the processor before a block-based encryption/decryption operation, execution of the encryption/decryption operation by the processor may have to wait until such time as the long-running operation completes. This may substantially lower overall throughput and reduce system performance. 
     SUMMARY OF THE INVENTION 
     The disadvantages of the prior art are overcome by providing a system and method for utilizing prioritized queues on a computer, such as a security appliance or a second storage system. Illustratively, a plurality of queues is organized on the computer to enable long-running operations to be loaded on (directed to) a long running operation queue, while faster, “short-running” operations are directed to a short running operation queue. The queues may be associated with one or more processors (e.g., processor cores) of the computer to thereby enable improved throughput. When an operation request (e.g., a tape compression operation, an encryption operation, a disk compression operation, etc.) is received at a processor intake of the computer, a determination is made whether the operation contained within the received request is a long-running operation, e.g., a tape compression operation. If so, the operation is placed in the long-running operation queue. The processor core that is associated with the long-running operation queue thereafter removes the operation from the queue and executes the operation. The status of the operation, e.g., operation complete, an error code, etc., is then loaded onto an outgoing long-running operation status queue. The status may subsequently be removed and reported to an initiator of the long-running operation. 
     Similarly, if a determination is made that the received operation is not a long-running operation, e.g., a compression operation, the operation is placed in a non-long running operation queue. The processor core associated with the short-running operation queue then removes the operation from the queue and processes the operation. Status information relating to that operation is then loaded onto a short-running status queue. The status may be subsequently removed from the queue and reported to the initiator of the operation. By utilizing a plurality of queues directed to different priorities of operation, overall system throughput may be increased by, among other things, reducing the number of short-running operations that are delayed. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numerals indicate identical or functionally similar elements: 
         FIG. 1  is a schematic block diagram of an environment including a security appliance in accordance with an illustrative embodiment of the present invention; 
         FIG. 2  is a schematic block diagram of a security appliance in accordance with an illustrative embodiment of the present invention semi; 
         FIG. 3  is a schematic block diagram illustrating an arrangement of software processes and modules executing on a security appliance in accordance with an illustrative embodiment of the present invention; 
         FIG. 4  is a schematic block diagram of an illustrative operation queue in accordance with an illustrative embodiment of the present invention; 
         FIG. 5  is a schematic block diagram of an exemplary status queue in accordance with an illustrative embodiment of the present invention; and 
         FIG. 6  is a flow chart detailing the steps of a procedure for utilizing prioritized queues in accordance with an illustrative embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT 
     The present invention provides a system and method for prioritized queues. Illustratively, a plurality of queues are organized to enable long-running operations to be directed to a long running queue operation, while faster operations are directed to a non-long running operation queue. Queues may be associated with one or more of a plurality of processor cores to therefore enable improved throughput operations. When an operation request is received, a determination is made whether it is a long-running operation, e.g., a tape compression operation. If so, the operation is placed in a long-running operation queue. When the processor core that is executing long-running operations is ready for the next operation, it removes an operation from the long-running operation queue and processes the operation. The status of the operation is then placed in an outgoing long-running operation status queue. The status may then be removed and reported to the initiator of the long-running operation. 
     Similarly, if a determination is made that the received operation is not a long-running operation, e.g., a compression operation, the operation is placed in a non-long running operation queue. The processor core executing non-long-running operations then removes the operation from the queue and processes the operation. Status information relating to the operation is then placed in a non-long-running status queue. The status may then be removed from the queue and reported back to the initiator of the operation. By utilizing a plurality of queues directed to different priorities of operation, overall system throughput may be increased and the number of non-long-running operations that are delayed may be reduced. 
     A. Security Appliance Environment 
       FIG. 1  is a schematic block diagram of an exemplary environment  100  including a multi-protocol security appliance  200  that may be advantageously used with an illustrative embodiment of the present invention. The security appliance  200  is illustratively coupled between one or more clients  102  and one or more storage systems  110  although, in alternative embodiments, the security appliance (or its functionality) may be embodied on/in the storage system to thereby provide a secure storage system. The security appliance  200 , which is configured to act as an encryption proxy, intercepts a data access request issued by client  102  and destined for the storage system  110 , wherein the data access request may be a read request to retrieve certain data stored on storage devices, such as disks  120 , coupled to the storage system  110  or a write request to store data on the disks. In the case of a write request, the security appliance  200  intercepts the request, encrypts the data associated with the request and forwards the encrypted data to the storage system  110  for storage at a specified location (address) on disk  120 . In the case of a read request, the security appliance intercepts the request and forwards it onto the storage system, which returns the requested data to the appliance in encrypted form. The security appliance  200  then decrypts the encrypted data and returns the decrypted data to the client  102 . 
     In the illustrative embodiment, the security appliance employs a conventional encryption algorithm, e.g., the Advanced Encryption Standard (AES) or other appropriate algorithms, to transform unencrypted data (cleartext) generated by the clients  102  into encrypted data (ciphertext) intended for secure storage, i.e., one or more cryptainers, on the storage system  110 . To that end, the security appliance illustratively uses a high-quality, software or hardware-based pseudo random number generation technique to generate encryption keys. The encryption and decryption operations are performed using these encryptions keys, such as a cryptainer key associated with each cryptainer. As described herein, the security appliance  200  uses an appropriate cryptainer key to encrypt or decrypt portions of data stored in a particular cryptainer. In addition to performing encryption and decryption operations, the security appliance  200  also performs access control, authentication, virtualization, and secure-logging operations. 
     Illustratively, the clients  102  may comprise application service providers, virtual tape systems, etc. Thus, in accordance with an illustrative embodiment of the present invention, clients  102  may send a plurality of types of operations to the security appliance  200 . For example, a client may send one or more block-based encryption/decryption operations directed to a logical unit number (lun) or may transmit one or more compression/decompression operations directed to a virtual tape stream. 
     B. Security Appliance 
       FIG. 2  is a schematic block diagram of the multi-protocol security appliance  200  that may be advantageously used with the present invention. As used herein, a security appliance denotes a computer having features such as simplicity of security service management for users (system administrators) and clients of network attached storage (NAS) and storage area network (SAN) deployments. The security appliance comprises one or more processors, e.g., central processing units (CPU  220   a,b ), a memory  210 , one or more network adapters  220   a,b , a storage encryption processor (SEP  270 ) and a card reader  230  interconnected by a system bus  240 , such as a conventional Peripheral Component Interconnect (PCI) bus. The SEP  270  is configured to perform all encryption and decryption operations for the security appliance in a secure manner; for example, the SEP is configured to protect plaintext encryption keys from system software executing on each CPU  202 . Accordingly, the SEP is illustratively embodied as a FIPS 140-2 level-3 certified module. 
     In accordance with the illustrative embodiment of the present invention, the SEP  270  includes a plurality of processor cores  275  A, B. It should be noted that two cores are shown for illustrative purposes only. In accordance with alternative embodiments of the present invention, the SEP  270  may have any number of the processor cores including, for example, a single processor core. As such, the depiction of the SEP  270  having two processor cores  275  A, B should be taken as exemplary only. Furthermore, while one SEP  270  is shown in  FIG. 2 , the principles of the present invention may be utilized in systems having any number of SEPs. As such, the description of a single SEP  270  should be taken as exemplary only. 
     Since the SEP  270  protects encryption keys from being “touched” (processed) by the system software executing on the CPU  202 , a mechanism is needed to load keys into and retrieve keys from the SEP. To that end, the card reader  230  provides an interface between a “smart” system card  250  and the SEP  270  for purposes of exchanging encryption keys. Illustratively, the system card is a FIPS 140-2 level-3 certified card that is configured with customized software code. The security appliance (and card reader  230 ) are further configured to support additional smart cards referred to as recovery cards  260   a,b . The security appliance illustratively supports up to 40 recovery cards with a default value of, e.g., 5 recovery cards, although any number of cards can be supported based on the particular security policy. 
     Operationally, encryption keys are exchanged between the SEP  270  and system card  250 , where they are “secret shared” (cryptographically assigned) to the recovery cards  260  as recovery keys, as described herein. These recovery keys can thereafter be applied (via the recovery cards) to the security appliance  200  to enable restoration of other encryption keys (such as cryptainer keys). A quorum setting for the recovery cards  260  may be provided such that the recovery keys stored on the recovery cards are backed up in a threshold scheme whereby, e.g., any 2 of the 5 default cards can recover the keys. 
     The network adapters  220  couple the security appliance  200  between one or more clients  102  and one or more storage systems  110  over point-to-point links, wide area networks, virtual private networks implemented over a public network (Internet) or shared local area networks. In a SAN environment configured to support various Small Computer Systems Interface (SCSI)-based data access protocols, including SCSI encapsulated over TCP (iSCSI) and SCSI encapsulated over FC (FCP), the network adapters  220  may comprise host bus adapters (HBAs) having the mechanical, electrical and signaling circuitry needed to connect the appliance  200  to, e.g., a FC network. In a NAS environment configured to support, e.g., the conventional Common Internet File System (CIFS) and the Network File System (NFS) data access protocols, the network adapters  220  may comprise network interface cards (NICs) having the mechanical, electrical and signaling circuitry needed to connect the appliance to, e.g., an Ethernet network. 
     The memory  210  illustratively comprises storage locations that are addressable by the processors and adapters for storing software programs and data structures associated with the present invention. The processor and adapters may, in turn, comprise processing elements and/or logic circuitry configured to execute the software programs and manipulate the data structures. An operating system  212 , portions of which is typically resident in memory and executed by the processing elements, functionally organizes the appliance  200  by, inter alia, invoking security operations in support of software processes and/or modules implemented by the appliance. It will be apparent to those skilled in the art that other processing and memory means, including various computer readable media, may be used for storing and executing program instructions pertaining to the invention described herein. 
     The operating system  212  illustratively organizes the memory  210  into an address space arrangement available to the software processes and modules executing on the processors.  FIG. 3  is a schematic diagram illustrating an arrangement  300  of software processes and modules executing on the security appliance  200  in accordance with the present invention. In the illustrative embodiment, the operating system software is a customized version of a Unix type operating system, although other operating systems may be utilized in alternate embodiments of the present invention. 
     For both NAS and SAN environments, data is received at a proxy  320  of the security appliance. The proxy  320  is a kernel module embodied as, e.g., the network protocol stack configured to interpret the protocol over which data is received and to enforce certain access control rules based on one or more policies. Each policy is served by a box manager  360  that is illustratively embodied as a database application process configured to manage a configuration repository or database (Config DB  370 ) that stores permissions, access control lists (ACLs), system-wide settings and encrypted keys. A socket server  380  provides interfaces to the box manager  360 , including (i) an HTTP web interface  382  embodied as, e.g., a graphical user interface (GUI) adapted for web-based administration, (ii) a SSH interface  384  for command line interface (CLI) command administration, and (iii) an SNMP interface  386  for remote management and monitoring. 
     Specifically, the box manager  360  supplies the permissions and encrypted keys to the proxy  320 , which intercepts data access requests and identifies the sources (clients  102 ) of those requests, as well as the types of requests and the storage targets (cryptainers) of those requests. The proxy also queries, using, e.g., an interprocess communication (IPC) technique, the box manager for permissions associated with each client and, in response, the box manager  360  supplies the appropriate permissions and encrypted key (e.g., a cryptainer key). The proxy  320  then bundles the data together with the encrypted key and forwards that information to a crypto process (layer)  330  that functions as a “wrapper” for the SEP  270 . As noted, the SEP resides on an interface card, which is hereinafter referred to a data crypto card (DCC  340 ). 
     Illustratively the DCC  340  cooperates with the crypto layer  330  to implement a plurality of prioritize queues, such as operation queues  400  and status queues  500  in accordance with an illustrative embodiment of the present invention. Each operation queue  400  is utilized by the crypto layer  330  to access the DCC  340  by, for example, supplying starting and ending points of data as well as offsets into the data along with the encryption keys used to encrypt data. In accordance with the illustrative embodiment of the present invention, the DCC  340  removes operations from the queue  400  and processes them before placing status indicators in status queue  500 . The crypto layer  330  may retrieve status information, e.g., operation complete, error codes, etc., from queue  500  and return the status information to the appropriate initiator of the operation. In an illustrative embodiment, operation requests are received by the crypto layer  330  and enqueued in an operations queue  400  before processing by one of the cores of the SEP  270 . The crypto layer or, in alternative embodiments, the DCC  340  determines whether the received operation request is a long running operation. If the operation contained in the received request is a long running operation, e.g., a compression operation, the operation is enqueued in a long running operation queue  400 . Otherwise, the operation is enqueued in a short running operation queue  400 . In accordance with alternative embodiments, there may be a plurality of long and short running operation queues (and associated status queues  500 ). Each of the queues may be associated with one or more processor cores in a predefined manner, established by, e.g., the DCC  340 , to enable optimized processing of operations. In alternative embodiments, the association of individual queues with specific cores may dynamically change depending on the type of operation mix being processed. It should be noted that in alternative embodiments, queues  400 ,  500  may be implemented in modules other than DCC  340 , e.g. queues  400 ,  500  may be implemented in crypto layer  330 . As such, the description of queues being implemented by DCC  340  should be taken as exemplary only. Furthermore, the method of associating processor cores with queues may vary as will be appreciated by one skilled in the art. Thus, in the illustrative embodiment, the decision as to which processor core  275  an operation is directed is made by the software executing on processor  202 . However, in alternative embodiments, this decision may be performed by other modules. As such, this description should be taken as exemplary only. 
     The crypto layer  330  interacts with the DCC  340  by accessing (reading and writing) registers on the DCC and, to that end, functions as a PCI interface. The DCC  340  includes one or more previously loaded keys used to decrypt the supplied encrypted keys; upon decrypting an encrypted key, the DCC uses the decrypted key to encrypt the supplied data. Upon completion of encryption of the data, the DCC returns the encrypted data as ciphertext to the proxy  320 , which forwards the encrypted data to the storage system  110 . 
     Notably, the security appliance  200  “virtualizes” storage such that, to a client  102 , the appliance appears as a storage system  110  whereas, from the perspective of the storage system, the security appliance appears as a client. Such virtualization requires that security appliance manipulate network addresses, e.g., IP addresses, with respect to data access requests and responses. Illustratively, certain of the customizations to the network protocol stack of the proxy  320  involve virtualization optimizations provided by the appliance. For example, the security appliance  200  manipulates (changes) the source and destination IP addresses of the data access requests and responses. 
     C. Prioritized Queues 
     The present invention provides a system and method for prioritized queues. Illustratively, a plurality of queues are organized to enable long-running operations to be directed to a long running queue operation, while faster operations are directed to a non-long running operation queue. Queues may be associated with one or more of a plurality of processor cores to therefore enable improved throughput operations. When an operation request is received, a determination is made whether it is a long-running operation, e.g., a tape compression operation. If so, the operation is placed in a long-running operation queue. When the processor core that is executing long-running operations is ready for the next operation, it removes an operation from the long-running operation queue and processes the operation. The status of the operation is then placed in an outgoing long-running operation status queue. The status may then be removed and reported to the initiator of the long-running operation. 
     Similarly, if a determination is made that the received operation is not a long-running operation, e.g., a compression operation, the operation is placed in a non-long running operation queue. The processor core executing non-long-running operations then removes the operation from the queue and processes the operation. Status information relating to the operation is then placed in a non-long-running status queue. The status may then be removed from the queue and reported back to the initiator of the operation. By utilizing a plurality of queues directed to different priorities of operation, overall system throughput may be increased and the number of non-long-running operations that are delayed may be reduced. 
       FIG. 4  is a schematic block diagram of an exemplary operation queue  400  in accordance with an illustrative embodiment of the present invention. The queue  400  is illustratively utilized by the crypto layer  330  to enqueue operations for processing by the SEP (or one or more cores) therein. Illustratively, a plurality of operation queues  400  may be utilized in accordance with the teachings of the present invention. Two pointers are illustratively maintained for each queue, e.g., a producer pointer and a consumer pointer. The producer pointer signifies the location where the next operation is to be enqueued, while the consumer pointer signifies the location of the operation that is to be next removed from the queue for processing. Thus, in the example shown in  FIG. 4 , four operations are enqueued, e.g., operations  405  A-D. The consumer pointer signifies that the next operation to be removed from the queue for processing is operation  405  A. Similarly, the producer pointer indicates that the next operation to be enqueued will be inserted as operation  405 E. It should be noted that illustratively the queue  400  is a “wrap around” queue so that when the producer and/or consumer pointer reaches the last the entry of the queue, it utilizes the first entry as a next entry. It should be noted that in alternative embodiments other implementations of an operations queue  400  may be utilized. As such, the description herein should be taken as exemplary only. 
       FIG. 5  is a schematic block diagram of an exemplary status queue  500  in accordance with an illustrative embodiment of the present invention. Similar to the operations queue  400 , the status queue includes a producer pointer and a consumer pointer. In relation to the status queue  500 , the producer pointer signifies the next the location where the next status information is to be inserted by the data crypto card  340 . The consumer pointer signifies the next status entry that is to be read and returned to the initiator in accordance with an illustrative embodiment of the present invention. As used herein, an initiator is a client or process that initiates an operation request and to which status information is directed. Thus, in the example shown in  FIG. 5 , there are two status entries  505 A, B enqueued within status queue  500 . The consumer pointer points to status entry  505 A, which will be the next status entry returned to an initiator. The producer entry points to a location where status entry  505 C will be enqueued once the next operation has been completed. 
       FIG. 6  is a flow chart detailing the steps of a procedure for utilizing prioritized queues in accordance with an illustrative embodiment of the present invention. The procedure  600  begins in step  605  and continues to step  610  where an operation request is received by, e.g., the crypto layer  330  of a computer, such as security appliance  210 . A determination is made in step  615  whether the received operation request contains a long-running operation. Illustratively, this determination is made by the crypto layer  330 ; however, in alternative embodiments, this determination may be made by other modules of the computer. 
     If the operation received is a long-running operation, the procedure continues to step  620  where the received operation is placed in a long-running operation queue. At a later point in time, the operation is removed from the long-running operation queue and processed by one or more cores of the SEP in step  625 . The status of the operation is then placed on a long-running status queue in step  630 . The status is then removed from the long-running status queue and reported to the initiator in step  635 . The procedure  600  then completes in step  640 . 
     However, if in step  615  it is determined that the operation is not a long-running operation, then the procedure branches to step  645  where the received operation is placed on a non-long-running (i.e., short-running) operation queue. A SEP core removes the operation from the queue and processes the operation in step  650 . The status of the processed operation is then placed on a non-long running status queue in step  655 . The status is then removed from the queue and reported to the initiator in step  660  before the procedure completes in step  640 . 
     To again summarize, the present invention enables a plurality of operation queues to be configured in a defined system arrangement with one or more processor cores. Upon receiving an operation request, the system enqueues the operation onto one of the queues based upon one or more characteristics of the operation. Illustratively, the characteristic is whether the operation is a long running operation. However, it should be noted that in alternative embodiments, additional and/or differing characteristics may be utilized. Once enqueued, the operation is subsequently processed by one of the processor cores that is illustratively configured to process (execute) operations having a certain characteristic. In alternative embodiments, the association of processor cores and queues may be dynamically modified depending on, e.g., the operation types and quantities that are being received by the system. 
     The foregoing description has been directed to specific embodiments of this invention. It will be apparent; however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. For instance, it is expressly contemplated that the procedures, processes, layers and/or modules described herein may be implemented in hardware, software, embodied as a computer-readable medium having executable program instructions, firmware, or a combination thereof. Accordingly this description is to be taken only by way of example and not to otherwise limit the scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.