Abstract:
One embodiment of the invention provides a disk-to-tape storage system including a front-end portion and a hack-end portion. The front-end portion ha, a first interface for receiving storage commands and data over a network from an application performing a backup or archive operation. The received storage commands conform to a standardised command format. The back-end portion has a second interface for transmitting storage commands and the received data for storage in a tape library. The disk-to-tape storage system is operable to transform the received storage commands from the standardised command format into an appropriate format for the transmitted storage commands so as to maintain direct accessibility by the application of the received data as stored in the tape library.

Description:
FIELD OF THE INVENTION  
       [0001]    The present invention relates to computer systems, and in particular to a disk-to-tape storage system which may be used for data backup, data archiving, and so on. 
       BACKGROUND OF THE INVENTION  
       [0002]    In many circumstances it is desirable to maintain a backup or archive copy of data from a computer system. For example, if a fault develops with a particular system that deletes or overwrites data or otherwise makes the data inaccessible, such a hardware failure on a disk drive or a malicious virus, etc, a backup copy of the data can be used to restore the operational system to its correct state. A backup system can also be used to ensure that important data is not lost should the operational system suffer from damage, such as through an earthquake, flooding, fire, and so on. 
         [0003]    Another motivation for backup reflects the ever-increasing amount of data that is stored on computer systems. For example, users accumulate growing amounts of email, and businesses generate huge volumes of data and documents, many of which exist only in soft copy form. It can be very expensive to maintain all this data on primary storage at a client and/or server. Nevertheless, in some industries, there can be a statutory duty to maintain information for a set time period, especially within the financial and pharmaceutical industries, while other industries may have their own business reasons for data retention. Accordingly, data may be moved out from primary systems to backup or archive systems. Such archive systems will generally have a lower cost per unit storage than the primary systems, although the time to retrieve data from such an archive system may be relatively large. The use of backup and archive systems is described in: “Storage Networks” by Daniel Worden, 2004, Apress, Springer-Verlag, ISBN 1-59059-298-0. 
         [0004]    Many backup and archive systems are based on tape storage, since the cost per unit storage for tape is generally lower than for any other storage medium. Such systems may be provided by an automated library containing many tape cartridges that can be loaded and unloaded automatically by a robotic system. 
         [0005]    In many early systems, backup and/or archiving was implemented by running a backup application on the primary system—i.e. the same system as the main application to be backed up—and a tape backup unit was directly attached to this primary system. The backup application therefore communicated directly with the local tape unit to perform the desired backup operation. Over the years, more complex configurations have been adopted, such as the development of storage area networks (SANs). These networks are dedicated to performing backup, archiving, and other such storage operations (rather than say being used for general email communications). A SAN can be used to connect multiple primary systems to a single backup storage system. This then avoids the need for a separate backup system for each primary system; rather a single set of backup facilities can be shared between multiple primary systems. 
         [0006]    One complication with the shared or common use of a backup system is that there is a risk of contention between the various primary systems that are using the backup system for backup or archive operations. This problem is particularly acute for backup applications that expect sole use of the relevant backup system (as a leftover from when each primary system had its own dedicated backup system). One way to address such concerns is to provide storage or media management products, such as the Automated Cartridge System Library Software (ACSLS) available from StorageTek (Storage Technology Corporation). Such a media manager acts as an interface between the applications requiring backup and the available storage device(s), such as a tape library. The media manager can then be used to regulate communications in a more orderly manner between the various primary systems and the backup system. 
         [0007]    Many applications and operating systems now include some intrinsic backup functionality, rather than requiring a separate backup application to perform this task. In some cases, this backup may be implemented using disk-to-disk backup, such as disk mirroring, rather than using tape. Nevertheless, many organisations still use tape as the (ultimate) backup storage medium, either instead of or in addition to such disk-to-disk backup. For example, the cost per unit storage of tape remains lower than for disk. In addition, the use of two different storage media can improve robustness of the backup or archive solution—e.g. a disk backup may be more vulnerable to a virus than a tape backup. 
         [0008]    In some systems, a two-stage backup approach has been adopted, where the lo first stage represents a backup from disk-to-disk, and the second stage a backup from the disk-to-tape. These two stages are generally decoupled, in that they represent two separate operations. The initial disk-to-disk backup may be performed in response to an interaction between the primary application and the backup application, whereas the subsequent disk-to-tape backup is normally performed under the exclusive control of the backup operation using internal (proprietary), procedures. The primary application has no knowledge or understanding of this disk-to-tape backup stage. For example, the primary application does not know on which tape a given piece of data is stored, and the data stored on tape can generally be accessed only via the backup application. 
         [0009]    The growing complexity of storage architectures has led to increasing problems with compatibility. For example, each media manager typically has its own interface format for receiving backup or archive data. An application or operating system that wants to utilise this media manager therefore has to support the relevant interface. In practice, many applications provide support for only one or two media managers. Accordingly, an organisation will normally select a media manager that is supported by its application(s) (or conversely will select applications that support its media manager). However, sometimes such flexibility does not exist. For example, a first organisation may acquire a second organisation, and find that the applications of the second organisation do not support the media manager that the first organisation has already installed. It will be appreciated that this lack of compatibility can therefore impact the ability of an organisation to implement a simple and cost-effective backup and archive strategy. 
         [0010]    A further problem is that even if an application or operating system does support an interface into a particular media manager, it is frequently the case that such support is only provided at a relatively minimal level. For example, an interface to a media manager system may provide some complex commands that allow a user to extract maximum benefit and efficiency from the storage system. However, application and operating system vendors may frequently opt to support only the simplest command interface to a media manager, since they may have multiple media managers to support, and often the quality of this support is not a major factor when it comes to a customer making purchase decisions for the application or operating system. Unfortunately, using this simple command interface may seriously degrade the performance of the media manager, compared to what could be achieved by using the more complex interface. 
       SUMMARY OF THE INVENTION  
       [0011]    Accordingly, one embodiment of the invention provides a disk-to-tape storage system comprising a front-end portion and a back-end portion. The front-end portion has a first interface for receiving storage commands and data over a network from an application performing a backup or archive operation. The received storage commands conform to a standardised command format. The back-end portion has a second interface for transmitting storage commands and the received data for storage in a tape library. The disk-to-tape storage system is operable to transform the received storage commands from the standardised command format into an appropriate format for the transmitted storage commands while still maintaining direct accessibility by the application of the received data as stored in the tape library. 
         [0012]    The support for a standardised command format, such as SCSI (or iSCSI), allows the disk-to-tape storage system to interact with the vast majority of available backup applications and functions, which all tend to support SCSI as a form of lowest common denominator. However, because the disk-to-tape storage system does not represent a real storage device itself (e.g. it is not a real SCSI device), but only emulates the behaviour of such a system, it avoids exposing the application to problems associated with real devices, such as contention for access to a physical device between different applications. 
         [0013]    Although the emulation of real storage devices is already provided by existing disk-to-disk backup systems (virtual tape systems), these systems do not generally provide tape backup, or if they do, the manner in which such tape backup is performed is proprietary to the application, and decoupled from the disk-to-disk backup. As a result, the application itself is unable to access any data backed up onto tape. Rather such data can only be accessed by the disk-to-disk backup system, and then made available to the application. The application is therefore locked in to the disk-to-disk backup system. 
         [0014]    In contrast, in accordance with one embodiment of the present invention, the tape backup phase maintains direct accessibility by the application of the data as stored in the tape library. In one particular embodiment, this is achieved by preserving volume information between the received storage commands from the application and the transmitted storage commands. In other words, if the application tries to write data to a given tape, then the disk-to-tape storage system ensures that the data is written onto a tape having an identity specified by the application. (N.B. It will be appreciated that the tape is not actually mounted on the physical device that the application thinks, but this does not impact subsequent retrieval). One benefit of this is that it ensures that data is allocated to various tapes in a manner that makes sense to the application (which understands the data), rather than according to the needs of the tape backup system itself. In addition, if the application subsequently needs to access a particular data item, it knows the identity of the tape on which this data item is stored. The application is therefore able to access the data item directly through the relevant tape library if so desired, rather than being reliant upon the tape backup system as an intermediary. This is particularly useful for archive systems, in which data may have to be retrieved from tape many years after the creation of the tape, when the tape backup system originally employed may no longer be extant. 
         [0015]    As mentioned above, existing disk-to-disk backup systems with tape support generally decouple the tape storage from the disk storage. In contrast, in accordance with one embodiment of the present invention, at least some of the transmitted storage commands have a direct correspondence to the received storage commands. Thus when an application sends a command to write data to tape, this may lead directly to the data being sent for storage to the tape library, subject to the general operational requirements of the system (e.g. caching, resolving conflict between different accesses to the same device, etc). As a result, although the application is interacting directly only with the front-end portion, the operations of the back-end portion are closely coupled to this interaction. The application is therefore still reasonably aware of when data has indeed been stored onto tape (which may in turn have a bearing on the operations of the application itself). 
         [0016]    Note that in some circumstances, the disk-to-tape storage system may be configured to do some transformation on the received data between receipt from the application and storage into the tape library—e.g. compression, or re-formatting. Although this may make it difficult for the original application to access the data stored in the tape library directly without using the disk-to-tape storage system as an intermediary, there may, be other benefits or motivations for such an approach. For example, an organisation may backup data from many different applications, and the disk-to-tape storage system can then be used to convert the data from a format particular to any given application to a standardised format determined by the organisation. The organisation may then have separate facilities to access tapes having this standardised format (which may or may not involve going through the disk-to-tape storage system). 
         [0017]    One embodiment of the disk-to-tape storage system includes an internal media manager, which can send storage commands to the tape library via the second interface. The disk-to-tape storage system may also include a media manager interface that interacts with an external media manager, which in turn acts as an intermediary between the disk-to-tape storage system and the tape library. It will be appreciated that the disk-to-tape storage system may be connected directly (i.e. without an external media manager) and/or indirectly (i.e. with an external media manager) to multiple different tape libraries or archives. It will also be appreciated that the tape library may itself be implemented as some form of virtual tape system (i.e. emulated by disk), although this will be transparent to the disk-to-tape storage system. 
         [0018]    In one embodiment, the disk-to-tape storage system includes one or more virtual devices. The virtual devices are responsive to storage commands received from the application to provide an emulated response to the application in accordance with the standardised command format. For example, if the application sends a SCSI command to the system, then the virtual device returns the appropriate SCSI response. The virtual devices are also responsive to the incoming storage commands from the application to initiate said transmitted storage commands directed to the real devices. In one particular implementation, the virtual devices include at least one virtual tape drive and at least one virtual medium changer. 
         [0019]    In one embodiment, the system includes a database, which can be used to maintain the state of the virtual devices. The database can also be used to hold information describing the operation of certain types of devices. The virtual devices can then be created and operated based on this information. 
         [0020]    In one embodiment, the disk-to-tape storage system further includes a stored set of business rules, which are used to translate high-level storage commands received from an application into a standardised command format. This can help to simplify use of the disk-to-tape storage system by applications, in that the backup functionality does not necessarily need to be written directly in accordance with the standardised command format (which may be relatively low level). Rather, the application may use high-level storage commands for communicating with the disk-to-tape storage system, with the disk-to-tape storage system then being responsible itself to perform the conversion into the corresponding commands in SCSI or such-like. 
         [0021]    In one particular implementation, the disk-to-tape storage system operates as a SCSI to media manager interface, and is located between storage initiators (such as a backup application) and real devices. The disk-to-tape storage system achieves backup independence by emulating SCSI devices, and also by managing a known set of physical SCSI resources. The disk-to-tape system therefore encapsulates knowledge of back-end tape drives, medium changers and volumes, and so allows any backup application that can control a SCSI device to utilise a proprietary media manager or tape library without the backup application having to know anything about that particular media manager or tape library. This is particularly suited to providing a consolidated storage platform within an enterprise storage environment. 
         [0022]    The invention also provides method and computer program product embodiments corresponding to the system. It will be appreciated that such computer program product and method embodiments may utilise and benefit from the same particular features as described above in relation to the method embodiment. 
         [0023]    Note that the computer program product may be provided as a set of instructions recorded onto a physical medium, such as a CD, a DVD, and so on, or encoded into a transmission medium on a wireless or wired network such as the Internet. In either case, such instructions can then be loaded into a computer system for execution. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS  
         [0024]    One or more embodiments of the invention will now be described in detail by way of example only with reference to the following drawings: 
           [0025]      FIG. 1  is a schematic diagram showing the environment of a disk-to-tape storage system in accordance with one embodiment of the invention; 
           [0026]      FIG. 1A  is a schematic diagram showing the command flow to and from the disk-to-tape storage system of  FIG. 1  in accordance with one embodiment of the invention; 
           [0027]      FIG. 1B  is a schematic diagram showing the command flow to and from the disk-to-tape storage system of  FIG. 1  in accordance with another embodiment of the invention; 
           [0028]      FIG. 2  is a schematic diagram showing the main components of the disk-to-tape storage system of  FIG. 1  in accordance with one embodiment of the invention; 
           [0029]      FIG. 3  is a schematic diagram showing the main components within the SQL database from the disk-to-tape storage system of  FIG. 2  in accordance with one embodiment of the invention; 
           [0030]      FIG. 4  is a schematic flowchart showing the configuration phase of the disk-to-tape storage system of  FIG. 2  in accordance with one embodiment of the invention; and 
           [0031]      FIG. 5  is a schematic flowchart showing the operational phase of the disk-to-tape storage system of  FIG. 2  in accordance with one embodiment of the invention. 
       
    
    
     DETAILED DESCRIPTION  
       [0032]      FIG. 1  illustrates a disk-to-tape storage system  110  and its environment in accordance with one embodiment of the invention. The disk-to-tape storage system  110  provides archive services for a range of servers  101 A,  101 B,  101 C,  101 D,  101 E. The servers  101  are assumed to represent the primary systems running applications that require backup or archiving of data by the disk-to-tape storage system  110 . 
         [0033]    Servers  101 A,  101 B,  101 C,  101 D,  101 E communicate with disk-to-tape system  110  via a variety of connections. In particular, servers  101 A and  101 B communicate with disc-to-tape storage system  110  via storage area network (SAN)  100  and Gigabit Ethernet  100 C; servers  101 C and  101 D communicate with disk-to-tape storage system  110  via FDDI ring  100 A; and server  100 E communicates with disk-to-tape storage system  110  via SCSI link  100 B. It will be appreciated that tie particular configuration of servers  101  and network connections shown in  FIG. 1  is by way of illustration only, and many other configurations may be supported by disk-to-tape storage system  110 . For example, there may be more or fewer servers than shown in  FIG. 1 , allocated as appropriate across the different network connections, and other forms of connection between the servers  101  and the disk-to-tape storage system  110  may also be supported, apart from those shown in  FIG. 1 . In addition, disk-to-tape storage system  110  may provide backup storage for machines other than servers  101 , such as clients attached to computer networks, and so on. 
         [0034]    The disk-to tape-storage system  110  is linked to a tape archive  131 , which may include a plurality of tape units  130 A,  130 B plus one or more SCSI medium changers  132 . In some configurations an external media manager  120  may be present as an intermediary between disk-to-tape storage system  110  and tape archive  131 , while in other cases the disk-to-tape storage system  110  may use its own internal media manager  120 A for communications with the tape archive  131 . In this latter case, disk-to-tape storage system  110  interacts directly with tape archive  131 , without using media manager  120  as an intermediary. In operation, disk-to-tape storage system  110  therefore forwards data received from servers  101  for storage into tape archive  131 . 
         [0035]    Again it will be appreciated that the particular configuration shown  FIG. 1  is for of illustration only, and that many other configurations may be employed with disk-to-tape storage system  110 . For example, there may be more than one tape library or archive  131  supported by disk-to-tape storage system  110  for storing backup data from servers  101  (such an additional tape archive is indicated schematically by tape archive  131 A in  FIG. 1 ). Furthermore, although  FIG. 1  illustrates two tape units  130 A,  130 B and two medium changers, it will be appreciated that any number of tape devices and medium changers may be used, depending upon the particular storage requirements of the primary systems being supported. 
         [0036]    In some configurations, one or more of the tape units  130 A,  130 B may be replaced or implemented by a virtual tape system (VTS), in which a disk storage unit emulates the behaviour of a tape unit. In other words, the VTS stores the data onto disk, but responds (e.g. in terms of SCSI messages) as if it were saving the data onto tape. In the present context, such a virtual tape system will be regarded as a form of tape unit, since it appears as such from the perspective of the disk-to-tape storage system  110 . 
         [0037]      FIG. 1A  is a schematic diagram illustrating command and data communications for performing backup and archive operations in the environment of  FIG. 1  in accordance with one embodiment of the invention. Server  101  is assumed to run at least one backup application  102 . Note that backup application  102  may comprise a specialised backup application that provides backup services to other programs running on server  101  (or other linked machines), or it may represent an application or operating system that incorporates its own backup functionality. 
         [0038]    The backup application  102  communicates with the disk-to-tape storage system  110  using a standardised protocol. In one embodiment of the invention, this standardised protocol comprises the small computer system interface (SCSI) protocol. Note that the precise version of SCSI used depends upon the particular network that connects server  101  to the disk-to-tape storage system  110 . For example, if a fibre channel network is used to connect a server  101  to the disk-to-tape storage system  110 , then conventional SCSI can be used, whereas if an Ethernet  101 C is used for this connection, then the TCP/IP based iSCSI protocol may be used. 
         [0039]      FIG. 1A  further illustrates that communications between disk-to-tape storage system  110  and media manager  120  use a format or protocol dependent upon the particular media manager  120 . In general, this format will conform to some proprietary interface defined by the manufacture of media manager  120 . Disk-to-tape storage system  110  therefore acts as an intermediary between server  101  and media manager  120 , in that it allows server  101 , and more especially application  102 , to utilise a standardised protocol, namely SCSI, for performing backup and archive operations in conjunction with tape archive  131 . It will be appreciated that in practice SCSI is supported at some level by all applications that can perform storage operations, and hence all applications  102  are expected to be able to interact with the disk-to-tape storage  110  using the SCSI protocol, irrespective of whether such applications also support any proprietary media manager protocols. 
         [0040]    The disk-to-tape storage unit  110  in turn interacts with external media manager  120  using the proprietary media manager interface for that particular external media manager unit  120 . Disk-to-tape storage unit  110  is designed to support a range of media manager interfaces, and to exploit such interfaces as effectively as possible. This is to be contrasted with many applications  102 , which even if they support a particular media manager interface, may only support a limited subset of the interface, and therefore fail to obtain the full performance available from the interface. 
         [0041]    The front end of disk-to-tape storage system  110  therefore provides connectivity to various backup applications  102  via appropriate system connections (e.g. SAN  100 , FDDI network  100 A). The back end components external to the disk-to-tape system  110  are those resources available for use by the disk-to-tape storage system  110  (such as tape archive  131 ). 
         [0042]    It will be appreciated that the architecture of  FIG. 1A  allows support for a given media manager  120  to be concentrated into a single system, namely disk-to-tape storage system  110 , rather than having such support incorporated separately into many different applications  102 . Hence it is generally, easier and more cost-effective for disk-to-tape storage system  110  to support a range of media manager interfaces, and to support them well, rather than to provide such support in application  102 . 
         [0043]    Accordingly, interposing disk-to-tape storage system  110  between server  101  and tape archive  131  enhances the interconnectivity between application  102  and tape archive  131 , since assuming that application  102  supports SCSI, application  102  can now interoperate with any tape archive  131  supported by disk-to-tape storage system  110 . Furthermore, efficiency and performance can be improved through the ability of disk-to-tape storage system  110  to exploit the proprietary interface of media manager  120  in an efficient manner. 
         [0044]    In some implementations, disk-to-tape storage system  110  can be regarded as a high-speed cache between media-managed libraries (such as tape archive  131 ) and related backup applications  102 . The disk-to-tape storage system  110  can emulate SCSI medium changers and tape drives on the front end (the part of the disk-to tape storage system visible to any back application), while operating and maintaining one or more external media-managers  120  and/or direct-connect SCSI medium changers  132  or tape drives  130 A,  130 B on the back-end (which then comprise the physical resources available for use by the disk-to-tape storage system). 
         [0045]      FIG. 1B  illustrates the backup data flow in a configuration where no external media manager  120  is present. In this situation, operations are generally the same as described above in relation to  FIG. 1A , except that disk-to-tape storage system  110  can be regarded as providing its own internal media manager functionality  120 A. The internal media manager  120 A communicates directly and in a suitable format (which may be proprietary) with tape archive  131  to backup data from application  102 . 
         [0046]      FIG. 2  illustrates some of the components of disk-to-tape storage system  110  in more detail, along with the connections to SAN  100  and media manager  120  (the other components from the configuration of  FIG. 1  are omitted for clarity). In one embodiment, the disk-to-tape storage system is implemented as a standalone embedded system running on the Linux platform. A standard single or dual-processor motherboard is couple with suitable network interfaces (such as SCSI or fibre channel. To externally connected SCSI initiators, the disk-to-tape storage system  110  operates just as an emulated physical device would—external applications are unable to differentiate between a physical device and a device as emulated by the disk-to-tape storage system  110   
         [0047]    In one implementation, the disk-to-tape storage system  110  comprises modular software objects and processes that utilise a standard database (such as an SQL database) to provide a central repository for system and device status and state. In one particular implementation, the disk-to-tape storage system executes as several daemon processes and threads, continually watching for and executing inbound SCSI commands, while maintaining status and state information for all the available resources within the database. 
         [0048]    Considering the components of the disk-to-tape storage system  110  in more detail in accordance with one particular embodiment of the invention, these include an SQL database  240 ; various listeners  202 ; a SCSI manager  230 ; various device emulators  245 ; a cache  270 ; a media manager interface  260 ; and a logging system  280 . These components support the use of virtual device objects and an event mechanism that allow disk-to-tape storage system  110  to act as an intermediary between primary systems  101  and media manager  120  or tape archive  131  (if no external media manager is present). The main components of disk-to-tape storage system  10  will now be described in more detail. 
         [0049]    SQL Database  240 —SQL database provides standard SQL database functionality for disk-to-tape storage system  110 . The SQL database  240  may be local or remote to the rest of the disk to tape system. In addition, the SQL database may be used by just one instance of the disk-to-tape storage system  110 , or it may be shared (globally) by any, or all, instances of the disk-to-tape storage system  110  in a given environment. Note that in other embodiments, this functionality might be implemented by any suitable form of database, not necessarily an SQL database (for example some form of object-oriented database). 
         [0050]    The SQL database  240  provides real and emulated device and media characteristics, shared state information, business rules, and administrative information and data mining (history, predictive analysis, and usage logging, as examples). The major database entities (as shown in  FIG. 3 ) are: 
         [0051]    Stock Devices  310 : stock devices are those devices available for emulation. SQL database  240  maintains stock device tables that provide details of known SCSI devices. The stock device details are based upon manufacturer documentation and observed real device behavior (which often differ). The stock device details are shipped with the disk-to-tape storage system  110 . 
         [0052]    Real Devices  320 : real devices are those devices that are discovered and made available in a customer&#39;s environment. SQL database  240  maintains real device tables that provide a view of the customer&#39;s physical SCSI devices which are available to the back-end of the disk-to-tape storage system  110  (e.g. via media manager  120 ). The available devices are assigned by a policy as specified by the customer. The real device database tables are populated at initial start-up, and periodically updated during operation of the disk-to-tape storage system. 
         [0053]    Real Media  330 : real media database tables are populated at system commissioning, and periodically thereafter. They provide a view of available and in-use media discovered within a customer&#39;s real devices, such as tape systems  130 A,  130 B, including state information. 
         [0054]    Virtual Devices  340 : virtual device tables contain the operational state and device specific details for devices that are being emulated by disk-to-tape storage system  110 . The virtual device tables relate to the stock device tables and are populated from stock device data. Once instantiated, a virtual device  340  emulates the stock device  310  that the virtual device was based upon. 
         [0055]    Virtual Media  350 : virtual media tables are used to store information representing the metadata and state of (emulated) media available for use in the tape drives  130  and medium changers  132  of the virtual devices  340 . 
         [0056]    Business Rules  360 : the business rules tables contain “executable” instructions for the SQL database  240 , wrapping database transactions in a higher-level command set. The business rules tables provide programming simplicity by allowing developers using the disk-to-tape storage system  110  to be independent from SQL syntax in module coding, and by providing a standard database access methodology. The business rules tables  360  in the SQL database  240  provide a simple, string-based interface to the functionality of the disk-to-tape storage system  110 . 
         [0057]    Event System Interface and Queue  370 : common database tools, scripts, and GUIs may all use the event mechanism to control and interrogate the disk-to-tape storage system. Commands such as “mount volume”, “change log level”, or “add new virtual device” may be deposited in the events queue, and disk-to-tape storage system  110  will find these commands and execute them. The use of SQL database  240  for the event system interface and queue helps to provide user access control, high-availability and fail-over capabilities, and also allows operation and administration via conventional database software tools. 
         [0058]    Policies  380 : the policies tables and method allow for the customisation of the behaviour of the disk-to-tape storage system  110 , whereby specific needs and options for a given customer can be supported (e.g. such as which real devices are available for use by the disk-to-tape storage system  110  at a given installation). 
         [0059]    Listeners  202 —the disk-to-tape storage system  110  utilises listeners  202  to receive SCSI commands via standard SCSI, iSCSI (LAN), and fibre channel interfaces. It will be appreciated that the particular listener(s) utilised in any given configuration depends on the nature of network that links disk-to-tape storage system  110  with servers  101  or other data sources. Listeners  202  interact with the operating system drivers of disk-to-tape system  110  to provide device-specific control of SCSI control and data. Four listeners are provided in the particular embodiment of  FIG. 2 : 
         [0060]    QLogic Fibre Channel Listener  220 : this listener supplies fibre channel connectivity for the disk-to-tape storage system  110 , and in one embodiment is responsible for transacting with the QLogic family of HBA PCI cards (available from QLogic Corporation—see www.qlogic.com). The fibre channel listener  220  interacts with a fibre channel network via the QLogic target device driver  218 . 
         [0061]    SCSI Listener  221 : the disk-to-tape storage system  110  allows a server  101  to connect directly to the disk-to-tape storage system via a conventional parallel SCSI connector  100 B. Support for this SCSI connection is provided by a SCSI device driver  219  and the SCSI listener  221 . 
         [0062]    iSCSI LAN Listener  215 : the iSCSI listener is responsible for listening for SCSI commands from an iSCSI initiator across network  100 C, and allows for low cost connectivity to the disk-to-tape storage system  110  from any LAN-enabled server  100  or other system. (SCSI commands may be transacted via iSCSI without needing a traditional SCSI or fibre channel interface). 
         [0063]    Events Listener  205 : the events listener  205  is a special listener for supporting the event mechanism within the disk-to-tape storage system  110 . The events listener  205  listens only to the events table  370  in SQL database  240  in order to interface the database-driven events system to the rest of the disc-to-tape storage system  110 . A periodic check of the events table  370  reveals never events to be processed. 
         [0064]    The combination of event listener  205  and event system interface and queue table  370  in SQL database  240  allows disk-to-tape storage system  110  to support an event system that enables functionality within the disk-to-tape storage system  110  to be triggered externally via standard SQL database tools. Events such as “change log level”, “mount tape”, and many others are supported. Via the events system, users and applications may initiate certain disk-to-tape storage functions without having to operate as SCSI initiators. The events system also allows for all processes within the disk-to-tape storage system  110  to communicate with one other. In addition, remote instances of the disk-to-tape storage system  110  may communicate with each other via the events system. The list of possible events is registered in events table  370  in the SQL database  240 , and is extensible. 
         [0065]    SCSI Manager  230 —the SCSI manager  230  is a software object that launches and maintains any number of emulated device objects, such as a virtual tape drives or virtual medium changers. The SCSI manager limits the amount of coding necessary to implement emulated devices by incorporating base functionality in a single software object. An object for a newly emulated device only needs to be constructed to handle the specific SCSI needs of the device. The SCSI manager  230  handles two basic virtual SCSI object types (though many different types are possible), namely Vchanger  240  and Vtape  250 . 
         [0066]    Device Emulators  245 —a device emulator object contains specific knowledge of the behaviour of an emulated device. Device emulator objects  245  are created and managed by the SCSI manager  230 . General device type emulation comes from base-class definitions, such as the Virtual Tape (Vtape) class  250  and the Virtual Changer (Vchanger) class  240 . Specific device objects (not shown in  FIG. 2 ) can then be implemented based on these generic device types. In one particular embodiment, the following specific device emulators are available:
       StorageTek L700 medium changer   IBM LTO series tape drives   Quantum DLT 4000/7000 tape drives       
 
         [0070]    Command Interpreters—the command interpreters in the disk-to-tape storage system  110  process incoming SCSI commands as received over network  100 . The command interpreters are therefore the jumping-off point for command execution within the disk-to-tape storage system  10 . The command interpreters are not specifically shown in  FIG. 2 , but are spread across the SCSI manager  230 , the Vchanger  240  and Vtape  250  general device emulators, as well as any specific device emulators. 
         [0071]    Some SCSI commands are common to all device types, some are generalized to device types, and most are particular to a specific instance of a device type—for example, an LTO tape drive responds differently from a DLT tape drive. General SCSI commands are handled in the base-class definitions, and commands to specific device types are handled in each device emulator as appropriate. 
         [0072]    Logging System  280 —the logging system in disk-to-tape storage system  110  reports all interaction, traces, and error conditions to the user and system administrators. All transactions with the disk-to-tape storage system  110  are logged and time-stamped. Logging levels may be dynamically changed, increasing or decreasing the detail level without restarting the disk-to-tape storage system daemon. Logging support includes email notification of critical events. 
         [0073]    Cache  270 —cache  270  provides an intermediate transfer buffer for SCSI data, and offers a significant performance advantage over conventional SCSI tape devices. Data travelling to/from a SCSI initiator via the disk-to-tape storage system  110  utilises cache  270  where possible, and the resulting performance gains when reading or writing cache can be significant. A policy driven mechanism allows for parametric control of cache behaviour. The policies controlling cache behaviour determine events and actions regarding when and where to send data. 
         [0074]    Additional advantages of cache  270  include flexible data manipulation and management. As data are transferred, in-path changes can be executed to allow for format conversions, both in the data itself (e.g. compression), and also in the metadata within the data stream (e.g. header format conversions, date changes). Passing the data via cache  270  also enables improved management of the data flow in accordance with the cache control policies. 
         [0075]    Media Manager Interface  260 —the media manager interface  260  is used to connect the back-end of the disk-to-tape storage system  110  to external SCSI medium changers or tape drives, via media manager  120 . The media manager interface  260  tracks tapes and volumes in the medium changers and tape drives that are specified in SQL database  240 . When disk-to-tape storage system  110  is connected to one or more external media managers  120 , such as the Automated Cartridge System Library Software (ACSLS) available from Storage Technology Corporation, the Direct Attached Storage (DAS) system available from Hewlett Packard Corporation, the  3494  automated tape library available from IBM Corporation, or systems compatible with the IEEE standard 1244 for media management systems, the disk-to-tape storage system operates with the external media manager  120  to maintain knowledge of tape and volume information. Direct control of the external media manager is achieved via the media manager interface  260 . 
         [0076]    The disk-to-tape storage system  110  also includes its own internal media manager  120 A. This can be used to connect the disk-to-tape storage system  110  directly to a tape archive  131 , without the intermediate services of an external media manager. In this case, the internal media manager  120 A is responsible for performing the general functionality of a media manager. 
         [0077]    Operation of the disk-to-tape storage system  110  can be divided into two main phases: configuration and normal operation. Configuration involves discovering the customer resources that are available to the disk-to-tape storage system  110 , and assigning these resources as appropriate. 
         [0078]      FIG. 4  is a flowchart showing operations performed during configuration of the disk-to-tape storage system  110  in accordance with one embodiment of the invention. Processing commences with initial start-up of the system ( 410 ), whereupon the disk-to-tape storage system then probes the real devices and real tapes in a customer&#39;s infrastructure ( 420 ). The information from this survey is stored into the real devices table  320  and the real media table  330  of the SQL database  240  ( 430 ). An operator then assigns any or all of the real resources (i.e. real devices and/or tapes) for use by the disk-to-storage system  110  ( 440 ). 
         [0079]    Note that the survey of real devices and tapes may be repeated periodically after start-up, in order to update the information stored in SQL database. Similarly, the assignment may be revised subsequently by performing a dynamic reconfiguration of the disk-to-tape storage system  110  (i.e. without having to stop and then restart the disk-to-tape storage system). 
         [0080]    The virtual devices are also configured as part of startup ( 450 ). Virtual devices originate in the stock device tables  340  and are shipped extant with the disk-to-tape storage system  110 . The virtual devices include medium changers and tape drives for which operational data has been compiled. Virtual devices  340  are added to the SQL database  240 , and thereafter start when disk-to-tape storage system  110  is started. (It will be appreciated therefore that configuration of the virtual devices does not have to occur after configuration of the real devices, as shown in  FIG. 4 , but may instead occur before or at the same time as configuration of the real devices). 
         [0081]    Normal operation of the disk-to-tape storage system begins after configuration. It is assumed that the customer has configured their application(s)  102  to use the virtual SCSI connections to the disk;-to-tape storage system  110 , after which backup operations may proceed via disk-to-tape storage system as for any other SCSI device from the perspective of application(s)  102 . The disk-to-tape storage system listens for inbound SCSI commands received over network  100  and destined for the virtual devices ( 510 ). This listening is performed by the relevant listener  202 , based on the type of network  100  and communication protocol involved. Note that event listener  205  listens for an event to appear in the event queue  370  of database  240 —if such an event is detected, then the system implements the appropriate action. 
         [0082]    Assuming that a SCSI command arrives at a front-end device driver (such as Qlogic device driver  218 ), this is transferred via the relevant listener to the SCSI manager ( 520 ). The SCSI manager  230  relates inbound SCSI commands by logical unit number (LUN) to the emulated virtual device and decides which object (instances of Vchanger or Vtape) to call. Commands for medium changers are routed to the Vchanger module  240 , while commands for tape drives are routed to the Vtape module  250 , and inbound data associated with a SCSI command is routed to the cache  270  for storage ( 530 ). Specific instances of a virtual device operate on the received SCSI command via the SCSI interpreter for that object. 
         [0083]    The Vchanger module  240  interacts with and maintains the virtual medium changer&#39;s gripper, import/export (CAP), storage (slots), and data transport (drives) elements. When necessary, the Vchanger module interacts with a real medium changer to move real volumes about ( 540 ). Similarly, the Vtape module  250  interacts with and maintains state and status information for the virtual tape drive ( 550 ), and also redirects data to the cache module. Multiple virtual tape drives allow simultaneous data transfers. Inbound commands are handled with each instance of a virtual tape drive. Inbound data is cached and a policy-driven cache operation allows a variety of cache management techniques. For example, customers may set cache policy, such as retention time as part of policies  380 . Return status information and data may be sent along the reverse path from the SCSI manager  230  back out via the appropriate device driver and back to the initiator ( 560 ). 
         [0084]    The disk-to-tape storage system  110  may therefore be used by any backup application  102  that can control SCSI controlled medium changers and tape drives. Applications can be configured for use with disk-to-tape storage system  110 , after which backup and archive operations may proceed via disk-to-tape storage system  110  as with any other SCSI device. Operational history and administrative tools such as logging system  280  allow access to performance and status information from the SQL database, and policy-based logging and history mechanisms support for various operational needs. 
         [0085]    In conclusion, although a variety of particular embodiments have been described in detail herein, it will be appreciated that: this is by way of illustration only. The skilled person will be aware of many further potential modifications and adaptations that fall within the scope of the claims and their equivalents.