Abstract:
A method, non-transitory computer readable medium and device for seamless storage import includes importing file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained during the importing. An import status of the requested one of the entries is determined. The obtained request is processed based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored.

Description:
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 61/302,864, filed Feb. 9, 2010, which is hereby incorporated by reference in its entirety. 
    
    
     FIELD 
     This technology generally relates to methods and devices for appartuses for importing file systems and, more particularly, to methods for seamless storage importing of file systems and devices thereof. 
     BACKGROUND 
     In a virtualized storage environment, the virtualization device maintains metadata comprising all file system entries for all virtualized storage devices. The initial inventory scan (storage import) of the backend file systems builds the metadata. This storage import involves the virtualization device walking the entire directory structure to enumerate every file and directory in the backend file systems. It is critical that the file system entries are accurate and no incorrect entries exist in the metadata, and that no entries are missed. Depending on the dataset, storage import can take from minutes to several days or even weeks to complete. 
     During this time period, client querying and client operations that may result in file system changes are sent to the virtualization device and are expected to be handled and responses to in a timely manner. However, due to the nature and requirements of the storage import process described above, certain restrictions have been imposed on the types of client operations that are permitted during the data storage import to ensure that the client modifications do not compromise the integrity of the metadata. 
     Currently, directory modifications, such as directory renames or directory deletions, as well as certain file operations, are not allowed during the storage import scan. This is non-optimal at best and non-workable at worst in certain environments where clients may lose access to view or modify their files for long periods of time during the storage import process, which as mentioned earlier may take anywhere from hours to days/weeks. Additionally, in case of metadata failure where a rebuild of the metadata is required, these restrictions could easily interrupt the normal client workflow and impact operations with potential substantial costs. 
     SUMMARY 
     A method for seamless storage importing includes importing with a storage management computing device file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained at the storage management computing device during the importing. An import status of the requested one of the entries is determined with the storage management computing device. The obtained request is processed with the storage management computing device based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored with the storage management computing device. 
     A non-transitory computer readable medium has stored thereon instructions for seamless storage importing comprising machine executable code which when executed by at least one processor, causes the processor to perform steps including importing file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained during the importing. An import status of the requested one of the entries is determined. The obtained request is processed based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored. 
     A storage management computing device includes a memory coupled to the one or more processors which are configured to execute programmed instructions stored in the memory including importing file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained during the importing. An import status of the requested one of the entries is determined. The obtained request is processed based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored. 
     This technology provides a number of advantages including providing more effective and efficient methods and devices for seamless storage importing of file systems. With this technology, file system operations are permitted during the storage import process. As a result, clients do not lose access to view or modify their files during the storage import process which could potentially take anywhere from hours to days/weeks. Additionally, while permitting files system operations during storage import, this technology still ensures that all imported file system and metadata entries are accurate. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is an environment with an exemplary storage management computing device for seamless storage import; 
         FIG. 2  is a flowchart of an exemplary method for seamless storage importing; 
         FIG. 3  is an example of a directory to import; 
         FIG. 4  is a flowchart of an exemplary method for processing of a directory; 
         FIG. 5  is a flowchart of an exemplary method for managing the start of processing of a directory; 
         FIG. 6  is a flowchart of an exemplary method for managing the completion of processing of a directory; 
         FIG. 7  is a flowchart of an exemplary method for updating a scan based on one or more client changes during importing; 
         FIG. 8  is a flowchart of an exemplary method for processing each entry in a directory; 
         FIG. 9  is a flowchart of an exemplary method for managing importation of a directory during a client operation; and 
         FIG. 10  is a flowchart of an exemplary method for processing the next component. 
     
    
    
     DETAILED DESCRIPTION 
     An environment  10  with an exemplary storage management computing device  14  for seamless storage import is illustrated in  FIG. 1 . The environment  10  includes the storage management computing device  14 , a client computing device  12 , data storage devices  16 ( 1 )- 16 ( 2 ), and a metadata storage device  18  which are all coupled together by one or more communication networks  20 , although this environment can include other numbers and types of systems, devices, components, and elements in other configurations. This technology provides a number of advantages including providing methods and devices for seamlessly importing file systems. 
     The storage management computing device  14  provides seamless storage import from the data storage devices  16 ( 1 )- 16 ( 2 ) and the metadata storage device  18 , although other numbers and types of systems could be used and other numbers and types of functions could be performed. The storage management computing device  14  includes a central processing unit (CPU) or processor  22 , a memory  24 , and an interface system  26  which are coupled together by a bus or other link, although other numbers and types of systems, devices, components, and elements in other configurations and locations can be used. The processor  22  in the storage management computing device  14  executes a program of stored instructions for one or more aspects of the present technology as described and illustrated by way of the examples herein, although other types and numbers of processing devices and logic could be used and the processor could execute other numbers and types of programmed instructions. 
     The memory  24  in the storage management computing device  14  stores these programmed instructions for one or more aspects of the present technology as described and illustrated herein, although some or all of the programmed instructions could be stored and executed elsewhere. A variety of different types of memory storage devices, such as a random access memory (RAM) or a read only memory (ROM) in the system or a floppy disk, hard disk, CD ROM, DVD ROM, or other computer readable medium which is read from and written to by a magnetic, optical, or other reading and writing system that is coupled to the processor  22  in the storage management computing device  14 , can be used for the memory  24  in the storage management computing device  14 . 
     The interface system  26  in the storage management computing device  14  is used to operatively couple and communicate between the storage management computing device  14  and the client computing device  12  as well as the data storage devices  16 ( 1 )- 16 ( 2 ) and the metadata storage device  18  via one or more of the communications networks  20 , although other types and numbers of communication networks or systems with other types and numbers of connections and configurations can be used. By way of example only, the one or more communications networks can use TCP/IP over Ethernet and industry-standard protocols, including NFS, CIFS, SOAP, XML, LDAP, and SNMP, although other types and numbers of communication networks, such as a direct connection, a local area network, a wide area network, modems and phone lines, e-mail, and wireless communication technology, each having their own communications protocols, can be used. 
     The client computing device  12  utilizes the storage management computing device  14  to conduct one or more operations with one or more of the data storage devices  16 ( 1 )- 16 ( 2 ) and the metadata storage device  18 , such as to store a file or directory, delete a file or directory, create a file or directory, or rename a file or directory by way of example only, although other numbers and types of systems could be utilizing these resources and other types and numbers of functions utilizing other types of protocols could be performed. Although one client computing device  12  is shown, other numbers and types of devices could be used to interact with the storage management computing device  14 , the data storage devices  16 ( 1 )- 16 ( 2 ) and the metadata storage device  18 . Similarly, other numbers and types of data storage devices and metadata storage devices could be used. Each of the data storage devices  16 ( 1 )- 16 ( 2 ) stores content, such as files and directories, although other numbers and types of storage systems which could have other numbers and types of functions and store other data could be used. The metadata storage server  18  stores metadata relating to the stored content, such as files and directories, on the data storage devices  16 ( 1 )- 16 ( 2 ). 
     The client computing device  12 , the data storage devices  16 ( 1 )- 16 ( 2 ), and the metadata storage device  18  each include a central processing unit (CPU) or processor, a memory, and an interface or I/O system, which are coupled together by a bus or other link, although each could comprise other numbers and types of elements and component, such as control logic. The client computing device  12 , in this example, may make requests for and send data to different data storage devices  16 ( 1 )- 16 ( 2 ) and interact with metadata storage device  18  via the storage management computing device  14 . Generally, the storage management computing device  14  process requests received from the requesting client computing device  12  for files or directories on one or more of the data storage devices  16 ( 1 )- 16 ( 2 ) using metadata stored in metadata storage server  18 . The data storage devices  16 ( 1 )- 16 ( 2 ) may provide data or receive data in response to the received request. 
     Although examples of the storage management computing device  14 , the client computing device  12 , the data storage devices  16 ( 1 )- 16 ( 2 ), and the metadata storage device  18  are described herein, each of these systems can be implemented on any suitable computer system or computing device. It is to be understood that the devices and systems of the examples described herein are for exemplary purposes, as many variations of the specific hardware and software used to implement the examples are possible, as will be appreciated by those skilled in the relevant art(s). 
     Furthermore, each of the systems of the examples may be conveniently implemented using one or more general purpose computer systems, microprocessors, digital signal processors, and micro-controllers, programmed according to the teachings of the examples, as described and illustrated herein, and as will be appreciated by those ordinary skill in the art. 
     In addition, two or more computing systems or devices can be substituted for any one of the systems in any embodiment of the examples. Accordingly, principles and advantages of distributed processing, such as redundancy and replication also can be implemented, as desired, to increase the robustness and performance of the devices and systems of the examples. The examples may also be implemented on computer system or systems that extend across any suitable network using any suitable interface mechanisms and communications technologies, including by way of example only telecommunications in any suitable form (e.g., voice and modem), wireless communications media, wireless communications networks, cellular communications networks, G3 communications networks, Public Switched Telephone Network (PSTNs), Packet Data Networks (PDNs), the Internet, intranets, and combinations thereof. 
     The examples may also be embodied as a computer readable medium having instructions stored thereon for one or more aspects of the present technology as described and illustrated by way of the examples herein, as described herein, which when executed by a processor, cause the processor to carry out the steps necessary to implement the methods of the examples, as described and illustrated herein. Exemplary methods for seamless storage import will now be described herein. 
     An exemplary method for seamless storage importing will now be described with reference to  FIGS. 1-10 . Referring more specifically to  FIG. 2 , in step  200  the storage management computing device  14  begins to process an exemplary directory illustrated in  FIG. 3 . This exemplary directory includes a plurality of files (file 1 -fileN) and a plurality of directories (dir 1 -dirN), although the directory could comprise other numbers and types of structures. 
     The exemplary processing of the directory in step  200  of  FIG. 2  is described in greater detail with reference to  FIGS. 4-6 . Referring more specifically to  FIG. 4 , in step  202  the processing of the directory by the storage management computing device  14 . Next, in step  204  the storage management computing device  14  determines whether processing of a new directory has started. If in step  204  the storage management computing device  14  determines processing of a new directory has been started, then the Yes branch is taken to B at the start of the flowchart shown in  FIG. 5 . 
     Referring to  FIG. 5 , in step  212  the storage management computing device  14  determines whether the directory is already being processed. If in step  212  the storage management computing device  14  determines the directory currently being imported is already being processed, then the Yes branch is taken to step  214 . In step  214 , the storage management computing device  14  skips initiating any new processing with respect to this directory. Next, in step  222  this portion of the processing the directory is done and the storage management computing device  14  returns to step  210  in  FIG. 4 . 
     If in step  212  the storage management computing device  14  determines the directory currently being imported is not already being processed, then the No branch is taken to step  216 . In step  216 , the storage management computing device  14  indicates that the directory currently being imported is now being processed and initiates tracking all changes to this directory, such as by a file operation received from the client device  12  by way of example only. 
     In step  218 , the storage management computing device  14  determines whether a processing structure for this directory currently being imported exists. If in step  218  the storage management computing device  14  determines a processing structure for this directory currently being imported does not exist, then the No branch is taken to step  220 . In step  220 , the storage management computing device  14  creates a processing structure for this directory currently being imported and then proceeds to step  222  where this portion of the step of processing the directory in step  200  in  FIG. 2  is done. If in step  218  the storage management computing device  14  determines a processing structure for this directory currently being imported does exist, then the Yes branch is taken to step  222 . Next, in step  222  this portion of the processing the directory is done and the storage management computing device  14  returns to step  210  in  FIG. 4 . 
     Referring back to  FIG. 4 , if in step  204  the storage management computing device  14  determines processing of a new directory has not been started, then the No branch is taken to step  206 . In step  206 , the storage management computing device  14  determines whether the directory currently being processed is restarting processing. If in step  206  the storage management computing device  14  determines the directory currently being processed is restarting, then the Yes branch is taken to step  208 . In step  208 , the storage management computing device  14  resets the processing back to the beginning of the directory and returns to step  204 . 
     If in step  206  the storage management computing device  14  determines the directory currently being processed is not restarting, then the No branch is taken to step  210 . In step  210 , the storage management computing device  14  determines whether processing of the directory currently being imported is done. If in step  210 , the storage management computing device  14  determines that processing of the directory currently being imported is done, then the Yes branch is taken to C at the start of the flowchart shown in  FIG. 6 . 
     Referring to  FIG. 6 , in step  240 , the storage management computing device  14  sets the pending processing notification for the directory currently being processed to a done state. In step  242 , the storage management computing device  14  determines whether the processing structure established for the directory currently being process is referenced by any other work threads for other directories. If in step  242 , the storage management computing device  14  determines the processing structure is referenced by another work thread, then the Yes branch is taken to step  246  where this portion of the step of processing the directory in step  200  in  FIG. 2  is done. 
     If in step  242 , the storage management computing device  14  determines the processing structure is not referenced by another work thread, then the No branch is taken to step  244 . In step  244 , the storage management computing device  14  deletes this processing structure for the directory which was currently being processed and then proceeds to step  246  where this portion of the step of processing the directory in step  200  in  FIG. 2  is done. 
     Referring back to  FIG. 4 , if in step  210 , the storage management computing device  14  determines processing of the directory currently being imported is not done, then the No branch is taken to A back at the start of the flowchart shown in  FIG. 2 . 
     Referring back to  FIG. 2 , after the processing of the directory in step  200  and as described in greater detail with reference to  FIGS. 4-6 , the storage management computing device  14  proceeds to step  300 . In step  300 , the storage management computing device  14  updates the scan based on any changes resulting from one or more file operations initiated by the client computing device  12 , although the changes can occur in other manners from operations initiated by other devices. 
     Referring more specifically to  FIG. 7 , in step  302  the storage management computing device  12  determines whether any changes have been made to the directory which currently is being processed by the client device  12  or other device or system. If in step  302 , the storage management computing device  12  determines one or more changes have not been made to the directory which currently is being processed, then the No branch is taken to step  308  where this portion of the step of processing the directory in step  300  in  FIG. 2  is done. 
     If in step  302 , the storage management computing device  12  determines one or more changes have been made to the directory which currently is being processed, then the Yes branch is taken to step  304 . In step  304 , the storage management computing device  14  determines whether this directory currently being processed has been marked or otherwise identified as requiring a restart because of a change by the client computing device  12  or other device. If in step  304 , the storage management computing device  14  determines this directory currently being processed has been marked or otherwise identified as requiring a restart, then the Yes branch is taken to D in  FIG. 4  where a directory restart process starts in step  206 . 
     If in step  304 , the storage management computing device  14  determines this directory currently being processed has not been marked or otherwise identified as requiring a restart, then the No branch is taken to step  306 . In step  306 , the storage management computing device  14  determines whether any of the parent directories to the directory currently being processed have any changes that require a restart of this directory. If in step  306 , the storage management computing device  14  determines none of the parent directories has any changes that requires a restart of this directory, then the No branch is taken to step  308  where this portion of the step of processing the directory in step  300  in  FIG. 2  is done. 
     If in step  306 , the storage management computing device  14  determines one or more of the parent directories has one or more changes that requires a restart of this directory, then the Yes branch is taken to D in  FIG. 4  where a directory restart process starts in step  206 . 
     Referring back to  FIG. 2 , after updating the scan based on any changes resulting from one or more file operations initiated by the client computing device  12  in step  300  and as described in greater detail with reference to  FIG. 7 , the storage management computing device  14  proceeds to step  400 . In step  400 , the storage management computing device  14  processes each entry in the directory. 
     Referring more specifically to  FIG. 8 , in step  402 , the storage management computing device  14  begins to process each entry in the directory currently being processed for importation. In step  404 , the storage management computing device  14  determines whether there is an importlock currently being held on the entry in the directory being processed. If in step  404 , the storage management computing device  14  determines there is an importlock currently being held on the entry in the directory being processed, then the Yes branch is taken to step  404  until the importlock is released or a programmed period of time expires. 
     If in step  404 , the storage management computing device  14  determines there is a not importlock currently being held on the entry in the directory being processed, then the No branch is taken to step  406 . In step  406 , the storage management computing device  14  sets an importlock for the entry currently being processed. 
     In step  408 , the storage management computing device  14  determines whether the client device  12  or other device or system has modified this entry or created a change list entry, although other manners for indicating a change to this entry can be used. If in step  408 , the storage management computing device  14  determines this entry has been modified or a change list entry has been created, then the Yes branch is taken to step  410 . In step  410 , the storage management computing device skips this entry. 
     If in step  408 , the storage management computing device  14  determines this entry has not been modified and a change list entry has not been created, then the No branch is taken to step  412 . In step  412 , the storage management computing device  12  creates metadata for this entry for the importation process. In step  414 , the storage management computing device  14  releases the importlock on this entry. 
     In step  416 , the storage management computing device  14  determines whether there are any additional entries in this directory to be processed. If in step  416 , the storage management computing device  14  determines there is an additional entry in this directory to be processed, then the Yes branch is taken back to step  402 . If in step  416 , the storage management computing device  14  determines there is not an additional entry in this directory to be processed, then the No branch is taken back to step  418  where this portion of the step of processing the directory in step  400  in  FIG. 2  is done. 
     Referring back to  FIG. 2 , after the client computing device  12  processes each entry in the directory in step  400  and as described in greater detail with reference to  FIG. 8 , the storage management computing device  14  proceeds to step  500 . In step  500 , the storage management computing device  14  determines whether the directory has been fully imported. If in step  500 , the storage management computing device  14  determines the directory has not been fully imported, then the No branch is taken back to step  200 . 
     If in step  500 , the storage management computing device  14  determines the directory has been fully imported, then the Yes branch is taken back to step  600 . In step  600 , the storage management computing device  14  signals that the seamless storage import process has been completed. 
     Referring to  FIG. 9 , an exemplary method for managing importation of a directory during a client operation is illustrated. In step  702 , the storage management computing device  14  monitors for and identifies a file operation on a path in a directory by the client device  12 . 
     In step  704 , the storage management computing device  14  determines whether the file operation is on a directory which has already completed an import scan. If in step  704 , the storage management computing device  14  determines the file operation is on a directory which has already completed an import scan, then the Yes branch is taken to step  716 . In step  716 , the storage management computing device  14  is ready to service the file operation requested by the client device  12 . If in step  704 , the storage management computing device  14  determines the file operation is on a directory which has not already completed an import scan, then the No branch is taken to step  706 . 
     In step  706 , the storage management computing device  14  determines whether all path and entry components exist for the requested file operation. If in step  706 , the storage management computing device  14  determines all path and entry components exist for the requested file operation, then the Yes branch is taken to step  716  as described earlier. If in step  706 , the storage management computing device  14  determines all path and entry components do not yet exist for the requested file operation, then the No branch is taken to step  708 . 
     In step  708 , the storage management computing device  14  determines whether all necessary components have been processed for the requested file operation. If in step  708 , the storage management computing device  14  determines all necessary components have been processed for the requested file operation, then the Yes branch is taken to step  716  as described earlier. If in step  708 , the storage management computing device  14  determines all necessary components have not been processed for the requested file operation, then the No branch is taken to step  710 . 
     Referring to  FIG. 10 , an exemplary method for processing all necessary components in step  710  in  FIG. 9  is illustrated. In step  718 , the storage management computing device  14  conducts a lookup of the entry for the requested file operation on all shares. 
     In step  720 , the storage management computing device  14  determines whether the entry exists in any share. If in step  720 , the storage management computing device  14  determines the entry does not exist on any share, then the No branch is taken back to step  712  in  FIG. 9 . If back in step  720  in  FIG. 10 , the storage management computing device  14  determines the entry does exist on one or more shares, then the Yes branch is taken to step  722 . 
     In step  722 , the storage management computing device  14  determines whether the entry exists in metadata for all shares the entry was found on. If in step  722 , the storage management computing device  14  determines the entry exists in metadata for all shares the entry was found on, then the Yes branch is taken back to step  712  in  FIG. 9 . If in step  722 , the storage management computing device  14  determines the entry does not exist in metadata for all shares the entry was found on, then the No branch is taken back to step  724 . In step  724 , the storage management computing device  14  imports the file system entry and then proceeds back to step  712  in  FIG. 9 . 
     In step  712 , the storage management computing device  14  determines whether the importation of the entry was successful. If in step  712 , the storage management computing device  14  determines the importation of the entry was not successful then the No branch is taken to step  714 . In step  714 , the storage management computing device  14  generates an error message which is returned to the client device  12 . If in step  712 , the storage management computing device  14  determines the importation of the entry was successful then the Yes branch is taken back to step  708  as described in earlier. 
     Accordingly, as illustrated and described herein this technology provides a number of advantages including providing more effective and efficient methods and devices for seamless storage importing of file systems. With this technology, file system operations are permitted during the storage import process. As a result, clients do not lose access to view or modify their files during the storage import process which could potentially take anywhere from hours to days/weeks. Additionally, while permitting files system operations during storage import, this technology still ensures that all imported file system and metadata entries are accurate. 
     Having thus described the basic concept of the invention, it will be rather apparent to those skilled in the art that the foregoing detailed disclosure is intended to be presented by way of example only, and is not limiting. Various alterations, improvements, and modifications will occur and are intended to those skilled in the art, though not expressly stated herein. These alterations, improvements, and modifications are intended to be suggested hereby, and are within the spirit and scope of the invention. Additionally, the recited order of processing elements or sequences, or the use of numbers, letters, or other designations therefore, is not intended to limit the claimed processes to any order except as may be specified in the claims. Accordingly, the invention is limited only by the following claims and equivalents thereto.