INVENTORYING AND COPYING FILE SYSTEM FOLDERS AND FILES

Described is a system and method that facilitates fast and reliable discovery, enumeration, and processing of network shared resources. A file system processing subsystem (date engine) operates in a combined discovery, enumeration, and processing manner to provide access and operation as directed by the client to effect data collection and copying. The discovery, enumeration, and action operations use parallel operation and I/O (input/outputA) pipelining. Multiple threads are used during this process to enumerate each object's children, and enqueues each child to be handled by a new thread. For each network object discovered, the subsystem creates an object embodying operation and context information, and queues that object as a self-contained, asynchronous work item for a process thread pool to handle.

DETAILED DESCRIPTION

One example of a suitable computing system environment on which the invention may be implemented is a Microsoft Windows™ Server operating in a Microsoft Windows™ network. This computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment be interpreted as having any dependency or requirement relating to the Microsoft Windows™ Operating System.

The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.

Inventorying, Copying, Deleting File System Folders and Files

The present invention is generally directed towards a system and method by which File systems on network storage devices are inventoried and/or copied. To this end, the present invention walks the file system tree nodes of the specified source file system node and creates file system object operations that are queued for execution from a pool of available threads, including subsequent processing of subfolders. This method avoids the dangers of actual recursion. As will be understood, the subsystem in this implementation provides for significant parallel operation and I/O (input/output) pipelining, and thus facilitates rapid file system processing, whereby the subsystem is referred to herein (for brevity) as the DataEngine subsystem101(FIG. 1).

In general, in this example implementation, the DataEngine subsystem101component enumerates the folders and files of the source file system node and creates folder operation and file operation requests (Folder Operators and File Operators), and potentially creates a folder on a target file system in the case of Copy with associated folder metadata (attributes, dates, etc.) and Access Control Lists (ACLs).

Each Folder Operator and File Operator is executed entirely asynchronously from other Folder Operator and File Operator operations, the order of operation being determined by the order of queuing as determined by prior operations and obviating the need for costly synchronizing callbacks between or among the operators, since folder operations that processed the parent folders have already completed and exited upon enumeration of its folders and files.

The DataEngine101is initiated by a calling program100with a command (in this implementation Copy, Sync, or Inventory), a Source folder path and name, and a Target folder path and name(ignored in the case of Inventory). The command will cause an access to Server A105as the source of the Copy, Inventory, or Delete command and in the case of Copy command, Server B104is also accessed as the destination location of the Copy command. After processing the file or folder according to the command, the result of the processing is sent to the logger program103to record the event. In the case of the Inventory command, an additional result is sent to the DataCollector102to record the File Object found by the processing. The result sent to the DataCollector102and Logger103is also sent back to the User interface100.

The DataEngine creates a Folder Operator object200from the Source folder and queues202this object to initiate the process.

The Folder Operator206enumerates the contents of the folder and creates and queues Folder Operators202and File Operators202to perform operations on these file system entities. The DataEngine Controller200manages the thread pool209and the queue202, assigning the next operator to the next available thread. The process terminates when the queue is empty and the threads are all idle.

In the case of Copy, a Folder Operator is an object that enumerates a folder and creates and queues Folder Operators202and File Operators202to perform operations on these file system entities. In the case of Inventory, a Folder Operator is an object that enumerates a folder and creates and queues Folder Operators202, and collects information on folders and files that is reported to the DataCollector2; no File Operators are created in an Inventory operation. All folder and File Operators report progress and error information to the Logger210. The choice of information logged is controlled by options that are passed to the operators.

FIG. 4shows the queuing and threading of the Folder and File Operators. An object is received by queuing process401, identified by priority402, and placed in the normal priority queue403or the high priority queue404. Objects are removed from the queues405,406and assigned to an available thread409and executed411. As threads exit, they are returned to the thread pool410.

A File Copy Operator copies a file from the source folder to the target folder using double-buffered, overlapped input/output operations in this implementation. This reduces the time to copy a file from 2n to n+1, where n is the number of reads or writes to span the entire file. A File Operator copies all file metadata and access control list contents from the source folder to the target folder. A File Operator logs progress and error messages, gated by the logging options.

There are multiple types of Folder Operators to accomplish various types of copy operations, including but not limited to Simple Copy, Update Copy, and Synchronization Copy. The selection of folders and files may be filtered by name, date and time, and/or attributes. Folder and File Filters may be specified to narrow the selection of file system objects that have operations applied to them by Folder and File Operators.FIG. 5ashows the common structure and process of all Folder Operators types. A folder's contents are read502, filters are applied to the list of sub-folders and the resulting folders have Folder Operators queued503. Similarly, filters are applied to the list of sub-files and the resulting files have File Operators queued504. The target folder's metadata are updated505, the summary data of the folder contents are collected506, and the enumeration of the folder is logged507.

The Simple Copy Folder Operator copies all folders and files with their associated metadata and ACLs from the source folder without regard to the content of the target folder. The Simple Copy Folder Operator queues Simple Copy Folder Operators for its subfolders.

The Update Copy Folder Operator copies all folders and files with their associated metadata and ACLs from the source folder when the file in the source folder is different from target folder in regard to size or date, or when the folder or file does not exist on the target. When a folder in the target does not exist, the Update Copy Folder Operator no longer queues Update Folder Operators but optimizes by queuing Simple Copy Folder Operators thus saving performing read operations on the target device for subsequent operations.

The Synchronization Copy Folder Operator copies all folders and files with their associated metadata and ACLs from the source folder to the target folder when the file in the source folder is newer306than the file in target folder or does not exist in the target folder301and copies all folders and files with their associated metadata and ACLs from the target folder to the source folder when the file in the target folder is newer306than the file in source folder or does not exist301in the source folder. To implement this processing there is only one Synchronization Operator and the parameters for source and target folders are swapped306when calling the Operator. When a folder in the source does not exist on the target302, the Synchronization Copy Folder Operator no longer queues Synchronization Folder Operators but optimizes by queuing Simple Copy Folder Operators305thus saving performing read operations on the target device for subsequent operations. Following each Copy action the activity is recorded in the Logger303followed by returning the Thread304.

The Folder Inventory Operator enumerates the folders and files to create a data record212for transmission to the DataCollector213and then sends a message to the Logger210. It creates206and queues202a Folder Inventory Operator for each of its subfolders.

The Folder Delete Operator enumerates the folders down to the leaf level where upon reading the folder size metadata and finding zero size or empty contents, the Operator will remove the current Folder.

FIG. 5bshows the process of the File Operator to accomplish file transfer from source to target. The operation begins by creating two input/output buffers, opening the source file, reading its metadata512. The file is read and written in a double-buffered, overlapped manner513,514,515until the end of the source file is reached516, and the last buffer written to the target517. The file metadata is updated518and the operation is logged519. The thread exits.

The File Delete Operator is queued by another Operator. In the case of Copy type Operator following a successful copy operation, and if the Delete Operator is active then Copy will queue a Delete Operator on that File and the Copy operation completes.

The Logger is a process running on its own thread that embodies a dual queue and a writer601. Folder and File Operators add messages602to the queue in memory603and the Logger asynchronously writes the log entries to a file604in this implementation. The process is non-blocking due to the use of two queues, active and inactive605. The thread shuts down upon DataEngine exit606.

The DataCollector is a process running on its own thread that embodies a dual queue and a writer701. Folder and File Operators add inventory data objects702to the queue in memory703and the DataCollector asynchronously writes the log entries to a database704in this implementation. The process is non-blocking due to the use of two queues, active and inactive705. The thread shuts down upon DataEngine exit706.