Project: duracloud
Issue DURACLOUD-732: Duplicate-on-Demand via Inspector/Executor flow
As noted in testing described here https://wiki.duraspace.org/x/8YzvAQ, the work of transferring very large numbers of files from one storage provider to another is not particularly well suited to hadoop. This task it to create an alternative content duplication method. The requirements are as follows: - Must be able to move content from any provider to any other provider - Must be able to be deployed onto multiple server instances in order to share the load and speed up processing It may be necessary for this to be a multi-step process. Perhaps something like: 1. Run a process with the source space and the number of instances that will be used for the job as parameters, and it splits up the work into an input file for each worker node. 2. Each worker node is spun up and initialized with the proper information to do the work, including their portion of the job, as defined in step 1. 3. After all nodes have completed step 2, a final process verifies the transfer by comparing the source and destination spaces. The goals of this work: 1. Provide a method which can reliably transfer large numbers of content files from one provider to another. 2. Provide a starting point for the further work of developing a large file processing capability within DuraCloud that does not rely on Amazon EMR.
Story points: 5