Project: duracloud
Issue DURACLOUD-191: Update the Image Conversion service to make use of Amazon's Elastic MapReduce service
In order to handle processing the volume of data being presented by our pilot partners, DuraCloud needs to be able to distribute compute work onto multiple machines. It appears that the most standardized way of doing this distribution is to make use of Apache Hadoop. Since Amazon provides the Elastic MapReduce service, which makes use of Hadoop, it makes sense to investigate that as the initial step towards being able to take advantage of scalable processing in the cloud. We already have data to convert (NYPL tiff images) and an existing Image Conversion service, this task is to allow the image conversion to run via MapReduce.
Story points: 16