Adding scratch space support via small EBS volume
Changed crawler to use shared scratch space (EBS volume) to store
temporary files. This fixes the previous memory issue (root cause: transmitting
full html back to master over parallelpython's framework) and allowed me
to bump the batch size back to 1000