Clean up harvest loading routines.

Summary:
 - make generic directory loading method
 - add mwsd flag to specify harvest extension
 - generalize harvest loading recursive behavior

Test Plan: Run mwsd, mws-index.

Reviewers: radu.hambasan, dhasegan

Reviewed By: radu.hambasan

Differential Revision: http://phab.code4fun.de/D34