Enhancements to sweeper to better support Gobblin data

Maybe add a comment, the sourceSubDir usually means 'hourly'