Enhancements to sweeper to better support Gobblin data

This loop will only remove the hourly/ folder, not the daily/ folder, do we need to clean the daily/ folder?