Make sure directories we are downloading to are empty

We already clear them on error anyway and usually they should be
empty, but just to be safe.