Actually remove files and empty directories