removes unneeded files and bans spiders