Fix robots text files by making deny all the default

Also remove duplicate robots files now that a default is given.