Fix for DS-1781. Also, log a warning if email cannot be determined, but still set to netid.
Cleanup formatting of robots.txt so there are no blank lines in a group
record
Merge pull request #663 from tdonohue/DS-1781

DS-1781 : Ensure LDAP "email_field" is not ignored entirely
Merge pull request #808 from tdonohue/DS-1936-cleanup

DS-1936 followup formatting fixes for robots.txt