Updated robots.txt.
Merge pull request #1044 from DirectEmployers/pd-569

[PD-569] Updated robots.txt.