updated robots.txt per Seth's request.
Merge pull request #954 from DirectEmployers/pd-569

[PD-569] updated robots.txt per Seth's request.