Nixed 'datetime' field, added robots.txt parsing.