Added a basic robots.txt, to prevent crawling by bots. Would need to change it once we enable sharing so it only looks at user pages, and nothing else.
Updated robots.txt so /u/ URLs are accessible, since this is what users will share.