add support for configuring robots.txt
Fixes #4832 adds support for dynamic robots.txt
display warning for robots.txt if Elgg is not in web dir root and search adds its pages to exclusion list
Merge pull request #5668 from cash/robots

Adds robots.txt support to Elgg