Meta robots

This commit allows the code to honor the meta attributes for nofollow and noindex. The system is separately configurable to honor nofollow and/or noindex. The default configuration is to honor noindex, but to ignore nofollow.
Sure, will change.