Several updates to robots features to hopefully fix the URL issues.
Versioning