API robots.yml updated with new configuration options, removal of obsolete options, and fixed to conform to naming conventions
Added documentation
Fixed indentation