minor update to robots.txt and add _formated_short_contents to notes