LV: Added Swiftype meta information markers for smarter crawling and categorization
LV: Allowed the swiftype crawler when robots would normally be disabled
LV: Changed how robots.txt is rendered
LV: Reverted robots.txt rendering