Add robots.txt, update crontab and fix comments in .htaccess