Database changes, daemon, queue setup for robots.txt checking
update script with comments, add more config
read TTL from config; read URL submit routing key from config
fix default destination
add size & mime checks, logging
Merge pull request #65 from openrightsgroup/features/robots-txt-rebase

Features/robots txt rebase