Add robots monitoring functiontality for the backend
Rename crawl_summary to summary
Add an extra small test for email addresses
Move sidebar to shared folder
Merge pull request #5 from MartijnSch/robots

Include a feature for monitoring robots.txt files.