Master now uses database to store crawled URLs.

Fixes #40