Updated the README.
Frontier + tests added. More tests added to the crawler. Etc.