Implementation of a basic web crawler interface.