Base API for programming web crawlers