Basic backend structure for controlling the crawl