deprecated urlcrawler functionality, added files cleanup