reconfigured crawlers to use the db tables