Changes done to adopt new design for controlling the crawler count and
execution