many tweaks to support dynamic crawl UI updates