@sjarvie -- fix some issues affecting the web-crawlers perf

@sjarvie -- fix some issues affecting the web-crawlers perf