Scan improvements 01

speed up scanning of the top 1M sites without overwhelming the machine and few other fixes