An efficient and scalable web spider/crawler implementation