recursive crawl & add site details