crawler for finding all URL's in a given domain and reporting them for creating webserver redirect statements