now expands url into list of urls to check, fixed up canonicalization