added logic to exclude off-site links from the sitemap check and considered some edge cases