refactor for general sitemap and common crawler