refactored a bit and implemented recursive crawling to fill entries that
are missing in sitemap