Be sure that the annotate does not take in count unpublished entries and do not show authors or categories without published entries in the admin filters
Review the comment about number of queries
Be sure that the sitemap does not take in count element without published entries
Mesuring the number of queries in sitemaps