Use `wp` filter to detect site map requests. Only invalidate main query and make sure $wp_query is set.