Progressively update/append contacts to the cache to avoid blocking UI