Never make a crawler wait for remote content, ever - depend now on the proactive cache, it's okay if that means some content has to wait a day before its text/title/whatever shows up in Google or the like