fix diffbot smoke tests. do not index spider replies
for custom crawls.