fix mem leak. always harvest links
for custom crawls.
Merge branch 'diffbot' into diffbot-testing