refactored to use a stream worker so that we can
be sure the docs are processed before we call our callback