Added a semaphore which limits the number of simultaneous crawls

Also, refactored closing bodies to make sure all content is read before close.