Revert "Revert "Added a lock around each spider""

This reverts commit 6f1aa38484a0d92edd3aab10b0d249d2f263c568.