refactors crawler, fixes #26