updated crawler code and helpers. Pending version bump for apache clients which are going fluent.