small refactorings, more specs for http crawler client