Unified crawler client platform