redesigned the spider to be more modular, allowed adding NTLM authentication, settings from UI