removed the WebUI code
Merge branch 'experimental' into no-webui
Merge branch 'experimental' into no-webui
Merge branch 'experimental' into no-webui
Merge branch 'experimental' into no-webuii
removed web executables
Added preliminary distributed crawling support

[Issue #207]

* Added a special RPC::Server::Spider class handling the distribution
* Updated RPC::Server::Framework to support the new Spider and handle the new asynchronous crawling model -- still needs to grab remote page queues and element list
* RPC::Server::Distributor: #spawn => #configure_and_run -- and got cleaned up and split amongst a few helpers
Distributed crawling

[Issue #207]

* Slight cleanup of preliminary code
* Updated to send element IDs per page and the page queues of each slave to the master
Merge branch 'experimental' into feature/distributed-crawling
RPC::Server::Spider#collect_sitemaps: now sorts the sitemap
* RPC::Server::Framework::Distributor
  o #prefered_dispatchers => #preferred_dispatchers
  o updated tests
* Cleaned up crawl distribution code

[Issue #207]
RPC::Server::Spider: Changed from master polling to slave signaling
updated the way element IDs are sent back to the master to make much fewer calls [Issue #207]
RPC::Server::Framework, RPC::Server::Spider: further distributed crawl code cleanup and signaling bugfixes [Issue #207]
RPC::Server::Framework#run: changed async dispatcher iterator from #map to #each
RPC::Server::Framework::Distributor#preferred_dispatchers: calls the block on empty dispatchers instead of returning
RPC::Server::Framework
  * #progress_data: @opts.datastore[:dispatcher_url] => master?
  * #set_master: the element ID hash is being pruned to avoid transferring useless data
Arachni::URI: backed-off the errors
RPC::Server::Spider#collect_sitemaps: @peers.empty? => !master?
RPC::Server::Framework: updated the way sitemaps are handled