Some refactoring to make the get and head requests use the same base
code.
Adding my notes to the cobweb code for reference
Adding module designed to be a mixin providing the requesting
functionality in the main cobweb class
Merge branch 'new_branch_name' into malcolm-testing-develop
initial refactor of the get/head request code to use the new CobwebRequest
module.

Moved all the request code from the cobweb class into its own
module. This makes the core cobweb class much cleaner
Adding code coverage support to the cobweb project
Adding deadlock detection and recovery to the locking code to make
the various locks work properly
Minor corrections so that tests will run correctly
Made cacheing a seperate class, mainly so that its easier to manage
our caching overall.

- Testing is now much easier too, since redis can
be turned off in tests easily testing the requesting section of code
Adding more tests for head testing
Switched built in http lib out to use typhoeus/libcurl.

Libcurl is much faster than the built in ruby http libraries and
should also make the request process as reliable as http can be
All tests now pass after switching to Typheous http lib.

The mocks have been changed to properly reflect the new library
All tests now pass after switching to Typheous http lib. pt2
- Removing puts statement that was left in accidentially for debugging
Adding tests and fixes for redis caching abstraction.

The caching code was extracted from the request code in order to simplify the request code
Adding more sample httml to make testing locally more effective
Crawler finds links in a single process now. A little slower, but is
always correct in crawl numbers and doesnt have to deal with locking
and concurrency issues.
Depth of te crawl from the base url is now controllable, to allow for
fine tuning
Fixed recursion bug that was causing duplicatte urls to be added to
crawl list
Adding support for processing url items that have been queued using
the SpiderJob. Work is handed to a processing class that is passed in
the options hash parameter "url_processor" that has a perform method.
Added some error checking for url_processor
Modified some default settings to work with the new crawl implementation