using xpath for link grabbing instread of CSS (faster)
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3
Redirect is only in the range of 300 to 307
normalize redirect links, otherwise we run into stuff like "http://www.example.org" (missing final slash)
go back to uniq'ing links in Page
create mongo storage and rewrite storage tests to use only Pages
added Rakefile

currenly only the autospec command works, 'rake spec' still gives errors.
list info about development
Allow the crawler to skip links with query parameters.

This is done by calling 'skip_query_string' on the yielded anemone object.
refactor local variable into private method
refactor another local variable into a method
improve readability
fix typo in Page#to_hash to get tests passing
Merge branch 'mongo' into next
add exceptions and storage base abstraction
save certain page fields as binary in mongo storage adapter
add redis storage
Merge branch 'fork' into next
refactor skip_query_link option
update README
add VERSION file