updated README
list info about development
update README
merge changes to track external links
Think I found the culprit - @opts not being sent as hash
Wooh boy. Code was working, but it froze during a complicated crawl, investigating options / issues from github
Imported performance increases from https://github.com/chriskite/anemone/pull/70
Speedy, super quick and reliable logging of all internal links and external links. Stops when it needs to with :force_threading . This particular fork didn't update it's tests, but since I've grouped together code from about 5 different repos, Ill need to write quite a few tests to guarantee performance
Removed legacy collection_name variable from MongoDB initializers; moved to options hash for clarity & simplicity.
We can now log urls, updating only those which are older than the desired freshness (:time_to_stale in @opts); since the time_to_stale is checked BEFORE visiting the URL, this could potentially improve performance quite a bit on slow sites.
Added dynamic url input; users (or whatever is seeding initial URL list) do NOT have to format their urls like 'http://' while the force_valid_http: option is true
Dropped parsing for the 'base' tag, as it's a waste of a method call (it may be in the w3 spec, but nobody uses it, AND we already have the base URL, why do we need to call a method for every URL being parsed... inside of that url?).

All is working and all is well, finally.
All is working as expected; tests to come.
Removed mongo.rb.orig - leftover from rebase
gemspec didn't update correctly in mergetool, my error
Rebased into master