Added dynamic url input; users (or whatever is seeding initial URL list) do NOT have to format their urls like 'http://' while the force_valid_http: option is true
option working fine in IRB - testing to come tomorrow
removed put calls
BSON Binary was corrupting data from html bodies; storing with BSON::Serialize now.
Removed BSON::Binary support - was corrupting data
working now, but i'm very weary about corrupt bson data