Working on postgres functions
Need to move @table into a PG table class for refactoring, and add test suite, haven't even tried functions yet, but it should have most features covered
Fixed unless condition on line 37
Added PG SQL to storage.rb
renamed website_id to blog_id
this is a huge mess.
Adapter working - just changed hash key calls form symbol (ie: hash[:body]) to strings (ie: hash['body'])
All is working well - items storing in PG; removing debugging puts; then check to see that crawl behavior stops when expected
Puts removed
All works, but doesn't stop; continues to visit pages even through data is fresh.
All seems to be working, this cafe's internet disconnects continually. will test at home
All works, added rescue for httperror (404 error)
This needs some serious refactoring - all is working on a few different sites; though, it makes a hell of a lot of db calls just to check that an item is A) up-to-date (ie: Freshness level), and B) Check to see if it should be added to the link/page queues. These operations are taken for granted in Mongo, but actually inserting/updating would quickly lead to a O(n^x) situation  where n is the number of links on a given page, and x is the number of threads operating on those links. Or, maybe I'll just re-look at the layout when it's not 3:00am on Sunday.