First crack at the dispatcher changes
Cleaned up some naming issues. And rewrote the algo a little.
Checkpoint during test writting.
Just about ready for pull request.
Verify that the getnow tag was cleared correctly.
dispatcher: no need to sort links

Having the ORDER BY was causing slowness and happens to not be needed
anyway since the links are sorted in the desired order by default.
Comment change.
Moved imin and round to functions outside dispatcher
Changed a timeout from 5ms to 100ms.

The go-routines that where handling the sigint weren't catching it
and hanging.
Wrote add algorithm using chris' (modified) algorithm.
Stop clearing the getnow flag
Cell no longer holds domain
Memory saving optimization, and comment purge.
Localize the setting of Config.Dispatcher config vars.
Remove commented out import.
Fixed spelling mistake in function name.
Lifted the definition of cell out of generateSegments
fetcher: do not exclude entire hosts for TCP error

Previously we assumed we shouldn't crawl a host if we couldn't connect
to it, but in some cases (ex. rackcdn.com) the subdomains could be
crawled. In the future we should do more to check the IP on a per-fetch
basis, but for now changed it so we still try to crawl the site.
Merge pull request #37 from iParadigms/DispatchUpdate

Dispatch update
First checkin