stewartmckee/cobweb — pull request #13 by rojotek (2012-09-09 22:14:56+00:00)
Title: Two improvements for you to look at here - inprogress + updating of setting the queued state

There has been added an inprogress state to help ensure that the crawl finished event occurs when the crawl is really finished. Previously it was possible for the crawl finished event to occur before all jobs were complete when running without any page limits set.

The queued state was only being set after the enqueue operation was complete. It would be possible for two different queues to then enqueue the same job. It's now set before the enqueue is performed. This still could use some double enqueueing of links, but the edge case is decreasing.