taganaka/polipus — pull request #37 by tmaier (2014-06-10 07:25:58+00:00)
Title: Refactor Crawler

As a result from #33, I reconsidered the current structure or `PolipusCrawler`.

Especially `PolipusCrawler#takeover` is a very long method where lots is going on at the same time.
`PolipusCrawler` itself has lots of methods and is responsible for *everything* whats going on in Polipus.

I consider this pull request more a proof of concept and a starting point for a discussion.

I would like to move all methods of `PolipusCrawler` to its own classes or plugins so that every class has its own responsibility.
For now, I moved most of `PolipusCrawler#takeover` to `Worker#run` and split itself again in smaller methods.
This would allow a more thorough testing of only specific features of polipus without running the full stack.

The delegator used in `Worker` is more a temporary solution.

We could allow plugins to hook into `should_be_visited?` and add then a robots plugin, a follow_links_like plugin and a store_pages plugin.

A statistics plugin would replace incr_pages and incr_errors and hook into `on_after_download` and `on_page_error`