add the ability to stop the crawl

I had the need to stop the crawl at will, and so I added this new feature.

There's a new method that client code can call from within a page block, and will make the crawl stop after the current page is processed. The crawl is stopped nicely, not by forcefully killing the tentacle threads, but by clearing the queues so that the natural ending process occurs.

Hope this helps someone out there.
Wow, I did this a long time ago, and I'm not currently in context to answer
this correctly.

I recall that I clear the queue because the way the stop_crawl function
works is that it cleans the queue so that the crawl ends up naturally (i.e.
the loop processing the queue can't find anything else queued so it ends in
a natural way, instead of ending as in a break-loop situation, if you know
what I mean.

Anyway, good to know that there's a fork keeping anemone alive. I liked
that gem and it was sad to see it discontinued. I'll try to work on
adapting the PR for the fork.

--
Ernesto
about.me/gnapse
@gnapse <https://twitter.com/gnapse>

On Sun, Dec 14, 2014 at 2:34 PM, Mauro Asprea <notifications@github.com>
wrote:
>
> In lib/anemone/core.rb
> <https://github.com/chriskite/anemone/pull/61#discussion-diff-21797711>:
>
> > @@ -175,12 +188,17 @@ def run
> >
> >          @pages[page.url] = page
> >
> > +        if @stop_crawl
> > +          page_queue.clear
>
> @gnapse <https://github.com/gnapse> why u clear the queue? What if u want
> to restart the processing? Would u submit this PR in the fork called
> Medusa <https://github.com/brutuscat/medusa>?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/chriskite/anemone/pull/61/files#r21797711>.
>