grosser/parallel — Issue #22: Starting with more than one thread
Author: robmiller
Created: 2011-11-15 23:52:54+00:00
State: open

Thanks for the amazing library!

Part of some code I'm writing features a web spider that crawls pages, with each hit updating an array that functions essentially as a queue. Semi-pseudocode:

@queue = []
@visited = []

def hit(url)
find_links.each { |link|
next if @visited.include? link
@queue << link
}

@visited << url
end

def crawl
Parallel.map(@queue, :in_threads => 4) { |url|
hit(url)
}
end

Using `in_threads` it all works perfectly; though the array begins with one element, the first run-through queues up some more and these are gradually processed by the worker until all the unique URLs are hit.

However, there's a slight issue. Because of this line in `map`:

size = [array.size, size].min

…since my array starts with only one element, only one worker thread is started, and so my code never uses more than one thread.

I can get around this by queuing the initial URL `x` times, where `x` is the number of threads I want to use; that fetches the initial URL `x` times and everything else approximately once, but that seems awfully hacky and generally suboptimal.

Is there a way to tell Parallel to use more than one worker thread even if the initial workload doesn't have more than one element?