Futures version of reading ArrayBasedDirectedGraph

ArrayBasedDirectedGraph was not very clear with `apply()` method on almost 200 lines. I've cut it into submethods and rewrite it to futures instead of ExecutorUtil's parallel works.
Ajeet,

1. This seems suitable for IO tasks. And it is as such written in the comments in 
https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/FuturePool.scala#L58

We are doing mainly IO here, so it seems fine.

2. How does the client (caller) set concurrency in general when all the client knows is the number of parallel cores available on the machine. so, e.g., lets say a piece of code like here is reading multiple files, and there could be one to hundreds of files. How does one typically set the concurrency parameter?



On Jun 26, 2014, at 11:30 AM, agrewal <notifications@github.com> wrote:

> In cassovary-core/src/main/scala/com/twitter/cassovary/graph/ArrayBasedDirectedGraph.scala:
> 
> >     */
> > -  def apply(iteratorSeq: Seq[ () => Iterator[NodeIdEdgesMaxId] ], executorService: ExecutorService,
> > -      storedGraphDir: StoredGraphDir) = {
> > +  def apply(iteratorSeq: Seq[() => Iterator[NodeIdEdgesMaxId]], executorService: ExecutorService,
> @szymonm No need to pass future pool around. FuturePool.unboundedPool is a lazy val on an object. i.e. it is constructed lazily, and there is only one instance. So you can unambiguously refer to it.
> 
> This is only a suggestion, but it is easier to reason if you only specify parallelism at the top level, indicating how many parallel futures you want to run. The number of threads is not controlled by you. For example, if the futures are doing IO, it might be that you can multiplex multiple futures onto a single thread.
> 
> So for example, if you have two functions with different concurrency requirements. The following would be the cleaner approach.
> 
> class Foo(concurrency: Int) {
> val sem = new AsyncSemaphore(concurrency)
> def asyncFoo() = {
> sem.acquireAndRun { FuturePool.unboundedPool { your code here } }
> }
> }
> 
> val foo1 = new Foo(10)
> val foo2 = new Foo(20)
> 
> foo1 and foo2 will now reuse threads between them, while you retain control over their concurrency.
> 
> —
> Reply to this email directly or view it on GitHub.
>