twitter/scrooge — pull request #152 by nshkrob (2014-09-25 23:28:09+00:00)
Title: Multithreaded scrooge-generator

This makes scrooge-generator run in parallel over multiple input files (via parallel collections).

This requires to change the output filenames and class names for constants. Scrooge currently produces a file Constants.scala for any file containing constants. To avoid the collision, we prepend the thrift filename to the output.

This still leaves the possibility of output file collision if the input is two thrift files with identical namespaces and basenames, but that would be a user error.

Benchmarking the multithreaded vs singlethreaded scrooge versions on a 24-core machine:

$ for i in $(seq 1000); do echo "service MyService$i { i32 getNumber(1: string someInput) }" > service$i.thrift; done

$ time java -jar ../scrooge-multithreaded.jar $files -d out-multithreaded 2> /dev/null

real	0m7.474s
user	1m16.588s
sys	0m7.143s
$ time java -jar ../scrooge-singlethreaded.jar $files -d out-singlethreaded 2> /dev/null

real	0m22.466s
user	0m26.486s
sys	0m1.690s
$ diff -r out-multithreaded out-singlethreaded
$