A lot of smallish improvements from profiling

I used trove instead of Set<Integer>, did some other things that were inspired by running a
profiler on the code.