Turns out it is faster to not use the hashmap and just recalculate everything many times