Optimized fitness calculation to reuse datastructures