Fix the weight averaging to be correct