This is all done automatically. The promise of RL is that designing a reward function will be easier than designing the behavior itself. In general, we cannot expect to apply these techniques to all multi-goal problems. If goals are so different that they effectively can't be interrupted or interleaved, the ideas in this thesis may not be of much help. It only discovers whether goal-interleaving is possible.
Competition can actually result in serial behavior in practice.
Imagine agents that have almost no areas of agreement, where one agent's best action is always the other's worst action, and vice versa. Their loss by not getting their goal started cannot compete with its loss if it loses the lead. So it will fight them off and run until completion. Then another agent gets a chance, and runs to termination. And so on.