Swapping around the schedule to fill space of dropped speaker