Move the cancel condition up again

Should also avoid making the next map vote timer if we deny loading.