You'll only lose if the person waiting for their turn catches up to you and punches you before you've finished with five rounds. It's not impossible though so be careful.