Remove the delayed call when timing out.