Make sure we do not retract on a perimeter when there is no infill.