Only release accept lock when required.