Fix race condition that might leave access queue unpurged (#12277)

Merge: no
Change-Id: Iee1012486906d8c2c46cef94cfcd6d2b399d7a6b