Fix race condition that might leave access queue unpurged (#12292)

Cherry-pick of 3a373efe (the original fix) and 2ff71f62 (fixing a
regression) that were marked to not be automatically merged from 7.1 to
master.

Change-Id: Iee1012486906d8c2c46cef94cfcd6d2b399d7a6b