agnosticbin: Avoid race condition while removing elements

Change-Id: I0a10e3b40f2e7a06b887f1be00c153e0e21532d2