Change removeMatching() impl to better deal with concurrency