Is it safe to call remove without a synchronized here?