Yeah, that can be done. Costs an extra reference. Since it doesnt need to be a very high performance HashMap, I think either periodic cleaning or extra ref + reference queue works. I am open to suggestions.