Do you really need a Set here? Do you expect to remove duplicate latches added to the set? If not, just use an array or ArrayList which consume less memory.