Given that this is a fairly small set that probably wont be use a ton something like Collections.synchronizedSet() can provide some simplicity to your code while doing ok on the performance side.  Dont use this for large sets