Switch to ConcurrentHashMap backed sets to avoid concurrent modification exceptions.