Have you tried different Set implementations?