Ensure enabledMap[color] is unset properly.

Previously a function would not unset the number of colors present yet still clearing them, this could result in unwanted closing of colors.