Keep reference graph unchanged and only work with color update