Only change the UserPaint when needed