Fix the color being set wrong in the async scoreboards sometimes.