Try to guess the color, based on the saved history