You must leave potatoes in the ground until their color turns yellow. The reason you need to dig them up earlier than that is because the color of potatoes is affected by other ingredients as well as the amount of sun light. For instance, if you store a lot of white potatoes in a pile under bright light conditions for longer periods of time, they'll turn dark yellow. This is why we do not like to store white tubers in bins with other tubers because many tubers will shade each other so that the root systems take on a darker color. Also, some varieties are actually lighter or darker when they're young but as they age more, they tend to turn yellow and this means it's time for harvest already!