You know it's time to dig up your potatoes when the flowers bloom begin to fade and the buds have dropped. Another sign to watch out for is seeing unopened flower buds drop to the ground from the plant. The leaves might be fading to yellow while some are still green.