When the leaves are yellow, but not brown, and the vines are not actively growing, the potatoes are ready to harvest.