Potato soups usually turn out grainy or gritty because of the potato variety. You may have used floury potatoes instead of waxy potatoes. Other reasons may also be unpeeled or overcooked potatoes or curdled dairy.