The main difference between sweet potato pie and sweet potato casserole is largely in the crust and toppings. Pie doesn’t typically have any toppings other than whipped cream or coconut whip. Meanwhile, sweet potato casseroles are often made with a variety of toppings but do not contain any crust.