No, you don't. I added the baby golds just to catch the excess cheese and to cook in the melted butter on the bottom of the pan. If this seems like an extreme amount of potatoes, feel free to skip them! Melted, baked on cheese at the bottom of the pan is just as delicious!