You can, but I don't recommend it. It changes the color and the flavor of the mashed potatoes. Even though I love olive oil, I highly recommend sticking with a non-dairy butter that you enjoy the taste of.