Yes, you can leave out the yogurt but do add in a splash of vanilla extract if you do. If you just want to avoid dairy, try using a dairy-free yogurt instead.