Not at all! You can use any milk—almond, soy, oat, or coconut milk all work well.