No, you don't have to. You can easily use almond flour instead, in the same amount.