Yes, you can. Just replace the milk with the same quantity of water in the recipe to make the batter.