You can, but I recommend taking the time to coat them with the icing since it helps to prevent them from getting dry quickly. And it makes the madeleines A LOT more delicious!