In my opinion, the longer the cake sits in the fridge the better the flavor. I recommend refrigerating it overnight so the milk mixture absorbs into the sponge cake.