If the sponge cake doesn't absorb the milk mixture after being refrigerated for several hours, it may have been over-mixed. This creates a denser cake. The sponge cake must be light and airy so the milk has ample places to settle.