Patience is a virtue. I suspect you didn't let the cake cool completely before making the holes.