If you have a cake ball with a good structure, you don't have to. However, refrigerating them for 10 minutes or so does help them firm up so they don't crumble.