Yes, you do need to allow the gelatin to set up by chilling the cake in the refrigerator. Otherwise, the gelatin will make the cake wet and soggy.