Yes, they should be refrigerated. The Jello will dissolve into the cake when left at room temperature for too long.