Yes. If you skip refrigeration the peach filling will still be soft and it will be more like a crumble. Putting it in the fridge is what allows the filling to cool and be sliced into neat pieces.