You do not have to refrigerate every layer after it is spread. It will set a little bit while you're prepping the next layer, and each layer will be easy to pour and spread without disrupting the layer below it too much. Layering them without refrigerating will also allow the layers to stick together well for the finished sliced fudge pieces.