Select fabrics carefully. Some fibers make you feel toasty and comfortable while others make you feel cold and clammy. Cotton absorbs perspiration and does not retain insulating properties when wet so is not the best choice for very cold riding. Silk doesn't retain soil or odor yet absorbs up to 33 percent of its weight in moisture without feeling damp. It is ideal for under layers because it is thin yet very warm.
Polyester is good as a thin under layer and fluffy insulation.
Down, depending on the grade, can be light, resilient, and provide good insulation. Cordura nylon is lightweight, durable, abrasion resistant, and rot and mildew proof so it is good for outer layers. Gore-Tex has pores that block water penetration but allow for perspiration evaporation. It is windproof, waterproof, and breathable. Plan for perspiration release.