Opt for a lightweight fleece and layer it over a moisture-wicking fabric like wool or cotton. This will help manage body heat better.