Some materials are better than others at resisting the transmission of heat. If you can help it, use those materials to surround your home. There are two primary ways that heat escapes buildings. Heat transfer directly through building materials is one way. The other is heat loss from air infiltration.
Some people claim that the typical American home loses half of it's heat from cold air infiltration.
Older houses are clearly worse than newer houses. Cold air is more dense than warm air. Cold air infiltration is evident when the wind is blowing hard, but it can also be felt on a calm night when the air is very cold. Just hold your hand in front of an electrical outlet. You may feel a small cold blast, depending on the weather conditions.