This is known as the thermosiphon effect. When the air handler is on, air leakage can increase threefold due to the high air pressure produced. In supply ducts conditioned air is pushed out through holes, while leaky return ducts suck in air. Figure 2 illustrates how ducts leaking in unconditioned zones influence the pressure inside the thermal envelope. Outside air will then rush to infiltrate the envelope through any existing leaks.
Energy is lost in several ways.
In winter, heated air will blow uselessly through supply leaks into the crawlspace and the warm ducts will lose heat to the cold surrounding air all along the duct run. At the same time, cold air will seep into the depressurized living space at an accelerated rate and the furnace will be working harder and longer to maintain a comfortable room temperature. The fan draws in unconditioned air through holes in the ducts, allowing less air to exit the room through the return register. Pressure builds up in the living space so that the conditioned air seeks other escape passages and squeezes its way to the outside. Imagine the energy losses in an air conditioned home that has leaky ducts in the attic.