The answer begins with understanding the types of building materials commonly used in construction. Gypsum wallboard is present on the interior side of most residential construction. Some EIFS construction systems also use gypsum wallboard in a semi-waterproofed state, to provide external sheathing as well as interior sheathing. Such systems are even more vulnerable to moisture and mold problems. It's the paper on the surface of the wallboard that is the problem.
Most fungi are unable to grow well in the alkaline medium created by the gypsum.
With only paper as food, certain filamentous fungi such as penicillium aurantiogriseum, aspergillus versicolor, stachybotrys chartarum, and others not only tolerate alkaline conditions but also out-compete other organisms. They become dominant, and the wallboard becomes covered with these molds. Normally, we are exposed to mold spores in significant quantities outdoors. Many of these molds are leaf-decomposing fungi. The mold species indoors in water-damaged buildings are often soil fungi, different from those prevalent outdoors.