If your siding has fungus, mildew or mold, it’s time for replacement. You can also watch for increased energy bills, peeling paint, severe fading, warping, rotting or cracking.