Common signs it may be time to replace your siding include: loose or cracked siding, warping and rot and/or hole or damage to siding. It may also be time to replace your siding if you notice fungus, mold or mildew, higher energy bills or your current siding is in need of frequent painting.