Dirt, debris and tree sap can accumulate on your vinyl siding over time. This layer provides mold and algae with a source of nutrients. That's when you start to notice green, brown and black spots forming outside your home. Some areas of your home exterior are more mold and algae-prone compared to others. Sections of vinyl siding that are not exposed to sunlight can be vulnerable to mold and mildew growth. So are the areas of your home that is close to trees and plants.