Yellow mold is only dangerous or toxic when ingested or if you have a huge infection happening in your house plants that may release spores into the air. However, it is best to remove yellow mold as some people may be allergic to any form of mold, and it is unsightly to see these yellow splotches in your plants.