In addition, stress decreases the body's lymphocytes — the white blood cells that help fight off infection. The lower your lymphocyte level, the more at risk you are for viruses, including the common cold and cold sores.