Try to begin the communication, and if you are not successful on your own, consider getting a therapist to help you. My husband had an affair but says it won't happen again. I have a 15-month old son and am pregnant. He also started using pot while he was with her. I cannot answer whether you should stay or leave, but everyone should explore every opportunity to improve their marriage before leaving.
Communication is key to improvement.
Parents should reassure children about the child's security after the divorce. Let them know what their living arrangements will be like. Let them know that their parents still love them and that the divorce is not their fault but a difficulty between the two adults. Of course it is not wrong to love anyone, especially an ex husband. However, if the love is not returned and you find yourself unable to move forward in your life, this can become a problem for you.