He's had a hard time with his wife. At any rate, we do know that his life is very much regulated by devotion to, and responsibilities for his mother. Then there is you. Think of yourself as the one woman in his life who is not trying to dominate him. Let his time with you be pleasure and laughter and no more serious discussions of all the problems that beset him.
Poor guy deserves some fun.
Do whatever it takes to bring some other friends into your relationship - another couple to lunch with occasionally. Find a way to get out to a movie, or rent a video and ask neighbours in for coffee, or whatever. In other words, try to have a social life with him, join a choir together, take up oil painting, volunteer at the hospital, anything, together, no matter how little spare time you have, get some fun with people into your life together. That man needs a break. You do too.