Staying home seemed like a bigger investment in our family's growing needs. It will help your relationship with your husband to actively decide and agree upon which things absolutely must be accomplished and by whom, which things can wait, and even which things someone else can do. Make a list, put it in writing on display, and both of you can sign it if necessary. Plan to reevaluate the list together weekly until you are both satisfied with your mutual expectations and limits. Someone put it in perspective for me when she held up a string that represented an 80-year life span and pointed to the tiny section when our children needed us most.
Housework never goes away, but childhood passes quickly.
Maybe there is something else bothering your husband and the money worries are just a symptom of other concerns. When I look around at the people I work with, I feel alone in being the sole provider for my family. It can be very scary being the only one that provides cash and health insurance. No job feels really safe today. Rumors of layoffs or buy-outs are common and can make a person work harder and longer just to keep what he has in order to provide for his family.