The problem that you describe is not that uncommon in many ways. If as you indicate your house is damp and moldy, then the respiratory health problems you describe for yourself and your children are consistent with exposure to mold. Given the severity of the respiratory health problems you report, it is likely that you are being exposed to very high airborne mold levels. Exposure to high airborne mold can result in chronic allergy symptoms, reoccurring sinus infections and in sensitive individuals asthma. Mold cannot cause pneumonia directly.
Illness caused by mold or other indoor contaminants depends on dose.
In such cases the highest dose is experienced by housewives and their young children. Young children are particularly at risk because their immune system is less well developed. There is also another aspect to husbands in such cases. It is called denial. They often don't want to believe there is a problem with a house.