This is a very good question. Women often seem to struggle with their weight while their male partners eat whatever they want in whatever quantities they want. A good part of it is the fact, as you mentioned, that men may have a higher level of physical activity in their days than women do.
Men also have more muscle mass than women, so their basal metabolism is higher.
They need more calories to stay alive. Women's bodies are designed to conserve calories in order to support fertility. You may cook the same food for both of you, but you need to take much smaller amounts of the protein and starch, and a lot more of the vegetables to fill up on. You need fewer calories, so use a smaller plate and do not go back for seconds. If cleanup is a problem, ask someone else to place leftovers in the refrigerator while you wash dishes.