That's a part of learning to work with other people. Be open to new ideas and look for ways to keep the positive energy flowing. Your pubescent son probably has no interest in a bed of flowers. Just because it's something you've always wanted around, don't lock him in. Perhaps a cactus garden of his own is more acceptable.
Flexibility is key and working together means finding ways to make working together easier.
It's okay to have big ideas, but it's always a good idea start small. A garden will take up your time and energy and the bigger it is, the more time and energy it will take. Yes, you are doing it together, but it shouldn't be a burden or chore. As your interests grow, so will the size of your garden. Also be ready to switch gears.