Obviously you can't please everybody all the time, but you can do your best to avoid known problem areas. Humour, for example, can be dangerous. Racial, religious or sexist jokes are definitely out. You may think the fuss over he or chairman is silly, but if members of your audience think it's an important issue, you'd best pay attention. Perhaps you agree that you should avoid masculine words when you're referring to a mixed group, but you think the alternatives are just as bad, if not worse.
Readers are quite justified in grinding their teeth in annoyance.
The solution should flow so naturally that it's not noticed, except perhaps by people like me who spot these things and smile in admiration at an elegant solution to a thorny problem. You may agree with that proposition, but think it's too hard to write elegant gender-neutral documents. After all, you have only so much time, and the first priority is for the document to be complete and correct. You don't have time for frills like elegant writing. A few years ago, most of us had to learn to change from writing in the passive to writing in the active voice.