Critics should not judge any type of music based on its worst performances but rather on either the best or the bulk of performances. A look at the full bodies of work by people like Roscoe Mitchell and Matthew Shipp shows that musicians in the avant-garde can develop and change significantly. And yes a good deal of avant-garde jazz is displeasing to the ears. Critics of this element incorrectly assume that abrasive sounds are a bad thing. In many cases, the intention of the artists is to express emotions outside of happiness or a state of calm.
Life is full of unexpected and unpleasant moments.
The emotions that these moments cause has always been a subject for jazz. Many in the avant-garde keep up this tradition in ways that hit too close to home for some listeners. You can not really argue with a person who says that they just do not like avant-garde jazz. You can only like what you like and every person should be allowed to express their reasons for their own opinions. But they need to do so honestly and by attempting to understand the music what the music is trying to do and evaluate it by those standards.