The pastor can be in the most vulnerable position. If there has been a failure to listen, the pain is the greater as the pastor may not have really comprehended the reasons for the difficulties. There are then other problems which disenchanted minority groups generate. The most obvious is the refusal to support the work of the church prayerfully and financially. It does not seem to be a problem to press on passionately with the cause and claim the right to speak at special church meetings even when giving has ceased and there is no useful ministry being exercised.
Critics are usually to be found in the arm chairs with little or no stake in the ministry.
Some will be there because the leadership does not want them involved. There may be something which you can learn and it may be a help. Do not ignore the concerns of the those who are usually supportive of you. It may not be expressed in the right way but that is no excuse for not paying attention. Try not to react personally.