I'll even get voted out over this one. Adultery and gossip are not equal. The reputation of the church is a consideration too. Forgiveness does not mean staying in leadership. Restoring a fallen person as a Christian precedes restoring them as a leader.
Full restoration is possible.
In fact, I believe even a fallen ordained minister can be fully restored to active ministry. Sure, some of the people may never fully receive their ministry. Not by ignoring or dismissing the sin. But by that person dropping out of leadership and dealing with it. I'd say, 'Drop out of active ministry for a year and submit to a restoration process.