Do not venture into the fray alone. You need an older, wiser, and better-connected advocate. In a small college, this may mean going outside your department. If the new rules are set by the administration, they are unlikely to favor you. Watch out for excessive secrecy.
Some secrecy is, of course, required to protect the confidentiality of evaluations.
But too much secrecy serves as a cover for incompetence or worse. It never serves you, the faculty member being evaluated. Also, question the need for any secrecy that is imposed on you personally. The result was that the administration's interpretation of the dossier was the only official version they ever heard. It's one of the most shattering experiences one can have in academia, and your colleague would greatly appreciate any words of support you can offer, even if you don't know anything about the specifics of the case.