This makes it more difficult for someone to falsely accuse you of improper conduct. Sometimes, students sincerely want to discuss something in private but you must make some other arrangement to allow it. These things must be taught in some context and any challenging class gives the teacher the opportunity to do so. Students need to get used to the idea that their interests are sometimes not in accord with those of others and that, sometimes, they are going to have to take the short end of the stick. You cannot please all your students.
People have different motives and interests, many of which conflict.
Almost any decision you make will run contrary to someone's interests. Always do the right thing as you see it. However, if the entire class is angry, you probably need to reexamine your actions. It does not mean that everyone has an equal chance at success. You have no control over the experience your students bring to class.