When you create your own group, if you choose to moderate it, you basically have decided you want to avoid something that was happening in other groups. As a result, it's only logical that you can establish the appropriate set of behaviors for the group you are moderating. In order to make sure this happens, you have 4 things you can do. You can boot people who break your rules, you can keep people out by making the group secret, or invisible, or restricted. Just because you choose to only do a subset of these doesn't mean that you're less entitled to your rules.
Some people seem to think that they have more, or as much, control of a group that the moderator.
This is only true if the moderator says so. It's not the moderators job to entertain you, or accommodate you. It's your job to either fit in to existing groups, or exercise your privilege to make your own. That may seem a little authoritarian, but remember that if any group is too authoritarian for you, you can create a less authoritarian one. If other people agree with you, they'll follow.