There is no built-in way to control followers on Clubhouse. However, you can remove followers by clicking the "remove" button on their profile.