An audio conference system comprising a controller, a primary microphone and a plurality of general microphones, and an antenna that makes the controller and the microphones communicate via radio communication, wherein the controller includes: an utterance permitting unit that, in response to utterance request information received from any microphone, is configured to transmit utterance permission information to the microphone requesting an utterance; an operating mode control unit that is configured to switch an operating mode from an utterance inhibiting mode to a normal mode or from the normal mode to the utterance inhibiting mode when the utterance inhibition request information is received and to maintain the switched operating mode, and an utterance inhibiting unit that is configured, under control of said operating mode control unit and in response to utterance inhibition request information received from the primary microphone, to transmit utterance inhibition information to the general microphones, each of the general microphones includes: an utterance stopping unit that is configured to stop transmission of a speech signal to the controller upon receiving the utterance inhibition information from the controller; wherein the operating mode control unit is configured to control the utterance inhibiting unit to transmit said utterance inhibition information to the general microphones, when switching from the normal mode to the utterance inhibiting mode; wherein when an operating mode maintained by the operating mode control unit is an utterance inhibition mode and utterance request information is received from a general microphone after the general microphone has stopped transmitting the speech signal, the operating mode control unit is configured to control the utterance permitting unit so that no utterance permission information is transmitted to the microphone requesting the utterance; and wherein when an operating mode maintained by the operating mode control unit is a normal operating mode and utterance request information is received from a general microphone, the operating mode control unit is configured to control the utterance permitting unit so that utterance permission information is transmitted to the microphone requesting the utterance. an utterance permitting unit that, in response to utterance request information received from any microphone, is configured to transmit utterance permission information to the microphone requesting an utterance; an operating mode control unit that is configured to switch an operating mode from an utterance inhibiting mode to a normal mode or from the normal mode to the utterance inhibiting mode when the utterance inhibition request information is received and to maintain the switched operating mode, and an utterance inhibiting unit that is configured, under control of said operating mode control unit and in response to utterance inhibition request information received from the primary microphone, to transmit utterance inhibition information to the general microphones, an utterance stopping unit that is configured to stop transmission of a speech signal to the controller upon receiving the utterance inhibition information from the controller;