Audio encoder for encoding audio input data to obtain audio output data, comprising: an input interface configured for receiving a plurality of audio channels, a plurality of audio objects and metadata related to one or more of the plurality of audio objects; a mixer configured for mixing the plurality of audio objects and the plurality of audio channels to obtain a plurality of pre-mixed channels, each pre-mixed channel comprising audio data of an audio channel and audio data of at least one audio object; a core encoder configured for core encoding core encoder input data; and a metadata compressor configured for compressing the metadata related to the one or more of the plurality of audio objects to obtain compressed metadata; and wherein the audio encoder is configured to operate in either a first mode or a second mode of a group of at least two modes comprising the first mode, in which the core encoder is configured to individually encode the plurality of audio channels and the plurality of audio objects received by the input interface as the core encoder input data without any interaction by the mixer, and the second mode, in which the core encoder is configured for receiving, as the core encoder input data, the plurality of pre-mixed channels generated by the mixer and for encoding the plurality of pre-mixed channels generated by the mixer, and an output interface for providing an output signal as the audio output data, the output signal comprising, when the audio encoder operates in the first mode, encoded audio channels and encoded audio objects as an output of the core encoder and the compressed metadata, and the output signal comprising, when the audio encoder operates in the second mode, the output of the core encoder without any meta data related to the one or more of the plurality of audio objects. an input interface configured for receiving a plurality of audio channels, a plurality of audio objects and metadata related to one or more of the plurality of audio objects; a mixer configured for mixing the plurality of audio objects and the plurality of audio channels to obtain a plurality of pre-mixed channels, each pre-mixed channel comprising audio data of an audio channel and audio data of at least one audio object; a core encoder configured for core encoding core encoder input data; and a metadata compressor configured for compressing the metadata related to the one or more of the plurality of audio objects to obtain compressed metadata; and wherein the audio encoder is configured to operate in either a first mode or a second mode of a group of at least two modes comprising the first mode, in which the core encoder is configured to individually encode the plurality of audio channels and the plurality of audio objects received by the input interface as the core encoder input data without any interaction by the mixer, and the second mode, in which the core encoder is configured for receiving, as the core encoder input data, the plurality of pre-mixed channels generated by the mixer and for encoding the plurality of pre-mixed channels generated by the mixer, and the first mode, in which the core encoder is configured to individually encode the plurality of audio channels and the plurality of audio objects received by the input interface as the core encoder input data without any interaction by the mixer, and the second mode, in which the core encoder is configured for receiving, as the core encoder input data, the plurality of pre-mixed channels generated by the mixer and for encoding the plurality of pre-mixed channels generated by the mixer, and an output interface for providing an output signal as the audio output data, the output signal comprising, when the audio encoder operates in the first mode, encoded audio channels and encoded audio objects as an output of the core encoder and the compressed metadata, and the output signal comprising, when the audio encoder operates in the second mode, the output of the core encoder without any meta data related to the one or more of the plurality of audio objects. the output signal comprising, when the audio encoder operates in the first mode, encoded audio channels and encoded audio objects as an output of the core encoder and the compressed metadata, and the output signal comprising, when the audio encoder operates in the second mode, the output of the core encoder without any meta data related to the one or more of the plurality of audio objects.